Find a way to let EiE update the data

[JenMysoc]

EiE have asked to be able to update the person data in SYE on a regular basis. The research that Gemma and Bec have done has found situations that back up this need to proactively update the data even if the official data isn't the same.

As these updates may be based on political knowledge but not necessarily anything in a newspaper or on a website - we will need to give clear instructions on how to do this if we expect it to go through wikidata (so that it doesn't get deleted)

[GeorgieBurr]

Short term: until the data can be updated by SYE. EiE will send mySoc a weekly update of changes they know need to be made to the data.

Longer term: mySoc to explore the Wikidata/ Commons approach and to coordinate a call between PTCIJ and EiE following a decision on the Concorde Legacy.

Georgie to send update on DC and look to get PTCIJ together

[GeorgieBurr]

Georgie to bring PTCIJ and SYE together to discuss Data maintenance plans

[emilydrk]

Estimate: 5 days. Could point them to existing docs on how to update Wikidata (edit directly). On the other side, could build an entire new tool, but that is verging into an entire new project.

[chrismytton]

I originally commented on the wrong issue, oops! #182 (comment)

---

We've just had a call about this. We think there are essentially three approaches we could take:

1. Give them a full editing interface for people and make them a data source

• In our experience of giving people control means, the data is badly modelled, because of linked data.
• This option is popit and pombola again - Chris not keen - this is our experience of trying to deliver easy
• This could then become a source for Wikidata?

2. Use Wikidata tools and the EveryPolitician data pipeline

• Could structure this via the verification page? Or teach them to edit Wikidata? (There is seemingly a lot of appetite for Wikidata in Nigeria)
• The Wikidata option will have a delay and usability is not great - are we forcing an option on them that supports the DC, but not their needs?

3. Maintain a local datasource which overrides certain bits of data

• Pull data in from ‘The true source’ (be that Wikidata or EP)
• Add overrides to the local datasource, this could be changing something about existing people or adding new people

---

I think that 3. is probably the best trade-off between good UX for EiE and using data from Wikidata.

EiE are already sending us bits of data to update. Initially we could have this as just-another-datasource which gets layered on-top of the existing data. Further down the line once we've established the kind of things that they want to update we might be able to translate it to some kind of admin interface.

---

See also follow-up comment from @crowbot #182 (comment), included below for completeness.

Some afterthoughts - if we use the SYE repo to store the 'true source' plus overrides in a verification-pages compatible CSV file, interested Wikimedians in Nigeria could use it to populate Wikidata with memberships via the verification-pages (we'll have set up the data model for Nigeria as part of Concorde). So EiE don't need to become expert in Wikidata, but other people who are can see their changes to current politicians, and choose to update Wikidata as a result.

[crowbot]

Additionally, I wonder if we could get a supercheap editing interface via https://help.github.com/articles/rendering-csv-and-tsv-data/

[chrismytton]

If we do #237 first then we'll have a better idea of what's needed for this.

[EiEResearch]

Since this issue is still opened, has there been any resolve to this? We are currently transitioning government following the concluded elections. How can we edit the peoples' data without worrying for it to be overridden by mySociety's data flow process?

[chrismytton]

@EiEResearch We made some progress on this in #269 and #270, which changed the datasource from EveryPolitician to local CSV files stored within this repository. This is a step in the right direction, but currently still requires someone with moderate technical skills to update the CSV files correctly.

[EiEResearch]

@chrismytton Can this csv update be rolled out to the governors' data-set too? How do we handle assets like images also, I see there is a shineyoureyes-images repo that holds all officials thumbnails.

[chrismytton]

@EiEResearch The Governors already come from a CSV file, it's the morph/nigeria-state-governors.csv file, using the same format as reps and senate.

The shineyoureye-images repository is hosted on GitHub Pages. So any changes made to that repo are automatically deployed to theyworkforyou.github.io/shineyoureye-images. The images were originally generated from EveryPolitician, but the data now comes directly from CSVs, so the script described in the README won't work with new CSV data.

This means you'll have to add them manually or using a new script to generate the images into the right place. The repository has a fairly simple layout, the directories are in /<house>/<person_id> format, with each person directory containing three files, original.jpeg, 250x250.jpeg and 100x100.jpeg, hopefully the filenames are self-explainatory.

You should be able to add the new images into the correct directories with the expected names and things will just work…however there is a slight gotcha at the moment, which is that you need to add something in the image_url column of the CSV file to make the image appear on the frontend.

This is because person_proxy_images.rb aborts if a person's image is nil, even though it builds its own image URL.

shineyoureye-sinatra/lib/person_proxy_images.rb

Line 25 in 3d105f8

return if image.nil?

And MembershipCSV::Person#image get the value from image_url

shineyoureye-sinatra/lib/membership_csv/person.rb

Lines 28 to 30 in 3d105f8

	def image
	person['image_url']
	end

There might be a nicer way to handle that, but just something to watch out for.