future of people scrapers

[jamesturk]

It's probably time to begin the long-delayed movement of people scraping.

Right now the pupa scrapers are disabled. There's a process to update these semi-automatically that involves:

• running a person scrape in openstates/openstates
• running ./scripts/to_yaml.py which puts data into incoming/
• resolving the merges with various merge.py options and manual work
• committing the data

At a minimum we should:

• move the person scrapers from openstates/openstates to either this or a new repo (a couple have been experimentally moved to scrape/). [Actually: should we? what about common data?]
• dramatically improve the incoming/merge process so that (under normal circumstances) if there are no site changes there are no edits to the data created by the run [done]

Eventually/nice to have:

• once the process is solid, evaluate automatically creating PRs from ~weekly runs of the scrapers
• this could be the path to bringing committees back if it goes smoothly

[jamesturk]

see https://github.com/openstates/people#updating-an-entire-state-via-a-scrape for the current process

[jamesturk]

plan in progress, see #85