IL: Add witness slip URLs

[mileswwatkins]

Witness slips allow residents to, on the official state legislative information system, post their support/opposition to a bill. We can add this URL to documents (probably on the event object rather than on the bill object).

Eg:

• List linked from an event page
• List itself

@jonrogoff or @justgosh, can you give a couple of examples (maybe one for each chamber?) of what a slip URL would look like, or (better yet!) what the URL scheme is for slips? You said that they were hosted on my.ilga.gov, rather than ilga.gov (which is where we currently scrape).

[jonrogoff]

The basic approach is to start by going to the hearing page for each chamber:
http://my.ilga.gov/Hearing/AllHearings?chamber=S
http://my.ilga.gov/Hearing/AllHearings?chamber=H
Then go to the appropriate tab (ajax), probably Month for upcoming hearings and Previous for past hearings.

Each hearing has a Hearing Detail page, that lists the bills/witness slips available. For example:
http://my.ilga.gov/Hearing/HearingDetail/14797?SCommittees5/22/2017-page=1&SCommittees5/22/2017-orderBy=&SCommittees5/22/2017-filter=&committeeid=0&chamber=S&nodays=30&_=1492915051408

Each Witness Slip has a Create page (form) and a Info page (totals). For example:
http://my.ilga.gov/WitnessSlip/Create/99262?committeeHearingId=14797&LegislationId=99262&SCommittees5%2F22%2F2017-page=1&committeeid=0&chamber=S&nodays=30&_=1492915051408
http://my.ilga.gov/Hearing/WitnessSlipInfo/99262?hearingId=14797&LegislationDocumentId=130273&SCommittees5%2F22%2F2017-page=1&committeeid=0&chamber=S&nodays=30&_=1492915051408

[jonrogoff]

Special case: There are also Subject Matter hearings that have Witness Slips tied directly to the hearing rather than to bills. These hearings have an extra Subject Matter attribute that other hearings do not have.
http://my.ilga.gov/Hearing/HearingDetail/14788?SCommittees5/22/2017-page=1&SCommittees5/22/2017-orderBy=&SCommittees5/22/2017-filter=&committeeid=0&chamber=S&nodays=30&_=1492915528587

[easherma]

@jamesturk + @mileswwatkins I am volunteering for an Illinois-based non-profit called https://www.statematters.org/

We are very interested in being able to more easily provide schedules of upcoming hearings. @jonrogoff could probably clarify, but I believe their project did end up writing a scraper for this, but I would love to hear more about the feasibility of adding it to Open States directly for greater visibility and so it might be easier for our volunteer devs to have some endpoints to hit.

I'm wondering if the system IL.gov is using is common enough that other states might benefit?

Either way, I'd like to check in and also offer to help get this integrated into Open States as that seems like the best long term home for this. Would appreciate any thoughts/suggestions/collaboration! :-)

[jonrogoff]

Just to update some of the info above, the basic data structures unique to witness slips are:

Hearings which have:
Type (bill vote or subject matter)
Description (subject matter only)
Committee
Date
Bills (with associated Votes)

Witness Slips which link a Bill to a Hearing and provide Signatures:
Bill
Hearing
Signatures

Signatures which have:
Name
Affiliation (self, lobbying group represented, etc)
Stance (Proponent, Opponent, No Position)

I may have misnamed or forgotten a field or two but the intention here is to indicate that this isn't simply a flat entity to be added into a notes field. I suppose it could be done in a json format or some such, but at least as far as our stories went that would result in very slow and inefficient queries.

[showerst]

@easherma @jonrogoff Thanks for reaching out!

Adapting your data to OS sounds like it shouldn't be too bad -- our data models are fairly similar. So far this year we haven't been feeding event data into the api, because we've been short on developer hours to keep the scrapers maintained, and we're trying to lower the volunteer workload of the project.

@jamesturk if we verify this works are you open to re-scraping events in some states? Or are the challenges to getting that data loaded into the API too much time?

If james comes back with a yes, here's the data model documentation. I've recently done a few event scrapers that you can check out the code for to get an idea, AK MD. We have a few third-parties that run the scrapers even if the data isn't currently getting loaded into Openstates, so we've been accepting patches to those scrapers.

As far as the unique bits to IL, i'm thinking that witness slips and possibly signatures can be added under the add_document model.

[jamesturk]

Sorry for the delay. We aren't exposing events via the API anymore, so we'd need to find some support to get that added. I could be convinced to re-enable scrapers but I think the API would be a bigger obstacle. If there's an appetite to work on that though I'd be glad to discuss what it'd take.

…

On Tue, Mar 5, 2019 at 11:20 AM showerst ***@***.***> wrote: @easherma <https://github.com/easherma> @jonrogoff <https://github.com/jonrogoff> Thanks for reaching out! Adapting your data to OS sounds like it shouldn't be too bad -- our data models are fairly similar. So far this year we haven't been feeding event data into the api, because we've been short on developer hours to keep the scrapers maintained, and we're trying to lower the volunteer workload of the project. @jamesturk <https://github.com/jamesturk> if we verify this works are you open to re-scraping events in some states? Or are the challenges to getting that data loaded into the API too much time? If james comes back with a yes, here's the data model documentation <https://opencivicdata.readthedocs.io/en/latest/data/event.html>. I've recently done a few event scrapers that you can check out the code for to get an idea, AK <https://github.com/openstates/openstates/blob/master/openstates/ak/events.py> MD <https://github.com/openstates/openstates/blob/master/openstates/md/events.py>. We have a few third-parties that run the scrapers even if the data isn't currently getting loaded into Openstates, so we've been accepting patches to those scrapers. As far as the unique bits to IL, i'm thinking that witness slips and possibly signatures can be added under the add_document model. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/openstates/openstates/issues/1667#issuecomment-469743653>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAfYncZvWO0iRWkVylKi_8z9BA4Ld8Iks5vTplIgaJpZM4NFSBV> .