added scrape meta for chula_vista_pd #94

[naumansharifwork]

added scrape meta for chula_vista_pd.

Sample Meta Json is attached

ca_chula_vista_pd.json

[newsroomdev]

Another user-agent change

[newsroomdev]

Tagging @stucka for a second set of eyes

[stucka]

There's only the one index page here -- I think this should include force = True to force a rescrape on each run.

[stucka]

I like this methodology a lot, except we should probably log an error if there is no desired_element. If the underlying HTML has changed in a way that breaks this part of the scraper, as written this will fail quietly, i think.

[naumansharifwork]

Good idea

[stucka]

Nice work! I found a couple little things in the scraper proper, but I think maybe the metadata might need some work. Or I'm hopelessly confused. =)

I'm looking at the contrib docs: https://github.com/biglocalnews/clean-scraper/blob/dev/docs/contributing.md

I think in these the title is going to be the human-friendly name of the document, which is going to be the a.string or a.text or a.content, whatever that is. The text of the anchor tag.

Where you've got the case_id as "officer-involved shootings" or whatever, I'd maybe put that in ['details']['case_type'] or some such.

The case_id I think should be coming from this hunk of text that's not part of the links, things like "Officer-Involved Shooting | 700 Monterey Avenue"

This is ... really ugly and maybe there's a better way to do it, but it's trying to split the cases up into chunks that have that <p before going into the links.

            sections = desired_element.find_all("div", class_="accordion-item")
            for section in sections:
                case_type = section.find("div", class_="title").get_text(strip=True)
                psplit = "<p "
                for i, case in enumerate(str(section).split(psplit)):
                    if i > 0:
                        case = psplit + case
                    case_holder = BeautifulSoup(case, "html.parser")
                    if psplit in str(case_holder):
                        case_id = case_holder.find("p").text
                        links = case_holder.find_all("a")
                        for link in links:
                            link_href = link.get("href", None)

[stucka]

Nice work! I found a couple little things in the scraper proper, but I think maybe the metadata might need some work. Or I'm hopelessly confused. =)

OK, much much much easier way to get the case ID. Where you're processing your links, you can throw in a find_previous("p").content:

    for link in links:
        link_href = link.get("href", None)
        link_case_id = link.find_previous("p").text

[stucka]

@naumansharifwork , can you take another look at this? I did a few things you might not be happy with and we can back them out if desired. You did excellent work and I've likely mangled it! =(

The splash / redirect URLs weren't actually working for me when I dropped 'em in the browser so I put in some code to extract them out. And that seemed succinct enough I killed off your function to process them. But ... maybe I'm wrong here.

I also incorporated the splash / redirect workflow into the other workflow.

I put in some generic handling to detect whether a link was relative and if so add in the base URL. Otherwise, it remains intact. (The last hunk of code there I think ignored the possibility of a third kind of URL, like, whatever, a link to an attorney general's server or something.)

There was a \somethingsomething making it through in I think title that I set up for a text replacement in a couple spots even where it's not needed.

Updated JSON is here

ca_chula_vista_pd.json

[naumansharifwork]

@stucka I have tested the youtube links from the previous meta file they were also working fine for me, but if you think this is a better method to do this then its fine as well.