added scrape meta for palm_springs_pd #88

[naumansharifwork]

Added The scrape meta for the palm spring police department

[naumansharifwork]

sample meta json
ca_palm_springs_pd.json

[newsroomdev]

tagging @stucka to take a look when you get a chance and see if this fits with your ideas on request.post helper to get JSON responses with asset urls

[newsroomdev]

One small change

[newsroomdev]

It looks like this is overwriting the default user-agent set by the method. Does removing this header break the scraper? We want to maintain the user agent Big Local News (biglocalnews.org) to avoid getting blocked by their servers

[stucka]

In this case also I think user-agent is getting set in a Python dictionary, and then it gets thrown to utils, which is trying to set a User-Agent. Looks like the actual HTTP(s) request is case-insensitive, but the Python dictionary could conceivably hold values for both user-agent and User-Agent and then leave requests to figure out the ambiguity when it converts the Python dict into the actual POST header format.

I'd suggest we should require User-Agent with the caps to resolve any ambiguities.

[naumansharifwork]

I have removed the user-agent and it still works, but in utils it is setting up an other Cap case User-Agent,
If we add the indent at line 388 then it will only add the user-agent where the headers are not passed, I think the better way of doing it is that when passing the headers do-not pass the user-agent if possible and change the utils to set the default user-agent even if the headers are available and user-agent is not in headers dict.

[stucka]

That sounds reasonable to me. Maybe since we already know the capitalization can be unpredictable, that section in utils can test for both and only include it if neither is set:

    if "headers" not in kwargs:
        kwargs["headers"] = {}
    if "user-agent" not in kwargs["headers"] and "User-Agent" not in kwargs["headers"]:
        kwargs["headers"]["User-Agent"] = user_agent

[naumansharifwork]

@stucka yes I agree.

[stucka]

Without an indent on line 338, User-Agent will always be overridden, and that may break scrapers.

[stucka]

This is the mirror image of #93 (comment) =)