Replies: 11 comments
-
|
Since this library is used by Home Assistant, we cannot have a headless browser due to https://github.com/home-assistant/architecture/blob/master/adr/0004-webscraping.md Possible alternatives that I can think of:
|
Beta Was this translation helpful? Give feedback.
-
|
Also take a look at #13 since it might give you ideas to do something similar. That one is reusing the device token as a cookie which is similar to my 2nd alternative. Apparently that expires after 2 years. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the info. I did look at that PR and noticed the 2FA addition. This does give me some ideas, because it's not FE this library integrates with, it's opower. So if I can identify how the utility authorizes the access token to opower, then maybe an option could be to add a last-resort "utility" that's configured with I'll look over the har I captured, and see what I can find. But if the access token has a short expiration, the only alternative might be a cookie combined with some kind of ongoing ping to keep the session alive. The FE website does auto-logout after an annoyingly short time (again, makes no sense to me why they're treating it like a bank account 🤷♂️) |
Beta Was this translation helpful? Give feedback.
-
|
Yes we should only need subdomain, opower_access_token, and optionally timezone (we could get that from the API response) to integrate with any utility. I haven't tried but the access token is likely to be short lived. If we somehow had the refresh token we could in theory get a new access token before it expires without needing username/password. |
Beta Was this translation helpful? Give feedback.
-
|
So, I've done some testing today, and I made a generic I get the I'm not sure how other utilities respond, but this is how fenj responds to customers: {
"customers": [
{
"id": ####,
"uuid": "u-u-i-d",
"legacyOpowerId": "ab-1-####",
"accountNumber": "####-####",
"accountName": "",
"address": {
"uuid": "u-u-i-d",
"streetNumber": "##",
"streetName": "xxxxx",
"subpremise": "####",
"postalCode": "#####",
"city": "xxxxx",
"country": "XX",
"state": "XX"
},
"type": null,
"utilityAccounts": [
{
"id": #####,
"uuid": "u-u-i-d",
"utilityAccountId": "#####",
"utilityAccountId2": null,
"servicePointId": #####,
"meterType": "ELEC",
"preferredUtilityAccountId": "#####",
"readResolution": "BILLING"
}
]
}
],I have to step away for now, just wanted to mention that in case this is a dealbreaker and not worth pursuing. I will work on converting this response into a fake list of forecasts, and just see how far I can get.
Yeah, the access token seems to be in the ~10-15 min range. Then I have to log back in and extract my access token manually. Even if this won't work for JCP&L/fenj, I can contribute the |
Beta Was this translation helpful? Give feedback.
-
|
If you go over the commit history you should find an earlier version that I was similarly getting the accounts but I ended up cleaning it up since it was unused and I wanted to avoid maintenance costs. For the same reason I'd suggest not committing the manual.py utility. If you do manage to leverage it with an addon that has a headless browser to login into hard to reverse engineer sites like yours I'd be happy to merge any changes, including getting the accounts differently for utilities that don't have forecasted data. |
Beta Was this translation helpful? Give feedback.
-
|
FYI, I added back the way getting accounts from customer instead of forecasts |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
Atlantic City Electric, see home-assistant/core#97814, is another utility that lacks forecast data and only provides monthly data. I did try to request hourly or daily data via the API but opower returns 500, same error you encountered. |
Beta Was this translation helpful? Give feedback.
-
|
I think I agree with your comment about it not being very useful if it can only pull monthly data 🤔 Interesting that opower even allows this, given that according to another thread here, their documentation specifically says that daily is the bare minimum required update frequency. But our meters also aren't actually smart, and reading through the usage history, it only gives us "actual" readings about once every 2-4 months. Other months in between are "estimated". |
Beta Was this translation helpful? Give feedback.
-
|
@jhansche I believe this is why FirstEnergy has such beefy security now. I used to have a DIY custom component that was scraping the json from them and if i remember right they used a SAML type login before the hack. Hopefully we can figure something out eventually, I do miss having the info feed into HA, it beats logging into the website to keep an eye on things. |
Beta Was this translation helpful? Give feedback.
-
First Energy corp is the parent company of my utility, JCP&L (as well as several others). I see that JCP&L is using opower on the
fenjsubdomain.However, the problem I'm having that's preventing contributing a jcpl utility is the login process. The login form itself seems simple enough, but there is an encrypted js file added to the login page that intercepts the form submit action and injects 6 hidden form fields with encrypted or hashed data to accompany the login/password fields. Since these inputs are added JIT before submitting the form, we can't just extract those values from the login/landing page. Examples of the injected inputs:
To make matters worse, it appears that there is also a tripwire attached to this honeypot, where if you don't send the exact required values, your IP is blocked for a period of time (seems to be 5-15 min or so).
Looking at the other utility providers, none of them appear to have anything close to this level of security involved (why FirstEnergy requires such security..? 🤷)
I haven't tried something like headless chrome, which I think that would likely work.. But I suspect that would not be a welcome addition to this library.
Are there any other recommendations for how this could be handled?
Beta Was this translation helpful? Give feedback.
All reactions