eTRV temperature monitoring support

[whaleygeek]

The message structure for the eTRV is as follows:

{'header': {'encryptPIP': xxxxx, 'mfrid': 4, 'productid': 3, 'sensorid': xxxx},
'recs': [{'length': 2,
'paramid': 116,
'paramname': 'TEMPERATURE',
'paramunit': 'C',
'typeid': 144,
'value': 20.69921875,
'valuebytes': [20, 179],
'wr': False}],
'type': 'OK'}

[whaleygeek]

Also note that we need to add the productId to Devices.py

I’ve noticed that the eTRV is not recognised. A message displays stating product unknown:
mfrid:0x4 prodid:0x3 sensorid:0x461
read TEMPERATURE C = 22.69921875
ADD device:0x461 Manufactuer:Energenie Product:UNKNOWN

[whaleygeek]

Ed has a pull request pending against this repo, that I hope to merge and re-test over Christmas. I understand from Ed that he has this working fine now with the eTRV's in his house.

[whaleygeek]

The eTRV's monitoring capability already works out of the box with monitor.py - I am busy with other physical layer stuff at the moment. If anyone wants to try eTRV control, work off of ed's pull request at the moment, I'll get to that part later.

[whaleygeek]

Once the device_classes branch merges back to master, this will be very easy to do (basically just write a Devices.MIHO013 device class and describe the capabilities of the device (including action commands to do things like set a setpoint, and getter commands to get readings for ambient temp etc)

[whaleygeek]

There is some eTRV stuff over here on Ed's fork, that might inform our thinking about how to model an eTRV in the new device classes.

#17

[whaleygeek]

Now the device_classes sprint and branch has completed, it might be really easy to just write the ENER013 device class that describes the eTRV.

[whaleygeek]

As noted in #78 this code from Energenie has a eTRV_menu program that will probably have all the necessary info about communicating with the eTRV

http://blog.energenie4u.co.uk/tech-gadgets/something-weekend-two-way-raspberry-pi-transceiver-board-etrv-guide/

[tonbut]

Hi David, thanks for all your work on this project. I've successfully used it to monitor power using MIHO006 and controlled a switch (actually controlling storage heaters). I'm now trying to get my MIHO013 radiator valves working and looking at the code I see there is some work to do. I'm happy to take a look at this. I see the above post from the energenie blog and I have the C code so i'll see what I can get working.

[tonbut]

I have the C code running but the download of ENER314-RTprograms from Energenie web site doesn't have the eTVR_menu directory only an eTVR directory. The hope executable in there does receive messages from MIHO013 but doesn't have the menu to send messages. Is there another download somewhere? Thanks

[whaleygeek]

Hi @tonbut well, there's been various chatter about it all over the place - in some issues, on some pull requests. Some of the chatter was over here I think:

#17

Also some work was going on over here:

#78

I'll try and pull everyone over onto this single issue, so we can take stock of where everyone has got to, and perhaps push this forward to a conclusion.

[pvanderlinden]

Unfortunately I hadn't had time lately, I have an experiment branch. The code is working, the code is really messy though, for all supported messages, see:
https://github.com/pvanderlinden/pyenergenie/blob/hack/src/send_on_recv.py

[gpbenton]

Hi @tonbut , I have mentioned it before, but I have working code talking to my eTRVs at https://github.com/gpbenton/engMQTTClient.

[tonbut]

Thank you guys. This is great - just what I need.

[tonbut]

Hi guys, sorry I've been quiet for a while I was tied up on other stuff but I've just managed to make a little bit of progress on getting the MIHO013 eTVR working. So I wanted to check with whaleygeek and check I'm doing things right.

These are some changes I've made to the code base:

1. added a handle_message() method to MIHO013 device that transmits a battery level request message
2. added a battery level message template to top of devices.py using same structure as existing
3. changed init.py loop() to break out of loop as soon as message is received

I needed change3 because otherwise it appears the message is sent to late for eTVR to be listening.

Now when I run test code I always get a second message from eTVR immediately after first temperature reading (every 5 mins) with the battery level. It doesn't look like the actual voltage is decoded yet.

I plan to try and get the other messages working too if I'm on the right track with this. Also I'll abstract the send to that the request is queued until next receive.

[tonbut]

Actually I spoke to soon - console message I see is debug of the battery level message being sent not received :-(

[pvanderlinden]

I really should get this in decent state, got everything working, it is just in a really messy branch at the moment. Receiving the messages from eTRV is not 100% reliable though (but that is probably because it is wireless), which is a bit annoying with a message rate of 1 in 5 minutes.

@tonbut if you are interrested in my hacky solution you can look at the full diff of this: https://github.com/pvanderlinden/pyenergenie/tree/hack as I made a bugfix in the send/receive logic of the radio module. But mainly this: pvanderlinden@74ffb77 (not 100% sure yet if it is needed, but had less problems after that)

[tonbut]

@pvanderlinden yes I'm very interested. I'll compare. Yes the 5 min period makes it very painful.

I'm really keen to get a solution within the framework of the main codebase as I want to use other devices at the same time.

[tonbut]

This has got me stumped now. :-)

@pvanderlinden I've merged your changes and tried running your test programs. test2.py looks like the right one. I create a JSON message with correct sensor id and it appears to run, receive a message and then transmit. However I can see no effect with the exercise command or battery level checks.

I've also had no success (in a similar way) with the energenie code in eTVR_menu.

I've also tried an alternate ENER314-RT and an alternate MIHO013 just in case.

So I have the following questions if anyone can help:

1. what is correct value for encryptPIP for sending? I see both 0x100 and 0xF2 in different code bases.
2. should I expect to see the MIHO013 led flash when I send a identify message?
3. should I expect a battery level response immediately after sending a request or 5 minutes later?

@gpbenton I think next step will be to bite the bullet and get your setup working and see what it does differently.

Thanks in advance.

[pvanderlinden]

@tonbut Sorry should have explained how that branch works as it is a mess with different experiments.

• For the working experiment you have to have pyzmq installed (you should be able to just pip install that, although it might need to be able to compile the module)
• in one console run bridge.py (with sudo as it needs access as the rest of the pyenergenie source)
• in the other console run send_on_receive.py (both are in src), doesn't need sudo, to send the right messages, add/remove messages from to_send, and adjust the value for the message. Also make sure to update the sensorid for the messages as these are hardcoded to mine.

the radio module in eTRV_menu is not working for some reason, it just helped me get all the different messages.

1. EncryptPIP doesn't matter, the encryption is basicly useless on the energenie devices
2. I think so, the clearest responses I could get were: get battery level, and exercise (as you can here the device move)
3. Battery response should be send immediately, but the device only receives messages really short after it sends the temperature report.

[gpbenton]

Just to answer the other question: the Identify command does flash the LED for about 30s when it receives the command (which as @pvanderlinden says has to be milliseconds after receiving the temperature report)

[tonbut]

Thanks for your answers.

I've spent a bit more time today trying to move this forward but not had any success.

@gpbenton I've tried to build your repo but I'm failing on one step I think - I've built log4c but it doesn't seem to get installed so that the include <log4c.h> in engMQTTClient.c finds it. Can you remember the trick? (I thought sudo make install would do it)

After trying all permutations of things I can think of, I thought maybe I'm missing something fundamental here. Maybe the MIHO013 needs to be registered properly and receive an ACK? Looking at the documentation it says it should blink red fast after a successful join but I can't get this to work using the setup_tool or discover_mihome.

I'm wondering if I need a energenie hub to set the device up properly?

Any more guidance welcome!

FYI this is the way I'm sending a message:
I've added a handle_message to MIHO013 class and whenever I receive a message I do this:

payload = OpenThings.Message(MIHO013_IDENTIFY)
payload.set(header_productid=self.product_id,
header_sensorid=self.device_id)
self.send_message(payload)

MIHO013_IDENTIFY = {
"header": {
"mfrid": MFRID_ENERGENIE,
"productid": PRODUCTID_MIHO013,
"encryptPIP": 30527,
"sensorid": 0 # FILL IN
},
"recs": [
{
"wr": True,
"paramid": OpenThings.PARAM_IDENTIFY,
"typeid": OpenThings.Value.UINT,
"length": 0,
"value": 0 # FILL IN
}
]
}

Tracing through the radio call I see this all appears to work and I even see the red led flash briefly. But still no reaction from the device.

[gpbenton]

I've built log4c but it doesn't seem to get installed so that the include <log4c.h> in engMQTTClient.c finds it

log4c.h should be installed in /usr/local/include. Did you do sudo make install for log4c?

I'm wondering if I need a energenie hub to set the device up properly?

I don't have an energenie hub, so this isn't the case.

[tonbut]

@gpbenton you're a saviour. I've finally got something that get the Pi to talk to the MIHO013 - engMQTTClient. Got the identify command working. Now to find what's different in the python code. Thanks for your patience so far.

[tonbut]

I've finally got this working as expected. It turned out that all the extra debug I'd put in slowed down my old pi too much and I think it must have missed the timing on the receive window of the MIHO013.

In tracking this down I went through all the code down to the radio level and understand it much better! One difference I notice @whaleygeek is that the C code uses a random int as bytes 4 and 5 and your python code uses a the constant encryptPIP. This appears to make no difference to behaviour but maybe to security?

I've placed a queue into the MIHO013 class which will now send a single message if one is available each time a message is received. This is working. Mapping this to the action methods on the interface is easy.

However I have a stylistic question on the requesting of battery level and diagnostics: What I think would make sense is to specify refresh period with defaults for automatically updating the values. I.e. maybe every 1 hour or 6 hours a request for these levels is put into the queue automatically and then the internal readings state will be updated and available when, for example, when get_battery_voltage() is called.

Unfortunately at the moment I see no way of reliably obtaining the pipe temperature or valve position. The depth of the diagnostic flags is really nice touch though.

Can anyone give any guidance on what low power mode does?

Hopefully when this work is done it can be merged back into the main repo.

[gpbenton]

Can anyone give any guidance on what low power mode does?

Here is the answer I got from Energenie technical support

2. In answer to your question on the Low Power mode the intention with this function is to enhance battery life by limiting the hunting of the actuator, ie it limits small adjustments to degree of opening, when the room temperature is close to the set point. A consequence of the Low Power mode is that it may cause larger errors in controlling room temperature to the set temperature.

In practice, I don't find it works very well - the temperature varies too much.

[devanubis]

Hi @tonbut, did you ever push your working MIHO013 queue code?

I've got some free time coming up this month (well, cutting back on overtime) and I'd like to try getting temperature logging and TRV controls running.

Thanks to everyone who's had the time to work on this so far BTW

[tonbut]

I've made pretty good progress on MIHO013 code. I've got all the main messages working except for the setTemperature command. This appears to be getting ignored by the eTVR. Currently I'm working around this by using the setValve open/close commands. I haven't commited anything yet as I was hoping to try again for a last push... I never cloned the original repo to start working, just downloaded a zip. Let me take a look at the state of things and I'll get back to you. I have had my code running for a month or so now with no issues so it can't be all bad.

[tonbut]

I've cloned this repository at: https://github.com/tonbut/pyenergenie and commit the changes to three files. The major change is that the MIHO013 class in Devices.py has a number of new implemented methods:

• def get_battery_voltage(self)
• def get_ambient_temperature(self):
• def get_diagnostics(self):
• def get_setpoint_temperature(self):
• def set_setpoint_temperature(self, temperature):
• def set_valve_position(self, position):
• def set_identify(self):
• def turn_on(self):
• def turn_off(self):
• def enable_thermostat(self):

By default voltage and diagnostics are requested once per hour asynchronously. When the code is first run and the first message is received from an MIHO013 the voltage and diagnostics are both requested immediately. However because of the slow nature and the fact that they don't listen continuously it can be 5 minutes before the corresponding methods return anything other than None.

Hope this works for you. Let me know how you get on.

[tonbut]

Yes I think this is right as the other example uses
36'C as 0x01, 0x20 = 288
and 288/8 = 36

One other thing to note is that you must call enable_thermostat() before setting temperature to make sure the valve is in the right mode. (At least I think you do, I can't see it documented)

[tonbut]

@beyond-code-github I spent a bit of time today testing the temperature setting and it definitely doesn't work as it stands as you said!

I have found one problem - the parameter type I used in MIHO013_SET_TEMPERATURE was 0x92 - this should be 0x90. The additional 2 is added by the encoding and represents the length. I made this change and tested but it still doesn't work - though the raw output now looks the same as that generated by the c code:

Send unencrypted: [14, 4, 3, 109, 73, 0, 3, 154, 244, 146, 0, 176, 0, 19, 142]’)
(unfortunately this is decimal rather than hex but i've check it's the same)

So any help to track this down is much appreciated. I've tried all sensible options. It's a shame it's just this one command that doesn't work.

[tonbut]

@gpbenton has the answers in https://github.com/gpbenton/engMQTTClient/blob/master/engMQTTClient.c

His code to set the temperature uses a completely different encoding for the temperature value than that used in the old C code. Basically the temperature is an integer value written into the first byte and the second byte is 0x00. This seems to be working now as the valve makes small movements up and down when I set different temperatures.

I'll get this committed once i've tested it a bit more and cleaned up all the mess i've made trying every permutation under the sun.

[gpbenton]

I think the problem with the encoding I have is that only allows temperature to be set in whole degrees. From what I remember (it was a few years ago), the second byte represents 1/8ths degrees. In practice, for me, whole degrees work just fine and I have never bothered to go back and make it more accurate.

[tonbut]

I'm committed the code with encoding used by @gpbenton to https://github.com/tonbut/pyenergenie/

Setting MIHO013 temperature should now work. Thanks to @beyond-code-github for motivating me to relook at this.

[beyond-code-github]

Ahhh just came back to pick this up again after the Xmas break to see all this progress, thanks folks! I am going to have a go at building a small MQTT python app to encapsulate all the comms with my pimote over the next week or so, will let you know how I go.

[beyond-code-github]

Hello again folks! I have managed to do some real world testing and we are quite close now but I think still having problems with the specific value being sent in the payload. I had set up to graph my room temperature as reported by the TRV while the valve was open, and then after pulling in the updated code, enabling thermostat mode and setting to 22 degrees, and then 20 degrees. I used the debug print statements to verify that the commands were succesfully transmitted.

Before the change, the room was easily reaching 26+ degrees with the door shut (oof!). After the change at 22, it's less but still reached 24 or so. There was no change in this pattern after setting the value to 20, so I'm wondering if I in fact have some kind of default setting applied and the values provided by the set temp command have not taken at all.

Here's the graph over the last day or so with the stat set to 20. Of course I'd probably expect the TRV to overshoot the target temp and the rad still puts out heat for a while after shutting off, but as you can see it still warms when I turn the heat on even if it's already over the target to begin with, which definitely doesn't seem right.

I'll try and drill a bit more into the various codebases to see if I can spot anything else, but if anyone else can triple check things, a couple more pairs of eyes would be useful! Thanks again @tonbut @gpbenton and @whaleygeek

[gpbenton]

By comparison, I have my last 24 hour period from my lounge trv. The blue line is the target temperature sent, the green line the temperature reported and the yellow line is the valve state.

I have never been able to rectify the difference between the sent temperature and the trv stable temperatures, so I adjust for it in the controlling software. The difference is not the same among the three trvs I have, so I assume it has something to do with the installation of the physical valve.

The heating is on 6:30-7:30, 12:00-1:00 and 5:30-10:30

[beyond-code-github]

Ahhh interesting, so it's possible that instead of the TRV tweaking the valve position incrementally to achieve a specific temperature, instead it might just have a vague temperature -> position kinda strategy? What I might do is set up another one in another room for comparison or perhaps reduce the temperature down further to see if there comes a point where it drops or not. Good to know your code comes with confidence from the data though too @gpbenton.

[gpbenton]

I don't think the trv firmware is as simple as setting the valve to one position, as I hear it adjusting periodically, and when the temperature is too high the valve closes completely.

It could be that the firmware is trying to adjust for the temperature in the room as opposed to the temperature by the radiator, but I have never had another temperature sensor to check elsewhere in the room.

I have long since given up on trying to understand it - my current settings work for me now :-)

[beyond-code-github]

One last thing @tonbut, could you explain the rationale behind the change here? tonbut@f4e4554#diff-f2ef18687f08317911f0617d270b6937R1093

payload.set(recs_TEMPERATURE_value=int(temperature*256))

Is that to adjust for the fact that it's actually a two-byte payload with the second byte representing the fractional degrees and so we need to pad the integer out first to fit (i.e a bitwise shift of an entire byte)?

[beyond-code-github]

PS it also transpires that although I'd updated my fork of pyenergenie with the latest changes, I'd neglected to pull in the updated dependency in my server app so all my previous testing was moot 🤦‍♂️

[beyond-code-github]

After a lot of trial and error, I have managed to conclude that the temperature set messages are now working, but don't seem to take effect reliably every time they are sent. I'm starting to think this is because it sometimes misses the receive window so I am wondering if there is any downside to me just changing the code to continually send the set_temp command every 5 minutes when the window opens? Are there any other considerations I might have missed here that could be causing this?

[tonbut]

I worry that if you send set_temp every five minutes you'll sometimes create a backlog of messages as the device only listens around every five minutes. Might be better to send less often or only when you want to change temperature.

payload.set(recs_TEMPERATURE_value=int(temperature*256))

Yes this line was just to put the integer temperature into the first byte as per @gpbenton code. It might have int(temperature)*256 to ensure lower byte is zero.

[beyond-code-github]

A little further update from me: I caved in a while back and bought a mihome gateway on sale, as it seemed like at least then I'd have some assurances that the setup was going to work.

I've still got the pi mote listening away though, and I've spotted messages like this coming through whenever my gateway sets a new target temp:

MIHO013 receive %s {'rxtimestamp': 1550642525.817312, 'header': {'sensorid': xxxx, 'encryptPIP': xxxx, 'mfrid': 4, 'productid': 3}, 'recs': [{'paramid': 116, 'typeid': 144, 'paramname': 'TEMPERATURE', 'paramunit': 'C', 'value': 16.0, 'length': 2, 'valuebytes': [16, 0], 'wr': True}], 'type': 'OK'}

Does anyone know if this message is the TRV sending a response/ack, or is this just catching the set temp messages that the gateway itself is sending? Is there any way to tell?

[jollytoad]

The gateway cannot send messages to the TRV, as the TRV isn't always listening. From what I understand the TRV broadcasts a report and then briefly listens for a response, this is the only opportunity the gateway (or pi radio) has to send commands to the TRV.
So i'd expect you'd see a pair of messages in quite succession, first from the TRV and then the response from the gateway.

BTW, I have a gateway and a few TRVs, and have a custom setup with a pi & arduino to manage switching the boiler on or off based on TRV demand - just using the remote mihome APIs. (It's a bit annoying/cold when the internet or mihome site is down - but that's quite infrequent luckily)
I also have a energenie radio board for the pi, but i've never got round to getting it working - i'm keen to hear how you get on.

[whaleygeek]

This ticket is still open waiting for me and others to have time to look into it. I suspect a better scheduler that allows a queued message to be sent immediately after a specific eTRV style message is received, would give better response time here.

[whaleygeek]

Partly related: #9

[Achronite]

I've now managed to add support for the eTRV to my node-red-energenie implementation after buying a couple from Amazon. Like others, I have had lots of problems getting the eTRV to receive messages; I've managed to get to about a 50% Rx hit rate now by the eTRV after migrating most of the code to C and lots of optimisation but the receive window is soooo small...

Has anyone found a parameter call to obtain the set (target) temperature from the device? I need to ensure it has received my command? Otherwise spamming the set temperature param seems like the only way to almost guarantee it is set. Any ideas anyone?

[gpbenton]

I have never found a way of getting the eTRV to report back its set temperature. But my driver is reliable when communicating with my devices, so I haven't found it a problem.

[whaleygeek]

@Achronite Have you any measure of how small the receive window is, it would help with my future plans for the realtime message scheduler that I am currently thinking about. Thanks. (time from end of it's report, to start and end of the receive window, would be most helpful to know).

[Achronite]

I don't have an exact measure, All I know is that a poll interval (SetInterval) of 1 second from my node-red flow sometimes results in messages not being delivered.
So I would say the Rx window is around 2 seconds or less.
The best way to test is using the REQUEST_TEMPERATURE or REQUEST_DIAGNOSTICS commands, as they both return a result. If I get chance tomorrow I'll mock up a quick test using a simple node.js program with a few different intervals.

[gpbenton]

The receive window is 200ms. I have uploaded the q and a pdf energenie sent me a few years ago.
eTRVQandA.pdf

[Achronite]

@gpbenton that would explain a lot. Your code must check the Rx buffer more frequently then!

@whaleygeek is there any way of checking if there is a message in the Rx buffer without this polling?

I'm thinking along the lines of the way websockets work, async blocking calls, callbacks etc. Is this possible with SPI?

[whaleygeek]

@Achronite The issue is more that the radio is half duplex, and it's receive FIFO is only 66 bytes wide. So in some circumstances, not all the payload will fit in one go and it has to be streamed out live over the SPI.

The Python code leaves the radio in receive mode as much as possible, but due to there currently being no timing scheduler, the receive window (despite it being wide) can easily NOT be in the right place for the eTRV response. Also the overhead of Python interpretation is probably an order of magnitude or two more than running native compiled C code.

So my advice is that we would get best performance once I add a timing scheduler into the C, and schedule a tx and receive window for the eTRV intelligently at the right place in time.

[whaleygeek]

Looking at my sample apps, the APP_DELAY in the main loop is typically 2 (seconds), which will give poor performance where a critical timing window is required, and tx times are not scheduled.

Longer term I hope to run the C code in a posix thread of it's own (to enable the addition of a time-slotted real-time packet scheduler) and plan for this to have 'hints' passed to it as to the next expected report time for each device, so it can self adapt to any given environment.

There's quite a bit of work to do that properly, so the best quick fix is to run your app loop as fast as possible (set APP_DELAY to 0 perhaps) to maximise the probability of the (unsynchronised) tx and rx windows overlapping with the eTRV better.

[whaleygeek]

The RFM69 does have a receive interrupt which can be routed to a pin on the module. I'd have to check if this is already routed on the Energenie RT board or not. But the radio would need to be in receive mode (not transmitting, as it is half duplex) for this to be useful. I'll stare at the circuit diagram next time I do work on this repo and see if that is an option

[whaleygeek]

Also we switch the radio configuration between OOK and FSK as well as switching between Tx and Rx, which adds some additional latency between modes (as a collection of registers have to be rewritten every time the modulation scheme and transfer direction change).

My feeling is that re-architecting the C into a separate thread, adding the interrupt request line, and writing a simple packet scheduler with hints from the app indicating next expected receipt or eTRV receive window time-slots will give us a significant boost in reliability and performance all round.

[Achronite]

I've already changed my C code so that it always switches back to FSK Rx mode if I have 'monitor' devices after a Tx is made. Doing this I generally do not loose any transmitted messages from the device.

The issue is timing, and actively monitoring the SPI at the time the eTRV transmits. I'm using a polling method at the moment from the node-red side; I've made it async to prevent blocking the node.js event loop, but it still uses the node.js worker threads and I need to be careful with any form of long polling using these threads.

I've also been considering creating a child process to do all of the radio interfacing to prevent missing the Rx window of the eTRV, but it will require additional coding to work this out. I believe I could even invoke a callback when a message is received from the thread back to node using N-API.