[zigpy-integration] Allowing Endpoint allocation on ConBee/RaspBee for Livolo, Schnider and fancy devices

[pipiche38]

When doing a zdp_active_endpoint_request to the deConz key , we get 3 Eps

0x00 - Ok ZDO
0x01 - ok ZCL
0x50 - what is that ?

cc: @puddly, @deufo

[Hedda]

@manup related to the previous discussion in dresden-elektronik/deconz-serial-protocol#17

[pipiche38]

@manup related to the previous discussion in dresden-elektronik/deconz-serial-protocol#17

I probably missed something I understood that deConz set 0x01 and 242 ( 0xf2 ), but he was not mentioning 0x50 ( 80 )

[puddly]

What does the 0x50 endpoint's simple descriptor look like?

[pipiche38]

After the startup, here is how I retreive the list of Ep

    ep_list = "".join(
        "%02x" % ep_id
        for ep_id in self.app.get_device(nwk=t.NWK(0x0000)).endpoints.keys()
    )

Here is what I got in the logs

2022-04-19 13:55:59,195 INFO    :[0x0000] Got Node Descriptor: NodeDescriptor(logical_type=<LogicalType.Coordinator: 0>, complex_descriptor_available=0, user_descriptor_available=1, reserved=0, aps_flags=0, frequency_band=<FrequencyBand.Freq2400MHz: 8>, mac_capability_flags=<MACCapabilityFlags.RxOnWhenIdle|MainsPowered|FullFunctionDevice|AlternatePanCoordinator: 15>, manufacturer_code=4405, maximum_buffer_size=71, maximum_incoming_transfer_size=43, server_mask=64, maximum_outgoing_transfer_size=43, descriptor_capability_field=<DescriptorCapability.NONE: 0>, *allocate_address=False, *is_alternate_pan_coordinator=True, *is_coordinator=True, *is_end_device=False, *is_full_function_device=True, *is_mains_powered=True, *is_receiver_on_when_idle=True, *is_router=False, *is_security_capable=False)
2022-04-19 13:55:59,196 INFO    :[0x0000] Discovering endpoints
2022-04-19 13:55:59,195 DEBUG   : [ ZigpyForwarder_6] decode8002_and_process ProfileID: 0000 NwkId: 0000 Ep: 00 Cluster: 8002 Payload: 0300000010400f3511472b0040002b0000
2022-04-19 13:55:59,197 DEBUG   :Tries remaining: 3
2022-04-19 13:55:59,198 DEBUG   : [ ZigpyForwarder_6] ===> zdp_decoders 0000 00 8002 0300000010400f3511472b0040002b0000
2022-04-19 13:55:59,198 DEBUG   :Sending Zigbee request with tsn 7 under 8 request id, data: b'070000'
2022-04-19 13:55:59,199 DEBUG   : [ ZigpyForwarder_6] buildframe_node_descriptor_response for 0000 with status 00 nwkid: 0000 SrcNwkId: 0000 Payload: 0300000010400f3511472b0040002b0000
2022-04-19 13:55:59,200 DEBUG   : [ ZigpyForwarder_6] buildframe_node_descriptor_response sqn: 03 nwkid: 0000 Manuf: 1135 MacCapa: 0f
2022-04-19 13:55:59,201 DEBUG   : [ ZigpyForwarder_6] decode8002_and_process return ZDP frame: 0180420022ff030000001135002b002b0040000f474010dd03
2022-04-19 13:55:59,202 DEBUG   :Frame received: 0x17260025001e002602000000020000000000058007000500000002015000afddcfb605001e
2022-04-19 13:55:59,203 DEBUG   :APS data indication response: [30, <DeviceState.APSDE_DATA_REQUEST_SLOTS_AVAILABLE|APSDE_DATA_CONFIRM|2: 38>, <DeconzAddress address_mode=ADDRESS_MODE.NWK address=0x0000>, 0, <DeconzAddress address_mode=ADDRESS_MODE.NWK address=0x0000>, 0, 0, 32773, b'\x05\x00\x00\x00\x02\x01P', 0, 175, 221, 207, 182, 5, 0, 30]
2022-04-19 13:55:59,204 DEBUG   :Received frame on uninitialized device <DeconzDevice model='ConBee II' manuf='dresden elektronik' nwk=0x0000 ieee=00:21:2e:ff:ff:05:fe:7f is_initialized=False> from ep 0 to ep 0, cluster 32773: b'\x05\x00\x00\x00\x02\x01P'
2022-04-19 13:55:59,205 DEBUG   : [       ZigpyCom_6] App - get_device ieee:None nwk:0x0000
2022-04-19 13:55:59,207 DEBUG   :'aps_data_indication' response from <DeconzAddress address_mode=ADDRESS_MODE.NWK address=0x0000>, ep: 0, profile: 0x0000, cluster_id: 0x8005, data: b'05000000020150'
2022-04-19 13:55:59,208 DEBUG   : [       ZigpyCom_6] App - get_device found device: <DeconzDevice model='ConBee II' manuf='dresden elektronik' nwk=0x0000 ieee=00:21:2e:ff:ff:05:fe:7f is_initialized=False>
2022-04-19 13:55:59,208 DEBUG   :Command Command.aps_data_indication (1, 1)
2022-04-19 13:55:59,209 DEBUG   :Send: 0x1727000800010001
2022-04-19 13:55:59,209 DEBUG   : [       ZigpyCom_6] handle_message from Controller Sender: 0x0000 Profile: 0000 Cluster: 8005 srcEp: 00 dstEp: 00 message: 05000000020150
2022-04-19 13:55:59,211 DEBUG   : [       ZigpyCom_6] handle_message device 1: <DeconzDevice model='ConBee II' manuf='dresden elektronik' nwk=0x0000 ieee=00:21:2e:ff:ff:05:fe:7f is_initialized=False> Profile: 0000 Cluster: 8005 sEP: 0 dEp: 0 message: 05000000020150 lqi: 221
2022-04-19 13:55:59,211 DEBUG   :Frame received: 0x1727050800010026

[puddly]

You can read the simple descriptor for each endpoint by sending a loopback request to 0x0000. Zigpy does this during initialization, even for the coordinator.

The reason I ask is because the Conbee has three endpoint "slots" and seems to return three types of responses when you read them:

1. Status.UNSUPPORTED, when the endpoint is not defined.
2. The endpoint's simple descriptor.
3. A simple descriptor filled with random values.

I need to be able to distinguish "real" endpoints from the "invalid" endpoint, which for me looks like this:

SimpleDescriptor(
    endpoint=184,
    profile=7329,
    device_type=19200,
    device_version=18,
    input_clusters=[],
    output_clusters=[],
)

You say that your Conbee has an endpoint 0x50, which is different from mine.

[manup]

0x50 - what is that ?

Hi, the 0x50 is a very old endpoint which was used for proprietary OTA, it isn't used anymore. You can safely reconfigure it.

[pipiche38]

I'm keen to reconfigure it, but Why is not the Zigpy layer doing by default ?

[manup]

Personally I'd do it in the application, different applications using zigpy might have different needs.
The downside is more work for the application.

[pipiche38]

Ok, sound fine with, but right now from zigpy standpoint I can only add_endpoints and not remove any

[Hedda]

Originally posted by @puddly in zigpy/zigpy-deconz#172 (comment)

Is it possible to also create a remove_endpoint method on all the radio libraries as well?

I don't think this is possible.

1. Only ZNP allows deleting endpoints, EZSP does not seem to have a command to do this.
2. If you override register_endpoints and register three endpoints, the coordinator will not define any unnecessary endpoints on startup so there's no reason to explicitly delete any.

@puddly so if you use register_endpoints to register three different endpoints those should override the existing on deconz?

[pipiche38]

Originally posted by @puddly in zigpy/zigpy-deconz#172 (comment)

Is it possible to also create a remove_endpoint method on all the radio libraries as well?

I don't think this is possible.

1. Only ZNP allows deleting endpoints, EZSP does not seem to have a command to do this.
2. If you override register_endpoints and register three endpoints, the coordinator will not define any unnecessary endpoints on startup so there's no reason to explicitly delete any.

@puddly so if you use register_endpoints to register three different endpoints those should override the existing on deconz?

This is what I understood, basically zigpy_deconz handle EndPoint slots, and so by default there is one slot left (slot 2)

[pipiche38]

@manup could you clarify, that ConBee can support only 2 EndPoints ? This is a very heavy limitation, as in fact no endpoint can be added ?

@puddly, how can be the limitation to 2 , while I was able to show 3 EndPoints ? Is the limitation not somewhere else ?

[manup]

Hi, I've checked the current firmware code, there is indeed a fixed setting of 2 endpoints.
While it's just a define for the size of an array, raising the number might be a bit trickier taking nvram store/restore into account.

Do you have an overview of how many/which endpoints and their clusters you're using?

[Hedda]

While it's just a define for the size of an array, raising the number might be a bit trickier taking nvram store/restore into account.

Does that mean you can not simply make and release a new firmware image for ConBee/RaspBee that support more endpoints?

[pipiche38]

Hi, I've checked the current firmware code, there is indeed a fixed setting of 2 endpoints. While it's just a define for the size of an array, raising the number might be a bit trickier taking nvram store/restore into account.

Do you have an overview of how many/which endpoints and their clusters you're using?

@manup, I would say the max , but I think if we can extend to one at least and probably 2 Eps would be the best. I don't expect an end user having all strange devices on the same setup.

As the clusters are concerned, on the TI / CCxxx / ZNP stack we are just having the Basic Cluster for those Endpoint, as the principle is anyhow all commands send are forwarded back to UART.

[Hedda]

I don't expect an end user having all strange devices on the same setup.

FYI, most users actually do put all their devices on the same setup, at least in my experience many users (but not every user) aim to keep all their Zigbee devices on a single Zigbee network if they know that consolidating them takes the best advantage of Zigbee's mesh networking technology. And another reason is that most Zigbee implementations only support a single Zigbee Coordinator, see example this discussion about having multiple coordinators in one setup at the same time -> zigpy/zigpy#673

[pipiche38]

I don't expect an end user having all strange devices on the same setup.

FYI, most actually do put all their devices on the same setup, at least in my experience many (but not all) aim to keep all their Zigbee devices on a single Zigbee network as consolidating them takes advantage of Zigbee's mesh networking technology. And another reason is that most Zigbee implementations only support a single Zigbee coordinaor, see discussion -> zigpy/zigpy#673

I do agree, but from a statistical point of view I do not expect a user having ALL this crasy devices which required as many EndPoints. I'm just taking in consideration the deConz memory limitation and indeed we just need to balance between a reasonable need comparing to an unlimited one. Of course unlimited or something like 8 to 10 additional Eps would be great, but I don't think this would make sense, while we might have only few users in the world needing it

[manup]

In regards of mixed networks with all kinds of devices, I can also confirm everything thinkable is put together by users :)
3–4 endpoints should be suitable to support all known devices I recon.

I'll try to extend two additionals possible endpoints, likely in a second segment in nvram to be up and backward compatible. I'd recommend to configure a few more clusters than Basic for at least one endpoint to support a wider range of devices: IAS ACE and Time server clusters, and IAS Zone, IAS WD client clusters. Some devices actively probe the coordinator for them.

[pipiche38]

@manup happy to test any new firmware

[pipiche38]

@manup any update on the firmware capable of addressing more than 2 Eps ?

[manup]

Hi @pipiche38 sorry got a bit occupied with bug fixing in various places.
I have a look this weekend. I've also noticed that the endpoints in the deCONZ endpoint list in my setup don't look as they should (bould could be a deCONZ issue).

[manup]

Can you please try the following test version 0x26790700. It provides four configurable endpoints with index 0..3, prior versions have only two endpoints 0..1. Initially the new last two endpoints aren't configured.

The new endpoints are placed in a different NVRAM section to be backward compatible for up- and downgrades.

deCONZ_ConBeeII_0x26790700.bin.zip

For a test I configured the endpoint 0x11 with Identify client cluster additionally to 0x01 and 0xf2.

[pipiche38]

@manup Thanks very much , and I do confirm that adding Ep 0x08 allows now to pair Livolo Switchs without any issue (which was not possible before)!

@puddly we need to to increase MAX_NUM_ENDPOINTS in zigpy-deconz/zigbee/application.py from 2 to 4 accordingly to the firmware

[manup]

Cool glad it works, I'll put this in the official firmware release.

Side note to be handle old and this version: There is no command to query how many endpoints are supported, but if the configuration is done for an endpoint ouside the valid index range UNSUPPORTED status is returned.

[pipiche38]

Thanks very much for your support.

PR for moving from 2 to 4 Max Eps submitted

• Allowing up to 4 Endpoints zigpy/zigpy-deconz#196

[puddly]

@manup I have a few questions about the endpoints implementation.

• What are the default values for these extra endpoints?

• Is it possible to disable unused endpoints?

• Finally, are all supported endpoints contiguous and will they always be readable?

  I'm trying to avoid hard-coding the total number of endpoints and would like to try something like this:

  num_endpoints = 0
  
  # Read the current endpoints
  for index in range(256):
      try:
          await self._api.read_parameter(NetworkParameter.configure_endpoint, index)
      except zigpy_deconz.exception.CommandError as ex:
          assert ex.status == Status.UNSUPPORTED
          break
  
      num_endpoints += 1

  I seem to recall there being some cases where reads did not succeed but writes did if one endpoints was not configured properly.

[manup]

@manup I have a few questions about the endpoints implementation.

* What are the default values for these extra endpoints?

By default the two extra endpoints with index 2 and 3 are not set, which is determined by an invalid application endpoint number as 0x00 or 0xFF, they are also not returned in an ZDP_Active_ep request.

* Is it possible to disable unused endpoints?

Yes, like the default configuration above, configuring an endpoint number 0x00 or 0xFF disables it.

* Finally, are all supported endpoints contiguous and will they always be readable?
  I'm trying to avoid hard-coding the total number of endpoints and would like to try something like this:
  ```python
  num_endpoints = 0
  
  # Read the current endpoints
  for index in range(256):
      try:
          await self._api.read_parameter(NetworkParameter.configure_endpoint, index)
      except zigpy_deconz.exception.CommandError as ex:
          assert ex.status == Status.UNSUPPORTED
          break
  
      num_endpoints += 1
  ```

Basically yes, in the firmware the endpoint configuration is just stored in an array, currently with the size 4.
So that a request to index [4] and above returns UNSUPPORTED. For older firmware versions index [3] returns UNSUPPORTED.

So I think your code here should work.

  I seem to recall there being some cases where reads did _not_ succeed but writes _did_ if one endpoints was not configured properly.

I haven't seen this yet, but if this happens it should be fixed in the firmware.

[pipiche38]

@manup do you have any estimated plan when the firmware will be released ?
Just because with the new-radio API completed we have move our plugin with deConz compatibility to beta fir now

[puddly]

@manup I just tried flashing your beta build of the Conbee II firmware and the extra endpoints indeed are present. However, I'm unable to get any other firmware to run! The Conbee is unresponsive until I flash this beta firmware again: I can't downgrade to either 0x26780700 or 0x26580700.

Is there any way to clear NVRAM or otherwise reset the Conbee so that I can use it with older firmwares?

[pipiche38]

@manup do you have any update if and when this will be released ?

[pipiche38]

@manup do you have any update if and when this will be released ?

[github-actions]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[pipiche38]

@manup is the firmware released ?