Losing all my witness because unable to send proof of POC

[ruria]

Hi,
I´m missing (almost) all witness, because I am unable to send proof of POC. My logs are flooded of:

@miner_onion_server:send_witness:{207,5} failed to send witness, max retry

I tried with UPnP, but seems to be not detected (at least, this is my understanding). When enabled and reboot my miner, I can see:

2022-04-09 09:57:29.643 6 [warning] <0.1723.0>@libp2p_nat_server:delete_mapping:{192,13} failed to delete port mapping {44158,44158}: no_nat
2022-04-09 09:57:29.643 6 [info] <0.1723.0>@libp2p_nat_server:handle_call:{76,5} using int port 44158 ext port 44158 
2022-04-09 09:57:47.481 6 [info] <0.1723.0>@libp2p_nat_server:handle_call:{87,13} failed to add port ({44158,44158}) mapping with upnp: no_nat

So I´m trying with manual NAT on 44158. It seems to me to be fine:

2022-04-09 10:00:08.058 6 [info] <0.1701.0>@libp2p_transport_tcp:record_observed_addr:{1042,29} received confirmation of observed address /ip4/91.126.175.245/tcp/44158 from peer /p2p/112Z6uNzKbR6KekbAeQCs9VqizoeM7dhKBwZv4xRv92Gm2fqCHEL
2022-04-09 10:00:08.059 6 [info] <0.1701.0>@libp2p_transport_tcp:record_observed_addr:{1046,29} attempting to discover network status for "/ip4/91.126.175.245/tcp/44158" using stungun with "/p2p/112Z6uNzKbR6KekbAeQCs9VqizoeM7dhKBwZv4xRv92Gm2fqCHEL"
2022-04-09 10:00:08.742 6 [info] <0.2008.0>@blockchain_sync_handler:handle_data:{132,5} adding sync blocks [1304206]
2022-04-09 10:00:08.851 6 [notice] <0.2016.0>@libp2p_multistream_server:handle_info:{108,13} Failed to handshake client "/ip4/98.115.134.105/tcp/32918": closed
2022-04-09 10:00:09.130 6 [info] <0.2008.0>@blockchain_sync_handler:handle_data:{142,13} Eagerly re-gossiping 1304206
2022-04-09 10:00:09.283 6 [info] <0.1701.0>@libp2p_transport_tcp:handle_info:{502,21} retrying stungun with peer "/p2p/11T3xwitqSaXafRJPnwvXMWKzWtsqmaugb38Eg6exid5SZZaBCn"
2022-04-09 10:00:09.866 6 [info] <0.1701.0>@libp2p_transport_tcp:handle_info:{612,13} Got dial back confirmation of observed address "/ip4/91.126.175.245/tcp/44158" 

But then:

2022-04-09 10:04:24.786 6 [info] <0.1846.0>@miner_onion_server:decrypt:{362,13} sending witness at RSSI: -119, Frequency: 867.3, SNR: -1.2
2022-04-09 10:04:24.787 6 [info] <0.1846.0>@miner_onion_server:decrypt:{372,13} could not decrypt packet received via radio: treating as a witness
2022-04-09 10:04:24.787 6 [info] <0.2329.0>@miner_onion_server:send_witness:{188,13} sending witness at RSSI: -119, Frequency: 867.3, SNR: -1.2
2022-04-09 10:04:24.896 6 [warning] <0.2329.0>@miner_onion_server:send_witness:{243,37} failed to dial challenger "/p2p/11ZM7qZ3oTNRbB7Z8WFEvpZmZdAzz5fwjfJywKxBxo41AooFdCS": not_found
...
2022-04-09 10:04:54.898 6 [info] <0.2329.0>@miner_onion_server:send_witness:{246,37} re-sending witness at RSSI: -119, Frequency: 867.3, SNR: -1.2
2022-04-09 10:04:55.002 6 [warning] <0.2329.0>@miner_onion_server:send_witness:{243,37} failed to dial challenger "/p2p/11ZM7qZ3oTNRbB7Z8WFEvpZmZdAzz5fwjfJywKxBxo41AooFdCS": not_found
...
2022-04-09 10:05:55.113 6 [info] <0.2329.0>@miner_onion_server:send_witness:{246,37} re-sending witness at RSSI: -119, Frequency: 867.3, SNR: -1.2
2022-04-09 10:05:55.212 6 [warning] <0.2329.0>@miner_onion_server:send_witness:{243,37} failed to dial challenger "/p2p/11ZM7qZ3oTNRbB7Z8WFEvpZmZdAzz5fwjfJywKxBxo41AooFdCS": not_found
...
2022-04-09 10:09:25.854 6 [error] <0.2329.0>@miner_onion_server:send_witness:{207,5} failed to send witness, max retry

This occurs every time, 10 or more times per day. In the last week, just one witness was sent to the chain, all others, "failed to send witness, max retry"

Is it as expected? Is there something I can (re)configure on my side?
My miner is updated to 2022.03.23.1

Thanks a lot!

[serbyxp]

Yeah it’s happening more now then 3 months ago, more miners now then before. And they are unreachable, I’ve spent the day looking at my logs. Well it’s sad, that the network is in this state. I know they don’t care because in 5 days or whatever the light hot spot goes live and all this changes. But even the beacons, are being messed up due to this. All my beacons get sent to me , but the challanger doesn’t receive any and I don’t get the reward for it. I know this is happening across the network. Hopefully the challanges coming from validators fixes this with light hotspot. But even with my fix script it’s failing to dial like 70% of my witness receipts.

[onicolaos]

This bug isn't about many miners because it happened somewhere between 27 feb and it coincides with the miner image update of that time. No matter how many seed servers being added the issue remains so it is a bugged release

[eserogta]

Yeah it’s happening more now then 3 months ago, more miners now then before. And they are unreachable, I’ve spent the day looking at my logs. Well it’s sad, that the network is in this state. I know they don’t care because in 5 days or whatever the light hot spot goes live and all this changes. But even the beacons, are being messed up due to this. All my beacons get sent to me , but the challanger doesn’t receive any and I don’t get the reward for it. I know this is happening across the network. Hopefully the challanges coming from validators fixes this with light hotspot. But even with my fix script it’s failing to dial like 70% of my witness receipts.

I got My hotspot 5 days live and he never ever
had broadcasted any beacon since day 1, and I check everything and all seems ok.

[serbyxp]

I don’t know what to tell you about what I don’t know. What I do know is that the epoch change to 975 epoch has an impact on the amount of beacons + the cap. If your manufacture has incorrect / some weird settings in their “firmware” or operating system. And /or hardware batch is bugged on their end. It has nothing to do with the miner image itself, and you can have everything set up accordingly. If theirs something inside the miner itself that you can’t access then , I mean it’s hard to debug the situation. If you bought them from someone second hand or have a “host” situation, god knows what custom software they put on there… if your running DIY software then theirs a slew of issues that can come into play, it’s the same slew of issues manufacture have to overcome. Just because they sent working hardware for approval doesn’t mean it’s the same hardware they sold …

if your witnessing and not beaconing… you atleast know that your connected to the network and your hardware works in RX. But in TX your having issues. If your globa_conf.json file is set up wrong has the wrong frequencies list / ranges set up. If that’s the case then when a challanger sends you a beacon to send your concentrator is going to reject it. Because of the settings in that file.

so if you acquired a miner for AU928 and are using it in US915… your TX frequencies ranges could be incorrect. Even if your receiving RX messages, the TX config could be wrong. If the manufacture didn’t put a script in the miners packer_forwarder container or software , that automatically changes the global_conf or local_conf.json file according to the Regions parameters. That can cause an issue too.

The only way to check to see that is to “tail” all your logs and look or search for “poc found” and go acouple lines under it if it says sending witness that’s a witness if it says something other then sending witness look at that… look for something that says rejected frequency not accepted ( something of the sort ) I don’t have any of those issues at the moment. I did once long time ago. And all you had to change was this in the config to fix it…
US
https://github.com/helium/sx1302_hal/blob/helium/hotspot/packet_forwarder/global_conf.json.sx1250.US915#L19+L21

AU

https://github.com/helium/sx1302_hal/blob/helium/hotspot/packet_forwarder/global_conf.json.sx1250.AU915#L19+21

Eu

https://github.com/helium/sx1302_hal/blob/helium/hotspot/packet_forwarder/global_conf.json.sx1250.EU868#LL19+L21

See difference between AU US EU… if those numbers are not right for your region .
You will get challenged but if the TX pk falls outside the range of min max then it won’t beacon, responding back with a failed response as mentioned above

[meowshka]

@ruria how are things now after the switch to Light Hotspots?
Also, moving this to the Discussions board.

[meowshka]

Didn't hear back, assume the issues is not happening anymore - closing.

[ruria]

Hi,
I had not time to fully review my logs last days, I´m sorry. I would like to do some deep digging on logs before make any assumption!

BTW, some facts:

1. "Failed to send witness, max retry" messages are gone!
2. Very (very) low witness rate last week (maybe is my fault/location, maybe not... I´m not sure about this)
3. My logs are full of crash reports last week
4. My logs are full of "fail to pass request"
   2022-05-17 19:26:40.782 [error] <0.22205.1>@libp2p_stream_relay:handle_server_data:{193,13} fail to pass request {libp2p_relay_bridge_cr_pb,"/p2p/11YMKW8H17kkdk1McgK9G9ZNdvGTVJw5ppPUyaxuHWBuKgCQ4vT","/ip4/107.5.210.145/tcp/44158"} Server not found