Too many lora_subnet:parse_netid:Undefined CRASH REPORT

[apelekoudas]

Rewards rate keeps droping, and the logs are fulll of parse_netid:Undefined CRASH REPORT errors like the below.
What is the root cause?
Thanks

2022-07-25 10:47:04.293 6 [info] <0.2159.0>@blockchain_txn:absorb:{593,21} took 78 ms to absorb blockchain_txn_poc_receipts_v2
2022-07-25 10:47:04.304 6 [info] <0.1935.0>@blockchain_txn:absorb:{593,21} took 54 ms to absorb blockchain_txn_poc_receipts_v2
2022-07-25 10:47:04.352 6 [info] <0.2159.0>@blockchain_txn:absorb:{593,21} took 59 ms to absorb blockchain_txn_poc_receipts_v2
2022-07-25 10:47:04.352 6 [info] <0.1935.0>@blockchain_txn:absorb:{593,21} took 48 ms to absorb blockchain_txn_poc_receipts_v2
2022-07-25 10:47:04.375 6 [info] <0.1798.0>@miner_lora_light:handle_udp_packet:{326,5} PUSH_DATA #{<<"rxpk">> => [#{<<"chan">> => 2,<<"codr">> => <<"4/5">>,<<"data">> => <<"QJuAXAGAhGgC3VUVpdqsdif+NFz/Bdn0f5GfekNudmJF8ftle3L/sPIB9VR9rw==">>,<<"datr">> => <<"SF12BW125">>,<<"foff">> => 13149,<<"freq">> => 868.5,<<"jver">> => 1,<<"lsnr">> => -2.8,<<"mid">> => 0,<<"modu">> => <<"LORA">>,<<"rfch">> => 1,<<"rssi">> => -115,<<"rssis">> => -119,<<"size">> => 46,<<"stat">> => 1,<<"time">> => <<"2022-07-25T10:47:04.308361Z">>,<<"tmms">> => 1342781242308,<<"tmst">> => 170774790}]} from 2657445937037915303 on 45594
2022-07-25 10:47:04.381 6 [info] <0.2183.0>@blockchain_state_channels_client:init:{137,5} blockchain_state_channels_client init with #{swarm => <0.1641.0>}  hanlder: blockchain_state_channel_handler
2022-07-25 10:47:04.381 6 [warning] <0.1676.0>@blockchain_event:terminate:{108,5} terminating remove_handler
2022-07-25 10:47:04.382 6 [error] <0.2106.0>@Undefined:Undefined:Undefined gen_server blockchain_state_channels_client terminated with reason: {'module could not be loaded',[{lora_subnet,parse_netid,[22839451],[]},{blockchain_state_channels_client,handle_route_by_netid,6,[{file,"blockchain_state_channels_client.erl"},{line,1134}]},{blockchain_state_channels_client,handle_cast,2,[{file,"blockchain_state_channels_client.erl"},{line,187}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,695}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,771}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
2022-07-25 10:47:04.383 6 [error] <0.2106.0>@lora_subnet:parse_netid:Undefined CRASH REPORT Process blockchain_state_channels_client with 0 neighbours crashed with reason: call to undefined function lora_subnet:parse_netid(22839451)
2022-07-25 10:47:04.384 6 [error] <0.1758.0>@lora_subnet:parse_netid:Undefined Supervisor blockchain_state_channel_sup had child blockchain_state_channels_client started with blockchain_state_channels_client:start_link(#{swarm => <0.1641.0>}) at <0.2106.0> exit with reason call to undefined function lora_subnet:parse_netid(22839451) in context child_terminated
2022-07-25 10:47:04.423 6 [info] <0.1935.0>@blockchain_txn:absorb:{593,21} took 68 ms to absorb blockchain_txn_poc_receipts_v2
2022-07-25 10:47:04.504 6 [info] <0.1935.0>@blockchain_txn:absorb:{593,21} took 78 ms to absorb blockchain_txn_poc_receipts_v2
2022-07-25 10:47:04.591 6 [info] <0.2159.0>@blockchain_txn:absorb:{593,21} took 239 ms to absorb blockchain_txn_poc_receipts_v2
2022-07-25 10:47:04.599 6 [info] <0.1935.0>@blockchain_txn:absorb:{593,21} took 95 ms to absorb blockchain_txn_poc_receipts_v2
2022-07-25 10:47:04.664 6 [info] <0.2159.0>@blockchain_txn:absorb:{593,21} took 72 ms to absorb blockchain_txn_poc_receipts_v2

[carsten0x51h]

Hi everyone, I have the same problem here since a couple of days:

call to undefined function lora_subnet:parse_netid(<some_net_id>)

I am currently running "miner-arm64_2022.07.14.0_GA". Would be great if someone could have a look. Please find below a few more details:

2022-07-25 21:09:55.931 [error] <0.141.1>@lora_subnet:parse_netid Supervisor blockchain_state_channel_sup had child blockchain_state_channels_client started with blockchain_state_channels_client:start_link(#{swarm => <0.1640.0>}) at <0.143.1> exit with reason call to undefined function lora_subnet:parse_netid(638282114) in context child_terminated
2022-07-25 21:09:55.931 [error] <0.143.1>@lora_subnet:parse_netid CRASH REPORT Process blockchain_state_channels_client with 0 neighbours crashed with reason: call to undefined function lora_subnet:parse_netid(638282114)
2022-07-25 21:09:55.930 [error] <0.143.1> gen_server blockchain_state_channels_client terminated with reason: {'module could not be loaded',[{lora_subnet,parse_netid,[638282114],[]},{blockchain_state_channels_client,handle_route_by_netid,6,[{file,"blockchain_state_channels_client.erl"},{line,1134}]},{blockchain_state_channels_client,handle_cast,2,[{file,"blockchain_state_channels_client.erl"},{line,187}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,695}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,771}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
2022-07-25 20:58:32.706 [error] <0.1628.0> Supervisor blockchain_sup had child blockchain_state_channel_sup started with blockchain_state_channel_sup:start_link(["/var/data"]) at <0.20855.0> exit with reason shutdown in context child_terminated
2022-07-25 20:58:32.701 [error] <0.20855.0> Supervisor blockchain_state_channel_sup had child blockchain_state_channels_client started with blockchain_state_channels_client:start_link(#{swarm => <0.1640.0>}) at <0.136.1> exit with reason reached_max_restart_intensity in context shutdown
2022-07-25 20:58:32.700 [error] <0.20855.0>@lora_subnet:parse_netid Supervisor blockchain_state_channel_sup had child blockchain_state_channels_client started with blockchain_state_channels_client:start_link(#{swarm => <0.1640.0>}) at <0.136.1> exit with reason call to undefined function lora_subnet:parse_netid(155006429) in context child_terminated
2022-07-25 20:58:32.700 [error] <0.136.1>@lora_subnet:parse_netid CRASH REPORT Process blockchain_state_channels_client with 0 neighbours crashed with reason: call to undefined function lora_subnet:parse_netid(155006429)
2022-07-25 20:58:32.700 [error] <0.136.1> gen_server blockchain_state_channels_client terminated with reason: {'module could not be loaded',[{lora_subnet,parse_netid,[155006429],[]},{blockchain_state_channels_client,handle_route_by_netid,6,[{file,"blockchain_state_channels_client.erl"},{line,1134}]},{blockchain_state_channels_client,handle_cast,2,[{file,"blockchain_state_channels_client.erl"},{line,187}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,695}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,771}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
2022-07-25 20:58:27.700 [error] <0.20855.0>@lora_subnet:parse_netid Supervisor blockchain_state_channel_sup had child blockchain_state_channels_client started with blockchain_state_channels_client:start_link(#{swarm => <0.1640.0>}) at <0.134.1> exit with reason call to undefined function lora_subnet:parse_netid(155006429) in context child_terminated
2022-07-25 20:58:27.699 [error] <0.134.1>@lora_subnet:parse_netid CRASH REPORT Process blockchain_state_channels_client with 0 neighbours crashed with reason: call to undefined function lora_subnet:parse_netid(155006429)
2022-07-25 20:58:27.699 [error] <0.134.1> gen_server blockchain_state_channels_client terminated with reason: {'module could not be loaded',[{lora_subnet,parse_netid,[155006429],[]},{blockchain_state_channels_client,handle_route_by_netid,6,[{file,"blockchain_state_channels_client.erl"},{line,1134}]},{blockchain_state_channels_client,handle_cast,2,[{file,"blockchain_state_channels_client.erl"},{line,187}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,695}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,771}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
2022-07-25 20:58:14.077 [error] <0.20855.0>@lora_subnet:parse_netid Supervisor blockchain_state_channel_sup had child blockchain_state_channels_client started with blockchain_state_channels_client:start_link(#{swarm => <0.1640.0>}) at <0.30921.0> exit with reason call to undefined function lora_subnet:parse_netid(155006429) in context child_terminated
2022-07-25 20:58:14.076 [error] <0.30921.0>@lora_subnet:parse_netid CRASH REPORT Process blockchain_state_channels_client with 0 neighbours crashed with reason: call to undefined function lora_subnet:parse_netid(155006429)
2022-07-25 20:58:14.076 [error] <0.30921.0> gen_server blockchain_state_channels_client terminated with reason: {'module could not be loaded',[{lora_subnet,parse_netid,[155006429],[]},{blockchain_state_channels_client,handle_route_by_netid,6,[{file,"blockchain_state_channels_client.erl"},{line,1134}]},{blockchain_state_channels_client,handle_cast,2,[{file,"blockchain_state_channels_client.erl"},{line,187}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,695}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,771}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}

[dpezely]

The lora library call that's currently undefined seems to be invoked from this PR:
helium/blockchain-core#1412

Since that feature requires a future chain-var that has been put on hold, we can revert the feature by reversing its diff:

wget https://github.com/helium/blockchain-core/pull/1412.diff
patch -p1 --reverse < 1412.diff

Later, we can track why that library call is no longer available.