Skip to content
This repository has been archived by the owner on Jun 12, 2023. It is now read-only.

Please help me understand what breaks with this #1901

Open
teodoryantcheff opened this issue Mar 18, 2023 · 3 comments
Open

Please help me understand what breaks with this #1901

teodoryantcheff opened this issue Mar 18, 2023 · 3 comments

Comments

@teodoryantcheff
Copy link

So, I have this Pisces P100 miner that has been working completely ok for the better part of last year. A month ago miner's dashboard started showing blank witness logs, miner height is also blank, it's constantly syncing :

image

image

yet it seems to be mining ok.

Tried clearing blockchain data, restarting, resetting - still the same.

This is what I can see, logging on the miner:

CONTAINER ID        IMAGE                                                   COMMAND                  CREATED             STATUS              PORTS               NAMES
752ab769951f        quay.io/team-helium/miner:miner-arm64_2023.02.07.0_GA   "/opt/miner/bin/mine…"   4 weeks ago         Up 2 weeks                              miner
099a33cf462d        gateway_mfr:arm64                                       "/opt/gateway_mfr/bi…"   19 months ago       Up 3 weeks                              provision
raspberrypi:~#
raspberrypi:~# docker logs miner
Exec: /opt/miner/erts-12.3.2.7/bin/erlexec -noinput +Bd -boot /opt/miner/releases/2023.02.07.0/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/miner/lib -config /opt/miner/releases/2023.02.07.0/sys.config -args_file /opt/miner/releases/2023.02.07.0/vm.args -- foreground
Root: /opt/miner
/opt/miner
dets: file "/var/data/blockchain_swarm/cache.dets" not properly closed, repairing ...
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Exec: /opt/miner/erts-12.3.2.7/bin/erlexec -noinput +Bd -boot /opt/miner/releases/2023.02.07.0/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/miner/lib -config /opt/miner/releases/2023.02.07.0/sys.config -args_file /opt/miner/releases/2023.02.07.0/vm.args -- foreground
Root: /opt/miner
/opt/miner
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Exec: /opt/miner/erts-12.3.2.7/bin/erlexec -noinput +Bd -boot /opt/miner/releases/2023.02.07.0/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/miner/lib -config /opt/miner/releases/2023.02.07.0/sys.config -args_file /opt/miner/releases/2023.02.07.0/vm.args -- foreground
Root: /opt/miner
/opt/miner
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Exec: /opt/miner/erts-12.3.2.7/bin/erlexec -noinput +Bd -boot /opt/miner/releases/2023.02.07.0/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/miner/lib -config /opt/miner/releases/2023.02.07.0/sys.config -args_file /opt/miner/releases/2023.02.07.0/vm.args -- foreground
Root: /opt/miner
/opt/miner
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Exec: /opt/miner/erts-12.3.2.7/bin/erlexec -noinput +Bd -boot /opt/miner/releases/2023.02.07.0/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/miner/lib -config /opt/miner/releases/2023.02.07.0/sys.config -args_file /opt/miner/releases/2023.02.07.0/vm.args -- foreground
Root: /opt/miner
/opt/miner
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Exec: /opt/miner/erts-12.3.2.7/bin/erlexec -noinput +Bd -boot /opt/miner/releases/2023.02.07.0/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/miner/lib -config /opt/miner/releases/2023.02.07.0/sys.config -args_file /opt/miner/releases/2023.02.07.0/vm.args -- foreground
Root: /opt/miner
/opt/miner
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Exec: /opt/miner/erts-12.3.2.7/bin/erlexec -noinput +Bd -boot /opt/miner/releases/2023.02.07.0/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/miner/lib -config /opt/miner/releases/2023.02.07.0/sys.config -args_file /opt/miner/releases/2023.02.07.0/vm.args -- foreground
Root: /opt/miner
/opt/miner
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Exec: /opt/miner/erts-12.3.2.7/bin/erlexec -noinput +Bd -boot /opt/miner/releases/2023.02.07.0/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/miner/lib -config /opt/miner/releases/2023.02.07.0/sys.config -args_file /opt/miner/releases/2023.02.07.0/vm.args -- foreground
Root: /opt/miner
/opt/miner
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Exec: /opt/miner/erts-12.3.2.7/bin/erlexec -noinput +Bd -boot /opt/miner/releases/2023.02.07.0/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/miner/lib -config /opt/miner/releases/2023.02.07.0/sys.config -args_file /opt/miner/releases/2023.02.07.0/vm.args -- foreground
Root: /opt/miner
/opt/miner
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Exec: /opt/miner/erts-12.3.2.7/bin/erlexec -noinput +Bd -boot /opt/miner/releases/2023.02.07.0/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/miner/lib -config /opt/miner/releases/2023.02.07.0/sys.config -args_file /opt/miner/releases/2023.02.07.0/vm.args -- foreground
Root: /opt/miner
/opt/miner
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Exec: /opt/miner/erts-12.3.2.7/bin/erlexec -noinput +Bd -boot /opt/miner/releases/2023.02.07.0/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/miner/lib -config /opt/miner/releases/2023.02.07.0/sys.config -args_file /opt/miner/releases/2023.02.07.0/vm.args -- foreground
Root: /opt/miner
/opt/miner
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Exec: /opt/miner/erts-12.3.2.7/bin/erlexec -noinput +Bd -boot /opt/miner/releases/2023.02.07.0/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/miner/lib -config /opt/miner/releases/2023.02.07.0/sys.config -args_file /opt/miner/releases/2023.02.07.0/vm.args -- foreground
Root: /opt/miner
/opt/miner
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
raspberrypi:~#

But I have no idea how to make sense of this log.

Please help me understand what's breaking and advise on how to proceed.

Many thanks in advance.
@teodoryantcheff
Copy link
Author

teodoryantcheff commented Mar 20, 2023 •
edited
Loading

Digging a bit more into this lead me to:

admin@raspberrypi:~ $ sudo docker exec miner miner info height
RPC to '[email protected]' failed: {'EXIT',
                                  {function_clause,
                                   [{blockchain,sync_height,
                                     [undefined],
                                     [{file,"blockchain.erl"},{line,521}]},
                                    {miner_cli_info,get_info,0,
                                     [{file,"miner_cli_info.erl"},{line,100}]},
                                    {miner_cli_info,info_height,3,
                                     [{file,"miner_cli_info.erl"},{line,105}]},
                                    {clique_command,run,1,
                                     [{file,"clique_command.erl"},{line,87}]},
                                    {blockchain_console,command,1,
                                     [{file,"blockchain_console.erl"},
                                      {line,14}]}]}}
admin@raspberrypi:~ $

I also saw lots of warnings in the logs:

admin@raspberrypi:~ $ sudo docker exec miner tail -F /var/data/log/console.log | grep warning
2023-03-20 16:14:36.446 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 16, uri: http://13.37.13.24:8080/, pubkey: 11afuQSrmk52mgxLu91AdtDXbJ9wmqWBUxC3hvjejoXkxEZfPvY, module: router
2023-03-20 16:14:36.746 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 9, uri: http://44.238.156.97:8080/, pubkey: 11w77YQLhgUt8HUJrMtntGGr97RyXmot1ofs5Ct2ELTmbFoYsQa, module: router
2023-03-20 16:14:38.130 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 16, uri: http://13.37.13.24:8080/, pubkey: 11afuQSrmk52mgxLu91AdtDXbJ9wmqWBUxC3hvjejoXkxEZfPvY, module: router
2023-03-20 16:14:38.431 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 9, uri: http://44.238.156.97:8080/, pubkey: 11w77YQLhgUt8HUJrMtntGGr97RyXmot1ofs5Ct2ELTmbFoYsQa, module: router
2023-03-20 16:14:40.902 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 16, uri: http://13.37.13.24:8080/, pubkey: 11afuQSrmk52mgxLu91AdtDXbJ9wmqWBUxC3hvjejoXkxEZfPvY, module: router
2023-03-20 16:14:41.202 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 9, uri: http://44.238.156.97:8080/, pubkey: 11w77YQLhgUt8HUJrMtntGGr97RyXmot1ofs5Ct2ELTmbFoYsQa, module: router
2023-03-20 16:14:42.120 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 16, uri: http://13.37.13.24:8080/, pubkey: 11afuQSrmk52mgxLu91AdtDXbJ9wmqWBUxC3hvjejoXkxEZfPvY, module: router
2023-03-20 16:14:42.421 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 9, uri: http://44.238.156.97:8080/, pubkey: 11w77YQLhgUt8HUJrMtntGGr97RyXmot1ofs5Ct2ELTmbFoYsQa, module: router
2023-03-20 16:14:48.754 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 16, uri: http://13.37.13.24:8080/, pubkey: 11afuQSrmk52mgxLu91AdtDXbJ9wmqWBUxC3hvjejoXkxEZfPvY, module: router
2023-03-20 16:14:49.054 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 9, uri: http://44.238.156.97:8080/, pubkey: 11w77YQLhgUt8HUJrMtntGGr97RyXmot1ofs5Ct2ELTmbFoYsQa, module: router
2023-03-20 16:14:51.764 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 16, uri: http://13.37.13.24:8080/, pubkey: 11afuQSrmk52mgxLu91AdtDXbJ9wmqWBUxC3hvjejoXkxEZfPvY, module: router
2023-03-20 16:14:52.065 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 9, uri: http://44.238.156.97:8080/, pubkey: 11w77YQLhgUt8HUJrMtntGGr97RyXmot1ofs5Ct2ELTmbFoYsQa, module: router
2023-03-20 16:15:01.573 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 16, uri: http://13.37.13.24:8080/, pubkey: 11afuQSrmk52mgxLu91AdtDXbJ9wmqWBUxC3hvjejoXkxEZfPvY, module: router
2023-03-20 16:15:01.875 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 9, uri: http://44.238.156.97:8080/, pubkey: 11w77YQLhgUt8HUJrMtntGGr97RyXmot1ofs5Ct2ELTmbFoYsQa, module: router
2023-03-20 16:15:12.919 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 16, uri: http://13.37.13.24:8080/, pubkey: 11afuQSrmk52mgxLu91AdtDXbJ9wmqWBUxC3hvjejoXkxEZfPvY, module: router
2023-03-20 16:15:13.219 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 9, uri: http://44.238.156.97:8080/, pubkey: 11w77YQLhgUt8HUJrMtntGGr97RyXmot1ofs5Ct2ELTmbFoYsQa, module: router
2023-03-20 16:15:13.225 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 16, uri: http://13.37.13.24:8080/, pubkey: 11afuQSrmk52mgxLu91AdtDXbJ9wmqWBUxC3hvjejoXkxEZfPvY, module: router
2023-03-20 16:15:13.565 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 9, uri: http://44.238.156.97:8080/, pubkey: 11w77YQLhgUt8HUJrMtntGGr97RyXmot1ofs5Ct2ELTmbFoYsQa, module: router
2023-03-20 16:15:15.157 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 16, uri: http://13.37.13.24:8080/, pubkey: 11afuQSrmk52mgxLu91AdtDXbJ9wmqWBUxC3hvjejoXkxEZfPvY, module: router
2023-03-20 16:15:15.458 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 9, uri: http://44.238.156.97:8080/, pubkey: 11w77YQLhgUt8HUJrMtntGGr97RyXmot1ofs5Ct2ELTmbFoYsQa, module: router
2023-03-20 16:15:17.709 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 16, uri: http://13.37.13.24:8080/, pubkey: 11afuQSrmk52mgxLu91AdtDXbJ9wmqWBUxC3hvjejoXkxEZfPvY, module: router
2023-03-20 16:15:18.010 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 9, uri: http://44.238.156.97:8080/, pubkey: 11w77YQLhgUt8HUJrMtntGGr97RyXmot1ofs5Ct2ELTmbFoYsQa, module: router
2023-03-20 16:15:23.886 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 16, uri: http://13.37.13.24:8080/, pubkey: 11afuQSrmk52mgxLu91AdtDXbJ9wmqWBUxC3hvjejoXkxEZfPvY, module: router
2023-03-20 16:15:24.187 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 9, uri: http://44.238.156.97:8080/, pubkey: 11w77YQLhgUt8HUJrMtntGGr97RyXmot1ofs5Ct2ELTmbFoYsQa, module: router
2023-03-20 16:15:24.371 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 16, uri: http://13.37.13.24:8080/, pubkey: 11afuQSrmk52mgxLu91AdtDXbJ9wmqWBUxC3hvjejoXkxEZfPvY, module: router
2023-03-20 16:15:24.672 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 9, uri: http://44.238.156.97:8080/, pubkey: 11w77YQLhgUt8HUJrMtntGGr97RyXmot1ofs5Ct2ELTmbFoYsQa, module: router

And the peer list is ...empty:

admin@raspberrypi:~ $ sudo docker exec miner miner peer book -s
+---------------------------------------------+--------------+----------+---------+------+----------+
|                   address                   |     name     |listen_add|connectio| nat  |last_updat|
+---------------------------------------------+--------------+----------+---------+------+----------+
|/p2p/112ECz8yR9Q3QzJJZQyafX7gZzxmx6nzGTYggLVk|able-sangria-c|    0     |    0    |unknow| 64.022s  |
+---------------------------------------------+--------------+----------+---------+------+----------+

admin@raspberrypi:~ $

Trying to list snapshots:

RPC to '[email protected]' failed: {'EXIT',
                                  {function_clause,
                                   [{blockchain,head_hash,
                                     [undefined],
                                     [{file,"blockchain.erl"},{line,448}]},
                                    {blockchain,head_block,1,
                                     [{file,"blockchain.erl"},{line,487}]},
                                    {blockchain,find_last_snapshots,2,
                                     [{file,"blockchain.erl"},{line,2241}]},
                                    {blockchain_cli_snapshot,snapshot_list,3,
                                     [{file,"blockchain_cli_snapshot.erl"},
                                      {line,214}]},
                                    {clique_command,run,1,
                                     [{file,"clique_command.erl"},{line,87}]},
                                    {blockchain_console,command,1,
                                     [{file,"blockchain_console.erl"},
                                      {line,14}]}]}}
admin@raspberrypi:~ $

Following the maker guide from here, LoRa checks show:

admin@raspberrypi:~ $ sudo docker exec miner tail -f /var/data/log/console.log | grep lora
2023-03-20 16:21:39.791 6 [info] <0.1760.0>@miner_lora_light:handle_udp_packet:{326,5} PUSH_DATA #{<<"rxpk">> => [#{<<"chan">> => 2,<<"codr">> => <<"4/5">>,<<"data">> => <<"gKqq8hCAbgAB2UpLDxOWhV29j9jsLrMj6qNpoAqx9eszgq18MgIc3U9cIzCyGkJDtS425RDq12uaaxk98zXKJQ==">>,<<"datr">> => <<"SF12BW125">>,<<"freq">> => 868.5,<<"lsnr">> => -3.0,<<"modu">> => <<"LORA">>,<<"rfch">> => 1,<<"rssi">> => -106,<<"rssis">> => -110,<<"size">> => 64,<<"stat">> => 1,<<"tmst">> => 3185641417}]} from 12273815315514654720 on 53338
2023-03-20 16:21:40.631 6 [info] <0.1760.0>@miner_lora_light:handle_udp_packet:{350,5} PULL_DATA from 12273815315514654720 on 53338
2023-03-20 16:21:46.897 6 [info] <0.1760.0>@miner_lora_light:handle_udp_packet:{326,5} PUSH_DATA #{<<"rxpk">> => [#{<<"chan">> => 1,<<"codr">> => <<"4/5">>,<<"data">> => <<"QIfnBAKAVAAO9UFWj1hWi2lYv36Buajl/60bs+Lt">>,<<"datr">> => <<"SF12BW125">>,<<"freq">> => 868.3,<<"lsnr">> => -8.0,<<"modu">> => <<"LORA">>,<<"rfch">> => 1,<<"rssi">> => -106,<<"rssis">> => -113,<<"size">> => 30,<<"stat">> => 1,<<"tmst">> => 3192754773}]} from 12273815315514654720 on 53338
2023-03-20 16:21:47.132 6 [info] <0.1760.0>@miner_lora_light:handle_udp_packet:{326,5} PUSH_DATA #{<<"rxpk">> => [#{<<"chan">> => 6,<<"codr">> => <<"4/5">>,<<"data">> => <<"4Ht07Tt7BUx/Y5HjuCtVFC8EDSkYm0vKs+gm1WOMV+EcxbnqmxHeEpsWz1QiXccl3NV3kA==">>,<<"datr">> => <<"SF12BW125">>,<<"freq">> => 867.7,<<"lsnr">> => -1.2,<<"modu">> => <<"LORA">>,<<"rfch">> => 0,<<"rssi">> => -97,<<"rssis">> => -100,<<"size">> => 52,<<"stat">> => 1,<<"tmst">> => 3192988611}]} from 12273815315514654720 on 53338
2023-03-20 16:21:50.632 6 [info] <0.1760.0>@miner_lora_light:handle_udp_packet:{350,5} PULL_DATA from 12273815315514654720 on 53338
2023-03-20 16:22:00.633 6 [info] <0.1760.0>@miner_lora_light:handle_udp_packet:{350,5} PULL_DATA from 12273815315514654720 on 53338
2023-03-20 16:22:00.886 6 [info] <0.1760.0>@miner_lora_light:handle_udp_packet:{326,5} PUSH_DATA #{<<"rxpk">> => [#{<<"chan">> => 0,<<"codr">> => <<"4/5">>,<<"data">> => <<"gB9zEBCAwgABQAEXnBkIMHuG1/qiYUJnVU7zpfiwxuMkcheZcx1J3aYfGM50LUk/rLmAWGUy8b93PM+o9kXJEg==">>,<<"datr">> => <<"SF12BW125">>,<<"freq">> => 868.1,<<"lsnr">> => -8.0,<<"modu">> => <<"LORA">>,<<"rfch">> => 1,<<"rssi">> => -110,<<"rssis">> => -117,<<"size">> => 64,<<"stat">> => 1,<<"tmst">> => 3206742901}]} from 12273815315514654720 on 53338
2023-03-20 16:22:05.109 6 [info] <0.1760.0>@miner_lora_light:handle_udp_packet:{326,5} PUSH_DATA #{<<"rxpk">> => [#{<<"chan">> => 0,<<"codr">> => <<"4/5">>,<<"data">> => <<"AMEorQEAe0kcRFUJFQghWgA+gA0UsJ0=">>,<<"datr">> => <<"SF12BW125">>,<<"freq">> => 868.1,<<"lsnr">> => -9.0,<<"modu">> => <<"LORA">>,<<"rfch">> => 1,<<"rssi">> => -110,<<"rssis">> => -118,<<"size">> => 23,<<"stat">> => 1,<<"tmst">> => 3210966694}]} from 12273815315514654720 on 53338
2023-03-20 16:22:05.832 6 [info] <0.1760.0>@miner_lora_light:handle_udp_packet:{326,5} PUSH_DATA #{<<"stat">> => #{<<"ackr">> => 100.0,<<"dwnb">> => 0,<<"rxfw">> => 6,<<"rxnb">> => 19,<<"rxok">> => 6,<<"time">> => <<"2023-03-20 16:22:05 GMT">>,<<"txnb">> => 0}} from 12273815315514654720 on 53338

@teodoryantcheff
Copy link
Author

The last thing that came to mind is to look at the "gateway" container, which seems to be constantly crashed :

raspberrypi:~# docker logs --since 30m -f provision
=INFO REPORT==== 20-Mar-2023::16:29:25.067585 ===
SIGTERM received - shutting down

=SUPERVISOR REPORT==== 20-Mar-2023::16:29:25.085367 ===
    supervisor: {local,gr_counter_sup}
    errorContext: child_terminated
    reason: killed
    offender: [{pid,<0.464.0>},
               {id,gr_lager_default_tracer_counters},
               {mfargs,{gr_counter,start_link,
                                   [gr_lager_default_tracer_counters]}},
               {restart_type,transient},
               {shutdown,brutal_kill},
               {child_type,worker}]
=SUPERVISOR REPORT==== 20-Mar-2023::16:29:25.086479 ===
    supervisor: {local,gr_param_sup}
    errorContext: child_terminated
    reason: killed
    offender: [{pid,<0.463.0>},
               {id,gr_lager_default_tracer_params},
               {mfargs,{gr_param,start_link,[gr_lager_default_tracer_params]}},
               {restart_type,transient},
               {shutdown,brutal_kill},
               {child_type,worker}]
Exec: /usr/local/lib/erlang/erts-10.7.1/bin/erlexec -noshell -noinput +Bd -boot /opt/gateway_mfr/releases/0.1.0/start -mode embedded -boot_var ERTS_LIB_DIR /usr/local/lib/erlang/lib -config /opt/gateway_mfr/releases/0.1.0/sys.config -args_file /opt/gateway_mfr/releases/0.1.0/vm.args -- foreground
Root: /opt/gateway_mfr
/opt/gateway_mfr

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@teodoryantcheff and others