bgpreader-2.3.0 Timed out FetchRequest in flight

[kuixiang]

we have builded bgpreader-2.3.0 from source code,and exe the command

/usr/local/bin/bgpreader -p routeviews-stream

about 1 minute you will meet the error about "Timed out ".

U|A|1708910793.689293|routeviews-stream|is-cc-bmp1|amsix|78.142.195.220|9002|80.249.209.216|24.152.116.0/23|80.249.209.216|6447 9002 37468 269194 270497 270497 270497 270497 270497|270497|||
U|A|1708910793.689299|routeviews-stream|is-cc-bmp1|amsix|78.142.195.220|9002|80.249.209.216|170.231.208.0/22|80.249.209.216|6447 9002 37468 267420|267420|||
U|A|1708910793.689305|routeviews-stream|is-cc-bmp1|amsix|78.142.195.220|9002|80.249.209.216|186.209.222.0/23|80.249.209.216|6447 9002 37468 267420|267420|||
%5|1708910939.065|REQTMOUT|rdkafka#consumer-1| [thrd:stream.routeviews.org:9092/bootstrap]: stream.routeviews.org:9092/0: Timed out FetchRequest in flight (after 61034ms, timeout #0)
%4|1708910939.065|REQTMOUT|rdkafka#consumer-1| [thrd:stream.routeviews.org:9092/bootstrap]: stream.routeviews.org:9092/0: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
2024-02-26 09:28:59 3614571: bs_transport_kafka.c:144: ERROR: Local: Timed out (-185): stream.routeviews.org:9092/0: 1 request(s) timed out: disconnect (after 260168ms in state UP)
%5|1708911000.563|REQTMOUT|rdkafka#consumer-1| [thrd:stream.routeviews.org:9092/bootstrap]: stream.routeviews.org:9092/0: Timed out FetchRequest in flight (after 61038ms, timeout #0)
%4|1708911000.563|REQTMOUT|rdkafka#consumer-1| [thrd:stream.routeviews.org:9092/bootstrap]: stream.routeviews.org:9092/0: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
2024-02-26 09:30:00 3614571: bs_transport_kafka.c:144: ERROR: Local: Timed out (-185): stream.routeviews.org:9092/0: 1 request(s) timed out: disconnect (after 61047ms in state UP)

[hanskuhn]

Hi,

I tried the same command and get a different result. I get a connection failed error from broker.bgpstream.caida.org.

This is the metadata service that's hosted at CAIDA that brokers the connection to stream.routeviews.org, which is the kafka server that provides the raw data feed.

stream.routeviews.org is currently working fine. If you can provide an accurate time and date for a time when you see the error you reported, I can look in the logs to see if there's further information.

@dandersen515 Are you the right person to look at broker.caida to see if it's performing nominally?

U|A|1708980810.213379|routeviews-stream|is-cc-bmp1|route-views4|128.223.51.15|24482|2405:4200:100::15|2a0d:2681:2600::/40|2405:4200:100::15|6447 24482 20473 210842|210842|20473:23 20473:4000 24115:20473 24115:65012 24482:2 24482:200 24482:11000 24482:11030 24482:11031 24482:65122||
U|A|1708980810.213383|routeviews-stream|is-cc-bmp1|route-views4|128.223.51.15|24482|2405:4200:100::15|2a0d:2681:600::/40|2405:4200:100::15|6447 24482 20473 210842|210842|20473:23 20473:4000 24115:20473 24115:65012 24482:2 24482:200 24482:11000 24482:11030 24482:11031 24482:65122||
U|A|1708980810.213395|routeviews-stream|is-cc-bmp1|route-views4|128.223.51.15|24482|2405:4200:100::15|2a0d:2681:4300::/40|2405:4200:100::15|6447 24482 20473 210842|210842|20473:23 20473:4000 24115:20473 24115:65012 24482:2 24482:200 24482:11000 24482:11030 24482:11031 24482:65122||
1708980811 HTTP ERROR: SSL connect error (35)
2024-02-26 12:53:31 14945: bsdi_broker.c:966: ERROR: Could not open https://broker.bgpstream.caida.org/v2/data?projects[]=routeviews-stream&intervals[]=1708980791,0&dataAddedSince=1708980780 for reading
2024-02-26 12:53:31 14945: bsdi_broker.c:951: WARNING: WARN: Broker request failed, waiting 1s before retry
1708980812 HTTP ERROR: SSL connect error (35)
2024-02-26 12:53:32 14945: bsdi_broker.c:966: ERROR: Could not open https://broker.bgpstream.caida.org/v2/data?projects[]=routeviews-stream&intervals[]=1708980791,0&dataAddedSince=1708980780 for reading
2024-02-26 12:53:32 14945: bsdi_broker.c:951: WARNING: WARN: Broker request failed, waiting 2s before retry
1708980814 HTTP ERROR: SSL connect error (35)
2024-02-26 12:53:34 14945: bsdi_broker.c:966: ERROR: Could not open https://broker.bgpstream.caida.org/v2/data?projects[]=routeviews-stream&intervals[]=1708980791,0&dataAddedSince=1708980780 for reading
2024-02-26 12:53:34 14945: bsdi_broker.c:951: WARNING: WARN: Broker request failed, waiting 4s before retry
1708980818 HTTP ERROR: SSL connect error (35)
2024-02-26 12:53:38 14945: bsdi_broker.c:966: ERROR: Could not open https://broker.bgpstream.caida.org/v2/data?projects[]=routeviews-stream&intervals[]=1708980791,0&dataAddedSince=1708980780 for reading
2024-02-26 12:53:38 14945: bsdi_broker.c:951: WARNING: WARN: Broker request failed, waiting 8s before retry
1708980827 HTTP ERROR: SSL connect error (35)

(Timestamps are PST)

[kuixiang]

I just tried it, and now there are no errors. It seems that this broker is not stable.

[mbUSC]

@hanskuhn I have the same problem

[hanskuhn]

@mbUSC if you want to send me an IP address and timestamp and I can look in the logs on the kafka server that provides the raw data.

That said, if the broker service provided by CAIDA is the source of the problem, I might not have any entries b/c the flow is:

client > broker.bgpstream.caida.org > stream.routeviews.org