rcl_logging_syslog is alternative logging backend implementation that can be used for ROS 2 application via rcl_logging_interface.
rcl_logging_syslog uses SYSLOG(3) to send the log data to rsyslog a.k.a rocket-fast system for log processing 🚀.
The main objective is that Enabling ROS 2 logging system with Cloud-Native Log Management and Observability.
see overview slide deck for more information.
The logging data is critical especially for entire system observability and status, so that application can alert the administrator or even give the feedback to the system with adjusting the parameter. This importance rises once it comes to robotics and robot application, especially distributed system such as ROS 2 or edge computing because we must be able to specify what went wrong in the 1st place with these logging data.
rsyslog is available in default Ubuntu distribution managed by system service, performative, and many configuration supported including log data pipeline. So that user can choose the logging configuration depending on the application requirement and use case, sometimes file system sink, sometimes forwarding to remote rsyslogd, or even FluentBit.
FluentBit is a Fast Log Processor and Forwarder part of Graduated Fluentd Ecosystem and a CNCF sub-project.
See how it works 🔥
- rsyslog / FluebtBit
rcl_logging_syslog_to_fluent-bit.mp4
- rsyslog / Fluentd / Loki / Grafana
output.mp4
Supported ROS Distribution
| Distribution | Supported | Branch |
|---|---|---|
| Rolling Ridley | ✅ | rolling (Development) |
| Jazzy Jalisco | ✅ | jazzy |
| Iron Irwini | ✅ | iron |
| Humble Hawksbill | ✅ | humble |
- rsyslog installation
rcl_logging_syslog requires rsyslog package, which Ubuntu should have already in default.
But if you are using container, the situation is bit different from host system since there is no system services or rsyslogd is running by default.
In the case of container, we need to install rsyslog packages in the container root file system.
Note
We can enable the container with host system privileges but that is NOT recommended, especially for security.
### Install rsyslog package
apt install rsyslog- create
rosdirectory forrsyslog.
The following commands require root permission.
/var/log/ros is the root directory for rcl_logging_syslog, all logs are expected to be stored under this root directory.
mkdir -p /var/log/ros
chown syslog:adm /var/log/rosWarning
For production, user needs to configure appropriate group if necessary but owner should be syslog. Otherwise rsyslogd may meet the permission problem for the file system.
- rsyslog service check
If you are running the application on host system, check the rsyslog system service.
systemctl status rsyslogIf you are running the application on container environment, start the rsyslogd daemon process.
/usr/sbin/rsyslogd -n -iNONEOr you prefer to use rsyslogd in the host system from the application in the container, you need to bind the /dev/log to the container via -v /dev/log:/dev/log, so that the application running in the container can open the Unix Domain Socket for rsyslogd running in the host system.
In this setting, all rsyslogd configuration needs to be set in the host system.
- fluent-bit installation
if you want to use fluent-bit to forward the logging data from rsyslogd, please follow Fluent Bit Installation Manual to install it.
### Check fluent-bit package is installed
dpkg -l fluent-bit
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-=================================
ii fluent-bit 3.1.6 amd64 Fast data collector for LinuxCurrently we need to build the source code if we want to use alternative logging backend with ROS 2. See more details for ros2/rcl#1178.
Please follow ROS 2 Official Development / Installation to build the rcl_logging_syslog package below.
### Clone the repository under the workspace
cd <YOUR_WORKSPACE>/src
git clone https://github.com/fujitatomoya/rcl_logging_syslog.git
### Build rcl_logging_syslog
export RCL_LOGGING_IMPLEMENTATION=rcl_logging_syslog
colcon build --symlink-install --cmake-clean-cache --packages-select rcl_logging_syslog rclFollowing to the previous build procedure, we need to copy ros2-test.conf to /etc/rsyslog.d/, and restart the rsyslog.
sudo cp ./config/ros2-test.conf /etc/rsyslog.d/Then we can start the colcon test:
This test creates and uses /var/log/ros/rcl_logging_syslog directory in temporary for the test, and it checks the file existence and contents using rcl_logging_syslog.
colcon test --event-handlers console_direct+ --packages-select rcl_logging_syslogSYSLOG(3) is really simple that does not have much interfaces to control on application side, it just writes the log data on rsyslog Unix Domain Socket.
So we need to configure rsyslog how it manages the log message with /etc/rsyslog.conf, for example file system sink and forward the message to fluent-bit.
| environmental variable | default | Note |
|---|---|---|
RCL_LOGGING_SYSLOG_FACILITY |
LOG_LOCAL1 |
syslog facility: either of LOG_CRON, LOG_DAEMON, LOG_SYSLOG, LOG_USER, and LOG_LOCAL0-7. (/etc/rsyslog.conf must be configured accordingly.) |
Examples:
export RCL_LOGGING_SYSLOG_FACILITY="LOG_LOCAL2"See more details for https://www.rsyslog.com/doc/index.html.
Copy ros2-logging.conf to /etc/rsyslog.d/, and replace <FLUENTBIT_IP> with your own IP address where fluent-bit listens on.
sudo cp ./config/ros2-logging.conf /etc/rsyslog.d/Check the configuration file to see if there is no error,
rsyslogd -N1
rsyslogd: version 8.2312.0, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.then, restart the rsyslog system service, or simply restart rsyslog daemon when using container.
### via systemd (not container environment)
sudo systemctl restart rsyslog
systemctl status rsyslog
### container environment
/usr/sbin/rsyslogd -n -iNONEWith above configuration, rsyslogd also sends log data to TCP port 5140 that fluent-bit listens with the following configuration.
Replace <FLUENTBIT_IP> with your own IP address in fluent-bit.conf.
### Create working directory
mkdir fluentbit_ws; cd fluentbit_ws
### Copy the original files to the local directory
cp -rf /etc/fluent-bit/fluent-bit.conf fluent-bit.conf
cp -rf /etc/fluent-bit/parsers.conf parsers.conf
### Modify the configuration as below
cat fluent-bit.conf
[SERVICE]
Flush 1
Parsers_File parsers.conf
[INPUT]
Name syslog
Parser syslog-rfc3164
Listen <FLUENTBIT_IP>
Port 5140
Mode tcp
[OUTPUT]
Name stdout
Match *
# Start fluent-bit
/opt/fluent-bit/bin/fluent-bit --config=./fluent-bit.confLogging messages from ROS 2 application will be sent to rsyslogd and forwarded to fluent-bit if that is configured.
The followings are example output for file system and fluent-bit.
Start ROS 2 demo application:
ros2 run demo_nodes_cpp talker
[INFO] [1724453270.445092186] [talker]: Publishing: 'Hello World: 1'
[INFO] [1724453271.444981999] [talker]: Publishing: 'Hello World: 2'
[INFO] [1724453272.445056310] [talker]: Publishing: 'Hello World: 3'
...Logging files under file system:
With example configuration and template above, file name is <PROGRAMNAME>_<PID>_<YEAR>-<MONTH>-<DATE>.log.
And log data is prefixed with hostname, facility name, log level and program name as well.
root@tomoyafujita:/var/log# find ros
ros
ros/tomoyafujita
ros/tomoyafujita/talker_31348_2024-08-28.log
root@tomoyafujita:/var/log# cat ros/tomoyafujita/talker_31348_2024-08-28.log
Aug 28 11:30:18 tomoyafujita local1.info: talker[31348]: [INFO] [1724869818.620827927] [talker]: Publishing: 'Hello World: 1'
Aug 28 11:30:19 tomoyafujita local1.info: talker[31348]: [INFO] [1724869819.620758249] [talker]: Publishing: 'Hello World: 2'
Aug 28 11:30:20 tomoyafujita local1.info: talker[31348]: [INFO] [1724869820.620777277] [talker]: Publishing: 'Hello World: 3'FluentBit output:
/opt/fluent-bit/bin/fluent-bit --config=./fluent-bit.conf
Fluent Bit v3.1.6
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
______ _ _ ______ _ _ _____ __
| ___| | | | | ___ (_) | |____ |/ |
| |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __ / /`| |
| _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / \ \ | |
| | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /.___/ /_| |_
\_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ \____(_)___/
[2024/08/23 16:02:00] [ info] [fluent bit] version=3.1.6, commit=, pid=1686882
[2024/08/23 16:02:00] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/08/23 16:02:00] [ info] [cmetrics] version=0.9.4
[2024/08/23 16:02:00] [ info] [ctraces ] version=0.5.5
[2024/08/23 16:02:00] [ info] [input:syslog:syslog.0] initializing
[2024/08/23 16:02:00] [ info] [input:syslog:syslog.0] storage_strategy='memory' (memory only)
[2024/08/23 16:02:00] [ info] [in_syslog] TCP server binding 43.135.146.89:5140
[2024/08/23 16:02:00] [ info] [sp] stream processor started
[2024/08/23 16:02:00] [ info] [output:stdout:stdout.0] worker #0 started
...
[0] syslog.0: [[1724844661.000000000, {}], {"pri"=>"142", "time"=>"Aug 28 11:31:01", "host"=>"tomoyafujita", "ident"=>"talker", "pid"=>"31348", "message"=>"[INFO] [1724869861.620769614] [talker]: Publishing: 'Hello World: 44'"}]
[0] syslog.0: [[1724844662.000000000, {}], {"pri"=>"142", "time"=>"Aug 28 11:31:02", "host"=>"tomoyafujita", "ident"=>"talker", "pid"=>"31348", "message"=>"[INFO] [1724869862.620773450] [talker]: Publishing: 'Hello World: 45'"}]
[0] syslog.0: [[1724844663.000000000, {}], {"pri"=>"142", "time"=>"Aug 28 11:31:03", "host"=>"tomoyafujita", "ident"=>"talker", "pid"=>"31348", "message"=>"[INFO] [1724869863.620773763] [talker]: Publishing: 'Hello World: 46'"}]
...