Opensuse sidewinderd service won't auto start on boot

[lurkynerd]

Everything works perfect when started manually but the sidewinderd service won't auto start on boot. I am not sure if it's apparmor issue. I run luks so I commented to wait extra in settings file. No logs show any issues. The pid file was empty that's about it, I added sidewinder process id inside but still no auto start on boot. Anything else I should try?

[tolga9009]

Hi there,

are you using full disk encryption with LUKS or just specific partitions, like your /home partition? The option encrypted_workdir in in the config file is only needed, if there is a possibility sidewinderd is launched, before your working directory becomes available (typically in your /home directory).

If there are specific needs for your boot order, please look into https://wiki.archlinux.org/title/systemd#Replacement_unit_files, which enables you to launch sidewinderd at a later moment, e.g. once your system is fully unlocked. You can edit the sytemd Service file entries After= and Requires= in order to determine the correct moment for your system. To further analyze your boot process, you can use systemd-analyze https://wiki.archlinux.org/title/Improving_performance/Boot_process.

The pid file is expected to be empty. It's just a file, which makes sure only one instance of sidewinderd is running. Usually, that file gets created on launch and deleted on stopping sidewinderd. Under certain conditions (e.g. kernel panic, sudden powerloss), the pid file can remain. Usually, a simple power cycle or manually deleting the file fixes the issue. But usually, it shouldn't lead to the issue you're facing - and it would also log in that case (https://github.com/tolga9009/sidewinderd/blob/master/src/process.cpp#L90).

Please keep us updated. Thanks!

Cheers,
Tolga

[lurkynerd]

Thanks for your wonderful works and all the tips @tolga9009 . I tried to delay start of the service and even do a sleep timer to start it but to no avail. But i did notice that I get this message when trying to start sidewinder manually from terminal: PID file could not be created. This might be the culprit, any idea how to fix it? I tried deleting the pid file but I still get same error.

[lurkynerd]

This is the log when calling sidewinderd from terminal.

Started sidewinderd.
Found device: 045e:0768
Can't open uinput
Can't open hidraw interface
Keyboard Constructor
Error getting HID feature report.
Error setting HID feature report.
Error getting HID feature report.
Error setting HID feature report.
Error getting HID feature report.
Error setting HID feature report.
Error getting HID feature report.
Error setting HID feature report.
Error getting HID feature report.
Error setting HID feature report.
Error getting HID feature report.
Error setting HID feature report.

Strangely, when starting the service manually everything works as intended. It just won't auto start at boot.

[tolga9009]

You're running sidewinderd from the terminal with sudo, right? There should be no difference between sudo systemctl start sidewinderd vs sudo sidewinderd in terms of functionality. It will need to be run as root; once everything is set up, it will drop permissions later on.

I have the feeling, we might have multiple issues at once here, so I'd say focus on what we know and tackle them one by one. First of all, stop and disable automatic sidewinderd startup: sudo systemctl disable --now sidewinderd. Reboot your system for a clean starting point.

Revert your changes to /etc/sidewinderd.conf by simply copying the contents of https://github.com/tolga9009/sidewinderd/blob/master/etc/sidewinderd.conf. Only change the entry user in the configuration file to your desired user. sidewinderd will set up the macros in user's home directory, which by default is root (bad design choice by me btw.). Also, once sidewinderd is done setting up everything, it will drop root permissions and resume with user permissions. For clarification: the program always needs to be started with root permissions (e.g. sudo or via systemd service file).

Now, the first step should be trying to get sidewinderd working from the commandline, i.e. by using sudo sidewinderd.

If this works, reboot for a clean starting point and try using sudo systemctl start sidewinderd. I expect this to work, since it already worked, but still try it just to be sure.

In the meantime, I will try to find the journalctl commands to introspect, what's going on with sidewinderd during automatic startup.

//Edit: the command is sudo journalctl -r -u sidewinderd.service. Once you get sidewinderd working from commandline and by manually starting the systemd service file, you can enable automatic startup again and analyze the output of the journalctl command above.

Of course, if you suspect AppArmor to be an issue, you can try disabling it. I have no in-depth experience with neither openSUSE nor AppArmor, thus can't give you specific directions.

[lurkynerd]

Thank you for helping me investigate this issue. When trying to run sidewinder with sudo I get "sudo: sidewinderd: command not found"

I tried all your resetting advise, started from blank slate and it still refuses to auto start on boot. I just have to manually start the service file on boot. I guess I could simply write a cron to start it. This might also be apparmor issue perhaps. Nothing useful from journalctl logs.

You may close this issue unless someone who uses opensuse wants to chime in.

Thanks for your hard work!

[jayfan0]

systemctl status sidewinderd.service
○ sidewinderd.service - Support for Microsoft SideWinder X4 / X6 and Logitech G105 / G710+
     Loaded: loaded (/etc/systemd/system/sidewinderd.service; enabled; preset: disabled)
     Active: inactive (dead)

Same issue here. Does the disabled preset have anything to do with this? Seems that prevents services starting on boot but I cant figure out how to enable it

[tolga9009]

Weird. I will install openSUSE in a virtual machine later today and try to figure out, if there is something special going on. Which openSUSE version are you using? cat /etc/os-release and uname -r should give you distro info and kernel version.

[jayfan0]

Sure thing. Here you go:

cat /etc/os-release

NAME="openSUSE Tumbleweed"
# VERSION="20240216"
ID="opensuse-tumbleweed"
ID_LIKE="opensuse suse"
VERSION_ID="20240216"
PRETTY_NAME="openSUSE Tumbleweed"
ANSI_COLOR="0;32"
# CPE 2.3 format, boo#1217921
CPE_NAME="cpe:2.3:o:opensuse:tumbleweed:20240216:*:*:*:*:*:*:*"
#CPE 2.2 format
#CPE_NAME="cpe:/o:opensuse:tumbleweed:20240216"
BUG_REPORT_URL="https://bugzilla.opensuse.org"
SUPPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org"
DOCUMENTATION_URL="https://en.opensuse.org/Portal:Tumbleweed"
LOGO="distributor-logo-Tumbleweed"

uname -r

6.7.4-1-default

[jayfan0]

Update: I found this post and gave it a go:

I added Restart=always to the service file and rebooted. Seems to have survived the reboot and loaded up before the desktop environment. Seems like a success but I'll try play around some more today before celebrating

Just to be sure, is this method safe and sensible for a third party service that has been manually installed? (I trust this one, I'm just asking for general good practise). Is this even addressing the root cause?

Full service file:

cat /etc/systemd/system/sidewinderd.service 

[Unit]
Description=Support for Microsoft SideWinder X4 / X6 and Logitech G105 / G710+
After=multi-user.target

[Service]
ExecStart=/usr/local/bin/sidewinderd
Restart=always

[Install]
WantedBy=multi-user.target

Edit: More info:
Similar to Lurkynerd above, sudo sidewinderd doesnt work, but sudo /usr/local/bin/sidewinderd does. Maybe a path issue

[tolga9009]

I guess it's okay as a temporary workaround, but it doesn't solve the root cause. You may also want to look into https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#RestartSec= as per default systemd will invoke a restart every 100ms. I think 5 seconds is a more reasonable value. There is a reason, why sidewinderd is failing in the first place (maybe interfaces not brought up in time, maybe permission issues), but is running fine when a certain amount of time has passed after boot (e.g. Restart=always or manual start)

Unfortunately, I didn't had the time to spin up the VM yet, but there seems to be additional dependencies on openSUSE for sidewinderd to operate properly. Most likely this can be fixed by updating the sidewinderd.service file and identifying the necessary dependencies or handling failures more gracefully in the source code.

Thanks to both of you for reporting this issue, I will take care of it!

//Edit1: the sudo sidewinderd command not found issue is openSUSE specific, due to their defaults for sudo:

## Use this PATH instead of the user's to find commands.
Defaults secure_path="/usr/sbin:/usr/bin:/sbin:/bin"

Per sane default, make install will install to /usr/local/bin, hence sidewinderd command cannot be found. Invoking sidewinderd by the user finds the command, but fails due to missing root permissions. First invoking sudo su to become root and then invoking sidewinderd works though.

These defaults and system settings are outside the scope of this project's source code and are not critical, as you can invoke the absolute path; the systemd service file uses the absolute path anyway. Still, this can be properly addressed in a distro-specific package (.deb/.rpm etc.) - at the moment I only packaged it for Arch Linux.

//Edit2: I was able to reproduce the issue, eventhough I'm not 100% sure, if it really is the same issue. Similar to the notes above, openSUSE somehow seems to ignore systemd files within the /usr/local/lib/systemd/system directory. Similar to @lurkynerd, systemctl start sidewinderd worked, but automatic startup on boot was ignored. I moved /usr/local/bin/sidewinderd to /usr/bin/sidewinderd and the systemd service file from /usr/local/lib/systemd/system/sidewinderd.service to /usr/lib/systemd/system/sidewinderd.service (plus fixed executable path in the systemd service file), everything worked as expected.

Still, I've noticed sidewinderd took around 5 seconds to load after I booted into the desktop environment, so it's possible there are more race conditions on your real systems, than what I was able to reproduce in my virtual environment.

I will further look into this on Monday/Tuesday, just to rule out a generic systemd issue on modern systems, rather than openSUSE specific issue/security measurement. And also to possibly find instructions to make it work form within /usr/local/ directories. Good thing though: this is naturally fixed by deploying openSUSE packages, as they will install everything to /usr instead of /usr/local. This is something I will consider supporting officially (plus more distros)

[lurkynerd]

@tolga9009 amazing detective work and thorough troubleshooting as always. I'm learning a lot about dev just from the way you approach problems.

I moved /usr/local/bin/sidewinderd to /usr/bin/sidewinderd and the systemd service file from /usr/local/lib/systemd/system/sidewinderd.service to /usr/lib/systemd/system/sidewinderd.service (plus fixed executable path in the systemd service file), everything worked as expected.

These same steps is what made sidewinderd finally start successfully on boot thanks to @jayfan0's clever troubleshooting. The service file was in the wrong place and his restart service hack fixed any lingering issues.