wsdd always needs restart.

[kristof-mattei]

Hi all,

OS: Ubuntu 23.04
needsrestart version: 3.6

I am using wsdd: https://packages.ubuntu.com/lunar/wsdd

This service is always marked as having to be restarted and I'd like to find out why that is.

In the case that it always needs to be restarted, is there a way to say: "if wsdd is the only one, don't prompt, just restart"?

Using Systemd!

Thanks!

[liske]

You can add wsdd to the blacklist option (using it's absolute path). To debug why needrestart decides that wsdd should always be restarted you can run needrestart -vrl.

[kristof-mattei]

[main] eval /etc/needrestart/needrestart.conf
[main] needrestart v3.6
[main] running in root mode
[Core] Using UI 'NeedRestart::UI::stdio'...
[main] systemd detected
[Core] #653 is a NeedRestart::Interp::Python
[Python] #653: source=/usr/sbin/wsdd                                               # <---------
[Core] #653 uses obsolete script file(s):                                          # <---------
[Core] #653  /usr/sbin/wsdd                                                        # <---------
[main] #653 is not a child                                                         # <---------
[Core] #718 is a NeedRestart::Interp::Python
[Python] #718: source=/usr/share/unattended-upgrades/unattended-upgrade-shutdown
[Core] #2185 is a NeedRestart::Interp::Python
[Python] #2185: could not get current working directory, skipping
[Core] #3133 is a NeedRestart::Interp::Python
[Python] #3133: could not get current working directory, skipping
[Core] #3953 is a NeedRestart::Interp::Python
[Python] #3953: could not get current working directory, skipping
[Core] #4215 is a NeedRestart::Interp::Python
[Python] #4215: could not get current working directory, skipping
[Core] #4523 is a NeedRestart::Interp::Python
[Python] #4523: could not get current working directory, skipping
[Core] #6301 is a NeedRestart::Interp::Python
[Python] #6301: could not get current working directory, skipping
[Core] #6861 is a NeedRestart::Interp::Python
[Python] #6861: could not get current working directory, skipping
[Core] #7429 is a NeedRestart::Interp::Python
[Python] #7429: could not get current working directory, skipping
[Core] #8353 is a NeedRestart::Interp::Python
[Python] #8353: could not get current working directory, skipping
[Core] #10447 is a NeedRestart::Interp::Python
[Python] #10447: could not get current working directory, skipping
[Core] #10448 is a NeedRestart::Interp::Python
[Python] #10448: could not get current working directory, skipping
[Core] #10449 is a NeedRestart::Interp::Python
[Python] #10449: could not get current working directory, skipping
[Core] #10640 is a NeedRestart::Interp::Python
[Python] #10640: could not get current working directory, skipping
[main] #653 exe => /usr/bin/python3.11                                             # <---------
[Core] #653 is a NeedRestart::Interp::Python                                       # <---------
[Core] #653 source is /usr/sbin/wsdd                                               # <---------
[main] #653 is wsdd.service                                                        # <---------
[ucode] using NeedRestart::uCode::AMD
[ucode] using NeedRestart::uCode::Intel
[uCode/AMD] #0 cpu vendor id mismatch
[uCode/Intel] #0 current revision: 0x00f0
+ + grepiucode_tool -oE --scan-system [^[:space:]]+$

+ sig=0x000506e3
+ [ -r /sys/devices/system/cpu/cpu0/microcode/processor_flags ]
+ cat /sys/devices/system/cpu/cpu0/microcode/processor_flags
+ filter=-s 0x000506e3,0x10
+ test -r /etc/needrestart/iucode.sh
+ . /etc/needrestart/iucode.sh
+ type bsdtar
+ IUCODE_TOOL_EXTRA_OPTIONS=
+ test -r /etc/default/intel-microcode
+ . /etc/default/intel-microcode
+ test  = no
+ [ -r /usr/share/misc/intel-microcode* ]
+ iucode_tool -l -s 0x000506e3,0x10 --ignore-broken -tb /lib/firmware/intel-ucode
+ grep 0x000506e3
[uCode/Intel] #0 available revision: 0x00f0
[Kernel] Linux: kernel release 6.2.0-20-generic, kernel version #20-Ubuntu SMP PREEMPT_DYNAMIC Thu Apr  6 07:48:48 UTC 2023
[Kernel/Linux] /boot/vmlinuz.old => 6.2.0-20-generic (buildd@lcy02-amd64-035) #20-Ubuntu SMP PREEMPT_DYNAMIC Thu Apr  6 07:48:48 UTC 2023 [6.2.0-20-generic]*
[Kernel/Linux] /boot/vmlinuz-6.2.0-20-generic => 6.2.0-20-generic (buildd@lcy02-amd64-035) #20-Ubuntu SMP PREEMPT_DYNAMIC Thu Apr  6 07:48:48 UTC 2023 [6.2.0-20-generic]*
[Kernel/Linux] /boot/vmlinuz => 6.2.0-20-generic (buildd@lcy02-amd64-035) #20-Ubuntu SMP PREEMPT_DYNAMIC Thu Apr  6 07:48:48 UTC 2023 [6.2.0-20-generic]*
[Kernel/Linux] Expected linux version: 6.2.0-20-generic

Running kernel seems to be up-to-date.

The processor microcode seems to be up-to-date.

Services to be restarted:
 systemctl restart wsdd.service

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.

@liske thanks for the reply. I've marked the lines above related to the process. This is after an update, a reboot, checking again for updates (none) and then running the command you have provided.

I'm not sure why it considers that /usr/sbin/wsdd uses obsolete script files.

[rogerdc]

I'm seeing this as well:

$ sudo needrestart -v -r l -m a
[main] eval /etc/needrestart/needrestart.conf
[main] needrestart v3.6
[main] running in root mode
[Core] Using UI 'NeedRestart::UI::stdio'...
[main] systemd detected
[Core] #1037 is a NeedRestart::Interp::Python
[Python] #1037: source=/usr/share/unattended-upgrades/unattended-upgrade-shutdown
[Core] #3014 is a NeedRestart::Interp::Python
[Python] #3014: source=/usr/share/system-config-printer/applet.py
[Core] #3040 is a NeedRestart::Interp::Python
[Python] #3040: source=/usr/share/hplip/systray.py
[Core] #3154 is a NeedRestart::Interp::Python
[Python] #3154: source=/usr/share/hplip/systray.py
[Python] #3154: use cached file list
[Core] #3157 is a NeedRestart::Interp::Python
[Python] #3157: source=/usr/share/hplip/systray.py
[Python] #3157: use cached file list
[Core] #3634 is a NeedRestart::Interp::Python
[Python] #3634: source=/usr/sbin/wsdd
[Core] #3634 uses obsolete script file(s):
[Core] #3634  /usr/sbin/wsdd
[main] #3634 is not a child
[Core] #3661 is a NeedRestart::Interp::Python
[Python] #3661: source=/usr/bin/protonvpn-app
[main] #3634 exe => /usr/bin/python3.11
[Core] #3634 is a NeedRestart::Interp::Python
[Core] #3634 source is /usr/sbin/wsdd
[main] #3634 is wsdd.service
[ucode] using NeedRestart::uCode::AMD
[ucode] using NeedRestart::uCode::Intel
[uCode/AMD] #0 cpu vendor id mismatch
[uCode/Intel] #0 current revision: 0x002f
+ iucode_tool --scan-system
+ grep -oE [^[:space:]]+$
+ sig=0x000206a7
+ [ -r /sys/devices/system/cpu/cpu0/microcode/processor_flags ]
+ cat /sys/devices/system/cpu/cpu0/microcode/processor_flags
+ filter=-s 0x000206a7,0x10
+ test -r /etc/needrestart/iucode.sh
+ . /etc/needrestart/iucode.sh
+ type bsdtar
+ IUCODE_TOOL_EXTRA_OPTIONS=
+ test -r /etc/default/intel-microcode
+ . /etc/default/intel-microcode
+ test  = no
+ [ -r /usr/share/misc/intel-microcode* ]
+ iucode_tool -l -s 0x000206a7,0x10 --ignore-broken -tb /lib/firmware/intel-ucode
+ grep 0x000206a7
[uCode/Intel] #0 available revision: 0x002f
[Kernel] Linux: kernel release 6.6.15-amd64, kernel version #1 SMP PREEMPT_DYNAMIC Debian 6.6.15-2 (2024-02-04)
[Kernel/Linux] /boot/vmlinuz-6.6.15-amd64 => 6.6.15-amd64 ([email protected]) #1 SMP PREEMPT_DYNAMIC Debian 6.6.15-2 (2024-02-04) [6.6.15-amd64]*
[Kernel/Linux] Expected linux version: 6.6.15-amd64

Running kernel seems to be up-to-date.

The processor microcode seems to be up-to-date.

Services to be restarted:
 systemctl restart wsdd.service

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.

needrestart.conf:

$ cat /etc/needrestart/needrestart.conf
# needrestart - Restart daemons after library updates.
#
# This is the configuration file of needrestart. This is perl syntax.
# needrestart uses reasonable default values, you might not need to
# change anything.
#

# Verbosity:
#  0 => quiet
#  1 => normal (default)
#  2 => verbose
#$nrconf{verbosity} = 2;

# Path of the package manager hook scripts.
#$nrconf{hook_d} = '/etc/needrestart/hook.d';

# Path of user notification scripts.
#$nrconf{notify_d} = '/etc/needrestart/notify.d';

# Path of restart scripts.
#$nrconf{restart_d} = '/etc/needrestart/restart.d';

# Disable sending notifications to user sessions running obsolete binaries
# using scripts from $nrconf{notify_d}.
#$nrconf{sendnotify} = 0;

# If needrestart detects systemd it assumes that you use systemd's pam module.
# This allows needrestart to easily detect user session. In case you use
# systemd *without* pam_systemd.so you should set has_pam_systemd to false
# to enable legacy session detection!
#$nrconf{has_pam_systemd} = 0;

# Restart mode: (l)ist only, (i)nteractive or (a)utomatically.
#
# ATTENTION: If needrestart is configured to run in interactive mode but is run
# non-interactive (i.e. unattended-upgrades) it will fallback to list only mode.
#
#$nrconf{restart} = 'i';

# Use preferred UI package.
#$nrconf{ui} = 'NeedRestart::UI::stdio';

# Change default answer to 'no' in (i)nteractive mode.
#$nrconf{defno} = 1;

# Set UI mode to (e)asy or (a)dvanced.
#$nrconf{ui_mode} = 'e';

# Print a combined `systemctl restart` command line for skipped services.
#$nrconf{systemctl_combine} = 1;

# Blacklist binaries (list of regex).
$nrconf{blacklist} = [
    # ignore sudo (not a daemon)
    qr(^/usr/bin/sudo(\.dpkg-new)?$),

    # ignore DHCP clients
    qr(^/sbin/(dhclient|dhcpcd5|pump|udhcpc)(\.dpkg-new)?$),

    # ignore apt-get (Debian Bug#784237)
    qr(^/usr/bin/apt-get(\.dpkg-new)?$),
];

# Blacklist services (list of regex) - USE WITH CARE.
# You should prefer to put services to $nrconf{override_rc} instead.
# Any service listed in $nrconf{blacklist_rc} will be ignored completely!
#$nrconf{blacklist_rc} = [
#];

# Override service default selection (hash of regex).
$nrconf{override_rc} = {
    # DBus
    qr(^dbus) => 0,

    # display managers
    qr(^gdm) => 0,
    qr(^kdm) => 0,
    qr(^nodm) => 0,
    qr(^sddm) => 0,
    qr(^wdm) => 0,
    qr(^xdm) => 0,
    qr(^lightdm) => 0,
    qr(^slim) => 0,
    qr(^lxdm) => 0,

    # networking stuff
    qr(^bird) => 0,
    qr(^network) => 0,
    qr(^NetworkManager) => 0,
    qr(^ModemManager) => 0,
    qr(^wpa_supplicant) => 0,
    qr(^openvpn) => 0,
    qr(^quagga) => 0,
    qr(^frr) => 0,
    qr(^tinc) => 0,
    qr(^(open|free|libre|strong)swan) => 0,
    qr(^bluetooth) => 0,

    # gettys
    qr(^getty@.+\.service) => 0,
    qr(^serial-getty@.+\.service) => 0,

    # systemd --user
    qr(^user@\d+\.service) => 0,

    # misc
    qr(^zfs-fuse) => 0,
    qr(^mythtv-backend) => 0,
    qr(^xendomains) => 0,
    qr(^lxcfs) => 0,
    qr(^libvirt) => 0,
    qr(^virtlogd) => 0,
    qr(^virtlockd) => 0,
    qr(^docker) => 0,

    # systemd stuff
    # (see also Debian Bug#784238 & #784437)
    qr(^emergency\.service$) => 0,
    qr(^rescue\.service$) => 0,
    qr(^elogind) => 0,

    # do not restart oneshot services, see also #862840
    qr(^apt-daily\.service$) => 0,
    qr(^apt-daily-upgrade\.service$) => 0,
    qr(^unattended-upgrades\.service$) => 0,
    # do not restart oneshot services from systemd-cron, see also #917073
    qr(^cron-.*\.service$) => 0,

    # ignore rc-local.service, see #852864
    qr(^rc-local\.service$) => 0,

    # don't restart systemd-logind, see #798097
    qr(^systemd-logind) => 0,
};

# Override container default selection (hash of regex).
$nrconf{override_cont} = {
};

# Disable interpreter scanners.
#$nrconf{interpscan} = 0;

# Ignore script files matching these regexs:
$nrconf{blacklist_interp} = [
    # ignore temporary files
    qr(^/tmp/),
    qr(^/var/),
    qr(^/run/),

];

# Ignore +x mapped files matching one of these regexs:
$nrconf{blacklist_mappings} = [
    # special device paths
    qr(^/(SYSV00000000( \(deleted\))?|drm(\s|$)|dev/)),

    # ignore memfd mappings
    qr(^/memfd:),

    # aio(7) mapping
    qr(^/\[aio\]),

    # Oil Runtime Compiler's JIT files
    qr#/orcexec\.[\w\d]+( \(deleted\))?$#,

    # plasmashell (issue #65)
    qr(/#\d+( \(deleted\))?$),

    # Java Native Access (issues #142 #185)
    qr#/jna\d+\.tmp( \(deleted\))?$#,

    # temporary stuff
    qr#^(/var)?/tmp/#,
    qr#^(/var)?/run/#,
];

# Verify mapped files in filesystem:
# 0 : enabled
# -1: ignore non-existing files, workaround for chroots and broken grsecurity kernels (default)
# 1 : disable check completely, rely on content of maps file only
$nrconf{skip_mapfiles} = -1;

# Enable/disable hints on pending kernel upgrades:
#  1: requires the user to acknowledge pending kernels
#  0: disable kernel checks completely
# -1: print kernel hints to stderr only
#$nrconf{kernelhints} = -1;

# Filter kernel image filenames by regex. This is required on Raspian having
# multiple kernel image variants installed in parallel.
#$nrconf{kernelfilter} = qr(kernel7\.img);

# Enable/disable CPU microcode update hints:
#  1: requires the user to acknowledge pending updates
#  0: disable microcode checks completely
#$nrconf{ucodehints} = 0;

# Nagios Plugin: configure return code use by nagios
# as service status[1].
#
# [1] https://nagios-plugins.org/doc/guidelines.html#AEN78
#
# Default:
#  'nagios-status' => {
#     'sessions' => 1,
#     'services' => 2,
#     'kernel' => 2,
#     'ucode' => 2,
#     'containers' => 1
#  },
#
# Example: to ignore outdated sessions (status OK)
# $nrconf{'nagios-status'}->{sessions} = 0;


# Read additional config snippets.
if(-d q(/etc/needrestart/conf.d)) {
      foreach my $fn (sort </etc/needrestart/conf.d/*.conf>) {
	      print STDERR "$LOGPREF eval $fn\n" if($nrconf{verbosity} > 1);
	      eval do { local(@ARGV, $/) = $fn; <>};
	      die "Error parsing $fn: $@" if($@);
      }
}

No .conf files in /etc/needrestart/conf.d.

# apt show wsdd
Package: wsdd
Version: 2:0.7.0-2.1
Priority: optional
Section: net
Maintainer: Matthew Grant <[email protected]>
Installed-Size: 114 kB
Depends: python3 (>= 3.7)
Homepage: https://github.com/christgau/wsdd
Download-Size: 33.6 kB
APT-Manual-Installed: no
APT-Sources: http://deb.debian.org/debian sid/main amd64 Packages
Description: Python Web Services Discovery Daemon, Windows Net Browsing
[...]

I have opened wsdd always marked as needing restart to attempt to diagnose this from both sides.

[christgau]

@liske My knowledge of Perl is rather limited, if non-existent, but doesn't that line

needrestart/perl/lib/NeedRestart.pm

Line 180 in 280c1ae

if(grep {!defined($_) || $_ > $ps->start + $tolerance} values %files) {

and digging through the other sources of the package mean that a process should be restarted, when the ctime of a (the) source file is more recent than the interpreter process itself?!

@rogerdc, @kristof-mattei You can verify yourself by doing

$ stat /usr/sbin/wsdd  # ("Change" will be the line of interest)
$ ps -o pid,user,args,start ${PID_OF_WSDD}

Maybe something is actually changing/touching the source file while the daemon is running...

[kristof-mattei]

@liske My knowledge of Perl is rather limited, if non-existent, but doesn't that line

needrestart/perl/lib/NeedRestart.pm

Line 180 in 280c1ae

if(grep {!defined($_) || $_ > $ps->start + $tolerance} values %files) {

and digging through the other sources of the package mean that a process should be restarted, when the ctime of a (the) source file is more recent than the interpreter process itself?!

@rogerdc, @kristof-mattei You can verify yourself by doing

$ stat /usr/sbin/wsdd  # ("Change" will be the line of interest)

$ ps -o pid,user,args,start ${PID_OF_WSDD}

Maybe something is actually changing/touching the source file while the daemon is running...

Quick test revealed that %files is a kvp of file path and its change time. The value is undef for key /usr/sbin/wsdd

To be continued.

[kristof-mattei]

Update: code checks the existence of the /proc/$pid/root//usr/sbin/wsdd to get its creation time, and then use that to see if it is before or after the change time of the potentially updated binary.

As that file doesn't exist it is always undef and thus needsrestart marks it as needing a restart. Not sure if it is a fail safe or whether there is other reasoning. Going through the code & history to see if I can find more.

To be continued.

[kristof-mattei]

Update: /usr/sbin/wsdd by default chroots into /run/wsdd which makes /proc/$pid/root to be empty, meaning needsrestart cannot find /proc/$pid/root//usr/sbin/wsdd.

needrestart/ex/needrestart.conf

Line 180 in 280c1ae

# -1: ignore non-existing files, workaround for chroots and broken grsecurity kernels (default)

actually shows the config for this.

For me I have $nrconf{skip_mapfiles} = -1; but that doesn't seem to help.

[christgau]

Update: /usr/sbin/wsdd by default chroots

Nitpicking, but just to be clear: No, it does not. You have to ask for chroot via command line argument. Without looking it up, it might be that the Debian package provides a systemd unit that asks for chroot.

[kristof-mattei]

Update: /usr/sbin/wsdd by default chroots

Nitpicking, but just to be clear: No, it does not. You have to ask for chroot via command line argument. Without looking it up, it might be that the Debian package provides a systemd unit that asks for chroot.

Something in between.

The default wsdd.service from wsdd has the chroot param: https://github.com/christgau/wsdd/blob/4bec7c92f8afb70b9f7d1866797c4edb4121099c/etc/systemd/wsdd.service#L18

[christgau]

The default wsdd.service from wsdd has the chroot param: https://github.com/christgau/wsdd/blob/4bec7c92f8afb70b9f7d1866797c4edb4121099c/etc/systemd/wsdd.service#L18

True. The service file from the Debian package does the same.

[christgau]

For me I have $nrconf{skip_mapfiles} = -1; but that doesn't seem to help.

Searching the code for skip_mapfiles, my conclusion is that is setting is not taken into account for inputs to interpreters

needrestart/perl/lib/NeedRestart.pm

Lines 171 to 174 in 280c1ae

	my $ps = nr_ptable_pid($pid);
	my %files = $interp->files($pid, \%InterpCache);
	foreach my $path (keys %files) {

And the code for getting the files used by the interpreter "blindly" walks the detected files

needrestart/perl/lib/NeedRestart/Interp/Python.pm

Lines 224 to 227 in 280c1ae

	my %ret = map {
	my $stat = nr_stat("/proc/$pid/root/$_");
	$_ => ( defined($stat) ? $stat->{ctime} : undef );
	} keys %files;

I also think the setting "skip_mapfile" is currently not intended to skip files used by chrooted interpreters. So, there appears to be a corner case for interpreted scripts, which run in an empty root, and which is not covered by needrestart at the moment.

[kristof-mattei]

The default wsdd.service from wsdd has the chroot param: https://github.com/christgau/wsdd/blob/4bec7c92f8afb70b9f7d1866797c4edb4121099c/etc/systemd/wsdd.service#L18

True. The service file from the Debian package does the same.

I didn't realize you are the author of wsdd.

[liske]

Update: code checks the existence of the /proc/$pid/root//usr/sbin/wsdd to get its creation time, and then use that to see if it is before or after the change time of the potentially updated binary.

As that file doesn't exist it is always undef and thus needsrestart marks it as needing a restart. Not sure if it is a fail safe or whether there is other reasoning. Going through the code & history to see if I can find more.

To be continued.

needrestart uses the process's current view on the filesystem using the /proc/$PID/root path. This works for "classic" chroot setups handled by init scripts but fails if chroot (or mount namespaces) are managed by the daemon itself and the binary or script files are no more available in this view.