Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Long list of false (?) positives #286

Open
Mi605 opened this issue Nov 1, 2023 · 10 comments
Open

Long list of false (?) positives #286

Mi605 opened this issue Nov 1, 2023 · 10 comments
Milestone

Comments

@Mi605
Copy link

Mi605 commented Nov 1, 2023

System just rebooted, after completely apt-upgraded. Still needrestart presents me a long list of processes which it considers to be in need of a restart:

$ needrestart -v
[main] eval /etc/needrestart/needrestart.conf
[main] needrestart v3.6
[main] running in user mode
[Core] Using UI 'NeedRestart::UI::stdio'...
[main] #7505 uses obsolete /usr/bin/bash
[main] #7565 uses obsolete /usr/bin/icewm-session
[main] #7571 uses obsolete /usr/bin/icewm
[main] #7580 uses obsolete /usr/bin/zzzfm
[main] #7621 uses obsolete /usr/bin/bash
[main] #7629 uses obsolete /usr/bin/bash
[main] #7634 uses obsolete /usr/bin/conky
[main] #7637 uses obsolete /usr/bin/bash
[main] #7642 uses obsolete /usr/bin/fbxkb
[main] #7645 uses obsolete /usr/bin/bash
[main] #7655 uses obsolete /usr/bin/wireplumber
[main] #7656 uses obsolete /usr/bin/pipewire
[main] #7657 uses obsolete /usr/bin/pipewire
[main] #7681 uses obsolete /usr/bin/bash
[main] #7722 uses obsolete /usr/bin/dbus-launch
[main] #7724 uses obsolete /usr/bin/dbus-daemon
[main] #7906 uses obsolete /usr/bin/volumeicon
[main] #8079 uses obsolete /usr/lib/firefox-esr/firefox-esr
[main] #8748 uses obsolete /usr/bin/bash
[main] #8801 uses obsolete /usr/bin/bash
[main] #8813 uses obsolete /usr/bin/mpv
[main] #9735 uses obsolete /usr/bin/gtkdialog
[main] #10002 uses obsolete /usr/bin/dash
[main] #10003 uses obsolete /usr/bin/dash
[main] #73825 uses obsolete /usr/bin/roxterm
Veraltete Prozesse:
aCSTV.sh[8748], conky[7634], dbus-daemon[7724], dbus-launch[7722],
 desktop-session[7505], devmon[7681], fbxkb[7642], firefox-esr[8079],
 gtkdialog[9735], icewm[7571], icewm-session[7565], mpv[8801, 8813],
 pipewire[7656], pipewire-pulse[7657], pipewire-start[7637, 7629, 7645],
 roxterm[73825], sh[10002, 10003], startup[7621], volumeicon[7906],
 wireplumber[7655], zzzfm[7580]

Nothing has been upgraded after the reboot, so in my understanding there should not show up any processes in need to be restarted.

Actually all the programs involved are present in most recent version and up to date when asking apt. Just for example one of them:

$ apt-cache policy firefox-esr
firefox-esr:
  Installiert:           115.4.0esr-1~deb12u1
  Installationskandidat: 115.4.0esr-1~deb12u1
  Versionstabelle:
 *** 115.4.0esr-1~deb12u1 500
        500 http://security.debian.org bookworm-security/main amd64 Packages
        100 /var/lib/dpkg/status
     102.15.1esr-1~deb12u1 500
        500 http://ftp.de.debian.org/debian bookworm/main amd64 Packages

And I made sure actually this version of firefox is running (By the way, there isn't any other version installed)

Still, after restarting firefox (closing and starting it again after I made sure it actually was no longer to be found in process list htop) needrestart again marks it to be in need of a restart immediately when checking.

Same goes for all the other processes. After restarting them they are still decried as obsolete and in need of a restart.

So I'm a bit confused now. I should add I do use needrestart the first time, so I can't tell apart whether this is an expected and normal behaviour of needrestart. Intuitively I'd say there is something wrong in the output.

System details:
antiX 23 runit 64-bit full

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 12 (bookworm)
Release:	12
Codename:	bookworm

$ uname -r
6.1.55-antix.1-amd64-smp

Hardware:

Graphics: AMD Cedar [Radeon HD 5000/6000/7350/8350 Series] vendor: ASUSTeK driver: radeon
Display: x11 server: X.Org v: 1.21.1.7 driver: X: loaded: radeon
CPU: Intel Core i3 540 family: 6 model-id: 0x25 (37) stepping: 5

Please let me know whether you need some more pieces of information about the system or the software installed.

@liske
Copy link
Owner

liske commented Mar 3, 2024

Does this only happen when running needrstart as non-root? Do you have any special filesystem setup?

@liske liske added the moreinfo label Mar 3, 2024
@Mi605
Copy link
Author

Mi605 commented Mar 9, 2024

Right now I have needrestart not installed any longer due to these false positives with which it came up constantly, asking e.g. to restart samba server again and again without any obvious need. And unfortunately I can't install it on the productive machine, no test system available currently . Sorry. So I can merely state from my recollection.

  • the same false results came up when running needrestart as root or sudoed.
  • this is a default ext4 file system.

Single peculiarity might be, this is a live system, run from a static linuxfs container file (which allows system updates on the fly and gets remastered for making them permanent). I have no means to check whether needrestart would conjure the same false result on a hard disk install.

$ mount | grep 'on / '
overlay on / type overlay (rw,relatime,lowerdir=/live/linux,upperdir=/live/aufs-ram/upper,workdir=/live/aufs-ram/work)

Does needrestart access the file system in an irregular way, what could explain the misbehaviour? When accessing the files in the system by the default methods (from within file manager or from console, including stat, ls, which or dpkg-query and apt) or using any other program or tool no issues with the overlay can be observed. All the files and folders can be accessed normally.

If so , then for reproducing the issue it might be worth to run the live ISO on a testing system, to understand what could cause the issue needrestart has then with it. It can be obtained from http://antixlinux.com/download/ I encountered the issue on the the 23.1 full runit versions, 32 and 64 bit both (other versions not in use here, probably same result). The live system can be run from e.g. an USB stick or a CD directly, so no lengthy install is needed.

If so, other Distros with Live overlays such as Knoppix might see the same issues with needrestart.

Some file system related output (Please note, system is meanwhile upgraded to antiX 23.1 full runit 64 bit) :

ps -aux returns e.g. for firefox the line (and some more of them, all showing the very same path to the executable) :

$ ps -aux | grep firefox
...
Robin    2015952 15.8  8.5 29190708 692072 ?     Ssl  11:41  12:55 /usr/lib/firefox-esr/firefox-esr
...

And from this output:

$ stat /usr/lib/firefox-esr/firefox-esr
 Datei: /usr/lib/firefox-esr/firefox-esr
 Größe: 691888    	Blöcke: 1352       EA Block: 4096   reguläre Datei
Gerät: 0/22	Inode: 9223372036854782162  Verknüpfungen: 1
Zugriff: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Zugriff: 2024-01-23 21:44:58.000000000 +0100
Modifiziert: 2024-01-23 21:44:58.000000000 +0100
Geändert: 2024-01-23 21:44:58.000000000 +0100
Geburt: -

$ ls -l /usr/lib/firefox-esr/firefox-esr
-rwxr-xr-x 1 root root 691888 23. Jan 21:44 /usr/lib/firefox-esr/firefox-esr

And:

$ which firefox-esr
/usr/bin/firefox-esr

$ ls -l /usr/bin/firefox-esr
lrwxrwxrwx 1 root root 30 23. Jan 21:44 /usr/bin/firefox-esr -> ../lib/firefox-esr/firefox-esr

The redirection is set up by default firefox esr package from the debian repos:

$ apt-cache policy firefox-esr
firefox-esr:
  Installiert:           115.8.0esr-1~deb12u1
  Installationskandidat: 115.8.0esr-1~deb12u1
  Versionstabelle:
 *** 115.8.0esr-1~deb12u1 500
        500 http://security.debian.org bookworm-security/main amd64 Packages
     115.7.0esr-1~deb12u1 500
        500 http://ftp.de.debian.org/debian bookworm/main amd64 Packages
        100 /var/lib/dpkg/status

There is nothing obvious what could explain needrestart not being able to access the files properly.

@Mi605
Copy link
Author

Mi605 commented Mar 11, 2024

Does this only happen when running needrstart as non-root?

Took a short scheduled maintenance downtime for an additional testing, installing needrestart temporarily before rebooting the machine from its default live medium again without it. Here the result, so you don't have to rely on my recollection merely:

$ sudo needrestart
Prüfe Prozesse...                                                                                                                                                                                             
Prüfe Kandidaten...                                                                                                                                                                                           
Scanning processor microcode...                                                                                                                                                                                
Prüfe Linux-Kernel...                                                                                                                                                                                         

Der laufende Kernel ist aktuell.

Dienste werden neu gestartet...
Dienste deren Neustart verschoben wurde:
 invoke-rc.d nmbd restart
 invoke-rc.d samba-ad-dc restart
 invoke-rc.d smbd restart

Es müssen keine Container neu gestartet werden.

Nutzer-Sitzungen mit veralteten Prozessen:
 Robin @ /dev/pts/0: bash[1785363], clipit[3732196]
 Robin @ /dev/pts/1: bash[542467]
 Robin @ /dev/pts/10: bash[3753229]
 Robin @ /dev/pts/11: bash[2750456]
 Robin @ /dev/pts/12: bash[2358667]
 Robin @ /dev/pts/13: bash[2128476]
 Robin @ /dev/pts/14: bash[3076838]
 Robin @ /dev/pts/15: htop[1964889]
 Robin @ /dev/pts/16: bash[2712065]
 Robin @ /dev/pts/17: bash[3263517]
 Robin @ /dev/pts/18: bash[1347625]
 Robin @ /dev/pts/19: bash[3573625]
 Robin @ /dev/pts/2: bash[623666], easyeffects[1648686]
 Robin @ /dev/pts/20: bash[3032698]
 Robin @ /dev/pts/3: bash[2719938]
 Robin @ /dev/pts/4: bash[2057043]
 Robin @ /dev/pts/5: bash[3057153]
 Robin @ /dev/pts/6: aCSTV.sh[1240398,1241815], bash[577912], cut[1241818], grep[1241817], gtkdialog[1241816], mpv[153230,153238], sh[149439,149440,150505,150506,153384,153385],
  sleep[2508398,2508399,2508403], tr[1241820], yad[153291,153295]
 Robin @ /dev/pts/7: bash[1474121], urxvt[1964867,1964868]
 Robin @ /dev/pts/8: bash[3735638]
 Robin @ /dev/tty2: bash[3962512]
 root @ /dev/pts/9: bash[2913292]
 root @ /dev/tty1: getty[7107]
 root @ /dev/tty2: login[7108]
 root @ /dev/tty3: getty[7106]
 root @ /dev/tty4: Xorg[484823]

No VM guests are running outdated hypervisor (qemu) binaries on this host.

Here again, nothing was installed after reboot (besides needrestart itself) so no processes can be outdated:

$ sudo apt-get install needrestart
...
Die folgenden NEUEN Pakete werden installiert:
  libintl-perl libmodule-find-perl libmodule-scandeps-perl libproc-processtable-perl libsort-naturally-perl needrestart
0 aktualisiert, 6 neu installiert, 0 zu entfernen und 0 nicht aktualisiert.
...

Would be great if we could use needrestart, since it provides a very much appreciated functionality.

$ apt-cache policy needrestart
needrestart:
  Installiert:           3.6-4+deb12u1
  Installationskandidat: 3.6-4+deb12u1
  Versionstabelle:
 *** 3.6-4+deb12u1 500
        500 http://ftp.de.debian.org/debian bookworm/main amd64 Packages
        500 http://ftp.de.debian.org/debian bookworm/main i386 Packages
        100 /var/lib/dpkg/status

System details: (new system: point release, new kernel, another boot medium to exclude arbitrary hardware issues.)
antiX 23.1 runit full 64 bit
running Live from USB stick, created in Full-featured mode (writable, not dd image mode) by MX/antiX Live USB Maker, ver 23.08.02

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 12 (bookworm)
Release:	12
Codename:	bookworm

$ uname -a
Linux antix1 6.1.60-antix.1-amd64-smp #1 SMP PREEMPT_DYNAMIC Tue Oct 31 01:27:40 EET 2023 x86_64 GNU/Linux

@liske liske added bug and removed moreinfo labels Mar 11, 2024
@liske
Copy link
Owner

liske commented Mar 11, 2024

Thesis: the overflay might be the trigger.

Needrestart checks the device and inode stats info from the filesystem with the values provided in /proc/$PID/maps. I'm going to reproduce and debug it, soon.

@liske liske added this to the v3.7 milestone Mar 11, 2024
@ant9000
Copy link

ant9000 commented May 6, 2024

I have the same problem on one small ARM embedded board with Debian 11.

Following your comment above, I compared the inode from /proc/PID/maps with the one given by stat command: can confirm that they are equal. So, the bug should lie somewhere else.

Here is an excerpt for ntfy, one of the services indicated as needing a restart:
image

If there is any further test I could do, I'm willing to help.

@ant9000
Copy link

ant9000 commented May 6, 2024

Increasing output, it might boil down to some missing feature in my kernel (which is not the standard Debian one):

root@arietta:~# needrestart -v -m a -r l -l
[main] eval /etc/needrestart/needrestart.conf
[main] needrestart v3.5
[main] running in root mode
[Core] Using UI 'NeedRestart::UI::stdio'...
[main] systemd detected
[main] #1 uses obsolete /usr/lib/systemd/systemd
[main] #1 is not a child
[main] #229 uses obsolete /usr/bin/dbus-daemon
[main] #229 is not a child
[main] #232 uses obsolete /usr/lib/systemd/systemd-logind
[main] #232 is not a child
[main] #234 uses obsolete /usr/sbin/wpa_supplicant
[main] #234 is not a child
[main] #298 uses obsolete /usr/sbin/openvpn
[main] #298 is not a child
[main] #375 uses obsolete /usr/sbin/sshd
[main] #375 is not a child
[main] #381 uses obsolete /usr/lib/systemd/systemd
[main] #381 is not a child
[main] #385 uses obsolete /usr/lib/systemd/systemd
[main] #385 is a child of #381
[main] #511 uses obsolete /usr/sbin/agetty
[main] #511 is not a child
[main] #532 uses obsolete /usr/sbin/NetworkManager
[main] #532 is not a child
[main] #1219 uses obsolete /usr/bin/ntfy
[main] #1219 is not a child
[main] #1220 uses obsolete /usr/sbin/agetty
[main] #1220 is not a child
[main] #1226 uses obsolete /usr/lib/systemd/systemd-timesyncd
[main] #1226 is not a child
[main] #1227 uses obsolete /usr/bin/udevadm
[main] #1227 is not a child
[main] #1229 uses obsolete /usr/lib/systemd/systemd-journald
[main] #1229 is not a child
[main] #1230 uses obsolete /usr/sbin/sshd
[main] #1230 is not a child
[main] #1 exe => /usr/lib/systemd/systemd
[main] #1 unexpected cgroup '/init.scope'
[main] trying systemctl status
[main] #1 running /etc/needrestart/hook.d/10-dpkg
dpkg-query: no path found matching pattern /usr/lib/systemd/systemd
[main] #1 running /etc/needrestart/hook.d/20-rpm
[main] #1 running /etc/needrestart/hook.d/30-pacman
[main] #1 running /etc/needrestart/hook.d/90-none
[main] #229 exe => /usr/bin/dbus-daemon
[main] #229 is dbus.service
[main] #232 exe => /usr/lib/systemd/systemd-logind
[main] #232 is systemd-logind.service
[main] #234 exe => /usr/sbin/wpa_supplicant
[main] #234 is wpa_supplicant.service
[main] #298 exe => /usr/sbin/openvpn
[main] #298 is [email protected]
[main] #375 exe => /usr/sbin/sshd
[main] #375 unexpected cgroup '/user.slice/user-0.slice/session-c1.scope'
[main] trying systemctl status
[main] #375 running /etc/needrestart/hook.d/10-dpkg
[main] #375 package: openssh-server
[main] #375 running /etc/needrestart/hook.d/20-rpm
[main] #375 running /etc/needrestart/hook.d/30-pacman
[main] #375 running /etc/needrestart/hook.d/90-none
[main] #381 exe => /usr/lib/systemd/systemd
[main] #381 part of user manager service: uid=0
[main] #511 exe => /usr/sbin/agetty
[main] #511 is [email protected]
[main] #532 exe => /usr/sbin/NetworkManager
[main] #532 is NetworkManager.service
[main] #1219 exe => /usr/bin/ntfy
[main] #1219 is ntfy.service
[main] #1220 exe => /usr/sbin/agetty
[main] #1220 is [email protected]
[main] #1226 exe => /usr/lib/systemd/systemd-timesyncd
[main] #1226 is systemd-timesyncd.service
[main] #1227 exe => /usr/bin/udevadm
[main] #1227 is systemd-udevd.service
[main] #1229 exe => /usr/lib/systemd/systemd-journald
[main] #1229 is systemd-journald.service
[main] #1230 exe => /usr/sbin/sshd
[main] #1230 is ssh.service

Services to be restarted:
 systemctl restart ntfy.service
 systemctl restart [email protected]
 systemctl restart ssh.service
 systemctl restart systemd-journald.service
 systemctl restart systemd-timesyncd.service
 systemctl restart systemd-udevd.service

Service restarts being deferred:
 systemctl restart NetworkManager.service
[main] using restart.d file dbus.service
 /etc/needrestart/restart.d/dbus.service
 systemctl restart [email protected]
 systemctl restart [email protected]
 systemctl restart systemd-logind.service
 systemctl restart wpa_supplicant.service

No containers need to be restarted.

User sessions running outdated binaries:
 root @ user manager service: systemd[381]
[main] run /etc/needrestart/notify.d/200-write
[/etc/needrestart/notify.d/200-write] skip session w/o tty
[main] run /etc/needrestart/notify.d/400-notify-send
[main] run /etc/needrestart/notify.d/600-mail

@liske
Copy link
Owner

liske commented Aug 10, 2024

I've wrote a little script containing most of needrestart's inode validation logic. Could you please give it a try?

The script expects a lines from /proc/$PID/maps, please use the lines reported as outdated for the pid.

#!/usr/bin/env perl

use strict;
use warnings;

while(<>) {
    chomp;
    my ($maddr, $mperm, $moffset, $mdev, $minode, $path) = split(/\s+/, $_, 6);

    print STDERR <<MAPS;
mdev=$mdev
minode=$minode
path=$path
MAPS

    # get on-disk info
    my ($sdev, $sinode) = stat($path);
    my @sdevs = (
	# glibc gnu_dev_* definition from sysmacros.h
	sprintf("%02x:%02x", (($sdev >> 8) & 0xfff) | (($sdev >> 32) & ~0xfff), (($sdev & 0xff) | (($sdev >> 12) & ~0xff))),
	# Traditional definition of major(3) and minor(3)
	sprintf("%02x:%02x", $sdev >> 8, $sdev & 0xff),

	# kFreeBSD: /proc/<pid>/maps does not contain device IDs
	qq(00:00)
    );

    # Don't compare device numbers on anon filesystems
    # w/o a backing device (like OpenVZ's simfs).
    my $major = (($sdev >> 8) & 0xfff) | (($sdev >> 32) & ~0xfff);
    $mdev = "00:00"
	if ($major == 0 || $major == 144 || $major == 145 || $major == 146);

    print STDERR <<EOL;
---
sdev=$sdev
sdevs=@sdevs
sinode=$sinode
major=$major
mdev=$mdev
EOL

    # compare maps content vs. on-disk
    unless($minode eq $sinode && ((grep {$mdev eq $_} @sdevs) ||
                      # BTRFS breaks device ID mapping completely...
                      # ignoring unnamed device IDs for now
                      $mdev =~ /^00:/)) {
	print STDERR "\n=> uses obsolete $path\n";
    }
    else {
	print STDERR "\n=> mapped $path is up-to-date\n";
    }

    print STDERR "\n";
}

@liske liske added the moreinfo label Aug 10, 2024
@liske liske modified the milestones: v3.7, v3.8 Aug 11, 2024
@ant9000
Copy link

ant9000 commented Aug 12, 2024

Hello, thanks a lot for looking into this!

I saved your script as test.pl and executed the following commands:

needrestart -v -m a -r l -l 2>&1 | tee needrestart.log
cat -n /proc/242/maps > proc.242.maps.txt
./test.pl proc.242.maps 2>&1 | tee ntfy.log

Here are the relevant logs:

needrestart.log
ntfy.log
proc.242.maps.txt

@liske
Copy link
Owner

liske commented Aug 12, 2024

Hello, thanks a lot for looking into this!

I saved your script as test.pl and executed the following commands:

needrestart -v -m a -r l -l 2>&1 | tee needrestart.log
cat -n /proc/242/maps > proc.242.maps.txt
./test.pl proc.242.maps 2>&1 | tee ntfy.log

Here are the relevant logs:

needrestart.log ntfy.log proc.242.maps.txt

Can you please remove the -n parameter from the cat command? The line numbering breaks the maps file parsing 🫣

@ant9000
Copy link

ant9000 commented Aug 13, 2024

Ops, sorry!

needrestart -v -m a -r l -l 2>&1 | tee needrestart.log
cat /proc/1616/maps > proc.1616.maps.txt
LC_ALL=C ./test.pl proc.1616.maps.txt 2>&1 | tee ntfy.log

(now ntfy is pid 1616)

needrestart.log
ntfy.log
proc.1616.maps.txt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants