Add plugin for monitoring GPU usage per user #947

htakano · 2018-09-25T07:41:55Z

It's similar to nvidia_gpu_, but shows each users' GPU usage graph.

sumpfralle

Thank you for such a nice awk plugin :)

Your plugin differs from the nvidia_gpu_ plugin at least by structure (using multigraph - yeah!) and probably also by content. Could you please describe the differences in the documentation header (e.g. USAGE)?

For bonus karma: maybe you want to upload example graphs below example-graphs/PLUGIN_NAME-day.png?

sumpfralle · 2018-09-25T11:44:30Z

plugins/gpu/nvidia_gpu_by_user

+	fi
+fi
+
+gpuUSERS=$(clean_fieldname "$(ls /home)" | tr "\n" " ")


This feels slightly dirty to me, since it uses the non-documented behaviour of clean_fieldname with regard to multi-line input. Please use something like the following instead:

find /home -maxdepth 1 -mindepth 1 -type d | while read -r dname; do clean_fieldname "$(basename "$dname")"; done

Btw.: is the underlying assumption (relevant users having a home directory below /home/ that matches their name) a good choice?
How about the following - in order to get at least around the directory/username match requirement?

getent passwd | cut -d : -f 1,6 | grep ":/home/" | cut -d : -f 1

(I cannot judge its portability - maybe you can comment)

In some environments "getent passwd" will not return users in LDAP (or even YP/NIS):

$ getent passwd | grep niclangf $ getent passwd niclangf niclangf:*:7181:10001:niclangf:/home/niclangf:/bin/bash

and in some environments "getent passwd" will return many many thousands of users.

What you find in /home is also very dodgy. Home directories can be mounted many other places in large environments.

So getting a "full catalog" of users is not very reliable or scalable. But if you have either a username or a UID this works

$ getent passwd niclangf niclangf:*:7181:10001:niclangf:/home/niclangf:/bin/bash $ getent passwd 7181 niclangf:*:7181:10001:niclangf:/home/niclangf:/bin/bash

If you want to provide a stable list of users to keep the graphs stable then persist the usernames... but I see I never got around to writing persistence functions for plugin.sh... There are good hints in Munin/Plugin.pm about what environment variables and such to use if you feel wizardly and want to write a file to persist something at least.

So what could be a good and portable approach for requesting "all non-system users" (I think, this is the goal)?
I have the feeling, that we cannot provide a really good and platform neutral implementation with the little effort that is justified for this task in this context. Thus I would be happy with anything that does not feel too dirty (maybe even the original proposal).

There isn't. Just graph for those that have been using the GPU and somehow persist it.

Thank you for good comments.
If there isn't suitable implementation for listing all non-system users,
then I think it is still good for users to be configurable.
The cpubyuser plugin has similar implementation.

sumpfralle · 2018-09-25T11:45:04Z

plugins/gpu/nvidia_gpu_by_user

+ }
+ return j;
+}
+


Please remove trailing whitespace.

Otherwise excellent plugin!

sumpfralle · 2018-09-25T11:46:16Z

plugins/gpu/nvidia_gpu_by_user

+END {
+ if (n < 0) {
+
+  print "No NVIDIA GPUs detected. Exiting."


Please print this to stderr and exit with an error.

sumpfralle · 2018-09-25T11:50:13Z

plugins/gpu/nvidia_gpu_by_user

+
+=head1 NAME
+
+gpubyuser - Plugin to monitor GPU memory usage by user


By convention, this is the full name of the plugin file.

htakano · 2018-09-26T02:27:54Z

I've modified some parts based on your feedback.

sumpfralle · 2018-09-26T09:11:05Z

It looks good to me, now!

Just one question:

Otherwise graph shows users that have been using GPUs only.

Currently gpuUSERS defaults to an empty string. This does not match the above description, or?
Since this plugin supports autoconf it should work with reasonable defaults without manual intervention. Thus I would suggest one of the following approaches:

try to find out, which accounts are using the GPU right now (ideally: persist this list below MUNIN_PLUGSTATE)
try to assemble a reasonable list of possible users (e.g. find /home -mindepth 1 -maxdepth 1 -type d -print0 | xargs -0 -r -n 1 basename)

htakano · 2018-09-27T02:09:22Z

Thank you for nice suggestions!

Otherwise graph shows users that have been using GPUs only.

Currently gpuUSERS defaults to an empty string. This does not match the above description, or?

It was not correct. How about this comment below?

If env.gpuusers is set, graph always shows listed users
(root, hideki in example) whether using GPU or not.
Otherwise, graph shows users that are using the GPU right now only.

try to find out, which accounts are using the GPU right now (ideally: persist this list below MUNIN_PLUGSTATE)
it should work with reasonable defaults without manual intervention. Thus I would suggest one of the following approaches:

For now, no suitable/portable method is found to list all non-system users automatically,
(find directories right under /home or getent would be at risk of listing thousands of accounts)
so I think it's better to be configurable.

sumpfralle · 2018-10-02T01:36:26Z

Otherwise, graph shows users that are using the GPU right now only.

Are you sure, that the plugin currently really does this?
As far as I understand the awk code, it looks like it will print no values at all, if the environment variable is empty. But maybe I misunderstand the code?

For now, no suitable/portable method is found to list all non-system users automatically,
(find directories right under /home or getent would be at risk of listing thousands of accounts)
so I think it's better to be configurable.

That sounds reasonable.
Please just keep in mind, that if the plugin supports autoconf, then it should work without a specific configuration (i.e. the graph should not be empty). I am not sure, whether this is currently its behaviour due to my question above.

sumpfralle · 2019-07-21T21:42:35Z

Maybe you want to comment on my last question?

Thank you!

sumpfralle · 2020-03-26T03:45:29Z

@htakano: ping?

github-actions · 2021-04-17T02:22:29Z

Stale pull request message

sumpfralle · 2021-07-14T20:42:19Z

Ping?

add nvidia_gpu_by_user plugin

614e7ed

htakano changed the title ~~Add plugin for monitoring GPU usage(memory) per user~~ Add plugin for monitoring GPU usage per user Sep 25, 2018

sumpfralle reviewed Sep 25, 2018

View reviewed changes

h-takano added 2 commits September 26, 2018 08:40

Modify nvidia_gpu_by_user plugin

9b9721a

Modify nvidia_gpu_by_user plugin

a6d2787

Modified nvidia_gpu_by_user plugin's usage

f41569c

github-actions bot added the no-pr-activity label Apr 17, 2021

github-actions bot closed this Apr 25, 2021

steveschnepp reopened this Apr 27, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add plugin for monitoring GPU usage per user #947

Add plugin for monitoring GPU usage per user #947

htakano commented Sep 25, 2018

sumpfralle left a comment

sumpfralle Sep 25, 2018

niclan Sep 25, 2018

sumpfralle Sep 25, 2018

niclan Sep 25, 2018

htakano Sep 25, 2018 •

edited

Loading

sumpfralle Sep 25, 2018

niclan Sep 25, 2018

sumpfralle Sep 25, 2018

sumpfralle Sep 25, 2018

htakano commented Sep 26, 2018 •

edited

Loading

sumpfralle commented Sep 26, 2018

htakano commented Sep 27, 2018 •

edited

Loading

sumpfralle commented Oct 2, 2018

sumpfralle commented Jul 21, 2019

sumpfralle commented Mar 26, 2020

github-actions bot commented Apr 17, 2021

sumpfralle commented Jul 14, 2021


		=head1 NAME

		gpubyuser - Plugin to monitor GPU memory usage by user

Add plugin for monitoring GPU usage per user #947

Are you sure you want to change the base?

Add plugin for monitoring GPU usage per user #947

Conversation

htakano commented Sep 25, 2018

sumpfralle left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

htakano Sep 25, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

htakano commented Sep 26, 2018 • edited Loading

sumpfralle commented Sep 26, 2018

htakano commented Sep 27, 2018 • edited Loading

sumpfralle commented Oct 2, 2018

sumpfralle commented Jul 21, 2019

sumpfralle commented Mar 26, 2020

github-actions bot commented Apr 17, 2021

sumpfralle commented Jul 14, 2021

htakano Sep 25, 2018 •

edited

Loading

htakano commented Sep 26, 2018 •

edited

Loading

htakano commented Sep 27, 2018 •

edited

Loading