tetra: Added dynamic log level change option

[PhilipSchmid]

A tetra loglevel subcommand was added, which allows one to change Tetragon's log level dynamically without restarting it.

Supported subcommands:

  get         Prints the current log level
  set         Set the log level

Fixes: #2545

tetra: Added dynamic log level change option

I manually tested it from a Linux and macOS client, issuing all sorts of tetra loglevel * commands. Worked flawlessly. Unfortunately, I don't have a Windows client, so I can't test it from there, but I'd expect the same results as it doesn't depend on local files or binaries.

[netlify]

✅ Deploy Preview for tetragon ready!

Name	Link
🔨 Latest commit	077b523
🔍 Latest deploy log	https://app.netlify.com/sites/tetragon/deploys/66ba3d991759dd0008438236
😎 Deploy Preview	https://deploy-preview-2643--tetragon.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[jrfastab]

Like the idea started reviewing before I noticed it was draft. But would be a good addition thanks for working on it.

[PhilipSchmid]

Added minimal doc:

[mtardy]

Hey, sorry I have not reviewed that until now.

As we discussed offline I have no strong feeling over that and thought that it was maybe a bit overkill to dedicate 3 RPCs for this, I would see how we could extend that in the future to do more debugging things:

• enabling gops
• enable pprof

Maybe we could factor all that into SetDebug and have more options for evolution so that we don't need many calls. Not sure Reset or Get is particularly needed but maybe I'm missing something.

So to recap my opinion would be to reduce this to 1 RPC that could be extended. But honestly, I'm not sure that's better so that would be cool to have other peoples' opinion or just merge this if it's satisfying to someone. It would be cool to add a test though!

Overall the PR is great and thanks for adding docs that's amazing.

[PhilipSchmid]

@mtardy Thanks a lot for your valuable input!

You're right. Adding three different gRPC endpoints solely for this is an overkill 😅 . Let's make it a bit more generic so that, e.g., gops and pprof could also be added in the future.
Furthermore, I'd say we can reduce that to two generic gRPC endpoints. Let's maybe call them GetDebug and SetDebug?

How about something like this:

// For now, we only want to support debug-related config flags to be configurable.
enum ConfigFlag {
	CONFIG_FLAG_LOG_LEVEL = 0;
}

enum LogLevel {
	LOG_LEVEL_PANIC = 0;
	LOG_LEVEL_FATAL = 1;
	LOG_LEVEL_ERROR = 2;
	LOG_LEVEL_WARN = 3;
	LOG_LEVEL_INFO = 4;
	LOG_LEVEL_DEBUG = 5;
	LOG_LEVEL_TRACE = 6;
}

message GetDebugRequest{
	ConfigFlag flag = 1;
}
message GetDebugResponse{
	ConfigFlag flag = 1;
	LogLevel level = 2;
}

message SetDebugRequest{
	ConfigFlag flag = 1;
	LogLevel level = 2;
}
message SetDebugResponse{
	ConfigFlag flag = 1;
	LogLevel level = 2;
}

service FineGuidanceSensors {
    ...
    rpc GetDebug(GetDebugRequest) returns (GetDebugResponse) {}
    rpc SetDebug(SetDebugRequest) returns (SetDebugResponse) {}

This could then be used within server.go like this:

func (s *Server) GetDebug(_ context.Context, req *tetragon.GetDebugRequest) (*tetragon.GetDebugResponse, error) {
	switch req.GetFlag() {
	case tetragon.ConfigFlag_CONFIG_FLAG_LOG_LEVEL:
		logger.GetLogger().Debugf("Client requested current log level: %s", logger.GetLogLevel().String())
		return &tetragon.GetDebugResponse{
			Flag:  tetragon.ConfigFlag_CONFIG_FLAG_LOG_LEVEL,
			Level: tetragon.LogLevel(logger.GetLogLevel()),
		}, nil
	default:
		logger.GetLogger().WithField("request", req).Warnf("Client requested unknown config flag %d", req.GetFlag())
		return nil, fmt.Errorf("client requested unknown config flag %d", req.GetFlag())
	}
}

func (s *Server) SetDebug(_ context.Context, req *tetragon.SetDebugRequest) (*tetragon.SetDebugResponse, error) {
	switch req.GetFlag() {
	case tetragon.ConfigFlag_CONFIG_FLAG_LOG_LEVEL:
		currentLogLevel := logger.GetLogLevel()
		changedLogLevel := logrus.Level(req.GetLevel())
		logger.SetLogLevel(changedLogLevel)
		logger.GetLogger().WithField("request", req).Warnf("Log level changed from %s to %s", currentLogLevel, changedLogLevel.String())
		return &tetragon.SetDebugResponse{
			Flag:  tetragon.ConfigFlag_CONFIG_FLAG_LOG_LEVEL,
			Level: tetragon.LogLevel(changedLogLevel),
		}, nil
	default:
		logger.GetLogger().WithField("request", req).Warnf("Client requested change of unknown config flag %d", req.GetFlag())
		return nil, fmt.Errorf("client requested change of unknown config flag %d", req.GetFlag())
	}
}

I basically gave this a try via 077b523.

IMO, this approach keeps our two new gRPC endpoints extendable for the future while not overengineering things for now. gops and pprof should be addressed in separate PRs, when required.
Furthermore, we're not directly depending on (complex?) 3rd-party data types like logrus.Level.

[PhilipSchmid]

(Fixed two failed CI tests with the latest two force pushes.)

[PhilipSchmid]

I just added basic tests for server.go. See a0c4b49. I didn't add tests for the tetra part as it would require some Mock setup, which, AFAICT, we currently don't have.

[PhilipSchmid]

Maps to https://github.com/sirupsen/logrus/blob/master/logrus.go#L93-L112

[mtardy]

Thanks, sorry for the delay I was on PTO 🏝️

[mtardy]

Amazing, it seems we have a first candidate to extend this debug RPC with process cache dump, Anastasios was thinking adding a RPC just for that might be overkill and we just have this now here, #2246. Thanks @PhilipSchmid for making this more generic, it seems it will be useful.

[kkourt]

See also: #2876