Improve doc build log notification

[doveye]

Need a better way to pull out errors and warnings from the MkDocs build, so we don't have to read the Jenkins console log to spot issues.

Possible solution: the Jenkins Log Parser plugin can produce a page that pulls out the errors and warnings in an easy-to-read way. It requires a file that defines the strings to look for in the Jenkins console log, and whether they are errors or warnings. This file can be created via the "Create file" step at the start of the job, and referenced later in the job, in a post-build step

It's possible to add notification (email or Slack) too.

Here's a text file used for the Log Parser plugin in a different MkDocs build. We could re-use this:

# Rules used by the Log Parser plugin to look for errors, warnings etc in the console log. This file is specified elsewhere in the build config.
# These rules are case sensitive and create quick access links in the "Parsed Console Output" (see link in left sidebar).
# The job can be set to fail on Error, be unstable on Warning

# Rule format is /<error_type>/<text_to_match>/    For more info, see https://wiki.jenkins.io/display/JENKINS/Log+Parser+Plugin

# Start of mkdocs build section (looks for comment output by shell script) for grouping errors and warnings found after the line.
# also creates a quick access link under "Info"
start /Build: Build mkdocs/

# Git errors, eg the specified branch doesn't exist, or incorrect/old credentials were used (git will report "fatal: xxx")
error /fatal/
# Error if MkDocs reports an ERROR
error /ERROR/
# Error if no file (eg MkDocs output) was transferred over SSH
error /SSH: Transferred 0 file(s)/
# Error if a file can't be found, eg a sed command is run on a file that doesn't exist
error /No such file or directory/

# Warn if MkDocs reports a WARNING (eg if files specified in links don't exist - broken link)
warning /WARNING/
# Warn if MkDocs finds files that aren't in the nav
warning /The following pages exist in the docs directory, but are not/

[SueChaplain]

Adam B has offered to set this up for us if we raise an issue in their queue.

We should collect together all the specific strings we want to report on. Here are some we can think of .....

• ERROR
• The following pages exist in the docs directory, but are not
• WARNING - Documentation file

On this job: https://ci.eclipse.org/openj9/view/Website-Doc/job/Build-Doc-Push_to_ghpages/

To be reported to slack. Standard slack channel used is Eclipse OpenJ9 "jenkins", but would be good if it could also go to one we use more regularly.

[doveye]

Copied into the description an example file containing strings to look for.

[SueChaplain]

Guess we would add this last one:

# Warn if MkDocs finds files in the nav that aren't in the docs dir
warning /WARNING - Documentation file/

[doveye]

I removed stuff that obviously wasn't applicable but left everything else just in case. The git one might still be useful, the SSH one might or might not be depending on what happens with the MkDocs output (I guess it just goes back into github rather than being SSHed about the place though), and not sure whether the build does anything with files (such as using sed) that could cause an error. We definitely want to see all the ERRORs and WARNINGs that MkDocs produces and it's useful to see the start of the actual MkDocs processing because you can skip all the Jenkins output before it.

[SueChaplain]

There are some Warnings that get produced for every build, which would be a pain to see. For example:

17:05:00 Warning: JENKINS-30600: special launcher org.jenkinsci.plugins.docker.workflow.WithContainerStep$Decorator$1@bd37423e; decorates RemoteLauncher[hudson.remoting.Channel@21028a4d:cent7-x64-3] will be ignored (a typical symptom is the Git executable not being run inside a designated container)

There is another too about sh security. I guess we don't see these on our other Jenkins builds, but we won't want to be spammed by them.

[doveye]

Yuck! Fair enough.

[AdamBrousseau]

I have not seen or used the log parser plugin before. It does look interesting but I worry that it is not very actively maintained and released so it may not even work for pipeline jobs.
https://github.com/jenkinsci/log-parser-plugin/releases
Could we try this out on a non-prod build/system first? Even hyc?

We have tested the Build failure analyzer plugin internally.
https://plugins.jenkins.io/build-failure-analyzer/
It works similar to but perhaps more simple than log parser. It will scan the console against the user defined issue list and will list out matching issues on the job's run page. The console will also be highlighted red on matching lines. It does have Slack notifications but we heave't tested it. One caveat is that it is enabled across Jenkins globally and there is one "problem list". This could get messy if more than just docs want to use it. Especially if we enable slack. The other caveat is that it will only scan non-green builds, so if your builds pass and you want to know about warnings then this wouldn't be a suitable solution.

cc @pshipton for awareness and @JasonFengJ9 since he has used it internally for dev builds.

Edit:

There are likely other plugin options too but another option would be to DIY. I don't think it's that much code to get/parse the console at the end of a build, compare against a list, and take action accordingly.

[SueChaplain]

@AdamBrousseau - thanks for the analysis.

It sounds like the Build failure analyzer plugin won't be suitable as our builds pass green but can have warnings.

If you don't want to use the log parser plugin, which I can understand, a DIY approach could work just as well. I guess that is just a post-processing step in the Jenkins job config.

[AdamBrousseau]

I'm not against trying the log parser plugin in a test env, like hyc-runtimes or better yet another internal instance (not openj9).
If we have to do the analysis ourselves it would be in a finally block of the pipeline job code but it shouldn't be that difficult to find keywords and send a Slack message with a summary.