Error requeueing or finishing unresponsive job messages filling the journal every 30s for unknown job ID

[miabbott]

Describe the bug

Error messages about "Error requeueing or finishing unresponsive job for an unknown job ID are being written to the journal every 30s. There's no obvious way to correct this error.

Environment

• OS version (/etc/os-release and /etc/redhat-release):

[core@localhost ~]$ cat /etc/os-release 
NAME="Red Hat Enterprise Linux"
VERSION="9.4 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.4"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.4 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.4
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.4"
[core@localhost ~]$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 9.4 (Plow)

• osbuild-composer version (rpm -qi osbuild-composer)

$ rpm -qi osbuild-composer
Name        : osbuild-composer
Version     : 101
Release     : 1.el9
Architecture: x86_64
Install Date: Tue Jun 18 08:06:06 2024
Group       : Unspecified
Size        : 18399
License     : Apache-2.0
Signature   : RSA/SHA256, Tue Mar  5 05:12:56 2024, Key ID 199e2f91fd431d51
Source RPM  : osbuild-composer-101-1.el9.src.rpm
Build Date  : Mon Feb 26 07:26:12 2024
Build Host  : x86-64-05.build.eng.rdu2.redhat.com
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Vendor      : Red Hat, Inc.
URL         : https://github.com/osbuild/osbuild-composer
Summary     : An image building service based on osbuild
Description :

A service for building customized OS artifacts, such as VM images and OSTree
commits, that uses osbuild under the hood. Besides building images for local
usage, it can also upload images directly to cloud.

It is compatible with composer-cli and cockpit-composer clients.

To Reproduce

Not sure how this happened...

Expected behavior

Maybe backoff the 30s timer after a while? Or stop trying to requeue after some time?

Additional context

The RHEL 9.4 host I am running osbuild-composer on had not seen activity since April (about 2 months), so when I started using it again, I had done a dnf update and rebooted the host. Everything seemed to start fine.

I did start a compose and then cancel it, but that cancellation appeared successful. See the list of composes and IDs:

$ sudo composer-cli compose list
ID                                     Status     Blueprint              Version   Type
3ab45919-8024-41a3-89d5-70c7d7dfc891   FINISHED   microshift-installer   0.0.1     edge-installer
45ff9187-c5dd-4c7c-b0a8-83943198c28d   FINISHED   microshift-installer   0.0.2     edge-installer
5ae97253-76f3-4659-a755-5c5deab8d71d   FINISHED   microshift             0.0.2     edge-container
906152c0-30a0-43cf-9dd3-36ee4d0e8fda   FINISHED   microshift             0.0.2     edge-commit
57d3a612-6094-4740-9f61-62321bf4bd0b   FAILED     vanilla-edge           1.0.0     edge-commit

The last compose in the list is the compose that I cancelled.

The contents of the logs:

Jun 18 08:08:37 localhost.localdomain systemd[1]: Started OSBuild Composer.
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="verbose flag is provided for backward compatibility only, current behavior is always printing the access log"
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="Loaded configuration:" func=main.main file="/builddir/build/BUILD/osbuild-composer-101/_build/src/github.com/osbuild/osbuild-composer/cmd/osbuild-composer/main.go:60"
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=warning msg="GLITCHTIP_DSN not configured, skipping initializing Sentry/Glitchtip" func=main.main file="/builddir/build/BUILD/osbuild-composer-101/_build/src/github.com/osbuild/osbuild-composer/cmd/osbuild-composer/main.go:89"
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="log_level = \"info\""
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="log_format = \"text\""
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="dnf-json = \"/usr/libexec/osbuild-depsolve-dnf\""
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="SplunkHost = \"\""
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="SplunkPort = \"\""
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="SplunkToken = \"\""
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="GlitchTipDSN = \"\""
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="[koji]"
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="  ca = \"\""
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="  enable_tls = true"
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="  enable_mtls = true"
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="  enable_jwt = false"
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="  jwt_ca_file = \"\""
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="  jwt_acl_file = \"\""
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="[worker]"
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="  ca = \"\""
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="  request_job_timeout = \"0\""
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="  base_path = \"/api/worker/v1\""
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="  enable_artifacts = true"
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="  pg_host = \"\""
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="  pg_port = \"\""
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="  pg_database = \"\""
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="  pg_user = \"\""
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="  pg_password = \"\""
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="  pg_ssl_mode = \"\""
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="  pg_max_conns = 0"
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="  enable_tls = true"
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="  enable_mtls = true"
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="  enable_jwt = false"
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="  jwt_ca_file = \"\""
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="  jwt_acl_file = \"\""
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="[weldr_api]"
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="  [weldr_api.distros]"
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="    [weldr_api.distros.\"fedora-*\"]"
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="      image_type_denylist = [\"iot-bootable-container\"]"
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="    [weldr_api.distros.\"rhel-*\"]"
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="      image_type_denylist = [\"azure-eap7-rhui\", \"azure-rhui\", \"azure-sap-rhui\", \"ec2\", \"ec2-ha\", \"ec2-sap\", \"gce-rhui\"]"
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="[distro_aliases]"
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="  rhel-7 = \"rhel-7.9\""
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="  rhel-8 = \"rhel-8.10\""
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="  rhel-9 = \"rhel-9.4\""
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:08:37 failed to parse distro ID string, using it as is: error when parsing distro name (rhel-8-beta): parsing major version failed, inner error:
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: strconv.Atoi: parsing "beta": invalid syntax
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:08:37 Loaded repository configuration file: /usr/share/osbuild-composer/repositories/rhel-8-beta.json
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:08:37 Loaded repository configuration file: /usr/share/osbuild-composer/repositories/rhel-8.10.json
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:08:37 Loaded repository configuration file: /usr/share/osbuild-composer/repositories/rhel-8.4.json
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:08:37 Loaded repository configuration file: /usr/share/osbuild-composer/repositories/rhel-8.5.json
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:08:37 Loaded repository configuration file: /usr/share/osbuild-composer/repositories/rhel-8.6.json
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:08:37 Loaded repository configuration file: /usr/share/osbuild-composer/repositories/rhel-8.7.json
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:08:37 Loaded repository configuration file: /usr/share/osbuild-composer/repositories/rhel-8.8.json
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:08:37 Loaded repository configuration file: /usr/share/osbuild-composer/repositories/rhel-8.9.json
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:08:37 Loaded repository configuration file: /usr/share/osbuild-composer/repositories/rhel-8.json
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:08:37 Loaded repository configuration file: /usr/share/osbuild-composer/repositories/rhel-9.0.json
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:08:37 Loaded repository configuration file: /usr/share/osbuild-composer/repositories/rhel-9.1.json
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:08:37 Loaded repository configuration file: /usr/share/osbuild-composer/repositories/rhel-9.2.json
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:08:37 Loaded repository configuration file: /usr/share/osbuild-composer/repositories/rhel-9.3.json
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:08:37 Loaded repository configuration file: /usr/share/osbuild-composer/repositories/rhel-9.4.json
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:08:37 Loaded repository configuration file: /usr/share/osbuild-composer/repositories/rhel-9.json
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:08:37 Starting metadata preload goroutines
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: Distro rhel-8-beta has repositories defined, but it's not supported. Skipping.
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: Distro rhel-8-beta has repositories defined, but it's not supported. Skipping.
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: Distro rhel-8-beta has repositories defined, but it's not supported. Skipping.
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: Distro rhel-8-beta has repositories defined, but it's not supported. Skipping.
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: Distro rhel-8-beta has repositories defined, but it's not supported. Skipping.
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: Distro rhel-8-beta has repositories defined, but it's not supported. Skipping.
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: Distro rhel-8-beta has repositories defined, but it's not supported. Skipping.
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: Distro rhel-8-beta has repositories defined, but it's not supported. Skipping.
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: Distro rhel-8-beta has repositories defined, but it's not supported. Skipping.
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: Distro rhel-8-beta has repositories defined, but it's not supported. Skipping.
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: Distro rhel-8-beta has repositories defined, but it's not supported. Skipping.
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: Distro rhel-8-beta has repositories defined, but it's not supported. Skipping.
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: Distro rhel-8-beta has repositories defined, but it's not supported. Skipping.
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: Distro rhel-8-beta has repositories defined, but it's not supported. Skipping.
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=warning msg="Sentry/Glitchtip not initialized, echo middleware was not enabled" func="github.com/osbuild/osbuild-composer/internal/cloudapi/v2.(*Server).Handler" file="/builddir/build/BUILD/osbuild-composer-101/_build/src/github.com/osbuild/osbuild-composer/internal/cloudapi/v2/server.go:92"
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:08:37-04:00" level=info msg="Worker (x86_64) registered" func="github.com/osbuild/osbuild-composer/internal/worker.(*Server).RegisterWorker" file="/builddir/build/BUILD/osbuild-composer-101/_build/src/github.com/osbuild/osbuild-composer/internal/worker/server.go:831"
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:08:37 POST /api/v1/blueprints/new
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: Distro rhel-8-beta has repositories defined, but it's not supported. Skipping.
Jun 18 08:08:37 localhost.localdomain osbuild-composer[31233]: Distro rhel-8-beta has repositories defined, but it's not supported. Skipping.
Jun 18 08:08:55 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:08:55 Finished preload of rhel-9.2 in 18.353013939s
Jun 18 08:09:12 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:09:12 POST /api/v1/compose
Jun 18 08:09:12 localhost.localdomain osbuild-composer[31233]: Distro rhel-8-beta has repositories defined, but it's not supported. Skipping.
Jun 18 08:09:12 localhost.localdomain osbuild-composer[31233]: Distro rhel-8-beta has repositories defined, but it's not supported. Skipping.
Jun 18 08:09:12 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:09:12 Please note that user customizations on "edge-commit" image type are deprecated and will be removed in the near future
Jun 18 08:09:16 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:09:16 Finished preload of rhel-9.4 in 38.69621956s
Jun 18 08:09:25 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:09:25 Finished preload of rhel-8 in 47.519430051s
Jun 18 08:09:28 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:09:28 Finished preload of rhel-9.1 in 50.882887415s
Jun 18 08:09:32 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:09:32 Finished preload of rhel-9.0 in 55.298201591s
Jun 18 08:09:50 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:09:50 Finished preload of rhel-9.3 in 1m13.037908874s
Jun 18 08:09:57 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:09:57 Finished preload of rhel-8.9 in 1m19.945445027s
Jun 18 08:10:17 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:10:17 Finished preload of rhel-9 in 1m40.125063939s
Jun 18 08:10:28 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:10:28 Finished preload of rhel-8.6 in 1m50.560688151s
Jun 18 08:10:54 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:10:54 Finished preload of rhel-8.7 in 2m16.986219263s
Jun 18 08:11:24 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:11:24 Finished preload of rhel-8.8 in 2m47.225307481s
Jun 18 08:11:38 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:11:38 Finished preload of rhel-8.5 in 3m1.494926916s
Jun 18 08:12:07 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:12:07 Finished preload of rhel-8.10 in 3m29.938202228s
Jun 18 08:12:23 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:12:23 Finished preload of rhel-8.4 in 3m46.190887797s
Jun 18 08:12:40 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:12:40 GET /api/v1/compose/queue
Jun 18 08:12:41 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:12:41 GET /api/v1/compose/finished
Jun 18 08:12:41 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:12:41 GET /api/v1/compose/failed
Jun 18 08:12:46 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:12:46 DELETE /api/v1/compose/cancel/57d3a612-6094-4740-9f61-62321bf4bd0b
Jun 18 08:13:31 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:13:31 POST /api/v1/blueprints/new
Jun 18 08:13:31 localhost.localdomain osbuild-composer[31233]: Distro rhel-8-beta has repositories defined, but it's not supported. Skipping.
Jun 18 08:13:36 localhost.localdomain osbuild-composer[31233]: 2024/06/18 08:13:36 POST /api/v1/blueprints/new
Jun 18 08:13:36 localhost.localdomain osbuild-composer[31233]: Distro rhel-8-beta has repositories defined, but it's not supported. Skipping.
Jun 18 08:13:53 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:13:53-04:00" level=info msg="Worker (x86_64) registered" func="github.com/osbuild/osbuild-composer/internal/worker.(*Server).RegisterWorker" file="/builddir/build/BUILD/osbuild-composer-101/_build/src/github.com/osbuild/osbuild-composer/internal/worker/server.go:831"
Jun 18 08:15:07 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:15:07-04:00" level=info msg="Removing unresponsive job: c0c35f43-27c4-4f91-9d83-a3922322a8da\n" func="github.com/osbuild/osbuild-composer/internal/worker.(*Server).WatchHeartbeats" file="/builddir/build/BUILD/osbuild-composer-101/_build/src/github.com/osbuild/osbuild-composer/internal/worker/server.go:135"
Jun 18 08:15:07 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:15:07-04:00" level=error msg="Error requeueing or finishing unresponsive job: error finishing job: job was canceled" func="github.com/osbuild/osbuild-composer/internal/worker.(*Server).WatchHeartbeats" file="/builddir/build/BUILD/osbuild-composer-101/_build/src/github.com/osbuild/osbuild-composer/internal/worker/server.go:150"
Jun 18 08:15:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:15:37-04:00" level=info msg="Removing unresponsive job: c0c35f43-27c4-4f91-9d83-a3922322a8da\n" func="github.com/osbuild/osbuild-composer/internal/worker.(*Server).WatchHeartbeats" file="/builddir/build/BUILD/osbuild-composer-101/_build/src/github.com/osbuild/osbuild-composer/internal/worker/server.go:135"
Jun 18 08:15:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:15:37-04:00" level=error msg="Error requeueing or finishing unresponsive job: error finishing job: job was canceled" func="github.com/osbuild/osbuild-composer/internal/worker.(*Server).WatchHeartbeats" file="/builddir/build/BUILD/osbuild-composer-101/_build/src/github.com/osbuild/osbuild-composer/internal/worker/server.go:150"
Jun 18 08:16:07 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:16:07-04:00" level=info msg="Removing unresponsive job: c0c35f43-27c4-4f91-9d83-a3922322a8da\n" func="github.com/osbuild/osbuild-composer/internal/worker.(*Server).WatchHeartbeats" file="/builddir/build/BUILD/osbuild-composer-101/_build/src/github.com/osbuild/osbuild-composer/internal/worker/server.go:135"
Jun 18 08:16:07 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:16:07-04:00" level=error msg="Error requeueing or finishing unresponsive job: error finishing job: job was canceled" func="github.com/osbuild/osbuild-composer/internal/worker.(*Server).WatchHeartbeats" file="/builddir/build/BUILD/osbuild-composer-101/_build/src/github.com/osbuild/osbuild-composer/internal/worker/server.go:150"
Jun 18 08:16:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:16:37-04:00" level=info msg="Removing unresponsive job: c0c35f43-27c4-4f91-9d83-a3922322a8da\n" func="github.com/osbuild/osbuild-composer/internal/worker.(*Server).WatchHeartbeats" file="/builddir/build/BUILD/osbuild-composer-101/_build/src/github.com/osbuild/osbuild-composer/internal/worker/server.go:135"
Jun 18 08:16:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:16:37-04:00" level=error msg="Error requeueing or finishing unresponsive job: error finishing job: job was canceled" func="github.com/osbuild/osbuild-composer/internal/worker.(*Server).WatchHeartbeats" file="/builddir/build/BUILD/osbuild-composer-101/_build/src/github.com/osbuild/osbuild-composer/internal/worker/server.go:150"
Jun 18 08:17:07 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:17:07-04:00" level=info msg="Removing unresponsive job: c0c35f43-27c4-4f91-9d83-a3922322a8da\n" func="github.com/osbuild/osbuild-composer/internal/worker.(*Server).WatchHeartbeats" file="/builddir/build/BUILD/osbuild-composer-101/_build/src/github.com/osbuild/osbuild-composer/internal/worker/server.go:135"
Jun 18 08:17:07 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:17:07-04:00" level=error msg="Error requeueing or finishing unresponsive job: error finishing job: job was canceled" func="github.com/osbuild/osbuild-composer/internal/worker.(*Server).WatchHeartbeats" file="/builddir/build/BUILD/osbuild-composer-101/_build/src/github.com/osbuild/osbuild-composer/internal/worker/server.go:150"
Jun 18 08:17:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:17:37-04:00" level=info msg="Removing unresponsive job: c0c35f43-27c4-4f91-9d83-a3922322a8da\n" func="github.com/osbuild/osbuild-composer/internal/worker.(*Server).WatchHeartbeats" file="/builddir/build/BUILD/osbuild-composer-101/_build/src/github.com/osbuild/osbuild-composer/internal/worker/server.go:135"
Jun 18 08:17:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:17:37-04:00" level=error msg="Error requeueing or finishing unresponsive job: error finishing job: job was canceled" func="github.com/osbuild/osbuild-composer/internal/worker.(*Server).WatchHeartbeats" file="/builddir/build/BUILD/osbuild-composer-101/_build/src/github.com/osbuild/osbuild-composer/internal/worker/server.go:150"
Jun 18 08:18:07 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:18:07-04:00" level=info msg="Removing unresponsive job: c0c35f43-27c4-4f91-9d83-a3922322a8da\n" func="github.com/osbuild/osbuild-composer/internal/worker.(*Server).WatchHeartbeats" file="/builddir/build/BUILD/osbuild-composer-101/_build/src/github.com/osbuild/osbuild-composer/internal/worker/server.go:135"
Jun 18 08:18:07 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:18:07-04:00" level=error msg="Error requeueing or finishing unresponsive job: error finishing job: job was canceled" func="github.com/osbuild/osbuild-composer/internal/worker.(*Server).WatchHeartbeats" file="/builddir/build/BUILD/osbuild-composer-101/_build/src/github.com/osbuild/osbuild-composer/internal/worker/server.go:150"
Jun 18 08:18:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:18:37-04:00" level=info msg="Removing unresponsive job: c0c35f43-27c4-4f91-9d83-a3922322a8da\n" func="github.com/osbuild/osbuild-composer/internal/worker.(*Server).WatchHeartbeats" file="/builddir/build/BUILD/osbuild-composer-101/_build/src/github.com/osbuild/osbuild-composer/internal/worker/server.go:135"
Jun 18 08:18:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:18:37-04:00" level=error msg="Error requeueing or finishing unresponsive job: error finishing job: job was canceled" func="github.com/osbuild/osbuild-composer/internal/worker.(*Server).WatchHeartbeats" file="/builddir/build/BUILD/osbuild-composer-101/_build/src/github.com/osbuild/osbuild-composer/internal/worker/server.go:150"
Jun 18 08:19:07 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:19:07-04:00" level=info msg="Removing unresponsive job: c0c35f43-27c4-4f91-9d83-a3922322a8da\n" func="github.com/osbuild/osbuild-composer/internal/worker.(*Server).WatchHeartbeats" file="/builddir/build/BUILD/osbuild-composer-101/_build/src/github.com/osbuild/osbuild-composer/internal/worker/server.go:135"
Jun 18 08:19:07 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:19:07-04:00" level=error msg="Error requeueing or finishing unresponsive job: error finishing job: job was canceled" func="github.com/osbuild/osbuild-composer/internal/worker.(*Server).WatchHeartbeats" file="/builddir/build/BUILD/osbuild-composer-101/_build/src/github.com/osbuild/osbuild-composer/internal/worker/server.go:150"
Jun 18 08:19:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:19:37-04:00" level=info msg="Removing unresponsive job: c0c35f43-27c4-4f91-9d83-a3922322a8da\n" func="github.com/osbuild/osbuild-composer/internal/worker.(*Server).WatchHeartbeats" file="/builddir/build/BUILD/osbuild-composer-101/_build/src/github.com/osbuild/osbuild-composer/internal/worker/server.go:135"
Jun 18 08:19:37 localhost.localdomain osbuild-composer[31233]: time="2024-06-18T08:19:37-04:00" level=error msg="Error requeueing or finishing unresponsive job: error finishing job: job was canceled" func="github.com/osbuild/osbuild-composer/internal/worker.(*Server).WatchHeartbeats" file="/builddir/build/BUILD/osbuild-composer-101/_build/src/github.com/osbuild/osbuild-composer/internal/worker/server.go:150"
...
...
...
...