Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenShift Compatibility for Jenkins Operator #826

Open
syanpriyajot opened this issue Apr 18, 2023 · 18 comments
Open

OpenShift Compatibility for Jenkins Operator #826

syanpriyajot opened this issue Apr 18, 2023 · 18 comments
Assignees
Labels
bug Something isn't working enhancement New feature or request not-stale
Milestone

Comments

@syanpriyajot
Copy link

Describe the solution you'd like
Since v0.6.0, it was mentioned that Jenkins Operator is not compatible with OpenShift.
https://github.com/jenkinsci/kubernetes-operator/releases/tag/v0.6.0

  • May I know why is the compatibility with OpenShift removed?
  • Can we please make this operator compatible with OpenShift?

Describe alternatives you've considered
Is there working documentation for OpenShift using which I can test the compatibility of latest release on OpenShift?
Or should exactly the same documentation as for Kubernetes work on OpenShift as well?

@brokenpip3

@syanpriyajot syanpriyajot added the enhancement New feature or request label Apr 18, 2023
@brokenpip3
Copy link
Collaborator

Generally speaking I don't think we have something specific that will let the operator to not run correctly in openshift.
I believe that in the past the team supported the jenkins version shipped by redhat which is different from the mainline jenkins lts that we use in our operator, so probably the removal was related to that.

Coming back to ocp: I belive that we may have some issue if we do not run the jenkins pod with an scc that will grants the anyuid, or the specific jenkins image id to be set as running user of the container. I'm not sure, we need to test it to understand if my guess will be true. But apart of this I do not expect any other critical issue in running vs ocp.

Now the real problem is: testing.
Minishift is stuck at ocp 3.11, we need to find a way to test it via a github actions in ocp 4.x.
Will try to build something after a small spring break

@brokenpip3 brokenpip3 self-assigned this Apr 21, 2023
@kumadee
Copy link

kumadee commented May 23, 2023

Generally speaking I don't think we have something specific that will let the operator to not run correctly in openshift. I believe that in the past the team supported the jenkins version shipped by redhat which is different from the mainline jenkins lts that we use in our operator, so probably the removal was related to that.

Coming back to ocp: I belive that we may have some issue if we do not run the jenkins pod with an scc that will grants the anyuid, or the specific jenkins image id to be set as running user of the container. I'm not sure, we need to test it to understand if my guess will be true. But apart of this I do not expect any other critical issue in running vs ocp.

Now the real problem is: testing. Minishift is stuck at ocp 3.11, we need to find a way to test it via a github actions in ocp 4.x. Will try to build something after a small spring break

@brokenpip3 I don't think RedHat will support minishift for ocp 4.x any longer. They have moved on to https://github.com/crc-org/crc for ocp 4.x, due to lot of changes in installation.

@brokenpip3
Copy link
Collaborator

yes I saw that but the crc will require 16gb of memory to run and the github runner cannot handle more than 7gb (and we do no have any free credits here to use the enterprise runners).
So we need to find a way to test the installation in openshift, if someone has other ideas I can take a look on ocp 4.x and fix all the issues but I cannot run it freely anywhere :(

@kumadee
Copy link

kumadee commented Jun 7, 2023

I followed the issue crc-org/crc#557 and it led to this issue crc-org/crc#2507, which aims to reduce the resource requirements of OCP 4.8. I think we are blocked until this issue has some meaningful progress.

There were some experimental changes done by crc team crc-org/snc#241 to do this but doesn't look ready for our use here.

@brokenpip3
Copy link
Collaborator

If someone can try to deploy it in openshift and send here the logs, decribe, evetns will be great, I can take a look and try a fix if needed (90% will be a scc issue)

@github-actions github-actions bot added the stale label Aug 27, 2023
@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Sep 6, 2023
@brokenpip3 brokenpip3 removed the stale label Oct 6, 2023
@brokenpip3 brokenpip3 reopened this Oct 6, 2023
@akilada
Copy link

akilada commented Oct 12, 2023

@brokenpip3
I have tried to implement jenkins operator v0.60 on openshift 4.11. However, the jenkins instance pod keeps getting terminated.

The following are my deployment steps:

However, the jenkins instance pod keeps getting terminated soon after initiation. The following are the logs from jenkins operator pod.

jenkins-operator-v0.60.log
events.log
jenkins-master-describe.log
jenkins-operator-describe.log

@brokenpip3
Copy link
Collaborator

@akilada could you please try by using the latest version (0.8.0)?
the 0.6.0 it's a very old version and contains issues that make impossible to run it everywhere

@github-actions github-actions bot added the stale label Dec 11, 2023
@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Dec 22, 2023
@brokenpip3 brokenpip3 reopened this Dec 24, 2023
@brokenpip3 brokenpip3 removed the stale label Dec 24, 2023
@github-actions github-actions bot added the stale label Feb 23, 2024
@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Mar 4, 2024
@BeastHook
Copy link

@brokenpip3
I have tried it with the latest operator version on a 4.13 Openshift cluster.
The master pod keeps terminating.

I used the example CR in your doc.

Here is the log of the terminated pod:
jenkins-example-jenkins-jenkins-master.log

The rest looks similar to @akilada logs

@jonesbusy
Copy link

This is similar to deployment of the helm chart on Openshift/OKD

You just need to adapt some environment vars for the plugin manager tool to download to a writtable location

Like jenkinsci/helm-charts#506 (comment)

@brokenpip3
Copy link
Collaborator

brokenpip3 commented Apr 19, 2024

in our case it's slightly better you can choose to:

  • change the env to a specific tmp dir and add a tmp dir in the volume/volumemounts
  • change the env to something like /tmp/dir as per @jonesbusy example
  • mount /usr/share/jenkins/ as tempdir

In any cases we should change our helmchart, or even better the code to always use /tmp or a specific emptydir to download the plugins, both in openshift or not openshift scenario.
Also I wish we can harden by default our securitycontext (with readonlyfs for instance) but in this period I do not have the amount of free time to add these changes, will try in the near future. In the mean time any PR is welcomed :)

@brokenpip3 brokenpip3 reopened this Apr 19, 2024
@brokenpip3 brokenpip3 added bug Something isn't working and removed needs triage stale labels Apr 19, 2024
@brokenpip3 brokenpip3 added this to the 0.9 milestone Apr 19, 2024
@BeastHook
Copy link

That worked out great.
I had to correct some plugins to higher versions because I use a newer LTS version of Jenkins.
The container in the pod comes up to the startup of Jenkins and terminates without any further action after the log line "Jenkins is fully up and running".

jenkins-example-jenkins-master .log

@brokenpip3
Copy link
Collaborator

Can you also attach the operator log?

@brokenpip3
Copy link
Collaborator

Also kubernetes events if possible

@BeastHook
Copy link

BeastHook commented Apr 19, 2024

Atm the Pod keeps getting a different problem:
INFO: I/O exception (java.net.SocketException) caught when processing request to {s}->https://ftp.halifax.rwth-aachen.de:443: Network is unreachable

Operator log:
jenkins-operator-6cdbd9f879-52nkg-jenkins-operator.log

I dont have a eventrouter on the Openshift running.
The events view shows:
image

@brokenpip3
Copy link
Collaborator

to me seems that kubelet that is terminating the pod

I dont have a eventrouter on the Openshift running.

you just need to run oc get ev -n <namespace or kubectl

also this part of the code is something I never touched, and will be hard to have a testing env, its seems kind of broken, I don't think we should handle the ocp route for the user, like we do with the ingress

2024-04-19T12:36:46.121Z DEBUG controller-jenkins Route API is available. Now creating route. {"cr": "msg"}
2024-04-19T12:36:46.132Z DEBUG controller-jenkins Jenkins Route is present {"cr": "msg"}

@BeastHook
Copy link

BeastHook commented Apr 22, 2024

Good morning,

this is what i get. not really meaningful...
Eventlog_jenkins.txt

Which parameter can I use to increase the log level?

The created route looks fine for me.

@brokenpip3
Copy link
Collaborator

brokenpip3 commented Apr 22, 2024

the issue is this one:

Jenkins master pod restarted by operator:; Jenkins pod security context has changed; Security context has changed

I don't know who is changing the security context of the pod.
Right now it seems that we have:

  • an issue with the plugin download directory that is not declared as emptyDir in the manifest
  • the operator that is trying all the time to create a route and say that is not present
  • a securitycontext change that will restart the pod all the time

I wish I can take a look but since I can't afford a ocp cluster it will be very hard to have a proper support for openshift 4.x.

@brokenpip3
Copy link
Collaborator

Good news: there may be a company that will provide us some ocp compute time to verify and implement the openshift compatibility, will share more news as soon I will have more info

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request not-stale
Projects
None yet
Development

No branches or pull requests

6 participants