[ci.jenkins.io] Use DigitalOcean VM agents instead of container agents

[dduportal]

Service(s)

ci.jenkins.io, DigitalOcean

Summary

The past 4 months events showed that using Kuberenetes agents in DigitalOcean for ci.jenkins.io is not the most efficient way to spend our credits and is also limited on some features.

• Challenge number 1: DigitalOcean does not allow autoscaling node pools to zero, which is a cost of 336$ per month ( 4k$ per year, for a credit of 18k$)

• Challenge 2, as seen in [DigitalOcean] Notice of SSH bruteforce activities #3541, the firewall of Kubernetes cannot be fine tuned

As pointed out by @halkeye months ago, we could use DigitalOcean with VM agents instead.

Major elements to consider:

• Building Custom Images: https://developer.hashicorp.com/packer/plugins/builders/digitalocean indicates that we should be able to create our own VM templates in jenkins-infra/packer-images
• Allow Jenkins to spin up VM agents "on the go": https://plugins.jenkins.io/digitalocean-plugin/ looks good
  • Looks like we can use custom images but to be verified
  • instanceCap attribute looks like configurable on both cloud and templates levels
  • usePrivateNetworking indicates that https://docs.digitalocean.com/products/networking/vpc/ could be used BUT it does not looks like that inbound protocol is a feature (ping @halkeye : is SSH required?) so we'll be stuck with Public IP to allow ci.jenkins.io to access the agent with SSH (not mentioning the need for 1/2 egress IPs)
  • Groovy snippet mentions an InitScript attribute: would be useful for setting up datadog agent (and custom stuff)

Reproduction steps

No response

[halkeye]

Allow Jenkins to spin up VM agents "on the go": https://plugins.jenkins.io/digitalocean-plugin/ looks good

uh oh, does that mean i actually have to help out on that plugin again? I'm a terrible plugin maintainer.

[dduportal]

Allow Jenkins to spin up VM agents "on the go": https://plugins.jenkins.io/digitalocean-plugin/ looks good

uh oh, does that mean i actually have to help out on that plugin again? I'm a terrible plugin maintainer.

[smerle33]

snapshot template have been created within digital ocean packer-1684165095 and is ready for testing @lemeurherve
for information, I have created a PR with the draft code that is able to build the snapshot image : jenkins-infra/packer-images#647

called locally with packer build -timestamp-ui -force -only="digitalocean.amd64"

[dduportal]

Delaying: no bandwidth on the upcoming milestone

[dduportal]

This task is not needed anymore since #3913 which started the move of ci.jenkins.io to an Azure (credits-sponsored) account, and now we do the same from Azure to AWS in #4313