Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scaling Node Pool via CLI Fails When No Nodes are Present in the Pool #449

Open
aakashnagpal4567 opened this issue Aug 14, 2024 · 5 comments

Comments

@aakashnagpal4567
Copy link
Contributor

aakashnagpal4567 commented Aug 14, 2024

Description

When there is no node in the node pool in the cluster, when we scale that node pool using Civo CLI, it's giving the error.
This functionality is working fine in the UI.

Steps to Replicate

  1. Create a Civo Kubernetes cluster with two node pools
  2. Scale one of the node pools to zero using the UI
  3. Using the CLI, scale the nodes in the node pool from zero to the desired amount with this command:
    civo kubernetes node-pool scale CLUSTER_NAME NODEPOOL_ID -n <NUMBER_OF_NODES>

For example, i tried to scale the node in the node pool that is empty, it gives an error:

$ civo kubernetes node-pool scale node-test 76a32cac-60a3-4a69-999d-251fac214c10 -n 2
Please check if you are using the latest version of CLI and retry the command 
If you are still facing issues, please report it on our community slack or open a GitHub issue (https://github.com/civo/cli/issues) 
Error: Unable to find "76a32cac-60a3-4a69-999d-251fac214c10" node pool inside "node-test" cluster

For reference, I'm attaching a video here:

node-pool-issue.mp4

Acceptance Criteria

  • Scaling the node pool to the zero nodes using Civo CLI should work.
  • Civo CLI should be able to successfully scale a node pool with zero nodes.
@Praveen005
Copy link
Contributor

Adding more context to it.

I tried recreating the issue, but I stumbled upon the following:

When you create the cluster using UI, it is not getting reflected in the CLI, and vice-versa.
That's why you got that error.

cluster on the UI:

Screenshot 2024-08-15 005650

CLI giving error:

Screenshot 2024-08-15 004010

cluster created on CLI, not reflecting the UI:

Screenshot 2024-08-15 010332

Nothing on the UI:

Screenshot 2024-08-15 010306

Quota rightly swelling up though:

Screenshot 2024-08-15 010242

Post cluster creation using the UI, I couldn't scale down to zero node node-pool as shown in your video, it gave an error. For n != 0 scaling up and down is going through as intended.

Screenshot 2024-08-15 005400

When you create the cluster using CLI, it gets created successfully, scaling up and down the node-pool also goes as intended, except when you want to scale down to zero nodes.

Screenshot 2024-08-15 010910

Summary:

  1. Scaling down to zero node is not happening either on the UI or the CLI. I am using the latest version of the CLI.
image
  1. Cluster created using UI is not reflecting in the CLI and vice-versa.

@fernando-villalba
Copy link
Contributor

@Praveen005 could it be you were on one region on the UI and in another on the CLI?

@aakashnagpal4567
Copy link
Contributor Author

aakashnagpal4567 commented Aug 15, 2024

@Praveen005 For this you need to create two node pools, please read the 1st point in "Steps to replicate". Currently, you can't scale the node pool to zero using CLI, it can only be done via the UI. We also have to add this functionality, if there are multiple node pools, we should be able to scale the node pool to zero, and also have to ensure that there should be at least one node in the cluster.

@Praveen005
Copy link
Contributor

Hi @fernando-villalba, You were right, it was a region mismatch. Thank you.

@aakashnagpal4567 I could recreate the issue now. Thank you.

@Praveen005
Copy link
Contributor

Hi Sir, @fernando-villalba

I wanted to update you on the issue, I worked on it,

And here is the gist:

  1. We can now scale up a node-pool which has zero nodes using CIVO CLI (fix explained below).
  2. While trying to scale node-pool down to zero-nodes using CIVO CLI, I am faced with the following error, for more context and to make informed changes, I need to see how it is being handled on the API side, to which I don't have access to.
image

Currently, scaling up a node pool with zero nodes using CIVO CLI results in an error because we are searching for the node pool in kubernetesFindCluster.Pools. However, when a node pool is scaled down to zero nodes, it is no longer active and cannot be found, leading to the error.

nodePoolFound := false
for _, pool := range kubernetesFindCluster.Pools {
if strings.Contains(pool.ID, nodePoolID) {
nodePoolID = pool.ID
nodePoolFound = true
}
}
if !nodePoolFound {
utility.Error("Unable to find %q node pool inside %q cluster", nodePoolID, kubernetesFindCluster.Name)
os.Exit(1)
}

To fix this, we can look for the pool in kubernetesFindCluster.RequiredPools. and add it to the nodePool. Like below:

image

This fix solves one part of the issue. I have tested it, and it works fine.

Result:

Initial state: one node pool with zero node

Screenshot 2024-08-15 151236

Can't scale up using CLI:

Screenshot 2024-08-15 152026

Post code changes:

Node pool rightly getting scaled up:

Screenshot 2024-08-15 152108

Change reflecting in UI:

Screenshot 2024-08-15 152202

Up & Running:

Screenshot 2024-08-15 152222

For the second part,

Even when I pass two node-pools in which one clearly has an active node , as evident below:

configKubernetes: &{ NYC1 1 [{ ad8793ba-dbfe-4286-a276-7167fc19ef38 1 g4s.kube.xsmall map[] [] false} { f805408a-de4c-4fb6-9989-fa1010b686d8 0 g4s.kube.xsmall map[] [] false}] }

kubernetesCluster, err := client.UpdateKubernetesCluster(kubernetesFindCluster.ID, configKubernetes)

I am getting the following error:

Screenshot 2024-08-16 152158

It needs to seen, how validation on the server side is happening.

Regards!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants