Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add functionality to get instance id from node name by regexp #55

Closed
wants to merge 1 commit into from

Conversation

mfranczy
Copy link
Contributor

@kubevirt-bot kubevirt-bot added the dco-signoff: yes Indicates the PR's author has DCO signed all their commits. label Apr 19, 2022
@kubevirt-bot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign nirarg after the PR has been reviewed.
You can assign the PR to them by writing /assign @nirarg in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@rmohr
Copy link
Member

rmohr commented Apr 19, 2022

@mfranczy did you see #48 ? Intuitively I would expect things to be solved by labels and not by regexp. I am however not entirely up-to-date regarding to this repo.

@mfranczy
Copy link
Contributor Author

mfranczy commented Apr 19, 2022

@mfranczy did you see #48 ? Intuitively I would expect things to be solved by labels and not by regexp. I am however not entirely up-to-date regarding to this repo.

yes I saw the issue, but this PR is not about load balancers and network traffic. It's about node initialization stage, once kubelet creates a node object the cloud provider has to find the node's instance (Virtual Machine), so based on the node name there is an attempt to fetch a VM.

Currently there is an assumption in the code

// instanceIDFromNodeName extracts the instance ID from a given node name. In
// case the node name is a FQDN the hostname will be extracted as instance ID.
// If the regexp is empty instance id will have the same value as a node name.
func instanceIDFromNodeName(nodeName string) string {
	data := strings.SplitN(nodeName, ".", 2)
	if instanceRegexp == "" {
	return data[0]

that works well with Gardener but not with other kubernetes providers like Kubermatic and probably could break at some point with Hypershift too.

For Kubermatic, a node name is equal to virtual machine name so I introduced this regexp field to offer some flexibility in terms of matching a node with virtual machine if other providers has a different VM naming pattern (other than node name = vm name).

@mfranczy
Copy link
Contributor Author

mfranczy commented Apr 19, 2022

@rmohr speaking about the issue with LB, I think we will have to write another controller that watches endpoints on a tenant cluster and updates labels accordingly on infra cluster to route the traffic correctly. Although this probably should be discussed on some meeting with you, however, I am currently preparing a PoC.

@rmohr
Copy link
Member

rmohr commented Apr 19, 2022

that works well with Gardener but not with other kubernetes providers like Kubermatic and probably could break at some point with Hypershift too.

For Kubermatic, a node name is equal to virtual machine name so I introduced this regexp field to offer some flexibility in terms of matching a node with virtual machine if other providers has a different VM naming pattern (other than node name = vm name).

I think I kind of understand the issue. Could you add examples of the node-name and vm name correlation? If possible for both, gardener and kubermatic. I still wonder if there would be a better way, e.g. based on labels to find correlating objects.

@mfranczy
Copy link
Contributor Author

mfranczy commented Apr 19, 2022

Could you add examples of the node-name and vm name correlation? If possible for both, gardener and kubermatic.

yes, I will provide examples

@rmohr
Copy link
Member

rmohr commented Apr 19, 2022

/cc @nirarg

Adding Nir, he spent more time on the repo than I did :)

@mfranczy
Copy link
Contributor Author

mfranczy commented Apr 19, 2022

I still wonder if there would be a better way, e.g. based on labels to find correlating objects.

you're right, seems like there will be a better way than regexp to identify a node. Based on labels it could work. One potential solution could be that k8s providers (once they start kubelet) could pass a label node.kubernetes.io/instance-id=<vm-name> (or something like that) to kubelet's --node-labels arg. Later in the cloud provider we could easily identify the vm. Altough let me check that and other ideas deeply.

@mfranczy
Copy link
Contributor Author

mfranczy commented Apr 29, 2022

After investigating the issue more deeply i discovered that cloud provider kubevirt implements Instances interface (that is for in-tree cloud providers).

Because of that we have some methods that are never called like CurrentNodeName (kubelet calls this only for in-tree).
We should drop the Instances and implement InstancesV2 which is an interface for external cloud providers.

I was wrong by proposing a regexp for node parsing.

Generally speaking by default kubelet registers nodes with name equal to the OS hostname or takes the value from --hostname-override argument.

Considering the above statement, I think we should follow the checks below:

  1. Node name is the same as a VM name - simple scenario we should just fetch a VM based on the node name
  2. Node name is the same as a VM hostname - in case somebody sets a spec.hostname in a VM template, the node will get registered with the name equal to the vm hostname. As @rmohr suggested we could filter VMs over a label, for instance over kubevirt.io/vm-hostname
  3. Node name is different than a VM name and hostname - that can happen if kubelet has a hostname-override argument set.
    We could fetch a vm over a custom node label (when kubelet register a node it allows to add custom labels over node-labels arg, so we could add node.kubernetes.io/instance-id=<vm-name> to the node)
  4. Anything different results in "Node instance not found"

@rmohr @nirarg do you have any concerns about things I wrote above? If not I will close this PR and start implementing InstancesV2 interface.

@nirarg
Copy link
Contributor

nirarg commented May 1, 2022

@rmohr @nirarg do you have any concerns about things I wrote above? If not I will close this PR and start implementing InstancesV2 interface.

Hi @mfranczy ,
That sound good for me, I think you can start with implementing InstancesV2

@mfranczy mfranczy closed this May 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dco-signoff: yes Indicates the PR's author has DCO signed all their commits. size/L
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants