Auth doesn't work between requester nodes

[wdbaruni]

AuthSecret only applies for compute nodes when they want to join a network, but requester nodes are still able to join others without authentication.

The reason is compute nodes join as NATS client and we are applying auth only for NATS clients. Requester nodes on the other hand are NATS servers and they join other Requesters using NATS cluster, which accepts a different configuration for auth.

This allows a hybrid node to join existing networks, and start accepting and running jobs without any auth. It works since the node will join as a NATS server where it will handle and forward messages, and at the same time it will run a NATS client that will connect to itself. This will allow the node to publish its NodeInfo to the whole network and handle AskForBid and all other messages.

Caution

Before jumping and adding the auth token to cluster routes, we need to understand how we can deploy a cluster in one go and without having to deploy the requester first, inspect the generated token, and then deploy the remaining nodes. Today I was able to deploying staging using NATS and it worked out of the box without pre-generation of auth token because of this security hole. The question are:

1. How should we deploy our existing networks and switch them to NATS?
2. How do we expect users to deploy their networks in one go?
3. How is bacalhau-tech network working today, and how is the auth key is generated?

[wdbaruni]

cc @simonwo @frrist

[frrist]

Interesting. Can you share the steps you followed so others can also reproduce this?

Perhaps we could target the cluster running on bacalhau.tech. It contains one requester node and two compute nodes. All of nodes use a token to authenticate:

• a token for authenticating to the requester API is required by clients.
• the requester requires a token from compute nodes to connect.

Before jumping and adding the auth token to cluster routes, we need to understand how we can deploy a cluster in one go and without having to deploy the requester first, inspect the generated token, and then deploy the remaining nodes.

We already understand this (I think) - we generate a token before deploying the cluster and configure the nodes to use the token. The nodes are then started with the token already present in their config file. This is how the new terraform works.

How should we deploy our existing networks and switch them to NATS

I'd propose we review the marketplace terraform which does this, and is deployed at bacalhau.tech which uses NATs as network transport. I am also happy to sync on this and walk through things - that might be easier than reading terraform.

How do we expect users to deploy their networks in one go?

At a high-level I think the answer is: "It depends" - what constitutes a network?

• Are all the compute nodes connected to a single requester node, or do different sets of compute nodes talk to their respective requester nodes?
• How do requester nodes communicate with each other?
• Can a compute node connected to requester A also talk to requester B?

How is bacalhau-tech network working today, and how is the auth key is generated?

When deploying you can provide tokens via the vars file. If you do not provide tokens they will be randomly generated. Before any nodes start their config files are defined to include tokens. They allows them to start up and connect to each other.

[frrist]

For what it's worth I am unable to reproduce this on bacalhau.tech - although I'll admit I'm not exactly sure what a successful reproduction looks like. Here is the config file I am using on a node I am running on my desktop.

node:
    network:
        cluster:
            name: global
            port: 6222
        orchestrators:
            - bacalhau.tech:4222
        port: 4222
        type: nats
    type:
        - compute
        - requester
    # added so I can target my node
    labels:
      - foo : bar

Running this job targeting bacalhau.tech as my API fails:

bacalhau docker run --selector=foo=bar ubuntu:latest echo hello world 
Job successfully submitted. Job ID: 2ef2f0dd-69c3-4914-add4-08e42ff3fef3
Checking job status... (Enter Ctrl+C to exit at any time, your job will continue running):

	Communicating with the network  ................  done ✅  0.0s
	   Creating job for submission  ................  err  ❌  0.0s

Error submitting job: not enough nodes to run job. requested: 1, available: 0. 
	Node QmTWp1LL: labels map[Architecture:amd64 Operating-System:linux git-lfs:true] don't match required selectors [foo = bar]
	Node QmfRfevR: labels map[Architecture:amd64 Operating-System:linux git-lfs:true] don't match required selectors [foo = bar]
Job Results By Node:

To get more details about the run, execute:
	bacalhau describe 2ef2f0dd-69c3-4914-add4-08e42ff3fef3

[wdbaruni]

To clarify, what I mean by auth not working is that a requester node can join even without auth key. Here is how you can reproduce this:

# hybrid node joining without token
bacalhau serve --node-type requester,compute --network nats --orchestrators=bacalhau.tech:4222  --cluster-peers=bacalhau.tech:6222

# calling node list against this node will print all nodes in the network
bacalhau node list

# you can even submit jobs and they will be orchestrated on the compute nodes in the network
bacalhau docker run --concurrency 3 ubuntu echo hello

We already understand this (I think) - we generate a token before deploying the cluster and configure the nodes to use the token. The nodes are then started with the token already present in their config file. This is how the #3089 works.

That is the ideal scenario, but we also generate a token if the user doesn't pass a token instead of assuming the network is open https://github.com/bacalhau-project/bacalhau/blob/main/cmd/cli/serve/util.go#L181

[simonwo]

Requester nodes on the other hand are NATS servers and they join other Requesters using NATS cluster, which accepts a different configuration for auth.

OK, I wasn't aware there was a difference. We can add the auth to cluster routes too.

Before jumping and adding the auth token to cluster routes, we need to understand how we can deploy a cluster in one go and without having to deploy the requester first, inspect the generated token, and then deploy the remaining nodes.

That is already possible – just create a token of your choosing (via any method) and add it to the config of each node when you deploy them.

How should we deploy our existing networks and switch them to NATS?

The terraform that Forrest has developed decides on an auth key ahead of time. We can do the same.

How do we expect users to deploy their networks in one go?

Users have a choice. For "unsophisticated" users who are hand-cranking a deployment, they can just use the output on stdout or the bacalhau.run as they do currently and receive secure NATS usage automatically. For more sophisticated users, they can pre-choose a token as above.

That is the ideal scenario, but we also generate a token if the user doesn't pass a token instead of assuming the network is open.

That is the desired behaviour. We should not support open networks where possible because there's more chance one will make it into production through crappy configuration. You can say that's the users fault if you like, but it'll be our support ticket to hear them complain about it and our reputational risk if our software is considered unsecure.

By requiring all networks to be secure, we turn an open network from a silent, invisible security risk into an up front problem that is immediately apparent to users when they can't connect.

[wdbaruni]

That is the desired behaviour. We should not support open networks where possible because there's more chance one will make it into production through crappy configuration. You can say that's the users fault if you like, but it'll be our support ticket to hear them complain about it and our reputational risk if our software is considered unsecure.

By requiring all networks to be secure, we turn an open network from a silent, invisible security risk into an up front problem that is immediately apparent to users when they can't connect.

I hear your point. I am just concerned that we are prematurely optimizing and over complicating the onboarding process at an early stage of the project. We are also giving a false assumption that it is secure by default with no user intervention required, where in-fact it is not production grade secure. We are printing the token in plain text in stdout and in bacalhau.run, and we have a single token to access the whole network, which won't allow us to authenticate nodeX is actually nodeX. Meaning someone who is deploying bacalhau in production should use a different auth mechanism like the ones explained in Node ACL and supported by NATS.

Most systems that I've played with are open by default (ElasticSearch, Nomad, Graphana, Spark, ...) which greatly simplified locally testing and playing with these solutions. Also someone is who is deploying and self-managing a bacalhau network in production is expected to have the level of understanding that their network is open and that they have to take additional steps to secure it, where they will have multiple options instead of us doing some magic on their behalf. I know I'll be surprised by some, but we can't check all the boxes and we need to make some tradeoffs.

In short, a single token for the whole network is not secure and doesn't solve authN. I understand it is better than wide open network, but it will give wrong assumptions that security is covered. Sometimes less is more!

What I think a good tradeoff is to introduce commands to help operators to secure their network and generate keys on-demand. NATS are doing something similar

Product's input is needed @aronchick

[simonwo]

I am just concerned that we are prematurely optimizing and over complicating the onboarding process at an early stage of the project.

Talking about Node ACLs at this point of the project is far more premature then authentication with a simple token. We have no requirement to apply access control on a node-by-node basis that has come from any actual use case. I deliberately decided against using NATS usernames and passwords because it only adds complexity to our existing setup.

Most systems that I've played with are open by default (ElasticSearch, Nomad, Graphana, Spark, ...) which greatly simplified locally testing and playing with these solutions.

The solution we already have makes the auth token part of the peer address, so there is zero extra complexity added. Could you tell me what the additional complexity is for testing with the existing token solution?

From my POV, forcing users to generate an auth token themselves and/or set config to choose between secure or insecure is adding complexity.

Also someone is who is deploying and self-managing a bacalhau network in production is expected to have the level of understanding that their network is open and that they have to take additional steps to secure it, where they will have multiple options instead of us doing some magic on their behalf.

Users should not need to be software engineers to understand how to run Bacalhau. My points above are about how the system fails when the user does not have that level of understanding. As I said elsewhere, you can't just chuck this sort of problem over the wall and say "the users are too stupid" – you have to take steps to make sure an misconfigured system is not a complete security write-off. It is much better to have secure and inaccessible system that needs to be reconfigured before use than have an accidentally and invisibly insecure system.

Also, you haven't seem to have acknowledged that there is already a way for users to set their own auth token via config. So they already have the multiple options that you specify.

In short, a single token for the whole network is not secure and doesn't solve authN.

This is plainly incorrect given the reasons you have set out. Do you have a better reason for believing this?

There is no such absolute as "secure" or "not secure" – there is only "secure enough". Using an auth token is secure enough for our networks today and doesn't introduce needless complexity into the setup experience.

[frrist]

Short cut here is to disable clustering of requester nodes.

[simonwo]

@wdbaruni Per the above I'm going to resolve this by adding the auth token to the cluster routes. Please shout if we need to talk about the solution further.

[wdbaruni]

@simonwo yeah go for it. This will break staging and demo networks as we don't pre-generate auth-tokens and they only connect today using this bug. But heads up that I am removing auto-generation of auth tokens and allowing open networks for now #3670

[wdbaruni]

Will address this as part of #3867