Secure bootstrap: Rewritten and improved

getting-started/installation/secure-bootstrap.markdown

@@ -5,116 +5,245 @@ published: true
 sorting: 20
 ---
 
-This guide presumes that you already have CFEngine properly installed
-and running on the policy hub, the machine that distributes the policy
-to all the clients. It also presumes that CFEngine is installed, but not
-yet configured, on a number of clients.
+This guide assumes you already have a working CFEngine hub (installed and bootstrapped), and you have installed CFEngine on a client you want to securely connect to the hub (bootstrap).
+See the [Getting started guide][Getting started] for an introduction to CFEngine and how to install it.
 
-We present a step-by-step procedure to securely bootstrapping a
-number of servers (referred to as *clients*) to the policy hub, over a
-possibly unsafe network.
+CFEngine's trust model is based on the secure exchange of keys.
+Since it's using mutual authentication, this trust goes in both directions.
+Both the client and the hub refuse to communicate with an unknown, untrusted host.
+Usually, when getting started with CFEngine, this step is automated as a dead-simple "bootstrap" procedure:
 
-## Introduction
+```command
+cf-agent --bootstrap <IP address of hub>
+```
+
+However, this is in the default configuration, and there are several limitations and implications of this;
+
+## Default configuration
 
-CFEngine's trust model is based on the secure exchange of keys. This
-exchange of keys between *client* and *hub*, can either happen manually
-or automatically. Usually this step is automated as a dead-simple
-"bootstrap" procedure:
+In the default configuration, the policy server (`cf-serverd`) on the hub machine trusts incoming connections from the same `/16` subnet.
+This means that:
 
-`cf-agent --bootstrap $HUB_IP`
+* Bootstrapping new clients will work as long as the 2 first numbers in the IP address are identical ([IPv4 dot decimal representation](https://en.wikipedia.org/wiki/Dot-decimal_notation)) .
+  The hub and client mutually accept each other's keys, automatically.
+* This applies to _all_ IP addresses within that range, not just the 1 IP address belonging to the client you are currently bootstrapping.
+* The hub will keep accepting new clients from those IP addresses until you change the configuration.
+* If you try to bootstrap a client where those 2 numbers in the IP address do not match the hub, it will fail.
 
-It is presumed that during this first key exchange, *the network is
-trusted*, and no attacker will hijack the connection. After
-"bootstrapping" is complete, the node can be deployed in the open
-internet, and all connections are considered secure.
+This situation, where the client and hub automatically transfer and trust each other's keys is called _automatic trust_ or _automatic bootstrap_.
+When using automatic trust, it is presumed that during this first key exchange, *the network is trusted*, and no attacker will hijack the connection.
+Below we will show ways to change the configuration and bootstrap your clients in more secure ways.
+The goal here is to illustrate the different approaches, explaining what is needed and the implications of each.
+In the end, you will not be running these commands manually, but rather putting them into a provisioning system.
 
-However there are cases where initial CFEngine deployment is happening
-over an insecure network, for example the Internet. In such cases we
-already have a secure channel to the clients, usually ssh, and we use
-this channel to *manually establish trust* from the hub to the clients
-and vice-versa.
+## Allowing only specific IP addresses / subnets
 
-## Locking down the policy server
+In order to specify and limit which hosts (IP addresses) are considered trusted and allowed to connect and fetch policy files, you can put the trusted IP addresses and subnets into the `acl` variable:
 
-We must change the policy we're distributing to fully locked-down
-settings. So after we have set-up our hub (using the standard procedure
-of `cf-agent --bootstrap $HUB_IP`) we take care of the following:
+```json
+[file=/var/cfengine/masterfiles/def.json]
+{
+  "variables": {
+    "default:def.acl": ["1.2.3.4", "4.3.2.1"]
+  }
+}
+```
 
-* `cf-serverd` must never accept a connection from a client presenting an
-  untrusted key. [Disable automatic key trust][Masterfiles Policy Framework#Automatic bootstrap - Trusting keys from new hosts with trustkeysfrom]
-  by providing an empty list for `default:def.trustkeysfrom`.
+**Important:** Replace `1.2.3.4` with the IP address of your hub, `4.3.2.1` with the IP address of your client, and extend the list with any additional IP addresses / subnets.
 
-## Bootstrap without automatically trusting
+If you are using CFEngine Build, you can use [this module](https://build.cfengine.com/modules/allow-hosts/), putting the IP addresses as module input, or add the json file above to your project.
+(Save it as a file called `def.json` and do `cfbs add ./def.json`).
 
-In order to securely bootstrap a host you must have the public key of the host
-you wish to trust.
+Once this is set, you are no longer using the default value explained above (the `/16` subnet).
+This variable controls 3 different aspects: IP addresses allowed to connect, IP addresses to automatically trust keys from, and IP addresses allowed to fetch policy files.
 
-Copy the hubs public key (`/var/cfengine/ppkeys/localhost.pub`) to the agent you
-wish to bootstrap. And install it using `cf-key`.
+At this point, you can run bootstrap on the client to the hub using automatic trust:
 
-```console
-[root@host001]# cf-key --trust-key /path/to/hubs/key.pub
+```command
+cf-agent --bootstrap 1.2.3.4
 ```
 
-**Note:** If you are using [protocol_version `1` or `classic`][Components#protocol_version]
-you need to supply an IP address before the path to the key.
+If the IP addresses are correct, keys will be automatically exchanged, and hosts will start using encrypted communication over TLS, with mutual authentication.
+At this point CFEngine works on your hub and client, even if they are not on the same `/16` subnet.
+If this is your first time testing CFEngine, feel free to stop reading here and test the various features of Mission Portal, start writing policy, etc.
+In the sections below, we will explore the security implications of this setup further, and show more secure approaches.
+
+**Tip:** Setting the variable to `["0.0.0.0/0"]` will open up your hub to all IPv4 addresses, the entire internet.
+This is generally not recommended, but can make sense if you disable automatic trust (shown below), need to support clients connecting from the public internet, and/or want to manage firewalling restrictions outside of CFEngine.
+
+## Disabling automatic trust - Locking down the policy server
 
-For example:
+In all cases, it is recommended to disable automatic trust when you are not using it.
+Either immediately after installation (if distributing keys through another channel, see below) or after you are done bootstrapping clients.
+You can edit the augments file to achieve this:
 
+```json
+[file=/var/cfengine/masterfiles/def.json]
+{
+  "variables": {
+    "default:def.trustkeysfrom": []
+  }
+}
 ```
-notice: Establishing trust might be incomplete. For completeness, use --trust-key IPADDR:filename
+
+If you are using CFEngine Build, you can achieve this by adding [this module](https://build.cfengine.com/modules/disable-automatic-key-trust/), or adding the json file above to your project.
+
+When combined with the variable above, you can create a very restricted setup:
+
+```json
+[file=/var/cfengine/masterfiles/def.json]
+{
+  "variables": {
+    "default:def.acl": ["1.2.3.4", "4.3.2.1"],
+    "default:def.trustkeysfrom": []
+  }
+}
 ```
 
-Next copy the hosts public key (`/var/cfengine/ppkeys/localhost.pub`) to the hub
-and install it using `cf-key`.
+Only those 2 IP addresses are allowed to connect, and they must use their existing keys, no new keys are automatically trusted.
+
+With what we've discussed up until now, we still need to _trust the network_ for limited periods of time, when we are bootstrapping new hosts.
+(Assuming that we are really communicating with the host we intend to, and that there aren't additional malicious hosts connecting from the same IP addresses / subnets).
+This is sometimes acceptable, especially if you are just testing CFEngine in a disposable and isolated environment.
+However, in a production setup it is recommended to exchange keys in the most secure / trusted method available.
+Below, we will show how.
+
+## Key location and generation
 
-```console
-[root@hub]# cf-key --trust-key /path/to/host001/key.pub
+If you are installing CFEngine using one of our official packages, keys are automatically generated and you can see them in the expected location:
+
+```command
+sudo ls /var/cfengine/ppkeys
+```
+```output
+ localhost.priv
+ localhost.pub
+'root-SHA=caa398e50c6e6ad554ea90e1bd5e8fee269ca097df6ce0c86ce993be16f6f9e3.pub'
 ```
 
-Now that the hosts trust each other we can bootstrap the host to the hub.
+The keypair of the host itself is always in the `localhost.pub` and `localhost.priv` files.
+Additional public keys from the hosts CFEngine is talking to over the network are in the other `.pub` files.
+The filename has a SHA checksum of the public key file - this is the CFEngine hosts unique ID (in Mission Portal, our API, PostgreSQL and LMDB databases, etc.).
+
+**Recommendation:** Don't copy, transfer, open, or share the private key (`localhost.priv`).
+It is a secret - putting it in more places is not necessary and increases the chances it could be compromised.
+When distributing keys for establishing trust, we are distributing the public keys (`.pub` files).
 
-```console
-[root@host001]# cf-agent --trust-server no --bootstrap $HUB
+If you are compiling CFEngine from source, or spawning a new VM based on a snapshot without keys inside, you can generate a new keypair:
+
+```command
+sudo cf-key
 ```
 
-## Manually establishing trust
+**Tip:** When using "golden images" to spawn machines with CFEngine already installed, ensure the keys in `/var/cfengine/ppkeys` are deleted before generating the snapshot, and generate / insert keys during provisioning.
+
+## Key distribution - boostrapping without automatically trusting
+
+To securely bootstrap a host to a hub, without trusting the network (IP addresses), you need to copy the 2 public keys across some trusted channel.
+Below we will be using SSH as the trusted channel, however the commands can easily be translated to however you are able to run commands and transfer files to your hosts.
+(This could be via memory stick, a management interface or some other out-of-band management solution).
+The same applies to passwordless sudo - we're using sudo commands without password prompts below, if you have configured password prompts for sudo, or another way you need to run privileged commands, please adjust accordingly.
 
-Get the hub's key and fingerprint, we'll them when configuring the host to trust
-the hub:
+Assuming you are sitting on a laptop / workstation, and have network and SSH access to both the client and the hub, first set up some variables for each of them:
 
-```console
-[root@hub]# HUB_KEY=`cf-key -p /var/cfengine/ppkeys/localhost.pub
+```command
+BOOTSTRAP_IP="1.2.3.4" HUB_SSH="[email protected]" CLIENT_SSH="[email protected]"
 ```
 
-### On each client we deploy
+Edit the 3 variables according to your situation, they represent:
 
-We will perform a *manual bootstrap*.
+* `BOOTSTRAP_IP` - The IP address of the hub, which you want `cf-agent` on the client to bootstrap to (connect to).
+* `HUB_SSH` - The username / IP combination you would use to connect to the hub with SSH.
+* `CLIENT_SSH` - The username / IP combination you would use to connect to the hub with SSH.
 
-* Get the client's key and fingerprint, we'll need it later when establishing
-  trust on the hub:
+### Trusting the client's key on the hub
 
-  ```console
-  [root@host001]# CLIENT_KEY=`cf-key -p /var/cfengine/ppkeys/localhost.pub`
-  ```
+Inspect the key:
 
-* Write the policy hub's IP address to `policy_server.dat`:
+```command
+ssh "$CLIENT_SSH" "sudo cat /var/cfengine/ppkeys/localhost.pub"
+```
+```output
+-----BEGIN RSA PUBLIC KEY-----
+MIIBCgKCAQEAt93D8fb+M7HGZxsVo+FnOhnLM9E0QCr046N369jOeePY65lPOhAD
+nlWlDPJrYqhnobEdnFr/uNp0ydqb1EASe4qjhQUDi1ujz5+T9dTwhZqUfx22RM6D
+CLulbdoXwImPOCNi157UBRIwYVJ6527rv0/TlTpS9iUQVStg0YCBEasGRcQfX/bU
+DKrL5Ei+ukJtSEx11NlZ9tRYNu22mJYPGGpNJ0FbiHvR+eu7mAuUZ1QeddcuYkGP
+H5/eIe0uTGOmFLXb4gUQymNLJUjQqxoO2l6Km4UpGj61871gCiqMGVTvvZWFbo+g
+1KR3RS6L/Gqv9U89msZTGQafpjFQyVbYnwIDAQAB
+-----END RSA PUBLIC KEY-----
+```
 
-  ```console
-  [root@host001]# echo $HUB_IP > /var/cfengine/policy_server.dat
-  ```
+It should have the format above, with `BEGIN RSA PUBLIC KEY`, the arbitrary data, and `END RSA PUBLIC KEY`.
+When you're scripting / automating the copying of keys, you can add some checks for this.
 
-* Put the hub's key into the client's trusted keys:
+Download the key:
 
-  ```console
-  [root@host001]# scp $HUB_IP:/var/cfengine/ppkeys/localhost.pub /var/cfengine/ppkeys/root-${HUB_KEY}.pub
-  ```
+```command
+ssh "$CLIENT_SSH" "sudo cat /var/cfengine/ppkeys/localhost.pub" > client.pub
+```
 
-### Install the clients public key on the hub
+Upload it to the hub:
+
+```command
+scp ./client.pub "$HUB_SSH":client.pub
+```
 
-* Put the client's key into the hub's trusted keys. So
-  on the hub, run:
+And use `cf-key` to trust the key:
+
+```command
+ssh "$CLIENT_SSH" "sudo cf-key --trust-key client.pub"
+```
+
+### Trusting the hub's key on the client
+
+Now, for the client we need to perform exactly the same steps:
+
+Inspect the key:
+
+```command
+ssh "$HUB_SSH" "sudo cat /var/cfengine/ppkeys/localhost.pub"
+```
+```output
+-----BEGIN RSA PUBLIC KEY-----
+MIIBCgKCAQEAt8Wti90sRjLiEhLbC5096nEhzV3fU0N4TrxiGPCb26KufavBrXGw
+vmzTeJoWnIFFSn7OYU1g59U7s4aViZwqQ647opc0gZo2dVjDTRFW8lB4dmS7SjAe
+t8NA3iXQigWY+45TbvPOalNHurhyrJ4g1+0ttdqwk/L1fVkK0u9wmHrgfo+UQR0D
+9P96GWnPKyzVp5PdMmfX0Sm6kMBurawRYeiFCq3gqGtkc0rj3FHr1afrM+8egP9D
+sWl43NmMlZ8B9Yt2bP0wdNsbXC7vouDZg8sIQVfvcxSkla+kGceGrEmNTDPuGFlx
+VknPhmpjMJ7XhvaXXR1btu3/PLjGLDj6SwIDAQAB
+-----END RSA PUBLIC KEY-----
+```
+
+Download the key:
+
+```command
+ssh "$HUB_SSH" "sudo cat /var/cfengine/ppkeys/localhost.pub" > hub.pub
+```
+
+Upload it to the client:
+
+```command
+scp ./hub.pub "$CLIENT_SSH":hub.pub
+```
+
+And use `cf-key` to trust the key:
+
+```command
+ssh "$CLIENT_SSH" "sudo cf-key --trust-key hub.pub"
+```
+
+### Start CFEngine on the client with a bootstrap command
+
+Now that keys are distributed, trust is established.
+We can run the normal bootstrap command with one crucial difference:
+`--trust-server no` tells the agent to **not** automatically trust an unknown key on the other end.
+This will start the normal CFEngine services (`cf-execd`, `cf-serverd`, etc.):
+
+```command
+ssh "$CLIENT_SSH" "cf-agent --trust-server no --bootstrap $BOOTSTRAP_IP"
+```
 
-  ```console
-  [root@hub]# scp $CLIENT_IP:/var/cfengine/ppkeys/localhost.pub /var/cfengine/ppkeys/root-${CLIENT_KEY}.pub
-  ```
+When we connect to the hubs IP address, if there is another server answering, a potential [man-in-the-middle attack](https://en.wikipedia.org/wiki/Man-in-the-middle_attack), it will not work.
+The agent on the client machine will refuse to communicate with the untrusted server.
+This is the main reason (security benefit) of doing mutual authentication and secure key distribution.