locked out after running debops

[schreon]

I just followed the instructions, ran debops, everything finished successfully. ( my goal was to install gitlab using this utility here ).

Now, whenever I want to ssh to my host I get the error message Permission denied (publickey).
Any tasks run via debops fail for the same reason.

Locally, some folders have been created in my project-folder ( ansible/secret/credentials, ansible/secret/dhparam, ansible/secret/pki )

I did not expect the default playbook of this package to instantly lock me out of my system.

What happened here? What can I do to access my host again?

[drybjed]

@schreon Hello. Have you seen the Getting Started Guide? You might have seen only the guide about the scripts, not the default playbook.

DebOps is designed for production environment in mind. Some of the things like use of SSH keys and firewall are enabled by default, however if you haven't changed anything, the various roles should pick up your SSH public key and the IP address you are connecting from and allow you to access the host.

In the past it was very common to be locked out by the firewall, but it shouldn't happen now in the default configuration. As for lack of public key... do you have an SSH key set up? Do you use ssh-agent and is your key added to it?

To get back to the host, you should try accessing it over the host console (if it's a VM, look at the VNC connection). Password to the root account is stored in secret/credentials/<host>/console/root/password file.

[schreon]

@drybjed Hi, thanks for the lightning fast reply. Indeed, I did not change anything except adding my host to the hosts file. And yes, I have a public key. I also tried to directly reference it using the ssh -i option - still, my connection is refused with that Permission denied (publickey) error. While running the debops command, I had an ssh connection open, which still worked after debops finished. However, I have not been able to connect via ssh again since I closed it.

I also tried to force password-authentication ( ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no ). This results in the same error.

I currently have no direct access to the host system the VM is running on, so I believe using VNC is no option (however I will contact an admin if I can not figure out how to gain access via ssh again)

[drybjed]

@schreon In that case you need to check the /var/log/auth.log on the server to see what might be the issue with sshd. You should also check if your SSH key was correctly configured on the admin account.

... Which, now that i think of it, might not have been. How was that VM prepared for you? Did you have an admin account set up with your SSH key? Do you use preseeding or is that something like a Digital Ocean VM? In that case you might want to check out debops.bootstrap role which can prepare a host to be managed with DebOps for you, including setting a host domain, preparing an admin account and setting up your SSH key.

[schreon]

We logged in via a host console. We discovered the following issues:

1. My local username and the remote username differ. However, ansible created a user with the same local username on the remote machine - with a password unknown to me. The password copied from the secret/credentials/<host>/console/root/password file did not work.
2. My previous remote username had not been added to the admins group, thus the AllowGroups section in sshd_config locked out my previous remote user.
3. Username/Password authentication had been turned off in sshd_config, not allowing me to fall back to that login strategy.

In the end, I ended up locked out because neither the previous remote user nor the new remote user could be used for logging in on the server.

Manually adapting the parameters in sshd_config finally fixed it for me.

I will grab another phyiscal machine now, put it under my desktop and try to "get started" there - allowing me to login directly if anything goes wrong :)

Thanks again for the quick replies.

[drybjed]

The different roles are configured to what I'm mostly using as defaults, an admin username the same as your local account name, mostly to avoide a static admin or ansible username and password authentication disabled by default since I configure new hosts using preseeding with my SSH key added automatically, you can use debops.bootstrap for that.

The admin account created by DebOps doesn't have a password because you are supposed to use SSH keys; passwords over SSH are disabled anyway. The password i pointed you to is for the root account which can be accessed over local console, if needed.

If you want to change the admin account name on remote server, you might need to set:

bootstrap_admin_name: '<account>'

in the Ansible inventory. Keep in mind that you should tell Ansible to use that as well, by setting ansible_ssh_user in the inventory.

[drybjed]

@schreon BTW, did you make GitLab work?

[schreon]

@drybjed I am still digesting your input and getting used to ansible / debops :) I will report back soon. Thanks a lot!

[drybjed]

@schreon Come over to #debops @ FreeNode if you like. We have cookies. :-)

[diarmaid-mcmanus]

I had a similar issue when using Vagrant and debops. The solution (thanks to drybjed) was to add the following variables:

users__accounts:
  - name: 'vagrant'
    groups: 'admins'

I'm adding this here, as this is the first result on Google, and it might help someone in the future!

[muelli]

FTR: There is debops/ansible-sshd#53 which probably fixed issues related locking out the ansible user.

[cyrrill]

I just launched a fresh install of Ubuntu 16.04 from AWS EC2 and experienced the same issue.

@drybjed having the user locked out of their machine is an unfortunate first step to get when running this program. It would be of great value if you could ensure sane defaults were set so that users weren't locked out with the default setup. If bootstrap needs to be run to ensure there is no lockout, why not just run it by default? There was nothing in the "getting started" wiki that warned about this.

[drybjed]

@cyrrill Recently the debops.core role had some changes that should have helped with the issue of an admin account not being present in the admins group. I suppose that it didn't work in your case for some reason. If you want, we could have an interactive session (most likely after holidays) to debug this further. It's hard to pinpoint the specific issue.

What "sane defaults" would like to see implemented in the project? Currently DebOps should detect the admin account by itself and add it to the admins group. I guess that the Getting Started guide could be expanded a bit and mention possible issues with admin account and SSH access, it was written some time ago.

The debops.bootstrap role is meant to be used in environments where you don't have control over the host initial deployment (DHCP/PXE/preseeding), so an AWS EC2 environment certainly applies here. In these environments you should run the bootstrap playbook on each new host so that it can setup your admin account, SSH keys and needed packages. Bootstrapping is mentioned in the guide, maybe it should be expanded or worded differently?

[cyrrill]

@drybjed Thanks for your outstanding work here, I forgot to mention that in the previous post.

The note in that page also says:

Bootstrapping a host this way is not needed if you already have an administrator account that can use sudo without a password.

Which actually applies in my case. AWS EC2 by default creates an instance where the "ubuntu" user can use passwordless sudo. You also already have SSH access by means of *.pem keys.

It wasn't clear to me that debops would alter my original account, or that there would be a risk of lockout. Clearing that up in bold somewhere would save a lot frustration to new users as myself.

I'm not sure which of these changes did the trick, but in compiling fixes from different threads, this diff now allows me to create a new instance, and not be locked out:

+++ b/inventory/group_vars/all/vars.yml
@@ -5,3 +5,13 @@ ntp__timezone: 'Etc/UTC'
 
 pki_internal: False
 pki_authorities: []
+
+core__admin_users: ["{{ ansible_user_id }}"]
+
+bootstrap_admin_name: 'ubuntu'
+ansible_ssh_user: 'ubuntu'
+
+
+users__accounts:
+  - name: 'ubuntu'
+    groups: 'admins'

Thanks again!

[cyrrill]

The debops.bootstrap role is meant to be used in environments where you don't have control over the host initial deployment (DHCP/PXE/preseeding),

Would be just so much simpler if this could happen by itself when needed.

Is there some programmatic way to determine if said bootstrapping has already occurred? Perhaps by testing the user groups?

Or is it possible to just make the bootstrap idempotent, so it doesn't affect it if run again?

[drybjed]

The above mention about debops.bootstrap only needed if you don't already have sudo access should be expanded upon in this case. I'll see what I can do.

I think that the variables above were needed before the debops.core handled the admin accounts in a more sane way. You could try and comment them out, and test on a new VPS to see if the current roles still have the problems that plagued them in the past.

The problem with determining if a user account is in groups that allow SSH access is that, if it isn't, you cannot connect to the host over SSH to test it. It's a chicken and egg problem which debops.bootstrap or a proper preseeding environment should solve. The problem with debops.bootstrap being in the main playbook is that in some cases you need to connect as root with a password, in others as root with an SSH key, and so on. The playbook for the bootstrap role does not impose any requirements, so that you are free to specify any combination of needed Ansible parameters that you need. The role is idempotent, but the environment it runs in changes after the bootstrapping.

But look at it in another way. Since you are using AWS EC2, does the VPS provider give you any way to configure the host during the deployment, like cloud-init? You should modify the defaults so that the admins group is present on the host and the admin accounts are included in it. That way you can solve the issue at the source, which IMO should be the proper way to handle it.

[cyrrill]

The problem with debops.bootstrap being in the main playbook is that in some cases you need to connect as root with a password

I think this should actually be disallowed? Some OS have this turned off in sshd config by default.

But look at it in another way. Since you are using AWS EC2, does the VPS provider give you any way to configure the host during the deployment, like cloud-init?

Not really, they just give you a choice of stock OS options, hardware +/- on CPU/RAM/Disk, firewall, etc.. but nothing really on the user side. All you get is the default "ubuntu" user with passwordless sudo, and root login disabled.

I could use some other tool to prepare the server, but I was hoping debops could take me from AWS launch to deploy...

You should modify the defaults so that the admins group is present on the host and the admin accounts are included in it.

This is not really possible with the default EC2 setup. Barring hardcoding this into debops, I would really hope there would be a simpler way.

Ubuntu is one of the most popular server choices, and EC2 one of the more common cloud choices. Maybe an "ubuntuops" fork can be maintained with some Ubuntu specific settings? I'd be willing to help maintain that if the best way of setting up is determined.

[drybjed]

As I said previously, at the moment DebOps should detect the correct admin account to which you are connecting and add it to the admins group.

In the AWS EC2 case, the debops.bootstrap role is a go-to role to prepare the host, so you should run it each time a new host is deployed. The role tries its best to detect the admin account, but if it's still wrong, you can just set the variables for these hosts in the inventory if needed, for example in a group all of them belong to. After you run the bootstrap role on the host, all needed changes should be present and debops.core should correctly detect the admin account as well.

I don't think that a separate fork for Ubuntu would be that useful. The special Ubuntu settings should be reviewed and if useful, implemented in the DebOps itself. It might even benefit Debian hosts, if the specific changes are good enough. Any specific configuration you have in mind?

[cyrrill]

@drybjed, I haven't reviewed the source enough to make exact sense as to how debops is handling the specifics of this, maybe once I spend some time I can make a more useful suggestion.
If you could make Ubuntu work out of the box in AWS, that would be awesome if it could all stay in debops original repo. I've had to make a private fork to include these changes in my project for now.

[drybjed]

@cyrrill Publish the repository, and I'll see what can be done. :-)

[cyrrill]

For now it contains some proprietary work stuff I haven't prepared for publishing publicly, however those changes are not relevant to this ticket; in terms of avoiding lockout, the vars in: #148 (comment) are pretty much what fixed it (albeit I don't know which one yet ;) )

[drybjed]

You should be able to just leave ansible_ssh_user and remove the others, then after running debops.bootstrap role check if the ubuntu user is in the admins UNIX group. If it's there, you should be good to go.

[cyrrill]

Once I make some progress in my work, I'll spend some time launching another instance to test. I'll report back.

[cyrrill]

Oh, and maybe could be useful to add the bootstrap note to the Installation guide of this repo. It's actually the one I initially used. Didn't read the fuller (and very complete) docs until I ran into problems.

https://github.com/debops/debops-tools#installation

[drybjed]

I'm working on cleaning up debops-tools repository (long overdue) and I'll remove the outdated guides in the process.

[cyrrill]

Had some more time today to run a test. Adding only ansible_ssh_user solves the issue of not being locked out.

Found that I don't even have to run the bootstrap. Was able to provision via the ec2_ ansible module, and then call debops directly.

I think we can close this issue?

[cyrrill]

Caveat lector:

Note
Ansible 2.0 has deprecated the “ssh” from ansible_ssh_user, ansible_ssh_host, and ansible_ssh_port to become ansible_user, ansible_host, and ansible_port. If you are using a version of Ansible prior to 2.0, you should continue using the older style variables (ansible_ssh_*). These shorter variables are ignored, without warning, in older versions of Ansible.

https://docs.ansible.com/ansible/playbooks_variables.html