[Impact]
For users who provide more than one unique SSH key to cloud-init and have a shared AuthorizedKeysFile configured in sshd_config, cloud-init 20.4 started writing all of these keys to such a file, granting all such keys SSH access as root. Previously, cloud-init did not handle such sshd_config configuration, and so would have written users' keys to the hardcoded `~/.ssh/authorized_keys` unconditionally.
[Test Case]
This test case has been written for use with the cloud-init integration testing framework:
```
ABSOLUTE_AUTH_KEYS_PATH = "/etc/ssh/authorized_keys"
BACKDOOR_KEYS = "/etc/backdoor_keys"
USER_DATA_TMPL = """\
#cloud-config
bootcmd:
- sed -i "s,#AuthorizedKeysFile.*,AuthorizedKeysFile {0} {1}," /etc/ssh/sshd_config
runcmd:
- |
cat << EOF > {1}
{{}}
EOF
""".format(ABSOLUTE_AUTH_KEYS_PATH, BACKDOOR_KEYS)
# This test checks that we do not write keys into an absolute
# AuthorizedKeysFile by default, because doing so presents a security issue.
#
# Because we're intentionally testing that we do not write authorized keys to
# the location configured for sshd, we cannot SSH into the system using keys in
# that path. We use runcmd to populate a second configured path to grant us
# access; runcmd will install the key after all the SSH key determination has
# been completed, therefore not affecting it.
def test_test(session_cloud, setup_image):
# We shouldn't write out absolute files until we've figured out how to do
# it right
user_data = USER_DATA_TMPL.format(
session_cloud.cloud_instance.key_pair.public_key_content
)
with session_cloud.launch(user_data=user_data) as client:
assert client.execute(
"test -f {}".format(ABSOLUTE_AUTH_KEYS_PATH)
).failed
```
[Regression Potential]
This update touches only cloud-init's SSH handling, by reverting a commit which modified its behaviour. It has been present in Ubuntu images since the 6th of January.
Users who have started relying on the new behaviour (cloud-init writing keys to non-standard AuthorizedKeysFile locations configured in sshd_config) in the past ~2 weeks will see this behaviour stop working.
[Original Report]
Starting on the 6th January 2021 we started observing SSH authentication issues in AWS AMI builds.
We have SSH configured with an absolute (i.e. rather than per-user) authorised keys file, e.g.
AuthorizedKeysFile /etc/ssh/authorized_keys
We observed that the file and parent folder permissions had been modified, to:
/etc/ssh - 0700
/etc/ssh/authorized_key - 0600
These permissions would be fine if the authorised keys file were in a users home directory, but not for a centrally owned absolute file.
We investigated and identified that between the 4th and 6th January 2021, cloud-init on Ubuntu 16.04 was upgraded from 20.3-2-g371b392c-0ubuntu1~16.04.1 to 20.4-0ubuntu1~16.04.1. The newer version included the following fix, that led to the problem, although prior commits set the scene for this commit to cause us trouble.
https://github.com/canonical/cloud-init/commit/b0e73814db4027dba0b7dc0282e295b7f653325c
While trying workarounds (e.g. oneshot service to revert permissions), we then ran into another change that appended exit(142) to the command option:
https://github.com/canonical/cloud-init/commit/e161059a18173e2b61c54dba9eab774401fb5f1f
which then meant, as root is disabled, that SSH would not work using the authorised key pair for any user. This is because cloud-init first writes the key for the user (e.g. ubuntu) and in our case writing the key to /etc/ssh/authorized_keys, then writes the key for the disabled root user to the same location, overwriting the previous write.
There are similarities to https://bugs.launchpad.net/cloud-init/+bug/1839061, but this is a different issue.
Hi Kevin,
Thanks for using cloud-init, and for taking the time to file such a detailed bug! Apologies for having broken your workflow.
I can reproduce the issue that you're seeing locally, and I'm going to start digging into what we need to do to address it.
I have one clarifying question: when launching an image with an older cloud-init and sshd_config as described, I don't see /etc/ssh/
Thanks!
Dan