EC2: token requests 400 due to redacted headers

Bug #1865882 reported by Chad Smith
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init
Fix Released
High
Chad Smith

Bug Description

On Focal EC2 cloud, Cloud-init v.20.1-5-g67c8e53c-0ubuntu1 recently redacted header values from logs for security, but the redact operation was operating on the source headers dictionary. Ec2 API token requests are mangled with a 'REDACTED' value as a result and cloud-init can no longer obtain a valid IMDSv2 token and falls back to IMDSv2 after 2 minutes of failed retries.

Commit introducing the regression:

https://github.com/canonical/cloud-init/commit/87cd040ed8fe7195cbb357ed3bbf53cd2a81436c#diff-a779470bb47168497ada0a33f7990b01R284

Error logs of interest
2020-03-03 13:27:47,839 - url_helper.py[DEBUG]: Read from http://169.254.169.254/latest/api/token (400, 341b) after 1 attempts
2020-03-03 13:27:47,839 - DataSourceEc2.py[WARNING]: Calling 'http://169.254.169.254/latest/api/token' failed [0/1s]: bad status code [400]
2020-03-03 13:27:47,847 - url_helper.py[DEBUG]: Please wait 1 seconds while we wait to try again
2020-03-03 13:27:48,848 - DataSourceEc2.py[DEBUG]: Refreshing Ec2 metadata API token
2020-03-03 13:27:48,848 - url_helper.py[DEBUG]: [0/1] open 'http://None/latest/api/token' with {'url': 'http://None/latest/api/token', 'allow_redirects': True, 'method': 'PUT', 'headers': {'User-Agent': 'Cloud-Init/20.1-5-g67c8e53c-0ubuntu1', 'X-aws-ec2-metadata-token-ttl-seconds': 'REDACTED'}} configuration
2020-03-03 13:27:48,849 - DataSourceEc2.py[WARNING]: Unable to get API token: None/latest/api/token raised exception HTTPConnectionPool(host='none', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc18d46cb10>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
2020-03-03 13:27:48,850 - url_helper.py[DEBUG]: [0/1] open 'http://169.254.169.254/2009-04-04/meta-data/instance-id' with {'url': 'http://169.254.169.254/2009-04-04/meta-data/instance-id', 'allow_redirects': True, 'method': 'GET', 'timeout': 50.0, 'headers': {'User-Agent': 'Cloud-Init/20.1-5-g67c8e53c-0ubuntu1'}} configuration
2020-03-03 13:27:48,869 - url_helper.py[DEBUG]: Read from http://169.254.169.254/2009-04-04/meta-data/instance-id (200, 19b) after 1 attempts
2020-03-03 13:27:48,869 - DataSourceEc2.py[DEBUG]: Using metadata source: 'http://169.254.169.254'
2020-03-03 13:27:48,869 - DataSourceEc2.py[DEBUG]: Refreshing Ec2 metadata API token

Chad Smith (chad.smith)
Changed in cloud-init:
importance: Undecided → High
assignee: nobody → Chad Smith (chad.smith)
status: New → Triaged
Chad Smith (chad.smith)
Changed in cloud-init:
status: Triaged → In Progress
Revision history for this message
Chad Smith (chad.smith) wrote :

A fix for this issue was committed to cloud-init upstream at https://github.com/canonical/cloud-init/commit/fa1abfec27050a4fb71cad950a17e42f9b43b478

Changed in cloud-init:
status: In Progress → Fix Committed
Revision history for this message
Chad Smith (chad.smith) wrote :

A version of cloud-init containing this fix was published to Ubuntu Focal (20.04) cloud-init 20.1-9-g1f860e5a-0ubuntu1.

If this is still a problem for you, please re-open this bug or submit a new bug with related context.

Changed in cloud-init:
status: Fix Committed → Fix Released
Revision history for this message
James Falcon (falcojr) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.