Comment 3 for bug 2056194

Between initramfs-tools 0.142ubuntu8 and 0.142ubuntu9, we've Replaced dhclient by dhcpcd (LP: #2024164).

When a DHCP server provides MTU settings to dhcpcd, it configures the routes with the appropriate mtu value (due to "option interface_mtu" in /etc/dhcpcd.conf), but it does not configure the MTU setting in the interface. So, we end up having a mismatch between interface and route MTU setting, as observed below:

root@(none):/# ip a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 02:00:17:05:ee:5a brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    inet 10.0.0.21/24 brd 10.0.0.255 scope global dynamic noprefixroute ens3
       valid_lft 86048sec preferred_lft 75248sec
    inet6 fe80::17ff:fe05:ee5a/64 scope link
       valid_lft forever preferred_lft forever
root@(none):/# ip route show
default via 10.0.0.1 dev ens3 proto dhcp src 10.0.0.21 metric 1002 mtu 9000
10.0.0.0/24 dev ens3 proto dhcp scope link src 10.0.0.21 metric 1002 mtu 9000
169.254.0.0/16 dev ens3 proto dhcp scope link src 10.0.0.21 metric 1002 mtu 9000

If we go back to initramfs-tools 0.142ubuntu8 (and, consequently to dhclient), the MTU settings will match:

root@(none):/# ip a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
    link/ether 02:00:17:05:ee:5a brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    inet 10.0.0.21/24 brd 10.0.0.255 scope global ens3
       valid_lft forever preferred_lft forever
    inet6 fe80::17ff:fe05:ee5a/64 scope link
       valid_lft forever preferred_lft forever
root@(none):/# ip route show
default via 10.0.0.1 dev ens3
10.0.0.0/24 dev ens3 proto kernel scope link src 10.0.0.21
169.254.0.0/16 dev ens3 scope link

Due to this mismatch when using dhcpcd, certain network activities might fail. For example, in Oracle instances, a curl to the DataSource will just hang:

curl --connect-timeout 10 --max-time 15 -H "Authorization: Bearer Oracle" -L http://169.254.169.254/opc/v2/instance/

If you either reduce the route MTU setting down to 1500 OR increase the interface MTU setting to 9000, curl works:

option 1 - Reduce route setting:

ip route delete 169.254.0.0/16 dev ens3 proto dhcp scope link src 10.0.0.21 metric 1002 mtu 9000
ip route add 169.254.0.0/16 dev ens3 scope link

option 2 - Increase interface setting:

ip link set mtu 9000 ens3

The correct MTU setting here is 9000. When you fully boot the instance, the MTU gets configured appropriately (outside of initramfs)

I also checked the Paravirtualized instances (uses virtio_net interfaces instead of mlx5_core) and the mismatch is also there, but curl still works, so the Paravirtualized instance is also exposed to the bug but more resistant to it.

While investigating this, I noticed that in past releases of dhcpcd, we had a hook 10-mtu, that would automatically configure the mtu setting based on information it received from the dhcp server. This file was removed by the following commit:

https://github.com/NetworkConfiguration/dhcpcd/commit/ca6cdf5847cda720b65793ea6827b1b373c62382

There are some discussion around why this was removed here:

https://github.com/NetworkConfiguration/dhcpcd/issues/67

Anyway, this causes such mismatch between interface and route settings, which cause curl to the IMDS to hang and cloud-init to fail.