IPv4 prefix isn't being registered

[pkqk]

Apologies if this isn't the correct place to ask this, I'm having trouble finding much documentation.

I'm running an Amazon Linux 2023 t4g.nano instance with the AMI ami-07832e309d3f756c8 (in us-east-1) which has:

amazon-ec2-net-utils.noarch            2.4.1-1.amzn2023.0.1

installed. I'm trying to work out how to get the IPv4 prefix delegation to be registered with the second network interface.

This is the configured interfaces on boot:

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 12:d5:d8:fa:a3:4d brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname eni-0fd505464293be38a
    altname device-number-0
    inet 10.6.70.16/24 metric 512 brd 10.6.70.255 scope global dynamic ens5
       valid_lft 3022sec preferred_lft 3022sec
    inet6 fe80::10d5:d8ff:fefa:a34d/64 scope link 
       valid_lft forever preferred_lft forever
3: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 12:43:38:d3:6b:45 brd ff:ff:ff:ff:ff:ff
    altname enp0s6
    altname eni-09fbb639e8a4e276c
    altname device-number-1
    inet 10.6.70.17/24 metric 522 brd 10.6.70.255 scope global dynamic ens6
       valid_lft 2616sec preferred_lft 2616sec
    inet6 fe80::1043:38ff:fed3:6b45/64 scope link 
       valid_lft forever preferred_lft forever

I also have a 10.6.70.32/28 prefix on the second ens6 interface

Pining an address in the /28 on the machine doesn't work from another instance or on the instance itself. However if I manually add an address to the interface using:

sudo ip -4 addr add 10.6.70.32/28 dev ens6
sudo ip -4 addr add 10.6.70.33/28 dev ens6

Then I can use ping those addresses.

I've read through most of setup-policy-routes.sh and lib.sh and it looks like it should be generating routes for the prefix.

-- Boot 0856ac531dbc4ede9bfd44e148daafaa --
Jul 10 04:14:19 ip-10-6-70-16 systemd[1]: Starting policy-routes@ens6.service - Set up policy routes for ens6...
Jul 10 04:14:19 ip-10-6-70-16 ec2net[1473]: Starting configuration for ens6
Jul 10 04:14:19 ip-10-6-70-16 ec2net[1476]: /lib/systemd/systemd-networkd-wait-online ens6
Jul 10 04:14:19 ip-10-6-70-16 ec2net[1511]: [get_meta] Querying IMDS for mac
Jul 10 04:14:19 ip-10-6-70-16 ec2net[1515]: Got IMDSv2 token from http://169.254.169.254/latest
Jul 10 04:14:19 ip-10-6-70-16 ec2net[1518]: [get_meta] Querying IMDS for network/interfaces/macs/12:43:38:d3:6b:45/device-number
Jul 10 04:14:19 ip-10-6-70-16 ec2net[1522]: Got IMDSv2 token from http://169.254.169.254/latest
Jul 10 04:14:19 ip-10-6-70-16 ec2net[1527]: Linking /run/systemd/network/70-ens6.network to /usr/lib/systemd/network/80-ec2.network
Jul 10 04:14:19 ip-10-6-70-16 ec2net[1535]: [get_meta] Querying IMDS for network/interfaces/macs/12:43:38:d3:6b:45/subnet-ipv6-cidr-blocks
Jul 10 04:14:19 ip-10-6-70-16 ec2net[1539]: Got IMDSv2 token from http://169.254.169.254/latest
Jul 10 04:14:19 ip-10-6-70-16 ec2net[1555]: [get_meta] Querying IMDS for network/interfaces/macs/12:43:38:d3:6b:45/subnet-ipv4-cidr-block
Jul 10 04:14:19 ip-10-6-70-16 ec2net[1559]: Got IMDSv2 token from http://169.254.169.254/latest
Jul 10 04:14:20 ip-10-6-70-16 ec2net[1564]: [get_meta] Querying IMDS for network/interfaces/macs/12:43:38:d3:6b:45/interface-id
Jul 10 04:14:20 ip-10-6-70-16 ec2net[1568]: Got IMDSv2 token from http://169.254.169.254/latest
Jul 10 04:14:20 ip-10-6-70-16 ec2net[1571]: [get_meta] Querying IMDS for network/interfaces/macs/12:43:38:d3:6b:45/device-number
Jul 10 04:14:20 ip-10-6-70-16 ec2net[1575]: Got IMDSv2 token from http://169.254.169.254/latest
Jul 10 04:14:20 ip-10-6-70-16 ec2net[1593]: Got IMDSv2 token from http://169.254.169.254/latest
Jul 10 04:14:20 ip-10-6-70-16 ec2net[1596]: [get_meta] Querying IMDS for network/interfaces/macs/12:43:38:d3:6b:45/ipv4-prefix
Jul 10 04:14:20 ip-10-6-70-16 ec2net[1600]: Got IMDSv2 token from http://169.254.169.254/latest
Jul 10 04:14:20 ip-10-6-70-16 ec2net[1624]: [get_meta] Querying IMDS for network/interfaces/macs/12:43:38:d3:6b:45/local-ipv4s
Jul 10 04:14:20 ip-10-6-70-16 ec2net[1632]: Got IMDSv2 token from http://169.254.169.254/latest
Jul 10 04:14:20 ip-10-6-70-16 ec2net[1641]: Called trap
Jul 10 04:14:20 ip-10-6-70-16 ec2net[1648]: Deferring networkd reload to another process
Jul 10 04:14:20 ip-10-6-70-16 systemd[1]: Finished policy-routes@ens6.service - Set up policy routes for ens6.

It logs that it checks the IMDS endpoint for ipv4-prefix on that interface so I think it should be generating routes for it.

Is there any setting I should be changing to get it to use them?

[pkqk]

It seems #58 is related, it's mentioned there that I could configure the extra addresses with drop in config files?

[pkqk]

I've worked out a bit more, I've got this script in the user-data cloud-init script.

TOKEN=$(curl -s -X PUT -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" "http://169.254.169.254/latest/api/token")
function imds () {
  curl -f -s -w "\n" -H "X-aws-ec2-metadata-token: ${TOKEN}" "http://169.254.169.254/${1}"
}

for mac in $(imds "latest/meta-data/network/interfaces/macs")
do
  mac=${mac%/} # remove trailing slash
  if pfx=$(imds "latest/meta-data/network/interfaces/macs/${mac}/ipv4-prefix")
  then
    # determine the interface name from the mac address
    # it is in the line before the mac address in the output of
    # `ip addr`
    iface="$(ip addr | grep -B1 $mac | head -1 | cut -d: -f2 | tr -d ' ')"
    # get the netmask (the number after the /)
    mask="$(echo $pfx | cut -d/ -f2)"
    # get the last octet of the address
    base="$(echo $pfx | cut -d/ -f1 | cut -d. -f 4)"
    # get the first 3 octects of the address
    net="$(echo $pfx | cut -d/ -f1 | cut -d. -f1,2,3)"
    # generate each possible address in the prefix
    # 32 - mask gives us the number of bits available for the host part of the address
    # 2**(bits) gives us the number of addresses in that range
    # each address will then be ${net}.${base + i}
    # $(( base + i )) is the bash expansion to evaluate maths expressions
    for ((i=0; i < 2**(32-mask); i++))
    do
      # The format for the .conf file is
      # [Address]
      # Address=1.2.3.4/24
      #
      echo -e "[Address]\nAddress=${net}.$((base + i))/24\n"
    done > "/etc/systemd/network/70-${iface}.network"
  fi
done

systemctl restart systemd-networkd

This sets up the ip addresses based on the configuration at boot time. I'm not sure how I could make it adapt to runtime configuration changes, such as a new ENI being added or updated. I think adding a udev rule is the right approach but I'm not sure what event to listen to and if systemd-networkd is still the right configuration to modify from there.

[dlim201]

@pkqk Running into the same issue, thanks a lot for the script. Reading #58 which you linked, it seems 1.xxx version of this repo had this supported, have you by any chance tried running this older version?

[pkqk]

Hi @dlim201 I wouldn't know how to run the version 1 branch. I'm using Amazon Linux 2023 as AL2 is approaching end of life so I don't want to build anything new on it.

We had tested this earlier on AL2 when examining if prefixes would work for us. We had this in the user script to enable it then:

echo export EC2PROVISIONPFXIPS=yes >> /etc/environment
for interface in `ip -o link show | awk -F': ' '{print $2}' | grep "eth[1-9]"`; do
  ec2ifdown $interface
  EC2PROVISIONPFXIPS=yes ec2ifup $interface
done