Skip to content

[Bug] karpenter: controller policy missing ec2:DescribeCapacityReservations, breaks ODCR support on Karpenter ≥ v1.6 #8748

Description

@candonov

Install Karpenter via the built-in karpenter: block in ClusterConfig and use an EC2 On-Demand Capacity Reservation by setting capacityReservationSelectorTerms on an EC2NodeClass.

Issue

The EC2NodeClass is stuck at Ready=Unknown (reason ReconcilingDependents) even though all sub-conditions (AMIsReady, SubnetsReady, SecurityGroupsReady, InstanceProfileReady, ValidationSucceeded, CapacityReservationsReady) are True.

Karpenter controller logs:

ERROR controller Reconciler error
controllerKind=EC2NodeClass  EC2NodeClass=gpu-inf
aws-error-code=UnauthorizedOperation
aws-operation-name=DescribeCapacityReservations
aws-status-code=403
error: ... User: arn:aws:sts::<acct>:assumed-role/eksctl-<cluster>-iamservice-role/...
is not authorized to perform: ec2:DescribeCapacityReservations because no
identity-based policy allows the ec2:DescribeCapacityReservations action

No nodes get provisioned, even though the referenced ODCR is healthy in EC2.

Root cause

The Karpenter controller IAM policy attached by eksctl is hardcoded in pkg/cfn/builder/karpenter.go and does not include ec2:DescribeCapacityReservations.

The action list currently granted is roughly:

ec2: CreateFleet, CreateLaunchTemplate, CreateTags, DescribeAvailabilityZones,
     DescribeInstanceTypeOfferings, DescribeInstanceTypes, DescribeInstances,
     DescribeLaunchTemplates, DescribeSecurityGroups, DescribeSubnets,
     DeleteLaunchTemplate, RunInstances, TerminateInstances, DescribeImages,
     DescribeSpotPriceHistory
iam: PassRole, CreateServiceLinkedRole, GetInstanceProfile, CreateInstanceProfile,
     DeleteInstanceProfile, TagInstanceProfile, AddRoleToInstanceProfile,
     RemoveRoleFromInstanceProfile, ListInstanceProfiles, ListInstanceProfilesForRole
eks: DescribeCluster
ssm: GetParameter
pricing: GetProducts
sqs: ... (only when withSpotInterruptionQueue: true)

Karpenter v1.3 added native ODCR support behind the ReservedCapacity feature gate, and v1.6 promoted that gate to Beta and enabled by default. The controller now calls ec2:DescribeCapacityReservations on every NodeClass reconcile in order to resolve capacityReservationSelectorTerms.

The upstream Karpenter CloudFormation reference already lists ec2:DescribeCapacityReservations under AllowRegionalReadActions in the KarpenterControllerResourceDiscoveryPolicy. The eksctl-bundled policy has not been updated to match.

Reproduction

cluster.yaml:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: ai-eks-docs
  region: us-east-2
  version: '1.35'
  tags:
    karpenter.sh/discovery: ai-eks-docs
iam:
  withOIDC: true
karpenter:
  version: '1.12.0'
managedNodeGroups:
  - name: managed-ng-1
    minSize: 1
    maxSize: 2
    desiredCapacity: 1
eksctl create cluster -f cluster.yaml

Apply an EC2NodeClass selecting an existing ODCR:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: gpu-inf
spec:
  role: "eksctl-KarpenterNodeRole-ai-eks-docs"
  amiSelectorTerms:
    - alias: al2023@latest
  subnetSelectorTerms:
    - tags: { karpenter.sh/discovery: ai-eks-docs }
  securityGroupSelectorTerms:
    - tags: { karpenter.sh/discovery: ai-eks-docs }
  capacityReservationSelectorTerms:
    - id: cr-xxxxxxxxxxxxxxxxx

Result: 403 UnauthorizedOperation on DescribeCapacityReservations, NodeClass stuck Ready=Unknown, no nodes launched.

Workaround

Manually attach the missing action to the controller role created by eksctl:

ROLE=$(kubectl get sa -n karpenter karpenter \
  -o jsonpath='{.metadata.annotations.eks\.amazonaws\.com/role-arn}' | awk -F/ '{print $NF}')

aws iam put-role-policy \
  --role-name "$ROLE" \
  --policy-name KarpenterCapacityReservations \
  --policy-document '{
    "Version":"2012-10-17",
    "Statement":[{
      "Effect":"Allow",
      "Action":["ec2:DescribeCapacityReservations"],
      "Resource":"*"
    }]
  }'

After this the next reconcile succeeds and the NodeClass goes Ready=True.

Suggested fix

In pkg/cfn/builder/karpenter.go:

  1. Add the action constant:

    ec2DescribeCapacityReservations = "ec2:DescribeCapacityReservations"
  2. Include it in the controller policy statement list alongside the other ec2:Describe* actions.

This is a read-only API and aligns with the upstream Karpenter CloudFormation policy, so it's safe to

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions