This page looks best with JavaScript enabled

An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation

 ·  ☕ 2 min read  ·  🤖 Alexander Chernov

When working with EKS under AWS, it’s possible that at some point you wanted to run a pod under a certain role, and you’ve encountered a following error:

1
An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity

What’s frustrating, is that by default AWS doesn’t provide you a lot of feedback of why that error happened.

So I’ve written down some debug steps for further reference:

Check that the service account is set up correctly by running an ephemeral pod

1
kubectl run -ti --restart=Never --rm debug --image=amazon/aws-cli --overrides='{ "spec": { "serviceAccount": "terraform-runner" }  }' -- sts get-caller-identity

This is what I got in response

1
2
3
An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity
pod "debug" deleted
pod runners/debug terminated (Error)

Which indicates that the link between service account and IAM role is not working as it should.

Next step was to check the Service account

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::00000000:role/terraform-runner
    kubectl.kubernetes.io/last-applied-configuration: |
            {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{"eks.amazonaws.com/role-arn":"arn:aws:iam::00000000:role/terraform-runner"},"labels":{"argocd.argoproj.io/instance":"runners"},"name":"terraform-runner","namespace":"runners"}}
  creationTimestamp: "2021-09-17T11:18:13Z"
  labels:
    argocd.argoproj.io/instance: runners
  name: terraform-runner
  namespace: runners
  resourceVersion: "34794913"
  uid: d84ceb7f-4fc7-41fa-af97-9b38c7510a94
secrets:
- name: terraform-runner-token-gtm4x

We can see that service account is correctly configured with addition of eks.amazonaws.com/role-arn,

Let’s check the iam role with aws cli aws iam get-role --role-name terraform-runner

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
    "Role": {
        "Path": "/",
        "RoleName": "terraform-runner",
        "RoleId": "AROATWMTMIX62NAAAJADH",
        "Arn": "arn:aws:iam::000000:role/terraform-runner",
        "CreateDate": "2021-09-17T09:26:19+00:00",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Sid": "",
                    "Effect": "Allow",
                    "Principal": {
                        "Federated": "arn:aws:iam::000000:oidc-provider/oidc.eks.eu-west-2.amazonaws.com/id/C2208D308EF0087F1B87E86008A8FAC2"
                    },
                    "Action": "sts:AssumeRoleWithWebIdentity",
                    "Condition": {
                        "StringEquals": {
                            "oidc.eks.eu-west-2.amazonaws.com/id/C2208D308EF0087F1B87E86500A8FAC2:sub": "system:serviceaccount:runners:ecr-publisher"
                        }
                    }
                }
            ]
        },
        "MaxSessionDuration": 3600,
        "RoleLastUsed": {}
    }
}

And there lies the problem, from the definition above we can see that we are permitting to assume this role to system:serviceaccount:runners:ecr-publisher while our service account is called terraform-runner (a copy/paste error in my terraform stack).

Once the error has been fixed, we can assume the role correctly

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
kubectl run -ti --restart=Never --rm debug --image=amazon/aws-cli --overrides='{ "spec": { "serviceAccount": "terraform-runner" }  }' -- sts get-caller-identity

If you don't see a command prompt, try pressing enter.
{
    "UserId": "AROATWMTM0000N6HAJADH:botocore-session-11111",
    "Account": "000000",
    "Arn": "arn:aws:sts::000000:assumed-role/terraform-runner/botocore-session-1111"
}
pod "debug" deleted

Note: if the command above works, and you pod still don’t, recreate the pod. Sometimes changes to the service accounts are not refreshed until the restart of the pod using it.


Alexander Chernov
WRITTEN BY
Alexander Chernov