EKS ALB is not to able to auto-discover subnets

Question

Background:

• I have a VPC with 3 public subnets(the subnets have access to an internet gateway)

• I have an EKS Cluster in this VPC, the EKS cluster is created from the console and not using eksctl

• I used this tutorial from the official aws documentation, I managed to set my ALB controller and the controller is running perfectly:

The cluster contains two node groups:

• First node group has one node of type: t3a.micro
• Second node group has one node of type: t3.small

$ kubectl get deployment -n kube-system aws-load-balancer-controller
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
aws-load-balancer-controller   1/1     1            1           60m

I used their game example and here is the manifest file:

---
apiVersion: v1
kind: Namespace
metadata:
  name: game-2048
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: game-2048
  name: deployment-2048
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: app-2048
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: app-2048
    spec:
      containers:
      - image: alexwhen/docker-2048
        imagePullPolicy: Always
        name: app-2048
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  namespace: game-2048
  name: service-2048
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
  type: NodePort
  selector:
    app.kubernetes.io/name: app-2048
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  namespace: game-2048
  name: ingress-2048
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - path: /*
            backend:
              serviceName: service-2048
              servicePort: 80

However when I describe ingress: I get the following messages

[email protected] MINGW64 ~/Desktop/.k8s
$ kubectl describe ingress/ingress-2048 -n game-2048
Name:             ingress-2048
Namespace:        game-2048
Address:
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host        Path  Backends
  ----        ----  --------
  *
              /*   service-2048:80 (172.31.4.64:80)
Annotations:  alb.ingress.kubernetes.io/scheme: internet-facing
              alb.ingress.kubernetes.io/target-type: ip
              kubernetes.io/ingress.class: alb
Events:
  Type     Reason            Age                From     Message
  ----     ------            ----               ----     -------
  Warning  FailedBuildModel  9s (x13 over 32s)  ingress  Failed build model due to couldn't auto-discover subnets: unable to discover at least one subnet

Here are the tags set on the 3 subnets:

And here are the route table for the subnets, as you can see they have an internet gw attached:

I searched everywhere and they all talk about adding the tags, I created a completely new cluster from scratch but still getting this issue, are there any other things I'm missing?

I checked this answer, but its not relevant because its for ELB not ALB,

================================

Update:

I explicitly added the subnets:

alb.ingress.kubernetes.io/subnets: subnet-xxxxxx, subnet-xxxxx, subnet-xxx

And now I got my external IP, but with some warning

$  kubectl describe ingress/ingress-2048 -n game-2048
Name:             ingress-2048
Namespace:        game-2048
Address:          k8s-game2048-ingress2-330cc1efad-115981283.eu-central-1.elb.amazonaws.com
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host        Path  Backends
  ----        ----  --------
  *
              /*   service-2048:80 (172.31.13.183:80)
Annotations:  alb.ingress.kubernetes.io/scheme: internet-facing
              alb.ingress.kubernetes.io/subnets: subnet-8ea768e4, subnet-bf2821f2, subnet-7c023801
              alb.ingress.kubernetes.io/target-type: ip
              kubernetes.io/ingress.class: alb
Events:
  Type     Reason             Age   From     Message
  ----     ------             ----  ----     -------
  Warning  FailedDeployModel  43s   ingress  Failed deploy model due to ListenerNotFound: One or more listeners not found
           status code: 400, request id: e866eba4-328c-4282-a399-4e68f55ee266
  Normal   SuccessfullyReconciled  43s  ingress  Successfully reconciled

Also going to the browser and using the external ip return: 503 Service Temporarily Unavailable

Solution 1

Ensure that --cluster-name in the aws-load-balancer-controller deployment is correct configured.

Use

kubectl get deployment -n kube-system aws-load-balancer-controller -oyaml |grep "cluster-name"

to get the cluster name in the deployment.

If it isn't correct, edit deployment with next command and rename it:

kubectl edit deployment -n kube-system aws-load-balancer-controller

Solution 2

In my case, it was because the I hadn't labeled the AWS subnets with the correct resource tags. https://kubernetes-sigs.github.io/aws-load-balancer-controller/guide/controller/subnet_discovery/

Edit - 5/28/2021

Public Subnets should be resource tagged with: kubernetes.io/role/elb: 1

Private Subnets should be tagged with: kubernetes.io/role/internal-elb: 1

Both private and public subnets should be tagged with: kubernetes.io/cluster/${your-cluster-name}: owned

or if the subnets are also used by non-EKS resources kubernetes.io/cluster/${your-cluster-name}: shared

Source: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.1/deploy/subnet_discovery/

Solution 3

If upgrading from v2.1 to v2.2 of the aws-load-balancer-controller, be aware you will get this same error as there are new IAM Permissions that are required. See the CHANGELOG here in the release for details / links to those new permissions: https://github.com/kubernetes-sigs/aws-load-balancer-controller/releases/tag/v2.2.0

The explicit link to the IAM Permissions: https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.2.0/docs/install/iam_policy.json

Comments

[Sabir Moglad]

Geez that was the issue! How come this was not set?

[Sabir Moglad]

My bad, I know what step I skipped: ii. Edit the saved yaml file. Delete the ServiceAccount section from the yaml specification. Doing so prevents the annotation with the IAM role from being overwritten when the controller is deployed and preserves the service account that you created in step 4 if you delete the controller. In the Deployment spec section set the --cluster-name value to your Amazon EKS cluster name.

[TlmaK0]

We have all made the same mistake :)

[sunsets]

this answer saved my day

[Blunderchips]

That link no longer exists (404)

[Andrew]

@Blunderchips fixed.

[Blunderchips]

Thanks @Andrew!

[Jerald Sabu M]

Thanks a lot, In my case, everything was correct except the clustername in the label kubernetes.io/cluster/${your-cluster-name}: owned and that fixed it :)