Ingress-Nginx + EKS Setup

After setting up workloads in EKS, the requirement was to expose the workloads outside the cluster. Here we can use a Kubernetes Ingress resource, which can be used to manage traffic to various Kubernetes Services. To manage the traffic using these ingress resources, we used Ingress-Nginx. It’s an open source ingress controller that’s widely used. Here’s the guide on how to set up ingress-nginx in an EKS cluster and use AWS Certificate Manager (ACM) for the TLS termination.

Ingress-Nginx Deployment#

We can deploy ingress-nginx using helm cli or kubectl (Kubectl Manifests). First we need to get the VPC details and the ACM Arn

Get VPC Cidr. Replace VPC_ID.

aws ec2 describe-vpcs --vpc-ids VPC_ID --query "Vpcs[0].CidrBlock" --output text```

Get the ACM Arn. Replace DOMAIN_NAME

aws acm list-certificates --query "CertificateSummaryList[?DomainName=='DOMAIN_NAME'].CertificateArn" --output text

Helm#

Add ingress-nginx helm repo

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

Use the following custom values file. Replace VPC_CIDR and ACM_ARN from the values we obtained before

1
controller:
2
  config:
3
    http-snippet: |
4
      server {
5
        listen 2443;
6
        return 308 https://$host$request_uri;
7
      }
8
    proxy-real-ip-cidr: VPC_CIDR
9
    use-forwarded-headers: "true"
10
  service:
11
    enabled: true
12
    external:
13
      enabled: true
14
    annotations:
15
      service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
16
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
17
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: ACM_ARN
18
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
19
      service.beta.kubernetes.io/aws-load-balancer-type: nlb
20
    type: LoadBalancer

Deploy the ingress-nginx chart

helm -n ingress-nginx upgrade --install ingress-nginx ingress-nginx/ingress-nginx -f ingress-nginx-values.yaml --create-namespace

Kubectl#

Get the kubectl deployment yaml for ingress-nginx with AWS support - Ref

curl -O https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.12.3/deploy/static/provider/aws/nlb-with-tls-termination/deploy.yaml

Change the values accordingly:

proxy-real-ip-cidr: XXX.XXX.XXX/XX - set value to VPC_CIDR
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-west-2:XXXXXXXX:certificate/XXXXXX-XXXXXXX-XXXXXXX-XXXXXXXX - set value in annotations to ACM_ARN

Deploy the kubectl yaml

kubectl apply -f deploy.yaml

Testing Ingress-Nginx with TLS#

Since ingress-nginx is deployed, we’ll need to wait for all the pods to come up.

kubectl -n ingress-nginx get pods

Get the NLB hostname that’s assigned to the ingress-nginx-controller kubernetes service. Add this hostname as the value of an A record in your DNS provider. The A record name should be the domain where you’ll need to access the kubernetes workload.

kubectl -n ingress-nginx get svc ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

Get the ingress class name used by ingress-nginx controller. By default the value will be nginx

kubectl get ingressclasses.networking.k8s.io

Now we can deploy the sample workload to test this

1
apiVersion: v1
2
kind: Pod
3
metadata:
4
  name: nginx
5
  labels:
6
    app: nginx
7
spec:
8
  containers:
9
    - name: nginx
10
      image: nginx:latest
11
      ports:
12
        - containerPort: 80
13
---
14
apiVersion: v1
15
kind: Service
16
metadata:
17
  name: nginx-service
18
spec:
19
  selector:
20
    app: nginx
21
  ports:
22
    - protocol: TCP
23
      port: 80
24
      targetPort: 80
25
  type: ClusterIP
26
---
27
apiVersion: networking.k8s.io/v1
28
kind: Ingress
29
metadata:
30
  name: nginx-ingress
31
spec:
32
  ingressClassName: nginx  # This must match the Ingress Controller's class name
33
  rules:
34
    - host: SAMPLE_DOMAIN  # Replace with your actual domain
35
      http:
36
        paths:
37
          - path: /
38
            pathType: Prefix
39
            backend:
40
              service:
41
                name: nginx-service
42
                port:
43
                  number: 80

Now try to access the domain at https://SAMPLE_DOMAIN. You’ll see the default nginx page. You can also notice how the ssl works at https://SAMPLE_DOMAIN.