...

1. Scale Down Nginx-Ingress Controller

   • Scale down the nginx-ingress controller to zero replicas.

     Code Block
     language none
     kubectl scale deployment nginx-ingress-controller --replicas=0 -n <namespace>

2. Monitor Kafka Lags

   • Monitor Kafka consumer lags until they reach zero to ensure no pending messages.

     Code Block
     language bash
     kafka-consumer-groups --bootstrap-server <kafka-broker> --describe --group <consumer-group>

     If latest monitoring is available, use Kafka-UI to monitor Kafka consumer lags.

     Code Block
     language none
     kubectl port-forward svc/kafka-ui 8080:8080 -n <namespace> # visit http://localhost:8080/kafka-ui to access dashboard

3. Scale Down Cluster Worker Nodes

   • Scale down the worker nodes from AWS Auto Scaling Groups, to prevent any further activities.

4. Backup EBS Volumes

   • Take snapshots of the EBS volumes attached to Persistent Volumes (PVs).

     • Kafka

     • Kafka - Infra (if available)

     • Zookeeper (if available)

     • Elasticsearch (data & master)

     • Elasticsearch - Infra (data & master) (if available)

5. Scale Down Cluster Worker Nodes

   • Scale down the worker nodes from AWS Auto Scaling Groups, to prevent any further activities.

Upgrading

1. Clone the DIGIT-DevOps repository.

   Code Block
   language bash
   git clone https://github.com/egovernments/DIGIT-DevOps.git

2. Navigate to the cloned repository and checkout the release-1.28-Kubernetes branch.

   Code Block
   language bash
   cd DIGIT-DevOps git checkout release-1.28-kubernetes

3. Check if the correct aws credentials are configured using aws configure list.
   Else run to aws configure to configure AWS CLI.

4. Open input.yaml file and fill in the inputs as per the regex mentioned in the comments.

5. Go to infra-as-code/terraform/sample-aws and run init.go script to enrich different files based on input.yaml.

   Code Block
   language bash
   cd infra-as-code/terraform/sample-aws go run ../scripts/init.go

6. Update EKS version under variable "kubernetes_version" in variables.tf file and update ami_id under module "eks" "worker_groups" in main.tf.
   Note: ami_id can be fetched using below cmd

   Code Block
   language bash
   aws ssm get-parameters --region <region> --names /aws/service/eks/optimized-ami/<eks_version>/amazon-linux-2/recommended/image_id

   
   Run below terraform commands to upgrade EKS.

   Code Block
   language bash
   terraform init # initializes terraform in current working directory terraform plan # creates an execution plan (verify this before runnind "apply") terraform apply # applies changes mentioned in execution plan

...