RDS Upgrade

Preparation

1. Bring Down Nginx-Ingress Controller

   • Scale down the nginx-ingress controller to zero replicas.

     kubectl scale deployment nginx-ingress-controller --replicas=0 -n <namespace>

2. Monitor Kafka Lags

   • Monitor Kafka consumer lags until they reach zero to ensure no pending messages.

     kafka-consumer-groups --bootstrap-server <kafka-broker> --describe --group <consumer-group>

     If latest monitoring is available, use Kafka-UI to monitor Kafka consumer lags.

     kubectl port-forward svc/kafka-ui 8080:8080 -n <namespace>
     # visit http://localhost:8080/kafka-ui to access dashboard

3. Backup RDS DB Instance

   • Create a snapshot of the RDS instance from AWS console or using below AWS CLI cmd.

     aws rds create-db-snapshot --region <region> --db-instance-identifier <db-instance-id> --db-snapshot-identifier <snapshot-id>

Upgrading

1. Clone the DIGIT-DevOps repository.

   git clone https://github.com/egovernments/DIGIT-DevOps.git

2. Navigate to the cloned repository and checkout the release-1.28-Kubernetes branch.

   cd DIGIT-DevOps 
   git checkout release-1.28-kubernetes

3. Check if the correct aws credentials are configured using aws configure list.
   Else run to aws configure to configure AWS CLI.

4. Open input.yaml file and fill in the inputs as per the regex mentioned in the comments.

5. Go to infra-as-code/terraform/sample-aws and run init.go script to enrich different files based on input.yaml.

   cd infra-as-code/terraform/sample-aws 
   go run ../scripts/init.go

6. Update engine version under module “db“ in main.tf file and run below terraform commands to upgrade RDS.

   terraform init    # initializes terraform in current working directory
   terraform plan    # creates an execution plan (verify this before runnind "apply")
   terraform apply   # applies changes mentioned in execution plan

Post Upgrade

1. Login to DB and Execute basic SQL Queries

   • Login using existing credentials and run basic SQL queries.

   • Make sure both Admin & Read credentials are working as expected.

2. Restart Services

   • Restart all services connected to the database to ensure they are utilizing the updated instance.

3. Scale Up Nginx-Ingress Controller

   • Scale up the nginx-ingress controller to required N replicas.

     kubectl scale deployment nginx-ingress-controller --replicas=N -n <namespace>

4. Verify System Health

   • Monitor the overall system health to ensure everything is functioning as expected.