Preparation
Bring Down Nginx-Ingress Controller
Scale down the nginx-ingress controller to zero replicas.
kubectl scale deployment nginx-ingress-controller --replicas=0 -n <namespace>
Monitor Kafka Lags
Monitor Kafka consumer lags until they reach zero to ensure no pending messages.
kafka-consumer-groups --bootstrap-server <kafka-broker> --describe --group <consumer-group>
If latest monitoring is available, use Kafka-UI to monitor Kafka consumer lags.
kubectl port-forward svc/kafka-ui 8080:8080 -n <namespace> # visit http://localhost:8080/kafka-ui to access dashboard
Backup RDS DB Instance
Create a snapshot of the RDS instance from AWS console or using below AWS CLI cmd.
aws rds create-db-snapshot --region <region> --db-instance-identifier <db-instance-id> --db-snapshot-identifier <snapshot-id>
Upgrading
Clone the DIGIT-DevOps repository.
git clone https://github.com/egovernments/DIGIT-DevOps.git
Navigate to the cloned repository and checkout the release-1.28-Kubernetes branch.
cd DIGIT-DevOps git checkout release-1.28-kubernetes
Check if the correct aws credentials are configured using
aws configure list
.
Else run toaws configure
to configure AWS CLI.Open input.yaml file and fill in the inputs as per the regex mentioned in the comments.
Go to infra-as-code/terraform/sample-aws and run init.go script to enrich different files based on input.yaml.
cd infra-as-code/terraform/sample-aws go run ../scripts/init.go
Update engine version under module “db“ in main.tf file and run below terraform commands to upgrade RDS.
terraform init # initializes terraform in current working directory terraform plan # creates an execution plan (verify this before runnind "apply") terraform apply # applies changes mentioned in execution plan
Post Upgrade
Login to DB and Execute basic SQL Queries
Login using existing credentials and run basic SQL queries.
Make sure both Admin & Read credentials are working as expected.
Restart Services
Restart all services connected to the database to ensure they are utilizing the updated instance.
Scale Up Nginx-Ingress Controller
Scale up the nginx-ingress controller to required N replicas.
kubectl scale deployment nginx-ingress-controller --replicas=N -n <namespace>
Verify System Health
Monitor the overall system health to ensure everything is functioning as expected.
Add Comment