Steps involved in eGov Deployment - Starter Kit

Create Kubernetes Cluster

...

Local Development K8S Cluster: Use development support environment for creating Kubernetes Cluster. Eg : Minikube.

...

Cloud Native K8S Cluster Services: Use Cloud Services to create Kubernetes Cluster. Eg : AWS EKS Service.

...

On-Premise K8S Cluster: Create your own Kubernetes Cluster with the help of master and worker nodes.

...

What does deploying the DIGIT (eGov) platform mean

• Provisioning the kubernetes Cluster in any of the cloud or private datacenter
• Setting up the disk volumes to attach to the stateful containers (Kafka, Elastic Search, ZooKeeper)
• Setting up the DB
• Preparing Deployment configuration using Jinja Template
  • K8s Secrets
  • K8s ConfigMaps
• Setting up Jenkins Job to deploy the components for the rolling updates.

High-level sequence of deployment

1. Create Kubernetes Cluster
2. DB Setup
3. Provision Persistant volumes
4. Setup Registry for build libraries & docker images
5. Services Lists
   1. Backbone
   2. Gateway
   3. eGov Platform
   4. Municipal services
   5. Infra services
   6. Frontend
6. Setup CI
7. Setup Deployment CI/CD

Prerequisites

• General
  
  • Understanding of Instances, LoadBalancers, SecurityGroups/Firewalls, ngnix, DB Instance, Data Volumes.
  • Experience of kubernetes, docker, jenkins, ngnix, Infra-as-code, Terraform
  • Permission and and account to provision the above resources.
• On-premise/private cloud requirements
  • Cloud Interface access to provision above infra
• Public cloud requirements
  • AWS or Azure account

Image Added

Infrastructure Design

• Image Added

Infrastructure Components

1. Basic Requirements (Size, Memory)
   
   • Public Cloud
     • Cloud native kubernetes engine like AKS, or EKS or GKE from AWS or Azure or GCP respectively
     • 6 k8s Nodes-
   m4large 
   • • with each 16GB RAM and 4 vCore CPUs
   • On-Prem/Data Center or custom 
     • 1 Bastion- t2micro (Gateway) 2GB RAM 1vCore CPU
     • 3 k8s Master- t2medium 4 GB RAM 2 vCore CPU
     • 6 k8s Nodes with each 16GB RAM and 4 vCore CPUs

2. DB Setup

   1. Use managed RDS Service with PostGres if you are using AWS, Azure or GCP

   2. Else provision a VM with PostGres DB and attach external volumes

3. Create Production ready Application Services Lists

   1. Infra Services - Dashboard, Kibana, Telemetry, Logging

   2. Backbone
      1. Infra: Elastic Search HA Cluster, Kafka HA Cluster, Zookeeper HA Cluster, Redis, Kafka Connect, Kibana, ES Curator
      2. Business: Elastic Search HA Cluster, Kafka HA Cluster, Zookeeper HA Cluster, Kafka Connect, Kibana, ES Curator

   3. eGov Platform Services

      1.  

   1. Municipal Services
      1. Public Grievance Redressal System
      2. Property Tax System
      3. Trade License System
      4. Accounting System
      5. Water & Sewerage Management
      6. Dashboards
      7. Fire No Objection Certificate (NoC)

   2. Frontend

      1. Web Citizen, Employee, etc

4. Keep data on volumes, Create persistant volums
   
   1. In care of Kubernetes volume, when a pod is deleted, all the data is lost too.
   2. Thus, we use Persistent Volumes, which will keep the data even if a pod is spawned.
   3. It stores the data on our local storage.
   4. eg: There is a Persistent Volume for elastic searchl. So, if the data inside the database will increase, the size of the local storage will also be needed to be increased.
   5. Thus, it is a best practice to keep database outside the kubernetes cluster.

5. Setup Registry for build libraries & docker images

   1. Create central container registry such as AWS EKS or Gitlab Registry or DockerHub or Artifactory.

   2. CI Tools will push the container image to the central container registry.

   3. Here we are using Nexus, DockerHub as shared repositories.

6. Setup CI

7. Create and configure shared repository for Continuous Push, Commit, etc.

8. Setup CI like Jenkins and create the Job that creates a pipeline from the JenkinsFile.
Platform services can be referred and forked from Infrastructure type (Services, DB servers, Storage, etc)

1. DB
2. Persistant volumes
3. LoadBalancer
4. DNS
9. Environments to set up (Development, Production, etc)
   1. Dev, QA, UAT and Prod
10. Services to be used on each Infrastructure type
    1. Backbone
    2. eGov Platform Services
    3. Municipal Services
    4. Infra Services
    5. Frontend

Deployment Architecture:

• Every code commit is well reviewed and merged to master branch through Pull Requests

• Each merge triggers a new CI Pipeline that ensures CodeQuality and CITests before building the artefacts.

• Artefact are version controlled and pushed to Artifactory like Nexus. https://repo.egovernments.org

• After successful CI Jenkins bakes the Docker Images with the Latest Artefacts and pushed the newly baked docker image to Docker Repo.

• Deployment Pipeline pulls the Image and pushes to the corresponding Env.

       Deployment Scripts:

• https://github.com/egovernments/

...

• eGov-

...

Install and configure continuous deployment tools for automatic build and test.

...

Builds are created from the Build management scripts, written inside InfraOps GitHub Repo (Private).

...

Setup CI/CD Pipeline

1. We are using Jenkins/Spinnaker as CI/CD Tool.

                        

Http Traffic/Routing:

...

• infraOps

• Python based Deployment script that reads the value from the Jinja 2 template and deploys into the cluster.       

• Each env will have one Jinja Template that will have the definition of services to be deployed, their dependancies like Config, Env, Secrets, DB Credentials, Persistent Volumes, Manifest, Routing Rules, etc.. 

 Image Added

Deployment Configurations

• eGov Services Integration & deployment

Let's Get Started

...

1. Create Kubernetes Cluster
   1. Local Development K8S Cluster: Use development support environment for creating Kubernetes Cluster. Eg : Minikube.
   2. Cloud Native K8S Cluster Services: Use Cloud Services to create Kubernetes Cluster. Eg : AWS EKS Service.
   3. On-Premise K8S Cluster: Create your own Kubernetes Cluster with the help of master and worker nodes.
   4. K8s Cluster Requirement
      1. Cloud native kubernetes engine like AKS, or EKS or GKE from AWS or Azure or GCP respectively
         • 6 k8s Nodes- m4large with each 16GB RAM and 4 vCore CPUs
      1. On-Prem/Data Center or custom 
         • 1 Bastion- t2micro (Gateway) 2GB RAM 1vCore CPU
         • 3 k8s Master- t2medium 4 GB RAM 2 vCore CPU
         • 6 k8s Nodes with each 16GB RAM and 4 vCore CPU

   5. Kubernetes Cluster Provisioning 

      1. Managed Kubernetes Engine:

      1. Choose your cloud provider (Azure, AWS, GCP or your private)

      2. Choose to go with the cloud provider specific Kubernetes Managed Engine like AKS, EKS, GKE. 

      3. Follow the Cloud provider specific instruction to create a Kubernetes Cluster (stable version 1.11 and Beyond) with 5 to 6 worker nodes with 16GB RAM and 4 vCore CPU (m4.xlarge)

      4. PostGres DB (Incase of AWS, Azure, GCP use the RDS) and have the DB server and the DB Credentials details.

      5. Provision the disk volumes for Kafka, ES-Cluster and ZooKeeper as per the below baselines and gather the volume ID details.

      6. Install Kubectl on DevOps local machine to interact with the cluster, setup kubeconfig with the allowed user credentials.

         1. Sample volumes to be provisioned

      Image Modified

            

            2. Private Cloud - Manually setup Kubernetes Cluster:

      • Create a VPC or Virtual Private Network with multi availability zones 

      • Provision the Linux VMs with any Container Optimised OS (CoreOS, RHEL, Ubuntu, Debian, etc) within the VPC Subnet.

      • Provision 1 Bastion Host that acts as proxy server to Kubernetes cluster Nodes.

      • 3 Master Nodes with 4 GB RAM 2 vCore CPU

      • 6 worker nodes with 16GB RAM and 4 vCore CPU

      • PostGres DB (Linux VM)

      • Provision the disk volumes for Kafka, ES-Cluster and ZooKeeper as per the below baselines and gather the volume ID details.

      • Create LoadBalancer or Ingress to talk to Kube API server that routes the external traffic to the services deployed on cluster.
      • Setup AuthN & AuthZ. 

      • Install Kubectl on DevOps local machine to interact with the cluster, setup kubeconfig with the allowed user credentials

      
      

             Useful Step-By-Step Links:    

      • Installing the Client Tools
      • Provisioning Compute Resources
      • Provisioning the CA and Generating TLS Certificates
      • Generating Kubernetes Configuration Files for Authentication
      • Generating the Data Encryption Config and Key
      • Bootstrapping the etcd Cluster
      • Bootstrapping the Kubernetes Control Plane
      • Bootstrapping the Kubernetes Worker Nodes
      • Configuring kubectl for Remote Access
      • Provisioning Pod Network Routes

Image Removed            Image Removed

Deployment Architecture:

• Every code commit is well reviewed and merged to master branch through Pull Requests

• Each merge triggers a new CI Pipeline that ensures CodeQuality and CITests before building the artefacts.

• Artefact are version controlled and pushed to Artifactory like Nexus.

• After successful CI Jenkins bakes the Docker Images with the Latest Artefacts and pushed the newly baked docker image to Docker Repo.

• Deployment Pipeline pulls the Image and pushes to the corresponding Env.

       Deployment Scripts:

• Python based Deployment script that reads the value from the Jinja 2 template and deploys into the cluster.       

• Each env will have one Jinja Template that will have the definition of services to be deployed, their dependancies like Config, Env, Secrets, DB Credentials, Persistent Volumes, Manifest, Routing Rules, etc..

 Image Removed     Image Removed

1. DB Setup
   1. Use managed RDS Service with PostGres if you are using AWS, Azure or GCP
   2. Else provision a VM with PostGres DB and attach external volumes
2. Keep data on volumes, Create persistant volums
   1. In case of Kubernetes volume, when a pod is deleted, all the data is lost too.
   2. Thus, we use Persistent Volumes, which will keep the data even if a pod is spawned.
   3. It stores the data on our local storage.
   4. eg: There is a Persistent Volume for elastic searchl. So, if the data inside the database will increase, the size of the local storage will also be needed to be increased.
   5. Thus, it is a best practice to keep database outside the kubernetes cluster.
3. Setup Registry for build libraries & docker images
   1. Create central container registry such as AWS EKS or Gitlab Registry or DockerHub or Artifactory.
   2. CI Tools will push the container image to the central container registry.
   3. Here we are using Nexus, DockerHub as shared repositories
4. Create Production ready Application Services Lists
   1. Backbone
      1. Infra: Elastic Search HA Cluster, Kafka HA Cluster, Zookeeper HA Cluster, Redis, Kafka Connect, Kibana, ES Curator
      2. Business: Elastic Search HA Cluster, Kafka HA Cluster, Zookeeper HA Cluster, Kafka Connect, Kibana, ES Curator
   2. Gateway
      1. ngnix
      2. zuul

         
         

         Image Added  Image Added

   3. eGov Platform Services
      1. egovio/employee
      2. egovio/citizen
      3. egovio/user-otp
      4. egovio/egov-accesscontrol
      5. egovio/egov-common-masters
      6. egovio/egov-filestore
      7. egovio/egov-idgen
      8. egovio/egov-indexer
      9. egovio/egov-localization
      10. egovio/egov-persister
      11. egovio/egov-searcher
      12. egovio/rainmaker-pgr
      13. egovio/egov-notification-sms
      14. egovio/egov-otp
      15. egovio/egov-user
      16. egovio/hr-masters-v2
      17. egovio/hr-employee-v2
      18. egovio/report
      19. egovio/tenant
      20. egovio/egov-mdms-service
   4. Municipal Services
      1. Public Grievance Redressal System
      2. Property Tax System
      3. Trade License System
      4. Accounting System
      5. Water & Sewerage Management (ERP)
      6. Dashboards
      7. Fire No Objection Certificate (NoC)
   5. Infra Services
      1. Fluentd,
      2. Kibana
      3. Telemetry
      4. Logging
   6. Frontend
      1. Web Citizen, Employee, etc.
5. Setup CI
   1. Create and configure shared repository for Continuous Push, Commit, etc.
   2. Setup CI like Jenkins and create the Job that creates a pipeline from the JenkinsFile.
   3. Platform services can be referred and forked from https://github.com/egovernments/egov-services
   4. Install and configure continuous deployment tools for automatic build and test.
   5. Builds are created from the Build management scripts, written inside InfraOps GitHub Repo (Private).
   6. With every deployment, a new container is deployed which saves the overhead of doing manual configurations.
   7. We can run pre-built and configured components in containers as a part of every release. 
   8. We provide all the kubernetes cluster configurations through .yaml files, which is usually called as the desired state of the cluster.
   9. Kubernetes offers autoscaling of workload based on CPU utilisation and memory consumption.
   10. Kubernetes allows vertical scaling-increasing the no. Of CPU’s for pods and horizontal scaling-increasing the no. of pods 
6. Upgrading the platform/installation/services
   1. We are using Jenkins/Spinnaker as CI/CD Tool
   2. Image Modified

Cluster/Service Monitoring 

• Monitoring

1. Prometheus / CloudWatch for node monitoring
2. Prometheus for pod level monitoring

...

1. Jaeger for distributed tracing
2. Traces are tagged with correlation-id

...

Multi-environment Cluster Orchestration and Management