Understanding DIGIT Infra, Services Integration & deployment

Deploying eGov services means that the respective services can be developed, packaged as docker containers and deployed into kubernetes Cluster by simply updating the "master deployment config File per environment" which is derived from the jina2 templates. This allows all the environments specific overrides, credentials and values.

From Code to Deploy flow

Instructions

1. Structure of a Sample service to be maintained in the Git Repo:

If a Micro Service or a Web/Mobile module is being developed it needs to have certain following key elements configured and checked into the Git Repo in the following structure.
1. Git Repo
  1. Micro Services: https://github.com/digit-egov/core-services
  2. Web: https://github.com/digit-egov/frontend
  3. business-services: https://github.com/digit-egov/business-services
  4. business-services: https://github.com/digit-egov/municipal-services
  5. MDMS: https://github.com/digit-egov/playground-mdms-data
2. Each service with have its 'Dockerfile' defining the base image, port and the dependencies, .jar or .war file to copy and start the service or it can refer to a common dockerfile.
3. If a service requires a DB migration it needs to be added in the following hierarchy along with the migration scripts and the seed files

2. Deployment templates in the Git Repo:

https://github.com/egovernments/eGov-infraOps/ OR https://github.com/egovernments/Train-InfraOps
Jinja 2 templates are created for all kubernetes manifests.
Jenkins jobs are used to deploy to cluster using these templates.
.

Environment Configuration (Dev.yml, QA.yml, etc...)
Environment configuration is used to configure the services which shall be deployed on an environment. It serves
1. List of services and their properties
2. Global configuration parameters, ex: db settings
3. Kafka topics and their partition/replication count
4. Endpoints that needs to be whitelisted
5. Encrypted Secrets # Kubernetes Secrets are used to store and serve secrets to various services
An example configuration can be found at InfraOps repo
Secrets.yml
Secrets are provided to various services through Kubernetes Secrets. These secrets are AES encrypted using a salt and configured on environment configuration manifests.
To encrypt
$ python encrypt.py <secret text>

To decrypt

$ python decrypt.py <secret text> | base64 -d

NOTE: Salt needs to be available as environment variable called "EGOV_SECRET_PASSCODE" which is also added in Jenkins Credentials to decrypt while deploying a service.
A separate secrets.yml is maintained to push the required secrets to environments.

Volumes.yml
Some services needs persistent storage to store the data. Volumes are configured through volumes.yml and applied whenever required on the environments.
Kubernetes supports many persistent storage volumes and can be configured in volumes.yml.
Namespaces.yml
eGov cluster consists of 4 major kubernetes namespaces.
- backbone
- egov
- es-cluster
- logging
- backbone
  Backbone namespace consists of
  1. kafka: Kafka is used extensively throughout eGov services platform. Kafka manifest consists of a Service definition and Statefulset definition. These manifests also support data volumes which are needed to persist data. Replicas of kafka nodes can be controlled by the environment manifests.
    1. zookeeper: Zookeeper manifest contains a service spec and Statefulset spec. Number of zookeeper replicas can be controlled through environment manifests.
    2. Redis: Redis is used to store app session data. Redis manifests contains a service definition and Statefulset definition.
    3. kafka-create-topic: By default, kafka creates topic (with 1 replication/ 1 partition) if does not exist when the first message is pushed to a topic by a producer. However, it does not create the topic with increased number of replication and partition based on the cluster setup. Hence, kafka-create-topic is used to create kafka topics. Topic names and their respective replication/partitions are provided through environment manifests. Kafka-create-topic is a Kubernetes Job definition which should be run by kubectl apply command.
  egov
  eGov services mainly follow the following pattern.
  1. Kubernetes Service definition
    1. Kubernetes Deployment definition
    2. Init containers for environment specific db migration if required
    3. Init containers for flyway db migration if required
    4. Service configuration parameters in the form of environment variables
    5. System resource limits
  Note: DB migrations may not be needed for web projects.
  es-cluster
  eGov uses Elasticsearch for storing some of persistent data. It is also used to aggregate all pods logs. es-cluster consists of
  1. es-client: Client ES service which all eGov services connects to
    1. es-data: Data node which stores all data. Number of replicas can be controlled through environment manifests.
    2. es-master: Master node to manage the es-cluster
    3. Kibana: Kibana visualization for both business data and logs
  logging
  eGov uses fluentbit to aggregate all logs and push to elasticsearch. Fluentbit runs as Daemonset on all minions which collects logs from all pods.
  A Kubernetes cronjob called delete-old-logs-in-es is created to cleanup all pods logs from elasticsearch cluster regularly. Number of days to keep the logs can be controlled through environment manifests.
  Addition to cronjob, logrotate cron job will be running on all minions which can rotate the logs including system logs. Frequency and number of days to rotate the logs can be controlled through environment manifests.
  other
  There are two more namespaces which can be deployed based on the need.
  - Playground namespace consists of a playground pod which can be used to execute commands on an environment.
  - Monitoring namespace used to configure monitoring solution for the cluster environment. Currently it is in alpha stage.