eGov Platform_Implementation CI_CD pipelines & it's structures

Contents:

1. Source Code Management (SCM): A Github Strategy

1.1. Git n Github
1.2. public & private repo (platform & implementation)
1.3. Git folder structure
1.4. Branching
1.5. merge with master
1.6. integration with jenkins

1. Jenkins:

2.1. Configuring job: github project
2.2. Implementation build
2.3. Job creation
2.4. Configuring job: approved based promotion
2.5. Build pipeline (ImplBuild, {dev/qa/uat}, {prod})
2.6. Deployments

1. Continuous Integration (CI)
2. Continuous Deliverables (CD)
3. Platform Build and it's stages
4. Customer Build and it's stages
5. Promoting the builds
6. Punjab Implementation
7. BPA Implementation
8. Kozhikode Implementation
9. AP Implementation
10. COC Implementation
11. NDMC Implementation

1. Github Strategy:
1.1 Git and Github -
Git: It is a Distributed Version Control System for tracking versions of files.
Github: It is web portal and cloud hosting service for your Git repositories.
Git allows you to track versions of your code in your local machine. However, if you want a remote backup of your code or want to publish your code to a community then you've to push it to Github.
1.2. Public & Private repo (platform & implementation) -
Public Repositories: All the eGov Platform related projects were kept open, and it's under open source.
Private Repositories: Usually our Implementation/Customer specific Projects goes under the private repo. With some exceptions eGov Platform projects were made private like our DIGIT-BPA-DCR.
1.3. Git folder structure -
Git as a SCM(Source Code Management), project folder structure designed and maintained as shown below.
> ERP Project git repository folder structure:
egov(Repository_Name/branch:master)| pom.xml (parent pom)||_egov-commons(Module_Name)| | pom.xml(Module_pom)| || |src||main|| |_java/org/egov|| ||| |resources|| |config/spring|| |db/migration|| |_main|| | V20160922112506.sql|| |_sample|| V20160922112506.sql||test||Module_Name| | pom.xml(Maven_File)| || |src||main|| |_java/org/egov|| ||| |resources|| |config/spring|| |db/migration|| |_main|| | Migration_File.sql|| |_sample|| Migration_File.sql||_test
> Microservice Project git repository folder structure:
egov-ap-services (Project)
|
|___asset (Modules)
| |
| |___asset-services (services)
| |___[src]
| |___Dockerfile
| |___pom.xml
|
|___web (Modules)
| |
| |___asset-web (services)
||___[app]
||___[resources]
||___Dockerfile
||___build.wrflo
|
|___Jenkinsfile
|___build.sh
1.4. Branching
1.5. merge with master
1.6. integration with jenkins -
Using github-webhooks, project is integrated to the jenkins by adding webhook details:

• Add Payload URL
• Add secret (if wants)
• Select events like to trigger webhook as Just the push event.
• And make trigger event Active
• Do Add Webhooks

3. Continuous Integration (CI) -
Maven as build tool:

• Developers to integrate code into a git repository several times
• Commit the changes to the repository
• Each check-in is then verified by an automated build.
• It pull out the dependencies, packages which are required and mentioned in pom.xml file.
• Make build self-testing using Junit Test
• With all binds and archives, it gets reading with EAR file to publish.

Jenkins as a CI-Server:

• monitors the repository and checks out changes when they occur
• builds the system and runs unit and integration tests
• releases deployable artifacts for testing
• assigns a build label to the version of the code it just built
• informs the team of the successful build
• If the build or tests fail, Jenkins alerts the team
• The team fixes the issue at the earliest opportunity
• Continue to continually integrate and test throughout the project

4. Continuous Deliverables (CD):
Continuous Delivery expands upon continuous integration by deploying all code changes to a environments (de/qa/uat) and/or a production environment after the build stage.

• When properly implemented, developers will always have a deployment-ready build artifact (EAR) that has passed through a standardized test process.
• This published Archive Artifacts (EAR) will promote to the environments (dev/qa)
• Every builds promoted to the environments will be tested this may include UI testing, load testing, integration testing, API reliability testing, etc.
• every code change is built, tested, and then pushed to a non-production testing (dev/qa) or staging environment (UAT).
• The difference between continuous delivery and continuous deployment is the presence of a manual approval to update to production. With continuous deployment, production happens automatically without explicit approval.
• Every revision that is committed triggers an automated flow that builds, tests, and then stages the update. The final decision to deploy to a live production environment is triggered by the Approver.

5. Platform Build and it's stages:

Fig.1.0: Build Stage
Code integration via SCM (Source Code Management) for the Platform and Implementation makes difference by branching out with the respective, from the master branch. And having separate build for the same. Code checkout, and building follows the build stage as shown in the block diagram (Fig.1.0). All the necessary packages, and dependencies where pushed/published in the package manager repository (Nexus). This contributes in the continuous integration. Here maven helps to build EAR(Enterprise Application aRchive) file.

Fig.2.0: CI Pipeline for Platform/Product


6. Customer Build and it's stages:
Customer/Implementation specific code integration goes to the implementation branch which is checked out from the main branch of SCM (Source Code Management).
Customer/Implementation related packages, and dependencies binds with Product specific packages, and dependencies by downloading from the Nexus repository on every build trigger. Maven builds to produce EAR file at the end for this process. This is also be a part of the continuous integration.

Fig.3.0: CI Pipeline for Customer/Implementation
7. Promoting the build to environments: It aims at building, testing, and releasing software. From the continuous integration part we published the EAR(Enterprise Application aRchive) file. Now we need to publish/deploy EAR file to different environments. This has to undergo deployment process for the build promotion to the different environments(dev/qa), staging, as well production. This promotion is manually done hence it contributes in the continuous delivery.

Fig.4.0: CI/CD Pipeline for Implementation
8. Punjab Implementation

1. Platform Build (Product)
2. Customer Build (Implementation)

Punjab Implementation Build illustration:

1. Impl build starts by Gtub push
2. Building remotely on jenkins server in private workspace called /punjab-impl-build
3. Fetching changes from the remote Git repository
4. Fetching upstream changes from git@github.com:egovernments/eGov-Punjab-Implementation.git
5. using GIT_SSH to set credentials
6. Checking out Revision
7. Notifier
8. Parsing POMs
9. Established TCP socket on 37441
10. checks for changes/updates for maven32-agent.jar, maven32-interceptor.jar, maven3-interceptor-commons.jar
11. JENKINS REMOTING CAPACITY = channel started
12. Executing Maven:

• Scanning for projects
• e-governments punjab erp
• egov-punjab-impl
• e-governments punjab erp ear

1. Building egov-punjab-impl PB-3.1.0-SNAPSHOT
2. Download packages, and dependencies from the nexus repo
3. BUILD SUCCESS
4. Archiving artifacts
5. Recording fingerprints
6. Notifier was triggered for: Always
7. Finished: SUCCESS

Next we have to promote the successful On Successful build, we get the punjab-impl-build number (ex: Build #1337). This build we have to promote to the required environments by Approving build promotion.

• Punjab git: deployments uses master branch
• Finance integration with the W&S module pipeline
  9. BPA Implementation:

1. Platform Build (Product)

• DIGIT BPA - Public
• DIGIT DCR - Private

Fig.5.0: CI/CD pipeline for BPA platform

1. Customer Build - Public

Fig.6.0: CI/CD Pipeline for BPA Implementation
10. Kozhikode Implementation: Its a direct customer specific implementation for BPA. * no RDS for kzkd.
Kozhikode CI/CD Pipeline Build: [Kozhikode Build]; [Deploy to AZ-Kozhikode-UAT]; [Deploy to Kozhikode PROD].
Kozhikode PROD cluster config: cloud provider = AWS, LB = Apache2, Wildfly instances = , Postgres DB = 1 Master, ES instances = , Filestore size = each,
11. Punjab Implementation:
Punjab W&S Pipeline: Punjab-ws-platform-build; punjab-ws-impl-build; Deploy to Punjab WS DEV; Deploy to Punjab WS UAT; Deploy to Punjab WS TRAINING; Deploy to Punjab WS PROD.
Punjab W&S cluster config: cloud provider =; LB = Apache2; Wildfly instances = 0; Postgres DB = 0 Master, 0 Slave, 0 backup; ES instances = 0; Filestore size = 0 each.
Punjab Coex pipeline: punjab-coexistence-impl-build; deploy-punjab-coe-dev; deploy-punjab-coe-qa; deploy-punjab-coe-uat; deploy-punjab-coe-uat
Punjab Coexistence cluster config: cloud provider =; LB = Apache2; Wildfly instances = 0; Postgres DB = 0 Master, 0 Slave, 0 backup; ES instances = 0; Filestore size = 0 each.
12. Digit BPA:
Digit-BPA-pipeline: digit-bpa-build; deploy-digit-bpa-dev; deploy-digit-bpa-dev
13. Bihar Implementation:
Bihar impl BPA pipeline: bihar-impl-build; deploy-bihar-uat
14. AP Implementation: * AP UAT and prod - is on cntrlS
cloud provider = AWS; LB = Apache2; Wildfly instances = 4; Postgres DB = 1 Master, 1 Slave, 1 backup; ES instances = 4, Filestore size = ,
15. COC Implementation:
16. NDMC Implementation (using only JBoss)