eGov ERP DevOps

Punjab-finance coex and punjab w&s prod deployment failure


Problem Statement:-

Title

Punjab-finance coex and punjab w&s prod deployment failure

Title

Punjab-finance coex and punjab w&s prod deployment failure

References

 

Description

  1. deploy-punjab-ws-prod: Wildlfy Management service not able to verify on the host (app1). Its throwing curl 28 error. Henceforth deployment is not happening on the prod.

  2. deploy-punjab-coe-prod: deployment is failing due to the ssh execution error. Henceforth deployment is not happening on the prod.

Occurrence Date

[29/07/2019]

Impact

Deployment failure, development stopped

RCA Owner

Manjunatha S


Observations/Analysis of the problem:-

  • Punjab WS prod has 1 Apache, 2 Wildfly servers.

    • deployment job in jenkins is ear-deployer

    • deployment using wildfly management console user is configured in this job

    • each management realm users are added in the individual wildfly app servers (i.e, app01 and app02)

    • method of deployment on wildfly server were type called hot deployment (management user does this job by curl://)

  • Punjab coex prod has 1 Apache, 1 Wildfly server.

    • ear-deployer config file has an server ssh details and for the hot deployment console user details in the encrypted form

    • ssh details created were not in the sudoer group, where docker runs on this machine for the deployment.

Snapshots:

Solution Approach:-

  1. Coex as its new deployment for the new instance, server user not able to perform the ear-deployment activity it required sudoers.

  2. Added the new server user to the sudoer group.

  3. For Punjab W&S - As temporary fix do manually deployment by keeping latest .ear file in the /wildfly/standalone/deployments/ and restart the wildfly.

  4. Punja WS deployment failing due to wildfly management service user verification is getting timedout with curl 28 error

  5. For the Punjab-coex deployment - add the ssh user to sudo group where ear-deployer is performing.


Timeline:-

Time

Involved Parties

Description

Time

Involved Parties

Description

[29/07/2019]

Manjunatha

Issue arises

[07/08/2019]

Manjunatha, and related Punjab team

Issue resolved


Conclusion:-

 


Related articles

Related articles:

 

DevOps as a Culture