Background
- Upon the peak memory utilisation of all the existing nodes in punjab cluster and implementation team's demand to match the need of the load, had to resize the cluster by adding better nodes. Of course upon customer approval & Demand.
- Punjab cluster resizing activity, causing customer dashboard to loose its visualisation and dashboard.
- While there was a peak load, the DB was bottleneck due to long running queries that cascaded into APIs and the system was slow.
- We had to fix the queries
Retrospective
CLUSTER:
- Though the cluster resize task as was successful, stateful pods (es-cluster) the way it is configured and the way kubernetes manages via headless service discovery resulted in ES-MASTER PODS being not connecting to each other, by the time the DevOps team troubleshoot and recover the ES-MASTER, Kibana lost its visualisation and dashboard indexes.
- No impact to the ES-Data, but kibana as lost its dashboard and visualisation indexes.
DB:
- APIs were slow during the peak load from the punjab system.
- Reports were taking much of the DB capacity due the inefficient joins, non-indexed data and Nested loop queries to DB, etc.
- We fixed DB Queries and brought down the
| Start doing | Stop doing | Keep doing |
|---|---|---|
|
|
|
Action items
- Gajendran C (Unlicensed), Abhishek Jain to figure out on how the service discovery strategy being followed at EkStep (Mathews) – Need to explore the Headless Service Discovery topic.
- Nithin DV (Unlicensed) Partitioning strategy for tables (Time & Tenant Based and on kafka topics)
- Ghanshyam Rawat Strategy plan to Index data based on columns of the tables, Query Optimisation
- Ghanshyam Rawat DB Script review process plan
- Abhishek Jain, Nithin DV (Unlicensed) To review what kind of data being captures and logged by Jaeger tracing, logs should be masked (Important)
- Gajendran C (Unlicensed) Production infra alert / dashboard (ES, Kafka, Zuul, DB) to the roadmap
- Nithin DV (Unlicensed), Tarun Lalwani To come up with the proof point on the need for synchronous calls to solve the specific use case.
- Gajendran C (Unlicensed) Setup the Platform PRDs prioritisation meeting with Abhishek Jain to keep it intact with the Operating plan strategy.