Self Contained Service Architecture (HLD)

Objective:

The objective of this module is to handle configurations of services in a centralised manner. The overall objective of the architecture is to simplify configuration handling and remove the need for manual pod restarts to refresh the configurations. The configurations that can be added by this handler are MDMS, pdf, perister, indexer, dss, reports, workflow and localisation.

Problems with existing approach:

Currently any developer who is working on a new service has to checkin configuration at 6 different places(includes reports, pdf template etc.). After checking in the configuration the developer has to restart all of these 6 pods. Once this is done, the developer needs cURL for workflow and localisation service to insert their configuration/data. The whole process is long, tedious and error prone(many times people forget to restart pods after adding config). This approach requires access to GitHub and jenkins which is not possible in the case of a sandbox environment.

API Details:

The handler will provide API endpoints to manage configurations. The structure of the API request will be as follows:

The request will contain an array of objects. Each object will contain the module name and the configuration that has to be added.

The handler will contain two APIs, _create and _update. The _create API will be used to add configuration for the first time. It will work in idempotent manner i.e if any configuration is already present it will ignore it (won’t add that particular config). Modules can integrate with this _create API. During application initialisation, the service will call this API to add the default configurations(if required). For any further changes to configuration the developer can call the _update API.

Moving Config to MDMS:

Currently services like persister, indexer, etc. use a git-sync container to load configuration files.If we run them as sidecar containers, each service will require one. We can remove this container from all these services by enhancing the MDMS service. This service(s) will now load configuration files by calling MDMS service. MDMS service will return the requested configuration files. Using this approach only MDMS service will be required to run the git-sync pod. 

Note: 

1. We will keep the existing resource loader logic to load from the resource folder. In addition to that we will provide another type of resource loader which will fetch from MDMS. Based on the environment variable it will select the loader.

2. Currently MDMS data is of JSON format. We need to add support(if not present) for YAML files to serve the configuration from MDMS.(Another way is to change the configuration format from YAML to JSON)

Design:

Open

The configuration handler will integrate with Git and the required DIGIT services (workflow, localisation etc.) to add/update configuration. Once the configuration is added in Git, the pods using this configuration should pull the latest changes. There are multiple approaches for achieving this. Below are the list of approaches with their pros and cons.

1. Periodic pull using git-sync:

The git-sync will run as sidecar container instead of init container and will periodically pull the Git repo based on configured frequency. We can configure the frequency around 1-2 mins (Note: git-sync checks the hash first and only pulls the data if hash has changed)

Pros:

Cons:

2. Modify git-sync code :

Understand and modify the git-sync code to expose an endpoint for manual pull. Whenever the config handler uploads configs on Git it will call this API of the modified git-sync container to initiate pull

Pros:

Cons:

Refreshing the configuration:

Once the configuration is pulled by git-sync the services should be notified to reload the configuration. Git-sync provides webhooks which are called once the new hash is synced. We will add one /config/_refresh API in each of these services and configure them in git-sync. Whenever git-sync pulls data successfully it will trigger _refresh API which will then refresh the configuration.

Since the _refresh API call goes to only one pod, only that pod will refresh the data. Because of this we cannot use local cache. We will need a cluster-aware caching implementation, which we can do using the Redis instance that is already deployed. By moving the configuration to redis, even if one pod refreshes the cache it will reflect in all the pods for that service.

For services like indexer and persister which load consumers from configuration, cache burst won’t work. We will have to use configuration versioning. In cache we will store configuration along with the hash associated with that version. Services like indexer and persister will maintain a variable called localOperatingConfigHash in its heap memory. Whenever it fetches data from redis cache it will compare the localOperatingConfigHash with the current hash(version) of the configuration, if it matches it will go ahead and perform the process, but if it is not same it will reload the consumers from the new configuration and update the localOperatingConfigHash to the value it got from redis cache

Open

Another optimised approach will be that persiter service fetches the configs when _refresh API is called. After fetching the config the service will get all the topics present in the configs and get a hash of that list of topics. This hash service will set in its local variable and add it in redis along with the configs. When another pod fetches the config and hash from redis it will compare the hash with its local hash and based on that it will restart consumers. In this way consumers will be only restarted when topics are added or removed.

Tracer Enhancements:

We will update the tracer library to automatically connect with the config handler module during initialisation. The tracer module will work by using a variable called tracer.config.path. This variable will contain the path where the configuration json is present. By default it will point to the resource folder inside the code base, but for production use cases it can be overwritten with Git url or any other valid file path as per requirement.

Open Points For Discussion:

1. Helm Charts:

2. Approaches to Deploy Config Handler (if we don’t want to provide _update API):

• Config handler code will be part of tracer

• We create a separate client for config handler

• We create separate service and run it as init container

If we want it to provide _update API as well then we need to deploy it as separate microservice