Architecture/VPC for central instance

Owned by Kaviyarasan P
Mar 14, 2022
5 min read

Single Instance Infrastructure Considerations

 

For CDG to be reality we need to consider multiple aspects of the infrastructure. CDG will be the central instance where the DIGIT platform will be hosted. Any state or city who is interested in DIGIT will use this central instance along with the custom services developed for the state/city. Each State will have a state specific domain registered with CDG. Every state enrolled here will have a separate implementation partner who takes care of the customization, deployment and maintenance of the state specific changes. 

 

In order to achieve this, it is necessary that the infrastructure is scalable and access rights can be granted at the right levels.The following are the broad objectives of the single instance of DIGIT, to cater to multiple States and ULBs:

 

  1. Ability to deploy State specific configurations, for e.g. MDMS data, localisation (each State may have different language requirements)

  2. Ability to extend baseline platform services for State specific implementations, for e.g. each state may sign up for different- SMS gateways, Property tax calculator, UI services, new services etc.

  3. Ability for each State on the single instance to have their own URLs for application access and ULB portal access

  4. Ability for the implementation/support partner for each State to make changes at will on their configurations and extensions.

  5. Ability for each State to have their own data stores, if absolutely necessary. If data stores are shared, need to ensure that the single instance implementation provides adequate segregation of the data for different States.

Various Infrastructure options

 

  1. Different VPCs for each State - Any state specific customization services will be deployed in this VPC and the state implementation team will have complete control over this environment. However, they will not have access to the central environment.

  2. Different namespaces for each state within the central VPC- Create a separate namespace for each state and have state specific custom services deployed in this namespace.

  3. Any other option?

 

Enablement Considerations:

 

Need to think about the following:

What is the process for a new State to be onboarded to the single instance? If a partner is implementing, what data needs to be provided to the single instance team to set up required infra for the State implementation partner to start their setup?

  • Can we define specific formats to be used by the State partners?

  • What is the SOP for the single instance administrator?

  • What is the SOP for troubleshooting support issues?

  • What level of access to the single instance deployment is required by the State partner team for troubleshooting issues?

  • Will the same kafka instance be used for all States? Or will each State have their own kafka instance? Other components? We need to think through...

 

As we all discussed in earlier meetings regarding central-instance, the deployment of multiple instance of services needs to be carried out which will be customized as per the clients needs such as calculators, UI, SMS, payment gateway etc. For the deployment of these services we need to create a protected environment so that the access can be restricted to only those people who are taking care of that environment. To enable this facility we have created the namespace based routing by changing the zuul discovery and adding an internal gateway which will route to any services in any environment(external/internal) as per config.

 

Pending works

 As of now we need to deploy the customizable services in a separate environment which is not shared  by the central services which are common to all. The Functionality has to be implemented this way because the states may even opt to carry their own environment for the services they customize. To enable this we need to set up these services in a separate environment and integrate it with the central instance server. 

 

Database:

If a state wants to use a separate database, we will need to deploy all the services which use the database on the state’s namespace. In case the states use the same database, we need to make sure that state A services are not modifying data of state B. Need to decide if we can use separate schemas for each state.

 

Similarly we will have to take a call on elasticsearch. If we are having the same elasticsearch DB for all states, we might need to upgrade DSS service to have role based access to charts. 

Also we will have to address the problem of access to kibana or elasticsearch,state should have access only to its data.

 

Also configuration of Redis database has to be taken care off. We will have to check when cache bursts are triggered. We might have to cache data at state level instead of service level. Might not be a priority task as it won’t hamper any flow or performance drastically, as we mostly cache configuration data only.

 

FileStore:

Will have to decide if we are going with a single bucket for all states or separate buckets for each state. We might have to enhance the service to support multiple buckets.

 

Deployment:

Following services needs to be deployed in two separate namespaces say pb and uk for testing Single Instance flow: 1. pt-calculator-v2

  1. sms-notification

  1. ws- calculator

  1. UI

  1. localization

  1. MDMS (Optional)

 

All other services should be deployed in a namespace called DIGIT. To start with and test the single instance use cases we can start with configuring a minimum of two VPCs.

 

Deployment of services like persister and indexer will depend on whether we are using single DB or multiple DB. 

 

Domain Routing:

Both uk and pb domains should be routed to the same singleInstance. For example https://uk.digit.gov.org and https://pb.digit.gov.org should be routed to same IP address

 

 

Kafka:

If we are using the same kafka for all states, we need to push data on different topics based on state tenantId. For example if we have customized property-service deployed on multiple state namespaces, in current architecture all services will push on the same kafka topic save-pt-property. Even if we have different persisters for all states, because of the same kafka backbone everyone will consume and persist each message. We can solve the problem by appending state tenantId to the kafka topic. In this use case pb will push data to pb-save-pt-property and uk will push on uk-save-pt-property. We will have separate persister configuration for both topics based on state customization.

 

Builds:

 pt-calculator-v2-db:develop-singleinstance-vpctest-a29e6286b-45

citizen:dev-new-instance-0e59d3495-496

Employee:dev-new-instance-0e59d3495-495

Egov-notification-sms:develop-86c0dec9-15