Migrating DIGIT from a Cloud Environment to a Data Center(DC)

Migrating an application from a cloud environment to a Data Center(DC) involves several steps, including planning, assessing dependencies, setting up the new environment, and performing the migration itself. Here's a step-by-step guide to help you with this process.

1. Planning and Assessment

Inventory Existing Resources

• Kubernetes (EKS): List out the current Kubernetes resources, including deployments, services, config maps, secrets, etc.

• PostgreSQL: Document the existing PostgreSQL databases, schemas, and tables.

• Kafka: List the Kafka topics and configurations.

• Elasticsearch: List the indices, mappings, and configurations.

• Redis: Document the existing Redis instances and their configurations.

Dependencies and Integrations

• Identify how different components interact with each other.

• List all external integrations that might be impacted by the migration.

Downtime and Rollback

• Assess acceptable downtime for each component.

• Develop a rollback strategy to revert to the original state in case of issues.

2. Setting Up the New Environment on DC

Infrastructure Provisioning

• Kubernetes Cluster: Set up a Kubernetes cluster in the new DC environment using tools like kubeadm, or managed Kubernetes services offered by the DC provider.

• PostgreSQL: Provision a PostgreSQL instance, ensuring the same version and configurations as the source.

• Kafka: Set up a Kafka cluster, ensuring replication factor and partition configurations match the source.

• Elasticsearch: Deploy an Elasticsearch cluster with the same version and settings.

• Redis: Set up Redis instances with the same configurations.

Networking and Security

• Configure VPCs, subnets, and security groups to mirror the source environment.

• Set up appropriate IAM roles, policies, and security measures to ensure data security.

3. Data Migration

PostgreSQL

1. Backup: Use pg_dump to create a backup of the PostgreSQL database.

   pg_dump -h source_host -U source_user -d source_db -F c -b -v -f backup.dump

2. Transfer: Copy the backup file to the new environment.

3. Restore: Use pg_restore to restore the backup in the new PostgreSQL instance.

   pg_restore -h target_host -U target_user -d target_db -v backup.dump

4. Verification: Verify the integrity of the data by comparing row counts, checksums, etc.

Kafka

1. MirrorMaker: Use Kafka MirrorMaker to replicate topics from the source to the target Kafka cluster.

   kafka-mirror-maker --consumer.config consumer.config --producer.config producer.config --whitelist ".*"

2. Verification: Ensure all topics and messages are replicated correctly.

Elasticsearch

1. Install Elasticdump: First, install elasticdump.

2. Export Data: Export the data from your source Elasticsearch instance.

   • Export Mappings:

     elasticdump --input=<http://source_host:9200/source_index> --output=mapping.json --type=mapping

   • Export Data:

     elasticdump --input=<http://source_host:9200/source_index> --output=data.json --type=data

   • Export Settings:

     elasticdump --input=<http://source_host:9200/source_index> --output=settings.json --type=settings

3. Transfer Data: Copy the exported JSON files (mapping.json, data.json, settings.json) to your target environment.

4. Import Data: Import the data into your target Elasticsearch instance.

   • Create Index with Settings:

     curl -X PUT "<http://target_host:9200/target_index>" -H 'Content-Type: application/json' -d @settings.json

   • Import Mappings:

     elasticdump --input=mapping.json --output=<http://target_host:9200/target_index> --type=mapping

   • Import Data:

     elasticdump --input=data.json --output=<http://target_host:9200/target_index> --type=data

5. Verify the Migration: After importing the data, verify that the data has been correctly migrated.

   • Check Index Settings:

     curl -X GET "<http://target_host:9200/target_index/_settings?pretty>"

   • Check Index Mappings:

     curl -X GET "<http://target_host:9200/target_index/_mapping?pretty>"

   • Check Data Count: Compare the document count between the source and target indices.

     • Source:

       curl -X GET "<http://source_host:9200/source_index/_count?pretty>"

     • Target:

       curl -X GET "<http://target_host:9200/target_index/_count?pretty>"

6. Cleanup: After verifying that the data has been correctly migrated, clean up any temporary files and configurations used during the migration process.

Redis

1. Dump: Use redis-cli to create a dump file.

   redis-cli -h source_host --rdb backup.rdb

2. Transfer: Copy the dump file to the new environment.

3. Load: Use redis-cli to load the dump file into the new Redis instance.

   redis-cli -h target_host --pipe < backup.rdb

4. Verification: Verify data integrity by comparing key counts and sampling values.

Copy files from AWS S3 Bucket to SDC File System:

To copy files from an AWS S3 bucket to your local system (SDC file system), you can follow the steps below:

1. Install AWS CLI:

If you haven't installed AWS CLI yet, you can follow these instructions:

a. Windows or macOS:

• Download the AWS CLI installer for Windows or macOS.

• Run the installer and follow the installation steps.

b. Linux:

• Run the following commands:

  curl "<https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip>" -o "awscliv2.zip" unzip awscliv2.zip sudo ./aws/install

You can verify that the installation was successful by running:

2. Configure AWS CLI:

Once the AWS CLI is installed, you need to configure it with your AWS credentials. Run the following command:

You will be prompted to enter the following:

• AWS Access Key ID: Your access key for the AWS account (you can create this in the AWS IAM console if you don’t have it).

• AWS Secret Access Key: The secret key associated with your access key.

• Default region name: The region where your S3 bucket is hosted (for example, us-east-1, ap-south-1).

• Default output format: Usually json, but you can choose other formats like text or table.

Example:

3. Copy Files from S3 Bucket to Local File System:

To copy files from an S3 bucket to your local system, use the aws s3 cp command. You can copy a single file or an entire directory.

a. Copy a Single File:

b. Copy an Entire Directory:

To copy all files from a specific directory on S3 to your local file system:

Example:

This command will recursively copy all the contents from the data directory in your S3 bucket to the /home/user/data/ directory on your local system.