/
Migrating DIGIT from a Cloud Environment to a Data Center(DC)

Migrating DIGIT from a Cloud Environment to a Data Center(DC)

Owned by Nikhil Mulinti
Last updated: Sept 09, 2024
4 min read

Migrating an application from a cloud environment to a Data Center(DC) involves several steps, including planning, assessing dependencies, setting up the new environment, and performing the migration itself. Here's a step-by-step guide to help you with this process.

1. Planning and Assessment

Inventory Existing Resources

  • Kubernetes (EKS): List out the current Kubernetes resources, including deployments, services, config maps, secrets, etc.

  • PostgreSQL: Document the existing PostgreSQL databases, schemas, and tables.

  • Kafka: List the Kafka topics and configurations.

  • Elasticsearch: List the indices, mappings, and configurations.

  • Redis: Document the existing Redis instances and their configurations.

Dependencies and Integrations

  • Identify how different components interact with each other.

  • List all external integrations that might be impacted by the migration.

Downtime and Rollback

  • Assess acceptable downtime for each component.

  • Develop a rollback strategy to revert to the original state in case of issues.

2. Setting Up the New Environment on DC

Infrastructure Provisioning

  • Kubernetes Cluster: Set up a Kubernetes cluster in the new DC environment using tools like kubeadm, or managed Kubernetes services offered by the DC provider.

  • PostgreSQL: Provision a PostgreSQL instance, ensuring the same version and configurations as the source.

  • Kafka: Set up a Kafka cluster, ensuring replication factor and partition configurations match the source.

  • Elasticsearch: Deploy an Elasticsearch cluster with the same version and settings.

  • Redis: Set up Redis instances with the same configurations.

Networking and Security

  • Configure VPCs, subnets, and security groups to mirror the source environment.

  • Set up appropriate IAM roles, policies, and security measures to ensure data security.

3. Data Migration

PostgreSQL

  1. Backup: Use pg_dump to create a backup of the PostgreSQL database.

    pg_dump -h source_host -U source_user -d source_db -F c -b -v -f backup.dump

  2. Transfer: Copy the backup file to the new environment.

  3. Restore: Use pg_restore to restore the backup in the new PostgreSQL instance.

    pg_restore -h target_host -U target_user -d target_db -v backup.dump

  4. Verification: Verify the integrity of the data by comparing row counts, checksums, etc.

Kafka

  1. MirrorMaker: Use Kafka MirrorMaker to replicate topics from the source to the target Kafka cluster.

    kafka-mirror-maker --consumer.config consumer.config --producer.config producer.config --whitelist ".*"

  2. Verification: Ensure all topics and messages are replicated correctly.

Elasticsearch

  1. Install Elasticdump: First, install elasticdump.

  2. Export Data: Export the data from your source Elasticsearch instance.

    • Export Mappings:

      elasticdump --input=<http://source_host:9200/source_index> --output=mapping.json --type=mapping

    • Export Data:

      elasticdump --input=<http://source_host:9200/source_index> --output=data.json --type=data

    • Export Settings:

      elasticdump --input=<http://source_host:9200/source_index> --output=settings.json --type=settings

  3. Transfer Data: Copy the exported JSON files (mapping.json, data.json, settings.json) to your target environment.

  4. Import Data: Import the data into your target Elasticsearch instance.

    • Create Index with Settings:

      curl -X PUT "<http://target_host:9200/target_index>" -H 'Content-Type: application/json' -d @settings.json

    • Import Mappings:

      elasticdump --input=mapping.json --output=<http://target_host:9200/target_index> --type=mapping

    • Import Data:

      elasticdump --input=data.json --output=<http://target_host:9200/target_index> --type=data

  5. Verify the Migration: After importing the data, verify that the data has been correctly migrated.

    • Check Index Settings:

      curl -X GET "<http://target_host:9200/target_index/_settings?pretty>"

    • Check Index Mappings:

      curl -X GET "<http://target_host:9200/target_index/_mapping?pretty>"

    • Check Data Count: Compare the document count between the source and target indices.

      • Source:

        curl -X GET "<http://source_host:9200/source_index/_count?pretty>"

      • Target:

        curl -X GET "<http://target_host:9200/target_index/_count?pretty>"

  6. Cleanup: After verifying that the data has been correctly migrated, clean up any temporary files and configurations used during the migration process.

Redis

  1. Dump: Use redis-cli to create a dump file.

    redis-cli -h source_host --rdb backup.rdb

  2. Transfer: Copy the dump file to the new environment.

  3. Load: Use redis-cli to load the dump file into the new Redis instance.

    redis-cli -h target_host --pipe < backup.rdb

  4. Verification: Verify data integrity by comparing key counts and sampling values.

 

Copy files from AWS S3 Bucket to SDC File System:

To copy files from an AWS S3 bucket to your local system (SDC file system), you can follow the steps below:

1. Install AWS CLI:

If you haven't installed AWS CLI yet, you can follow these instructions:

a. Windows or macOS:

  • Download the AWS CLI installer for Windows or macOS.

  • Run the installer and follow the installation steps.

b. Linux:

  • Run the following commands:

    curl "<https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip>" -o "awscliv2.zip" unzip awscliv2.zip sudo ./aws/install

You can verify that the installation was successful by running:

aws --version

2. Configure AWS CLI:

Once the AWS CLI is installed, you need to configure it with your AWS credentials. Run the following command:

aws configure

You will be prompted to enter the following:

  • AWS Access Key ID: Your access key for the AWS account (you can create this in the AWS IAM console if you don’t have it).

  • AWS Secret Access Key: The secret key associated with your access key.

  • Default region name: The region where your S3 bucket is hosted (for example, us-east-1, ap-south-1).

  • Default output format: Usually json, but you can choose other formats like text or table.

Example:

aws configure AWS Access Key ID [None]: <Your_Access_Key_ID> AWS Secret Access Key [None]: <Your_Secret_Access_Key> Default region name [None]: us-east-1 Default output format [None]: json

3. Copy Files from S3 Bucket to Local File System:

To copy files from an S3 bucket to your local system, use the aws s3 cp command. You can copy a single file or an entire directory.

a. Copy a Single File:

aws s3 cp s3://<bucket-name>/path/to/your-file.txt /local/path/to/save/

b. Copy an Entire Directory:

To copy all files from a specific directory on S3 to your local file system:

aws s3 cp s3://<bucket-name>/path/to/directory/ /local/path/to/save/ --recursive

Example:

aws s3 cp s3://my-bucket/data/ /home/user/data/ --recursive

This command will recursively copy all the contents from the data directory in your S3 bucket to the /home/user/data/ directory on your local system.

 

 

Related content