How to Backup DynamoDB Data to Different AWS Account

Table of Contents

• Introduction
• Export DynamoDB Table to S3
• Destination Bucket Setup
• AWS Configure
• Execute the Command
• Migrate Data from S3 to DynamoDB
• Conclusion

Introduction

This blog shows how to migrate DynamoDB data between different AWS accounts. You can migrate DynamoDB by using different approaches: 1. AWS S3 with AWS Glue. 2. Data Pipeline. 3. Amazon EMR. Here, we'll use S3 and focus on how to move DynamoDB backup data into a different AWS account.

Following is the process of handling migration from one AWS account to another account and the requirements:

• Have an AWS account to use as the source account.

• The source account has a DynamoDB table and the S3 bucket the DynamoDB table will export to.

• Have an AWS account to use as the target account, and the S3 bucket which the source account will copy backup DynamoDB files into.

Export DynamoDB Table to S3

Here, we'll export the DynamoDB table into the S3 bucket. Log in to AWS and go to the DynamoDB section.

Enable Point-in-Time Recovery

Before backing up the DynamoDB table to S3, you have to enable PITR (point-in-time recovery). After enabling PITR, click Export to S3, then choose the source table you want to back up to the same account's S3.

Enter the destination S3 bucket name and click the export button. The export should now start and will copy all the backup files for the DynamoDB table into the S3 bucket. Note that you can back up to a different AWS account directly, but it's not covered in this blog.

Add Bucket Policy to Source Bucket

After DynamoDB has been backed up to the S3 bucket, you'll need to modify the bucket policy. Replace ACCOUNT-B-ID with account B's ID, replace ACCOUNT-B-USER with account B's user who will later perform the S3 file sync. ACCOUNT-A-Bucket-Name should be replaced with the bucket that has the DynamoDB backup content.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::{ACCOUNT-B-ID}:user/{ACCOUNT-B-USER}"
            },
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::{ACCOUNT-A-Bucket-NAME}/*",
                "arn:aws:s3:::{ACCOUNT-A-Bucket-NAME}"
           ]
        }
    ]
}

Destination Bucket Setup

Setup Custom Policy

For the destination bucket setup, we need to set up a user who has permission to perform a bucket sync. The following policy grants access to both buckets.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::{source bucket name}/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::{destination bucket name}/*"
            ]
        }
    ]
}

Create a User and Attach the Policy

In the destination AWS account, create a new IAM user (or use an existing one) that will be used to run the S3 sync command. Attach the custom policy you created above to this user. Make sure to generate access keys for this user, as you will need them in the next step to configure the AWS CLI.

AWS Configure

Here, you'll run aws configure and use the credentials of the user who has the custom policy attached.

Execute the Command

At this point, both buckets should be ready and we can run the command to copy files from account A's bucket to account B's bucket.

aws s3 sync s3://{source bucket name} s3://{target bucket name} --source-region {source region} --region {target region}
// following is example
aws s3 sync s3://my-dynamodb-backup s3://my-dynamodb --source-region us-west-2 --region us-west-1

Migrate Data from S3 to DynamoDB

Now the S3 bucket should have the backup DynamoDB tables. There are a few ways you can handle ETL to migrate data into DynamoDB. You can use a Glue Job to read the files from the S3 bucket and write them to the target DynamoDB table, or write a Lambda function to read the S3 files and write them into the DynamoDB table. Following are a few possible ways to migrate data from S3 to DynamoDB, but they are not part of this blog.

• Write a Glue Job

• Data Pipeline

• Write a Lambda function

• Use Amazon EMR

Conclusion

Migrating DynamoDB data between AWS accounts involves exporting the table to S3, setting up cross-account bucket policies, and syncing the data to the destination account's S3 bucket. From there, you can use tools like AWS Glue, Data Pipeline, Lambda, or Amazon EMR to load the data into the target DynamoDB table. While there are several steps involved, following this process ensures a reliable and secure migration.