How CQLs talk

How to Bulk Upload JSON Data into DynamoDB

This is an in-depth how-to article to solve a common use case to insert JSON records in DynamoDB. I will show how to use a command-line interface (CLI) to run a NodeJS application to insert JSON data.

This is a generic application that supports any of the JSON documents and you don’t have to make any changes to make it work for your use case.

Create an AWS IAM user with permission to write to DynamoDB Table

You can skip this step if you already have a user with DynamoDB write access.

Open AWS Console and access IAM. Create a user with write access to DynamoDB. Note the AccessKey ID and Secret Access Key of the newly created user.

Setup CLI (Download > Install > AWS Configure)

Download CLI package from the AWS site — Link

E.g. I have a Linux machine so I download and installed using the following commands:

curl “https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o “awscliv2.zip”
unzip awscliv2.zip
sudo ./aws/install

After installation, run “aws configure” on the terminal/command line. Enter the Access Key ID and Secret Access Key when prompted. This will set-up your CLI with AWS credentials so that you can interact with AWS resources directly from the command line.

Create a DynamoDB table

Open AWS Console and Access DynamoDB interface. Create a DynamoDB table for your JSON Data.

Create a NodeJs application

This is the key component of the implementation. The outline of the implementation goes as follows:

Load the JSON data file using the file system module of NodeJS

const fs = require('fs');
let rawdata = fs.readFileSync('data.json');
let json_data = JSON.parse(rawdata);

Build the request payload in the required DynamoDB format

// Get DynamoDB records from json
const getDynamoDBRecords = function (data) {
let dynamoDBRecords = data.map(entity => {
entity = attr.wrap(entity)  //Important - Else Insert Error
let dynamoRecord = Object.assign({ PutRequest: { Item: entity } })
dynamoDBRecord = attr.wrap(dynamoRecord)
return dynamoRecord
})
return dynamoDBRecords
}

Create batches of requests that will be sent to DynamoDB

// JSON - Insert to Dynamo Table
const insertToDynamoTable = async function (json) {
try {
     let dynamoDBRecords = getDynamoDBRecords(json) //call above fn
     let batches = [];
     
     // create batch of 25 records
     while (dynamoDBRecords.length) {
          batches.push(dynamoDBRecords.splice(0, 25));
     }
     // Call DynamoDB API
     await callDynamoDBInsert(batches)
  } catch (error) {
      return error;
  }
}

Submit batches and wait for the completion

// Async function to insert batch records
const callDynamoDBInsert = async function(batches){
     const dynamoTableName = 'sample-table'
     return Promise.all(
           batches.map(async (batch) => {
              let requestItems = {}
              let requestItems[dynamoTableName] = batch
              const params = {
                   RequestItems: requestItems
              };
             await dbclient.send(new BatchWriteItemCommand(params))
          })
    );
}

Step 6: Run the NodeJs application on the command line

Complete downloadable implementation

copy and paste this code into the local machine to run the app

Sample JSON Data:

Next Steps

In the next story, I will show you how to automate a load of JSON files in DynamoDB with AWS Lambda and AWS S3.