How CQLs talk

How to Bulk Upload JSON Data into DynamoDB

This is an in-depth how-to article to solve a common use case to insert JSON records in DynamoDB. I will show how to use a command-line interface (CLI) to run a NodeJS application to insert JSON data.

This is a generic application that supports any of the JSON documents and you don’t have to make any changes to make it work for your use case.

You can skip this step if you already have a user with DynamoDB write access.

Open AWS Console and access IAM. Create a user with write access to DynamoDB. Note the AccessKey ID and Secret Access Key of the newly created user.

Download CLI package from the AWS site — Link

E.g. I have a Linux machine so I download and installed using the following commands:

curl “https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o “awscliv2.zip”
unzip awscliv2.zip
sudo ./aws/install

After installation, run “aws configure” on the terminal/command line. Enter the Access Key ID and Secret Access Key when prompted. This will set-up your CLI with AWS credentials so that you can interact with AWS resources directly from the command line.

Open AWS Console and Access DynamoDB interface. Create a DynamoDB table for your JSON Data.

This is the key component of the implementation. The outline of the implementation goes as follows:

Load the JSON data file using the file system module of NodeJS

const fs = require('fs');let rawdata = fs.readFileSync('data.json');let json_data = JSON.parse(rawdata);

Build the request payload in the required DynamoDB format

// Get DynamoDB records from jsonconst getDynamoDBRecords = function (data) {let dynamoDBRecords = data.map(entity => {entity = attr.wrap(entity)  //Important - Else Insert Errorlet dynamoRecord = Object.assign({ PutRequest: { Item: entity } })dynamoDBRecord = attr.wrap(dynamoRecord)return dynamoRecord
})
return dynamoDBRecords}

Create batches of requests that will be sent to DynamoDB

// JSON - Insert to Dynamo Tableconst insertToDynamoTable = async function (json) {try {     let dynamoDBRecords = getDynamoDBRecords(json) //call above fn     let batches = [];

// create batch of 25 records
while (dynamoDBRecords.length) {
batches.push(dynamoDBRecords.splice(0, 25));
}
// Call DynamoDB API
await callDynamoDBInsert(batches)
} catch (error) {
return error;
}
}

Submit batches and wait for the completion

// Async function to insert batch records
const callDynamoDBInsert = async function(batches){
const dynamoTableName = 'sample-table' return Promise.all( batches.map(async (batch) => {
let requestItems = {}
let requestItems[dynamoTableName] = batch
const params = {
RequestItems: requestItems
};
await dbclient.send(new BatchWriteItemCommand(params))
})
);
}

Step 6: Run the NodeJs application on the command line

copy and paste this code into the local machine to run the app

Sample JSON Data:

Sample JSON Doc. Application is NOT restricted to this JSON only.
Sample JSON Doc. Application is NOT restricted to this JSON only.

In the next story, I will show you how to automate a load of JSON files in DynamoDB with AWS Lambda and AWS S3.

Cloud Native App Developer