TheLazy.Dev
Portfolio GitHub

Blog

aws |serverless |dynamodb |nosql

DynamoDB OneTable API Overview

embedthis

Michael O'Brien

Posted on July 5, 2021

DynamoDB OneTable API Overview

DynamoDB OneTable (OneTable) is an access library for DynamoDB applications that use one-table design patterns with NodeJS. OneTable makes dealing with DynamoDB and one-table design patterns dramatically easier while still providing easy access to the full DynamoDB API.

We've worked very hard to make the OneTable API expressive, terse and elegant to use. I hope you will find that it makes your DynamoDB developments proceed faster and more easily than every before.

This article takes a quick tour through the OneTable API and demonstrates basic calls and patterns when using the API.

OneTable Overview

OneTable

A good way to read this article is to checkout the OneTable Overview working sample and follow along in VS Code. If you are using TypeScript, use the TypeScript sample instead.

Installation 

npm i dynamodb-onetable
Enter fullscreen mode Exit fullscreen mode

Quick Tour

Import the OneTable library. If you are not using ES modules or Typescript, use require to import the libraries.

import {Table} from 'dynamodb-onetable'
Enter fullscreen mode Exit fullscreen mode

If you are using the AWS SDK V2, import the AWS DynamoDB class and create a DocumentClient instance.

import DynamoDB from 'aws-sdk/clients/dynamodb'
const client = new DynamoDB.DocumentClient(params)
Enter fullscreen mode Exit fullscreen mode

This version includes prototype support for the AWS SDK v3.

If you are using the AWS SDK v3, import the AWS v3 DynamoDBClient class and the OneTable Dynamo helper. Then create a DynamoDBClient instance and Dynamo wrapper instance.

import {DynamoDBClient} from '@aws-sdk/client-dynamodb'
import Dynamo from 'dynamodb-onetable/Dynamo'
const client = new Dynamo({client: new DynamoDBClient(params)})
Enter fullscreen mode Exit fullscreen mode

Initialize your your OneTable Table instance and define your models via a schema. The schema defines your single-table entities, attributes and indexes.

import MySchema from './Schema'

const table = new Table({
    client: client,
    name: 'MyTable',
    schema: MySchema,
})
Enter fullscreen mode Exit fullscreen mode

Schemas

Schemas define your models (entities), keys, indexes and attributes. Schemas look like this:

export default {
    indexes: {
        primary: { hash: 'pk', sort: 'sk' },
        gs1:     { hash: 'gs1pk', sort: 'gs1sk', project: ['gs1pk', 'gs1sk', 'data'] },
    },
    models: {

        Account: {
            pk:         { type: String, value: 'account#${id}' },
            sk:         { type: String, value: 'account#' },
            id:         { type: String, uuid: true, validate: Match.ulid },
            name:       { type: String, required: true, unique: true, validate: Match.name },
            balance:    { type: Number, default: 0 },

            //  Search by account name or by type
            gs1pk:      { type: String, value: 'account#' },
            gs1sk:      { type: String, value: 'account#${name}${id}' },
        },

        User: {
            pk:         { type: String, value: 'account#${accountId}' },
            sk:         { type: String, value: 'user#${email}' },
            accountId:  { type: String, required: true },
            id:         { type: String, uuid: true, validate: Match.ulid },
            name:       { type: String, required: true, validate: Match.name },
            email:      { type: String, required: true, validate: Match.email, crypt: true },

            address:    { type: Object, default: {}, schema: {
                street: { type: String, /* map: 'data.street' */ },
                city:   { type: String, /* map: 'data.city' */ },
                zip:    { type: String, /* map: 'data.zip' */ },
            } },

            status:     { type: String, required: true, default: 'active', enum: ['active', 'inactive'] },
            balance:    { type: Number, default: 0 },

            //  Search by user name or by type
            gs1pk:      { type: String, value: 'user#' },
            gs1sk:      { type: String, value: 'user#${name}#${id}' },
        },

        Product: {
            pk:         { type: String, value: 'product#${id}' },
            sk:         { type: String, value: 'product#' },
            id:         { type: String, uuid: true, validate: Match.ulid },
            name:       { type: String, required: true },
            price:      { type: Number, required: true },

            //  Search by product name or by type
            gs1pk:      { type: String, value: 'product#' },
            gs1sk:      { type: String, value: 'product#${name}#${id}' },
        },

        Invoice: {
            pk:         { type: String, value: 'account#${accountId}' },
            sk:         { type: String, value: 'invoice#${id}' },

            accountId:  { type: String, required: true },
            date:       { type: Date, default: () => new Date() },
            id:         { type: String, uuid: true },
            product:    { type: String },
            count:      { type: Number },
            total:      { type: Number },

            //  Search by invoice date or by type
            gs1pk:      { type: String, value: 'invoice#' },
            gs1sk:      { type: String, value: 'invoice#${date}#${id}' },
        }
    }
Enter fullscreen mode Exit fullscreen mode

This schema has models for: Account, User, Product and Invoice.

The schema also defines your indexes. The index key values are defined for each model and can derive their values from other attributes via value templating.

Most interactions with the OneTable API are via model objects that correspond to each schema model. For example,

const Account = table.getModel('Account')
Enter fullscreen mode Exit fullscreen mode

For typescript, we use a slightly different formulation that provides type safety for your model references at compile time.

type AccountType = Entity<typeof Schema.models.Account>
const Account = table.getModel<AccountType>('Account')
Enter fullscreen mode Exit fullscreen mode

Create a Table

Normally you would create your tables separately, hopefully via infrastructure-as-code. But for development, you can create a table directly from the schema via:

await table.createTable()
Enter fullscreen mode Exit fullscreen mode

Creating an Item

Once the table is prepared, you can create an item:

let account = await Account.create({name: 'Acme Airplanes'})
Enter fullscreen mode Exit fullscreen mode

This will write the following to DynamoDB:

{
    pk:         'account:8e7bbe6a-4afc-4117-9218-67081afc935b',
    sk:         'account:98034',
    name:       'Acme Airplanes',
    id:         '8e7bbe6a-4afc-4117-9218-67081afc935b',
    balance:    0,
    created:    '2021-07-05T04:58:55.136Z',
    updated:    '2021-07-05T04:58:55.136Z',
}
Enter fullscreen mode Exit fullscreen mode

This API will return the following, and omit the underlying hidden properties: pk, sk, gs1pk, gs1sk.

{
    name:       'Acme Airplanes',
    id:         '8e7bbe6a-4afc-4117-9218-67081afc935b',
    balance:    0,
    created:    '2021-07-05T04:58:55.136Z',
    updated:    '2021-07-05T04:58:55.136Z',
}
Enter fullscreen mode Exit fullscreen mode

To return the hidden properties, set {hidden: true} in the params:

let account = await Account.create({name: 'Acme Airplanes'}, {hidden: true})
Enter fullscreen mode Exit fullscreen mode

Context Properties

If creating a multi-tenant application, it us helpful to have the account ID be part of the primary key. Rather than having each API specify this account ID, it is best to provide this via a context property that ensures it is never omitted. Context properties are blended with the API properties and take precedence.

table.setContext({accountId: account.id})
Enter fullscreen mode Exit fullscreen mode

Now when we create a user, it will automatically set the user account ID to the context property value.

let user = await User.create({name: 'Road Runner', email: 'roadrunner@acme.com'})
Enter fullscreen mode Exit fullscreen mode

Getting Items

To get an item, we provide the properties require to construct the key. In the schema, the Account keys were specified as:

{
    pk: { type: String, value: 'account#${id}' },
    sk: { type: String, value: 'account#' },
}
Enter fullscreen mode Exit fullscreen mode

So we need to provide the account id and then OneTable will construct the keys for you.

This will retrieve an account, using the primary key.

account = await Account.get({id: '8e7bbe6a-4afc-4117-9218-67081afc935b'})
Enter fullscreen mode Exit fullscreen mode

Using a secondary index

We can also fetch items using a secondary index which allows us to retrieve items by other attributes such as the Account name or User name. An overloaded secondary index can be used to retrieve different items using alternative properties. The schema defines a Global Secondary Index (GSI) and then each model entity uses different values from their value templates into the GSI key fields.

For example, the User type constructs the GSI sort key using the user name and user ID.

{
    gs1pk: { type: String, value: 'user#' },
    gs1sk: { type: String, value: 'user#${name}#${id}' },
}
Enter fullscreen mode Exit fullscreen mode

So we can retrieve a user by name from the GSI 1 using:

user = await User.get({name: 'Road Runner'}, {index: 'gs1', follow: true, log: true})
Enter fullscreen mode Exit fullscreen mode

This will use the secondary index and construct a query to search for users that have the gs1sk set to user#Road Runner. Because the user ID portion is not provided, OneTable searches for a user that begins with the provided string.

To save storage costs, the GSI in this example only projects the gs1pk, gs1sk and data attributes to the GSI. That means the other user attributes are not returned when querying only from the GSI. OneTable addresses this by providing the follow: true parameter which follows the GSI read with a read from the primary index to fetch the full user record.

The {log: true} is useful to see the actual DynamoDB request logged to the console.

Finding items

To find all the users for an account:

let users = await User.find()
Enter fullscreen mode Exit fullscreen mode

Remember, the account ID is provided by the context, so we don't need to explicitly provide it.

If we want to filter items, we can add non-key properties to use as filters. Here we find the users that have a role of 'admin'.

let adminUsers = await User.find({accountId: account.id, role: 'admin'})
Enter fullscreen mode Exit fullscreen mode

Alternatively, we can use a where clause. This will find users with an account balance over $100. Where clauses are useful for more complex filtering expressions. A where clause supplies property references by enclosing the name in ${}. Similarly, values are enclosed in {} without the dollar.

users = await User.find({}}, {
    where: '${balance} > {100}'
})
Enter fullscreen mode Exit fullscreen mode

Note: these are not Javascript string templates, so do not use back-ticks when specifying.

Updating Items

To update items, you can provide the keys and new values via the properties. Remember that when updating you cannot change an item's keys in DynamoDB.

user = await User.update({email: 'roadrunner@acme.com', balance: 0, role: 'admin'})
Enter fullscreen mode Exit fullscreen mode

To help prevent accidental creation when updating, OneTable adds a constraint to ensure your items already exist and you don't accidentally create new items by providing incorrect key ingredient properties.

To update nested properties, provide them via the params.set. This will update just the property you specify.

user = await User.update({email: 'roadrunner@acme.com'}, {set: {'address.zip': '{"98034"}'}})
Enter fullscreen mode Exit fullscreen mode

You can also use expressions to atomically update your attributes. For example:

user = await User.update({email: 'roadrunner@acme.com'}, {add: {balance: 10}})
user = await User.update({email: 'roadrunner@acme.com'}, {set: {balance: '${balance} - {20}'}})
Enter fullscreen mode Exit fullscreen mode

Item Collections

A key power of DynamoDB is to fetch multiple related items in a single efficient request.

To fetch an account, the account users and invoices in one request, use table.fetch.

let collection = await table.fetch(['Account', 'User', 'Invoice'], {pk: `account#${account.id}`})
Enter fullscreen mode Exit fullscreen mode

This returns an object map with lists for Account, User and Invoice.

In this request, we need to provide the actual primary key value as each model type may have different value templates to construct their pk value.

Batch Adding

To efficiently add many items, DynamoDB has batch operations. OneTable makes these easy. This code snippet creates 200 users in batches of 25.

let batch = {}
let i = 0, count = 0
while (i++ < 200) {
    User.create({name: `user${i}`, email: `user${i}@acme.com`}, {batch})
    if (++count >= 25) {
        await table.batchWrite(batch)
        batch = {}
        count = 0
    }
}
Enter fullscreen mode Exit fullscreen mode

Transactions

OneTable transactions are expressed similarly to batches.

let transaction = {}
await Account.update({id: account.id, status: 'active'}, {transaction})
await User.update({id: user.id, role: 'user'}, {transaction})
await table.transact('write', transaction)
Enter fullscreen mode Exit fullscreen mode

Pagination

The DynamoDB API has limits on the maximum size of request response data. So you often need to use pagination to step through large result sets.

OneTable uses the start param on request APIs and in result sets to mark the current reading position.

let start
do {
    users = await User.find({}, {start, limit: 25})
    //  Operate on this batch of 25 users ...
    start = users.start
} while (users.start)
Enter fullscreen mode Exit fullscreen mode

Summary

Checkout the full overview sample at:

You can read more in the detailed documentation at:

Why OneTable?

DynamoDB is a great NoSQL database that comes with a steep learning curve. Folks migrating from SQL often have a hard time adjusting to the NoSQL paradigm and especially to DynamoDB which offers exceptional scalability but with a fairly low-level API.

The standard DynamoDB API requires a lot of boiler-plate syntax and expressions. This is tedious to use and can unfortunately can be error prone at times. I doubt that creating complex attribute type expressions, key, filter, condition and update expressions are anyone's idea of a good time.

Net/Net: it is not easy to write terse, clear, robust Dynamo code for one-table patterns.

Our goal with OneTable for DynamoDB was to keep all the good parts of DynamoDB and to remove the tedium and provide a more natural, "JavaScripty / TypeScripty" way to interact with DynamoDB without obscuring any of the power of DynamoDB itself.

Other Working Samples

SenseDeep with OneTable

At SenseDeep, we've used the OneTable module extensively with our SenseDeep serverless developer studio. All data is stored in a single DynamoDB table and we extensively use one-table design patterns. We could not be more satisfied with DynamoDB implementation. Our storage and database access costs are insanely low and access/response times are excellent.

Contact

You can contact me (Michael O'Brien) on Twitter at: @mobstream, or email and read myBlog.

To learn more about SenseDeep and how to use our serverless developer studio, please visit https://www.sensedeep.com/.

Links

💖 💪 🙅 🚩
embedthis
Michael O'Brien

Posted on July 5, 2021

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

DynamoDB OneTable API Overview
aws DynamoDB OneTable API Overview

July 5, 2021

Using DynamoDB with TypeScript and OneTable
aws Using DynamoDB with TypeScript and OneTable

May 5, 2021

DynamoDB Attribute Packing for Single-Table Designs
aws DynamoDB Attribute Packing for Single-Table Designs

April 8, 2021

Evolving DynamoDB Designs
aws Evolving DynamoDB Designs

March 1, 2021