TheLazy.Dev
Portfolio GitHub

Blog

aws |nosql |dynamodb |serverless

DynamoDB OneTable

embedthis

Michael O'Brien

Posted on January 20, 2021

DynamoDB OneTable

DynamoDB OneTable (OneTable) is an access library for DynamoDB applications that use one-table design patterns with NodeJS. OneTable makes dealing with DynamoDB and one-table design patterns dramatically easier while still providing easy access to the full DynamoDB API.

OneTable is used by the SenseDeep Serverless Troubleshooter for all DynamoDB access. OneTable is provided open source (MIT license) from GitHub OneTable or NPM OneTable.

History and Credits

Rick Houlihan

After watching the famous Rick Houlihan DynamoDB ReInvent Video, we changed how we used DynamoDB for our SenseDeep serverless troubleshooter to use one-table design patterns. However, we found the going tough and thus this library was created to make our one-table patterns less tedious, more natural and a joy with DynamoDB.

A big thank you to Alex DeBrie and his excellent DynamoDB Book. Highly recommended. And thanks also to Jeremy Daly for his Off by None Blog blog, posts and his DynamoDB Toolbox which pointed out a better way for us to do a number of things.

OneTable Overview

OneTable

OneTable is not an ORM. Rather it provides a convenience API over the DynamoDB APIs. It offers a flexible high-level API that supports one-table design patterns and eases the tedium of working with the standard, unadorned DynamoDB API.

OneTable can invoke DynamoDB APIs or it can be used as a generator to create DynamoDB API parameters that you can save or execute yourself.

OneTable is not opinionated (as much as possible) and provides hooks for you to customize requests and responses to suit your exact needs.

Here are some of the key features of OneTable.

  • Schema supported one-table access to DynamoDB APIs.
  • Efficient storage and access of multiple entities in a single DynamoDB table.
  • High level API with type marshaling, validations, and extended query capability for get/delete/update operations.
  • Bidirectional conversion of DynamoDB types to Javascript types.
  • Option to invoke DynamoDB or simply generate API parameters.
  • Generation of Conditional, Filter, Key and Update expressions.
  • Schema item definitions for attribute types, default values, enums and validations.
  • Powerful field level validations with required and transactional unique attributes.
  • Easy parameterization of filter and conditional queries.
  • Multi-page response aggregation.
  • Compound and templated key management.
  • Encrypted fields.
  • Support for Batch, Transactions, GSI, LSI indexes.
  • Intercept hooks to modify DynamoDB requests and responses.
  • Controllable logging to see exact parameter, data and responses.
  • Simple, easy to read source to modify (< 1000 lines).
  • Safety options to prevent "rm -fr *".
  • No module dependencies.
  • Support for the AWS SDK v3

Installation 

npm i dynamodb-onetable
Enter fullscreen mode Exit fullscreen mode

Quick Tour

Import the OneTable library. If you are not using ES modules or Typescript, use require to import the libraries.

import {Table} from 'dynamodb-onetable'
Enter fullscreen mode Exit fullscreen mode

If you are using the AWS SDK V2, import the AWS DynamoDB class and create a DocumentClient instance.

import DynamoDB from 'aws-sdk/clients/dynamodb'
const client = new DynamoDB.DocumentClient(params)
Enter fullscreen mode Exit fullscreen mode

This version includes prototype support for the AWS SDK v3.

If you are using the AWS SDK v3, import the AWS v3 DynamoDBClient class and the OneTable Dynamo helper. Then create a DynamoDBClient instance and Dynamo wrapper instance.

import {DynamoDBClient} from '@aws-sdk/client-dynamodb'
import Dynamo from 'dynamodb-onetable/Dynamo'
const client = new Dynamo({client: new DynamoDBClient(params)})
Enter fullscreen mode Exit fullscreen mode

Initialize your your OneTable Table instance and define your models via a schema.

const table = new Table({
    client: client,
    name: 'MyTable',
    schema: MySchema,
})
Enter fullscreen mode Exit fullscreen mode

This will initialize your your OneTable Table instance and define your models via a schema.

Schemas

Schemas define your models (entities), keys, indexes and attributes. Schemas look like this:

const MySchema = {
    indexes: {
        primary: { hash: 'pk', sort: 'sk' }
        gs1:     { hash: 'gs1pk', sort: 'gs1sk' }
    },
    models: {
        Account: {
            pk:          { value: 'account:${name}' },
            sk:          { value: 'account:' },
            id:          { type: String, uuid: true, validate: /^[0-9A-F]{32}$/i, },
            name:        { type: String, required: true, }
            status:      { type: String, default: 'active' },
            zip:         { type: String },
        },
        User: {
            pk:          { value: 'account:${accountName}' },
            sk:          { value: 'user:${email}', validate: EmailRegExp },
            id:          { type: String },
            accountName: { type: String },
            email:       { type: String, required: true },
            firstName:   { type: String, required: true },
            lastName:    { type: String, required: true },
            username:    { type: String, required: true },
            role:        { type: String, enum: ['user', 'admin'], required: true, default: 'user' }
            balance:     { type: Number, default: 0 },

            gs1pk:       { value: 'user-email:${email}' },
            gs1sk:       { value: 'user:' },
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Schemas define your models and their attributes. Keys (pk, gs1pk) can derive their values from other attributes via templating.

Alternatively, you can define models one by one:

const Card = new Model(table, {
    name: 'Card',
    fields: {
        pk: { value: 'card:${number}'}
        number: { type: String },
        ...
    }
})
Enter fullscreen mode Exit fullscreen mode

To create an item:

let account = await Account.create({
    id: '8e7bbe6a-4afc-4117-9218-67081afc935b',
    name: 'Acme Airplanes'
})
Enter fullscreen mode Exit fullscreen mode

This will write the following to DynamoDB:

{
    pk:         'account:8e7bbe6a-4afc-4117-9218-67081afc935b',
    sk:         'account:98034',
    id:         '8e7bbe6a-4afc-4117-9218-67081afc935b',
    name:       'Acme Airplanes',
    status:     'active',
    zip:        98034,
    created:    1610347305510,
    updated:    1610347305510,
}
Enter fullscreen mode Exit fullscreen mode

Get an item:

let account = await Account.get({
    id: '8e7bbe6a-4afc-4117-9218-67081afc935b',
    zip: 98034,
})
Enter fullscreen mode Exit fullscreen mode

which will return:

{
    id:       '8e7bbe6a-4afc-4117-9218-67081afc935b',
    name:     'Acme Airplanes',
    status:   'active',
    zip:      98034,
}
Enter fullscreen mode Exit fullscreen mode

To use a secondary index:

let user = await User.get({email: 'user@example.com'}, {index: 'gs1'})
Enter fullscreen mode Exit fullscreen mode

To find a set of items:

let users = await User.find({accountId: account.id})

let adminUsers = await User.find({accountId: account.id, role: 'admin'})

let adminUsers = await User.find({accountId: account.id}, {
    where: '${balance} > {100.00}'
})
Enter fullscreen mode Exit fullscreen mode

To update an item:

await User.update({id: userId, balance: 50})
await User.update({id: userId}, {add: {balance: 10.00}})
Enter fullscreen mode Exit fullscreen mode

To do a transactional update:

let transaction = {}
await Account.update({id: account.id, status: 'active'}, {transaction})
await User.update({id: user.id, role: 'user'}, {transaction})
await table.transaction('write', transaction)
Enter fullscreen mode Exit fullscreen mode

Why OneTable?

DynamoDB is a great NoSQL database that comes with a steep learning curve. Folks migrating from SQL often have a hard time adjusting to the NoSQL paradigm and especially to DynamoDB which offers exceptional scalability but with a fairly low-level API.

The standard DynamoDB API requires a lot of boiler-plate syntax and expressions. This is tedious to use and can unfortunately can be error prone at times. I doubt that creating complex attribute type expressions, key, filter, condition and update expressions are anyone's idea of a good time.

Net/Net: it is not easy to write terse, clear, robust Dynamo code for one-table patterns.

Our goal with OneTable for DynamoDB was to keep all the good parts of DynamoDB and to remove the tedium and provide a more natural, "Javascripty" way to interact with DynamoDB without obscuring any of the power of DynamoDB itself.

More?

You can read more in the detailed documentation at: GitHub OneTable or NPM OneTable.

Future

We'll be releasing additional components that we use in-house like a migration tool and DynamodDB maintenance scripts to remove orphaned items and legacy attributes. If you have suggestions or ideas for how to improve OneTable please let us know. All feedback, contributions and bug reports are very welcome.

SenseDeep with OneTable

At SenseDeep, we've used the OneTable module extensively with our SenseDeep serverless troubleshooter. All data is stored in a single DynamoDB table and we extensively use one-table design patterns. We could not be more satisfied with DynamoDB implementation. Our storage and database access costs are insanely low and access/response times are excellent.

Contact

You can contact me (Michael O'Brien) on Twitter at: @SenseDeepCloud, or email and ready my Blog.

To learn more about SenseDeep and how to use our troubleshooter, please visit https://www.sensedeep.com/.

Links

💖 💪 🙅 🚩
embedthis
Michael O'Brien

Posted on January 20, 2021

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

OneTable CLI for DynamoDB Migrations
aws OneTable CLI for DynamoDB Migrations

February 20, 2021

DynamoDB OneTable
aws DynamoDB OneTable

January 20, 2021