This is a practical code walkthrough of Idempotency, including Dockerfiles you can pull down to run and test your code on.

Do note, this article is intended to be illustrative - not production ready code, and we will be taking some shortcuts for the sake of brevity.  With that said, let’s get started.

In this post, we’ll be focused on preventing duplicates and fault tolerance through a mechanism called Idempotency. We'll demonstrate this through implementing a REST API, making multiple calls to the same endpoint an idempotent operation.

What is Idempotency?

Idempotence loosely means that no matter how many times we attempt to execute an operation, it will only be processed once when there are multiple identical requests.  The same result should be returned to the client.

Idempotence thus allows REST API calls (or any API calls for that matter) to be retried safely. Networks are unreliable and fail.  When they do fail, information is lost.  An API call being duplicated or not recorded correctly is a nuisance and it leads to poor customer experience at best, and regulatory action at its worst.

An Example

Let’s examine a scenario and ground it in a real life use case.

John tries to make a $5 purchase at his local fast food restaurant. The restaurant has spotty internet.

Scenario 1: The initial POST request fails as the point of sale system tries to connect to a server
Scenario 2: The connection fails while the server is fulfilling the request.
Scenario 3: The call succeeds, but the point of sale system never receives the response from the server.
Scenario 4: The cashier, frustrated with the point of sale system for being so slow, hits the send button twice.

When the first request fails, imagine what the multiple requests might look like:

HTTP Body:
{ 
  "merchantName": "McDonalds",
  "transactionDateTime": "2023-02-14T18:30:00.000Z",
  "amount": "500",
}

HTTP Body:
{ 
  "merchantName": "McDonalds",
  "transactionDateTime": "2023-02-14T18:30:00.000Z",
  "amount": "500",
}

Without an implementation for idempotent requests, it’s hard to tell what happened with the checkout.  Did it fail? Did it succeed? Did it create a duplicated transaction?

While I’ve demonstrated the issue here with a toy example, there are many applications for a solution to this problem. One such example is when you need to update a bank account balance.

I also want to note that this issue compounds itself in distributed systems.  Failures are common, and you will need a strategy for handling dropped network connections.

The How

Okay, let’s outline a high level algorithm for how we can handle the issue we noted above and turn this into an idempotent operation.

Before trying to make any external calls that can fail, generate a key - X-Idempotency-Key that will be used for all requests to charge John for his food.

Send a request to the payment processor with X-Idempotency-Key as a header.

If the call fails, retry again with the same key, knowing that the processor will only process any given X-Idempotency-Key once.

The server takes an X-Idempotency-Key and registers an IdempotentRequest

The server processes the transaction as requested.

The server stores the response and response code for that IdempotentRequest

Any subsequent calls with that X-Idempotency-Key (the same idempotency key) will receive the same result as the first call.

The HTTP Requests examined earlier will now look like this instead:

HTTP Headers: 
{ 
  "X-Idempotency-Key": "dbcfc06c-...-0f13e7a987ca"
}
HTTP Body:
{ 
  "merchantName": "McDonalds",
  "transactionDateTime": "2023-02-14T18:30:00.000Z",
  "amount": "500",
}

HTTP Headers: 
{ 
  "X-Idempotency-Key": "dbcfc06c-...-0f13e7a987ca"
}
HTTP Body:
{ 
  "merchantName": "McDonalds",
  "transactionDateTime": "2023-02-14T18:30:00.000Z",
  "amount": "500",
} 

Ah, the server now knows it’s the same transaction because the X-Idempotency-Key header is the same! These are now idempotent operations.

With this said, let’s dive into implementation.  If you’re interested in following along or implementing yourself, you can checkout the repo here.

To implement yourself, checkout this commit hash before starting: cde73d369ecf2060d5fca60a89fc7b172229cb11

I'll be linking the GitHub commits as we move along as well.

Implementation

I’m using AdonisJs as my framework of choice, Dockerized with Postgres, but feel free to implement in a language and framework of your choosing.

First let’s get up and running.  This will install the dependencies and start our docker containers.

cd api && npm install
cd .. && docker compose up

Next, let’s create our model and migration for the database that will drive this functionality.

npm run ace make:model IdempotentRequest -m

Now that we have our model and migration existing, let’s add the columns we’ll need.

We'll need the responseBody and responseStatusCode to respond directly to the client when we have already handled a specific key.  resourcePath is not strictly needed, but you'll likely find it handy.

export default class IdempotentRequest extends BaseModel {
  /// Existing code here ...

  @column()
  public idempotencyKey: string

  @column()
  public resourcePath: string

  @column()
  public responseBody: string

  @column()
  public responseStatusCode: string
}

export default class extends BaseSchema {
  protected tableName = 'idempotent_requests'
  public async up () {
    this.schema.createTable(this.tableName, (table) => {
      // Existing code here ...

      table.string('idempotency_key')
      table.string('resource_path')
      table.jsonb('response_body')
      table.string('response_status_code')
    })
  }

Next up is creating our middleware, and wiring it to Adonis so the framework knows to always check for an Idempotency Key.  This code will apply the middleware to every incoming HTTP Request.

node ace make:middleware Idempotency

Then the wiring, registering with Adonis:

Server.middleware.register([
  () => import('@ioc:Adonis/Core/BodyParser'),
  () => import('App/Middleware/Idempotency'),
])

We actually haven't implemented any logic yet, we've just set ourselves up for success. Now... it's time.

Let's grab the X-Idempotency-Key header, as well as the resourcePath.  If we don't see an Idempotency Key, we can skip the idempotency logic altogether and still allow the requests to flow through to our controllers (this may not always be desirable behavior if you do not want to allow non idempotent operations).

export default class Idempotency {
  public async handle(
    { request, response }: HttpContextContract, 
    next: () => Promise<void>
  ) {
    const idempotencyKey = request.header('X-Idempotency-Key')
    const resourcePath = request.url(false)

    if (!idempotencyKey) {
      await next()
      return
    }
  }
}

Okay - let's check for an existing IdempotentRequest with the same key, or create one if it doesn't exist yet (because this is the first request!)

public async handle(
    { request, response }: HttpContextContract, 
    next: () => Promise<void>
  ) {
    // Code referenced above...

    const idempotentRequest = await IdempotentRequest.firstOrCreate(
      { idempotencyKey: idempotencyKey },
      {
        idempotencyKey,
        resourcePath: resourcePath,
      }
    )
  }

Not quite at the finish line yet, we need to either save the response body and status code, or return the cached response (if it's not the first request with the X-Idempotency-Key)

public async handle(
    { request, response }: HttpContextContract, 
    next: () => Promise<void>
  ) {
    // Code referenced above...

    // This is a property from the Lucid ORM - $isLocal is true if it was just created with the firstOrCreate.  
    if (idempotentRequest.$isLocal) {
      response.response.on('finish', () => {
        idempotentRequest.responseBody = JSON.stringify(response.getBody())
        idempotentRequest.responseStatusCode = response.getStatus()
        idempotentRequest.save()
      })
      await next()
    } else {
      return response
        .status(idempotentRequest.responseStatusCode)
        .send(idempotentRequest.responseBody)
    }
   }

There's also some housekeeping in this commit, like changing the responseStatusCode type to a number, telling our test runner to migrate our database before the tests run, and writing our first test to make sure the logic works.  For the sake of brevity, I'll point you to the commit here.

Alright, now that we’ve written the code, let’s try it out for real.

docker compose up (If you haven't already)
npm run ace migration:run  (Behind the scenes, this is just a alias to node ace ... from the AdonisJS documentation, you can see this in the package.json)

In your console, let's test this by making multiple identical requests to make sure that the result is the same, and this is now an idempotent operation.

curl -X POST \
  -H "Content-Type: application/json" \
  -H "X-Idempotency-Key: 333b2841-e854-4ba7-892f-e01787333049" \
   http://127.0.0.1:3333/authorizations

There's also a handy script in the package.json if you'd like to get into the database to see the idempotent_requests table: npm run docker-db.

I also would really encourage you to add some more tests! I have a few more commits to clean things up and flesh out the tests.

Edge Cases

Remember when I said we'll be taking some shortcuts? Yeah... about that.

Let's force some processing time here, and make a second request before the first one completes.

export default class AuthorizationsController {
  public async process({}: HttpContextContract) {
    await this.sleep(5000)
    return {
      id: uuid(),
      status: 'success',
    }
  }
  private async sleep(ms) {
    return new Promise((resolve) => setTimeout(resolve, ms))
  }
}

Run the curl command twice in quick succession (remember to use a new key!):

curl -X POST \
  -H "Content-Type: application/json" \
  -H "X-Idempotency-Key: a-new-idempotency-key" \
   http://127.0.0.1:3333/authorizations

{
    "message": "Invalid status code: null",
    "stack": "RangeError: Invalid status code: null\n    at new NodeError (node:internal/errors:371:5)\n    at ServerResponse.writeHead (node:_http_server:274:11)\n    at Response.flushHeaders (/home/node/app/node_modules/@adonisjs/http-server/build/src/Response/index.js:422:23)\n    at Response.endResponse (/home/node/app/node_modules/@adonisjs/http-server/build/src/Response/index.js:113:14)\n    at Response.writeBody (/home/node/app/node_modules/@adonisjs/http-server/build/src/Response/index.js:193:18)\n    at Response.finish (/home/node/app/node_modules/@adonisjs/http-server/build/src/Response/index.js:837:32)\n    at Server.handleRequest (/home/node/app/node_modules/@adonisjs/http-server/build/src/Server/index.js:125:26)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)",
    "code": "ERR_HTTP_INVALID_STATUS_CODE"
}

Now, this is not really the end of the world.  This is happening because the second request is seeing the database object before the first request gets a chance to save it's response.  If you retry again after our first request succeeds, you'll see all works as expected, in an idempotent manner.

Ideally, in a production system you'll have some sort of retry policy to adhere to, and the client should only be retrying with a duplicate request after a reasonable amount of time (when we'd expect the first request to be complete).

There's some ways around this, depending on the path you want to go down.

1. Hold on to the second request, looping until the response body and response code is present.  This is somewhat resource intensive (as it holds an extra connection open) and dangerous.  If your first request never completes, this request (and every request after) will be held until theres a timeout.

2. Return 429 - Too Many Requests to the client.  This is also a bit dangerous, for the same reason as above.  If the first request never completes, every request will return a 429.

3. Return 429 and add a locked_at column.  After a set amount of time, we assume the first request failed, and allow the second request to process the request as if it was the first.  This is a bit better, but in a real system you'd need to make sure all your external calls (if there are any) are also idempotent-aware.

Let's implement the 3rd option.

npm run ace make:migration alter_idempotent_requests_add_locked_at

export default class extends BaseSchema {
  protected tableName = 'idempotent_requests'

  public async up() {
    this.schema.alterTable(this.tableName, (table) => {
      table.timestamp('locked_at', { useTz: true })
    })
  }

  public async down() {
    this.schema.dropTable(this.tableName)
  }
}

npm run ace migration:run

Then let's modify our logic to accommodate.

public async handle(
    { request, response }: HttpContextContract, 
    next: () => Promise<void>
  ) {

    /** Code above **/

    if (idempotentRequest.$isLocal) {
      response.response.on('finish', () => {
        idempotentRequest.responseBody = JSON.stringify(response.getBody())
        idempotentRequest.responseStatusCode = response.getStatus()
        idempotentRequest.save()
      })
      await next()

      // There is an existing idempotent request.
    } else {
      const thirtySecondsAfterOriginalRequest = idempotentRequest.lockedAt.plus({ seconds: 30 })
      const now = DateTime.fromJSDate(new Date())

      // The first request is not done processing yet!
      if (!idempotentRequest.responseBody && thirtySecondsAfterOriginalRequest > now) {
        return response.status(429)
      } else if (!idempotentRequest.responseBody) {
        response.response.on('finish', () => {
          idempotentRequest.responseBody = JSON.stringify(response.getBody())
          idempotentRequest.responseStatusCode = response.getStatus()
          idempotentRequest.save()
        })
        await next()
        return
      }

      /** Code below **/
  }

Boom! We're done here.

Build Upon It

Some ideas if you’d like to play with this implementation further and build upon it.

1. Require the same request to be passed with the same key.

2. Do not allow failures to be retried, store them so each key truly is always the same operation, and each call has the same effect.

3. Consider implementing webhooks.

4. Improve your developer experience around webhooks.