Real world testing recipes: Node service that calls an external API

Originally published at coreycleary.me. This is a cross-post from my content blog. I publish new content every week or two, and you can sign up to my newsletter if you'd like to receive my articles directly to your inbox! I also regularly send cheatsheets, links to other great tutorials (by other people), and other freebies.

This is the next post in my series on knowing what to test in your Node applications by applying recipes to real-world scenarios.

In the first one, we went over scenarios you should cover in your tests when calling a database from a Node service.

In this post, we'll cover another very common real-world application: a Node service that calls an external REST API/endpoint. "External" means it is an application outside of our own - think the Twitter REST API, Google Maps API, or even an API internal to your company, but not part of your application.

In case you're new to the series...

If you've ever gone through a JavaScript or Node unit testing tutorial, it can be difficult to find anything that shows you what you should be testing - what scenarios to cover, what to write tests for, etc.

It's easy to find tutorials that take you through very basic examples - things like how to write tests for addition functions or tests for checking object properties - but more difficult to find ones that go beyond the basics as well as cover real-world scenarios.

As a developer you know you "should" be writing tests if you want to be considered a "good developer". But if you don't know the kinds of test scenarios you should be looking out for, it can be hard to write them in the first place. And if you're completely new to writing tests, it's even more frustrating.

When you've got a feature you need to implement at work, deadlines are looming, and you're stuck when it comes to the tests, usually those tests don't get written at all.

Application of testing recipes

When I was learning how to write tests for my code, I faced this frustration too. But I learned what things to cover by a couple different means:

getting bug reports for scenarios my tests should have covered
reading through lots of tests for existing codebases, both work codebases as well as open source ones

I eventually noticed that a lot of tests covered a similar set of scenarios. Depending on what part of an application you're working on, there are things to look out for to make sure your tests - and by extension, code - cover so you can be sure any new changes introduced into the application later on will catch anything if it breaks.

These scenarios are distilled from what I've found. You can use these as a starting point for when you encounter a similar application.

Ultimately, not only will you know what tests to write, the tests will also help inform the design/implementation of your code.

Our real-world example

We'll be writing tests for a library application that allows you to search for library books by title.

The API we will be interacting with will be the Open Library Search API.

The complete code (with tests!) can be downloaded here, but I recommend following along here first. After all, the point of this post is to help you identify scenarios to cover rather than just understand the code.

And with that, let's start to get into our recipes...

Scenario 1: Does our Node service call the external API successfully?

Here's our initial code to actually call the API. In our Node service - book.service.js:

const request = require('superagent')

const fetchBooks = async (query) => {
  return await request
      .get('http://openlibrary.org/search.json')
      .query({ q: query }) // query string
}

module.exports = {
  fetchBooks
}

So what scenario should the test cover?

const chai = require('chai')
const chaiAsPromised = require('chai-as-promised')
const nock = require('nock')

const { booksFixture } = require('./books.fixture')

const { fetchBooks } = require('../src/book-service')

const expect = chai.expect
chai.use(chaiAsPromised)

describe('Book Service', () => {
  describe('fetchBooks', () => {
    it('should return list of books based on search string', async () => {
      nock('http://openlibrary.org')
        .get('/search.json')
        .query(true)
        .reply(200, booksFixture)

      const {body} = await fetchBooks('lord of the rings')
      expect(body).to.deep.equal({
        docs: [
          {title_suggest: 'The Lord of the Rings', cover_edition_key: 'OL9701406M'},
          {title_suggest: 'Lord of the Rings', cover_edition_key: 'OL1532643M'},
          {title_suggest: 'The Fellowship of the Ring', cover_edition_key: 'OL18299598M'}
        ]
      })
    })
  })
})

What we test here is that we get back a successful response from the API (200 status code) and we get our book results back. Easy, right?

Because this is our first test we've added, so let's quickly go over how we're testing it.

Mocking

Making use of mocks in tests could be an entire book by itself, but here we use it in a pretty simple way with the npm module, nock.

nock will listen for calls made to the url we specify - in this case the Open Library REST API - and will "intercept" those calls. So instead of actually calling the real Open Library REST API, we specify the fake response to return.

We do this because:

Calling a real HTTP API introduces latency
Latency slows our tests down
We have less control over the data returned
The data returned might be slightly different, which would break our tests

Fixtures

And the fake response we return? That is our fixture. Just like mocks, fixtures are too big a concept to cover entirely here. But ultimately they're pretty easy.

This is what the real response looks like from the Open Library REST API:

Each of those properties like isbn and text are arrays with potentially hundreds of items. Can you imagine if we had to reproduce that response by hand? We could copy and paste the JSON response, but even then it would take up the whole test file and be really difficult to read.

Instead of reproducing the whole response, we only reproduce a subset of it. This gives us enough data to test what we need without having to clutter up our tests.

And that data goes in our books.fixture.js file:

const booksFixture = {
  docs: [
    {title_suggest: 'The Lord of the Rings', cover_edition_key: 'OL9701406M'},
    {title_suggest: 'Lord of the Rings', cover_edition_key: 'OL1532643M'},
    {title_suggest: 'The Fellowship of the Ring', cover_edition_key: 'OL18299598M'}
  ]
}

module.exports = {
  booksFixture
}

Scenario 2: What if the external API is down?

That's awesome if the API is running functionally, our current code and tests cover that.

But what if the API is down? How will our code handle that?

Let's take care of it in our code first. In book.service.js, let's modify our fetchBooks function:

const fetchBooks = async (query) => {
  let response

  try {
    response = await request
      .get('http://openlibrary.org/search.json')
      .query({ q: query }) // query string
  } catch(e) {
    response = e.status
  }

  if (response.status === 500) throw new Error('Open Library service down')
  else return response
}

Cool, let's add the test for that:

    it('should throw an error if the service is down', async () => {
      nock('http://openlibrary.org')
        .get('/search.json')
        .query(true)
        .reply(500)

      await expect(fetchBooks('lord of the rings')).to.be.rejected
    })

I chose to throw an error here in our service, which the controller which calls this service would then have to catch and handle. But we could just as easily return null or an empty array. This more just depends on your requirements.

Scenario 3: What if the external API finds nothing for our query?

If the API is up, but our search returns nothing, we'll get a 404 response code from the API. So let's handle that too:

const fetchBooks = async (query) => {
  let response

  try {
    response = await request
      .get('http://openlibrary.org/search.json')
      .query({ q: query }) // query string
  } catch(e) {
    response = e.status
  }

  if (response.status === 404) return null
  if (response.status === 500) throw new Error('Open Library service down')
  else return response
}

And the test:

    it('should return null if query returns a 404', async () => {
      nock('http://openlibrary.org')
        .get('/search.json')
        .query(true)
        .reply(404)

      const response = await fetchBooks('aksdfhkahsdfkhsadkfjhskadjhf')
      expect(response).to.be.null;
    })

Easy!

Scenario 4: What if there is a problem with our request?

There are several things that could be wrong with our request:

We could have accidentally forgotten to add the query string
We could have a bad character in the query
We could be missing the appropriate authentication tokens/headers

Fortunately, the Open Library API does not require any authentication tokens. It's... well... "open".

But if you did have a service that required a JWT token for example, or Basic Auth, it would be good to cover the scenario in which that is missing or improperly formatted.

Let's modify fetchBooks again:

const fetchBooks = async (query) => {
  let response

  try {
    response = await request
      .get('http://openlibrary.org/search.json')
      .query({ q: query }) // query string
  } catch(e) {
    response = e.status
  }

  if (response.status === 404) return null
  if (response.status === 500) throw new Error('Open Library service down')
  if (response.status >= 400) throw new Error('Problem with request')
  else return response
}

Because there are lots of different HTTP response codes we could cover, and we could write lots of conditional checks to handle each one, here we just specify if (response.status >= 400) to catch all of the Bad Request 400-level codes.

And the tests:

    it('should throw an error if there is a problem with the request (i.e. - 401 Unauthorized)', async () => {
      nock('http://openlibrary.org')
        .get('/search.json')
        .query(true)
        .reply(401)

      await expect(fetchBooks('lord of the rings')).to.be.rejected
    })

    it('should throw an error if there is a problem with the request (i.e. - 400 Bad Request)', async () => {
      nock('http://openlibrary.org')
        .get('/search.json')
        .query(true)
        .reply(400)

      await expect(fetchBooks('lord of the rings')).to.be.rejected
    })

Scenario 5: What if our application doesn't need the whole response?

What if our application doesn't need the whole response? What if we just need, say, the book titles?

We would need a filter/format function. In book.service.js, let's add a getBookTitles function and add it to the book service exports:

const getBookTitles = (searchResults) => {
  return searchResults.map(({title_suggest}) => title_suggest)
}

module.exports = {
  fetchBooks,
  getBookTitles
}

And the test:

  describe('getBookTitles', () => {
    it('should filter down response object to just book titles', () => {
      const titles = getBookTitles(booksFixture.docs)
      expect(titles).to.deep.equal([
        'The Lord of the Rings',
        'Lord of the Rings',
        'The Fellowship of the Ring'
      ])
    })
  })

Wrapping up

Here is the list of tests in their entirety:

const chai = require('chai')
const chaiAsPromised = require('chai-as-promised')
const nock = require('nock')

const { booksFixture } = require('./books.fixture')

const { fetchBooks, getBookTitles } = require('../src/book.service')

const expect = chai.expect
chai.use(chaiAsPromised)

describe('Book Service', () => {
  describe('fetchBooks', () => {
    it('should return list of books based on search string', async () => {
      nock('http://openlibrary.org')
        .get('/search.json')
        .query(true)
        .reply(200, booksFixture)

      const {body} = await fetchBooks('lord of the rings')
      expect(body).to.deep.equal({
        docs: [
          {title_suggest: 'The Lord of the Rings', cover_edition_key: 'OL9701406M'},
          {title_suggest: 'Lord of the Rings', cover_edition_key: 'OL1532643M'},
          {title_suggest: 'The Fellowship of the Ring', cover_edition_key: 'OL18299598M'}
        ]
      })
    })

    it('should throw an error if the service is down', async () => {
      nock('http://openlibrary.org')
        .get('/search.json')
        .query(true)
        .reply(500)

      await expect(fetchBooks('lord of the rings')).to.be.rejectedWith('Open Library service down')
    })

    it('should return null if query returns a 404', async () => {
      nock('http://openlibrary.org')
        .get('/search.json')
        .query(true)
        .reply(404)

      const response = await fetchBooks('aksdfhkahsdfkhsadkfjhskadjhf')
      expect(response).to.be.null;
    })

    it('should throw an error if there is a problem with the request (i.e. - 401 Unauthorized)', async () => {
      nock('http://openlibrary.org')
        .get('/search.json')
        .query(true)
        .reply(401)

      expect(fetchBooks('lord of the rings')).to.be.rejectedWith('Problem with request')
    })

    it('should throw an error if there is a problem with the request (i.e. - 400 Bad Request)', async () => {
      nock('http://openlibrary.org')
        .get('/search.json')
        .query(true)
        .reply(400)

      await expect(fetchBooks('lord of the rings')).to.be.rejectedWith('Problem with request')
    })
  })

  describe('getBookTitles', () => {
    it('should filter down response object to just book titles', () => {
      const titles = getBookTitles(booksFixture.docs)
      expect(titles).to.deep.equal([
        'The Lord of the Rings',
        'Lord of the Rings',
        'The Fellowship of the Ring'
      ])
    })
  })
})

Next time you're writing a Node application that calls out to an external REST API, use this recipe as a starting point for writing your tests. It will make it much easier to know what to cover and help you figure out how to write your code.

One last thing!

Testing took me a while to figure out - there's not a lot of good tutorials out there explaining both the how and what of testing.

I'm trying to help make it easier. Because as soon as I figured out how to write just a few tests, it "snowballed" from there, and writing tests became much easier. And maybe... even... fun?

There are more testing posts on the way - if you want to learn not only how to test but what to test, sign up to my newsletter to be notified as soon as the next post is released!

Blog