Service example to sync the database with data from a third party API (with tests)

Situation

Imaging you have an app which creates data in your database base on a third party API (this doesn’t mean that you replicate the data from the third party in your database, is more like if something changes in the third party app, then you create something in your DB base on that). This means that when the data changes in the third party, probably you will need to delete or create data in your database (model).

I’m going to explain you one approach for doing this with a list (array) of data. It includes the tests for the functions.

UI Example

I made an App with Next.js that consumes this services. Basically it fetch Pokemons from the https://pokeapi.co and saves them in a global variable that would simulate a database. It has a number input used as an “until this id” to fetch Pokemons, so we can simulate a change in a third party API that will trigger our sync service. Also, it shows a list with the current Pokemons saved and on the right side it shows which Pokemons were delete or create by the sync service.

DEMO: https://sync-db-app.vercel.app

Repository: https://github.com/georgexx009/sync-DB-app

Functions

We are going to divide our service in 4 functions. One it is going to coordinate the whole process, would be like the root of the service (syncDb). Also another to check which entities are going to be add to the database and which ones are going to be deleted, something like reducing an array (reduceSyncList). And the last two are for deleting and adding to our database.

Reduce sync list

This function should has two parameters. One is going to be your data list and the other is going to be the third party data list. If it possible, define the interface of how it looks the data from the 3rd party API, this will make your coding more easy.

interface Params {
    internalData: InternalData;
    externalData: ExternalData;
}

interface ReducedSyncList {
  toDelete: InternalData[]
  toAdd: ExternalData[]
}

export const reduceSyncList = ({ internalData, externalData }: Params): ReducedSyncList => {}

Observe that the property toDelete has the interface from your internal data. This is because the data that is going to be deleted comes from your database and the data that is going to be added comes from an external source.

You could add a mapper function to help you with this. What I would do is to map the external data to reduce it to only what I need. Also will help me to differentiate each type of data.

The first step in our reduce sync list function is to create an object, map each element from the internal data, place as key the externalID and as value the element it self. This would help us when we search each external element with the internal elements, making time complexity of O(N) instead of O(N^2).

const mappedInternalData = internalData.reduce((obj, el) => {
        return {
            ...obj,
            [el.externalId]: el
        }
    },{})

The second step is to find which external elements doesn’t exist in our DB (internal data). We are going to use a reduce and make the comparison statement searching the id from the external in the object that we created. Observe that this search takes O(1).

If it exists, we are going to delete the property from the object. Because we use spread to created the new object, we are not pointing to the same space of memory, we use a new one. If it doesn’t exists, we are going to add it to the new array with the elements that are going to be added.

const toAddList: ReducedSyncList['toAdd'] = externalData.reduce((syncLists, el) => {
    if (mappedInternalData[el.id]) {
      delete mappedInternalData[el.id]

      return syncLists
    }
    return [el ,...syncLists]
  }, [])

The third step is to get the ones that are going to be deleted. If they remain in the mappedInternalData, that means that doesn’t exist in the external source and if it is want you need, we are going to delete it.

We use Object.values which returns an array of the values from an object.

const toDeleteList: InternalData[] = Object.values(mappedInternalData)

And finally we return the object with the results.

return {
        toAdd: toAddList,
        toDelete: toDeleteList
  }

Create and delete DB record

I separate this to have more clean clode but is up to you. These functions are for using the ORM from your preference and interacting with your DB. Since this post is more focus on the sync service instead of saving the data in the DB, I’m going to mock this but you can use Sequelize, typeORM, mongoose or whatever works for you.

Im going to return a promise with a setTimeOut simulating a query to the DB.

const createDbRecord = ({ externalElement }: { externalElement: ExternalData }): Promise<InternalData> => {
    // simulating being connected with a database
    // use any orm of your preference
    const newID = generatorID();
    const mapped = mapExternalToInternal({ id: newID, externalData: externalElement });
    // SAVE IN DB
    return new Promise((resolve) => {
        setTimeout(() => {
            resolve(mapped)
        }, 200)
    });
}

const deleteDbRecord = ({ id }: { id: number }): Promise<boolean> => {
    // use try/catch, sometimes ORMs like Sequlize only returns a confirmation
    // if there is an error, return false
    return new Promise((resolve) => {
        setTimeout(() => {
            resolve(true)
        }, 200)
    })
}

Sync DB function

This function coordinates everything. We are going to have an interface that represents the results from our operations. I use this to only now the status of my operations and have a goog log or even repeat the operation for the ones that failed.

interface ResultOperation {
  id: number
  name: string
  status: boolean
}

let deleteResults: ResultOperation[] = []
let createResults: ResultOperation[] = []

Then we call our reduce function and we are going to iterate through the results to make our queries to our database. The only thing that I think should be said here is to remember how the array of promises works. We created the array with the map method, and then we use Promises.all() to retrieve the results.

const syncList = reduceSyncList({ internalData, externalData });

    if (syncList.toAdd.length > 0) {
        const arrayPromises = syncList.toAdd.map(async (el) => {
            const elCreated = await createDbRecord({ externalElement: el })
            return {
                id: el.id,
                name: el.name,
                status: elCreated ? true : false
            }
        });

        createResults = await Promise.all(arrayPromises);
    }

    if (syncList.toDelete.length > 0) {
        const arrayPromises = syncList.toDelete.map(async (el) => {
            const elDeleted = await deleteDbRecord({ id: el.id })
            return {
                id: el.id,
                name: el.name,
                status: elDeleted
            }
        });

        deleteResults = await Promise.all(arrayPromises);
    }

Finally we return again the results from the syncronization service.

Tests

I have a function to create the test data because I don’t want to have a file with the data.

I’m going to use the poke API, and from there create the data, internal and external data. Only I’m going to map the internal data because I’m simulating that I already process it and it is what I need to save in my database. And the external data will be as it comes from the API.

const createTestData = async (): Promise<{ externalData: ExternalData[], internalData: InternalData[] }> => {
    const generatorID = generateID({ initialID: 1 });
    const promises = [1,2,3,4,5].map(async (i) => {
        const res = await fetch('https://pokeapi.co/api/v2/pokemon/' + i);
        const data = await res.json();
        const newID = generatorID()
        return {
            internal: mapExternalToInternal({ id: newID, externalData: data }),
            external: data
        }
    })

    const data = await Promise.all(promises);

    return data.reduce((result, data) => {
        return {
            internalData: [...result.internalData, data.internal],
            externalData: [...result.externalData, data.external]
        }
    }, {
        externalData: [],
        internalData: []
    })
}

If you see, I have a fun function called generateID, I created it to generate my IDs (just numbers) to maintain the code simple. This is a clousure, that receives an initial ID number and from there every time the function returned is called, it increments the number count and returns it.

export const generateID = ({ initialID = 1 }: { initialID?: number } = {}) => {
    let ID = initialID;
    return () => {
        ID = ID + 1;
        return ID;
    }
}

Then, the tests consist on the escenarios to add a new element, to delete one and when there is no changes.

Depending on the test case, im going to add a mocked element to set the right context. The full test suites are in the repo.

describe('reduce sync list', () => {
        let externalData: ExternalData[];
        let internalData: InternalData[];

        beforeAll(async () => {
            const testData = await createTestData();
            externalData = testData.externalData;
            internalData = testData.internalData;
        });

        test('there is one to be deleted', () => {
            const internalDataWithOneInactive = [
                ...internalData,
                deletedInteralPokemon
            ];

            const result = reduceSyncList({ internalData: internalDataWithOneInactive, externalData });
            const { toAdd, toDelete } = result;
            expect(toAdd.length).toBe(0);
            expect(toDelete.length).toBe(1);
        });

        test('there is one to be added', () => {
            const externalDataWithOneNew = [
                ...externalData,
                newExternalPokemon
            ];
            const result = reduceSyncList({ internalData, externalData: externalDataWithOneNew });
            const { toAdd, toDelete } = result;
            expect(toAdd.length).toBe(1);
            expect(toDelete.length).toBe(0);
        });

        test('there is no changes', () => {
            const result = reduceSyncList({ internalData, externalData });
            const { toAdd, toDelete } = result;
            expect(toAdd.length).toBe(0);
            expect(toDelete.length).toBe(0);
        })
    })