The idea that I had for my Flatiron School Phase 1 Project seemed simple enough: a Single Page Application that would allow a user to search for an online chess player by username and would display their stats and recent games. The user would also be able to view each game through an iframe that would be embedded in the page.

As I took the first steps to create this program, I quickly discovered that it was more complicated than it initially seemed.

I started out by looking into the API for Chess.com, the most widely used online chess platform. However, I almost immediately ran into a big problem: The ID for a game's iframe did not match the game ID. As far as I could tell, the iframe ID was nowhere to be found in any of the API's responses, and without that ID, there would be no way to dynamically imbed a game's iframe.

I moved on to Lichess, the second most popular online chess platform. Luckily, Lichess's iframe ID and game ID matched. Problem solved, right?

Wrong.

As I looked over the Lichess API, I noticed something unfamiliar to me. A fetch request to https://lichess.org/api/games/user/{username} returned content in NDJSON (Newline Delimited JSON) format instead of in JSON format - the only format with which I was really familiar. Requests to other endpoints that I would be using - such as player information and games filtered by the opening played in the game, all returned data in JSON format - but I still needed a way to access and list a player's recent games.

Unsurprisingly, ignoring the offending 'Accept': "application/x-ndjson" and hoping to convert to JavaScript using .then(res => res.json()) did not work.

I then realized I had to figure out what NDJSON was before I could proceed. According to ndjson.org, NDJSON is "a convenient format for storing or streaming structured data that may be processed one record at a time." The main difference between the NDJSON and JSON formats seemed to be that NDJSON data is separated into separate lines using \n, while JSON data is not. Each separate line of NDJSON is valid JSON. In my case, each line of NDJSON would represent a chess game.

Here was my general idea for the function that would get a list of a user's games:

function listUserGames(e) {
    let username = e.target.querySelector('#username').value

    fetch(`https://lichess.org/api/games/user/${username}?max=8`, {
        method: 'GET',
        headers: {
            'Accept': "application/x-ndjson"
        }
    })
    .then(res => 
        // somehow convert res from NDJSON to JavaScript
    )
    .then(games => {
        if (games.length > 0) {
            games.forEach(game => displayGame(game))
        } else {
            // indicate that the user has never played a game
        }
    })
}

Google helped me find a working solution based on this repo. After some trial and error, I was left with the following code:

.then(res => res.text())
.then(body => {
    let str = "[" + body.replace(/\r?\n/g, ",").replace(/,\s*$/, "") + "]"
    return JSON.parse(str)
})

This worked, and I (mostly) understood why. It was converting the response to text, adding brackets to make the text like an array, replacing newline characters with commas to make each line like an array element, and removing whitespace before converting it to a JavaScript object. However, I felt uncomfortable using this. I didn't feel like I completely understood the syntax and I was not quite sure why whitespace was being removed. I couldn't have reproduced it. I moved on, eventually completing the application, but this block of code kept bothering me, even though it worked just fine.

I returned to this code block later and broke down what needed to be done.

1. Convert the response to text. (Done with .then(res => res.text()))
2. Make each line into an element in an array.
3. Convert each line to a JavaScript object

I knew how to split a string into an array, so I started there, using data.split('\n').

Now I had an array of strings. How could I go about converting each one to an object? I knew that each one was valid JSON, as stated in the NDJSON documentation, and I realized that the map() method would be perfect for this situation.

After implementing this method, I had the following code:

.then(res => res.text())
.then(data => {
    return data.split('\n').map(game => JSON.parse(game))
})

I tried it out, and... it didn't work. This error was displayed in my console when I ran my code:

Upon closer inspection, I found the problem. Running console.log(data.split('\n')) displayed an array with nine elements. I only asked for 8 games.

I finally found out why the whitespace needed to be removed.

It was time to try again. I used the filter() method to remove the empty string:

.then(res => res.text())
.then(data => {
    return data.split('\n').filter(item => item.length > 0).map(game => JSON.parse(game))
})

This still felt a bit inelegant, but it worked!

In the process of writing this blog, I realized I could make the code simpler by trimming the whitespace at the end of the string before splitting it into an array.

This left me with the following code:

.then(res => res.text())
.then(data => {
    return data.trim().split('\n').map(game => JSON.parse(game))
})

This is what I wanted: a way to convert NDJSON into JavaScript that I fully understood. This block of code turns the NDJSON to text, trims the whitespace, and makes each line of an element in an array. It then returns an array of JavaScript objects.

In retrospect, this solution seems obvious. However, as someone new to programming, a new format seemed much more intimidating than it actually was. I appreciate the process I went through now, as I have a much better understanding of what happens when my code is executed. A lot of debugging and examination of the network tab in Chrome DevTools allowed me see every change that my data underwent in the reformatting process. After this project, I can confidently say that I understand this process and can reproduce it. My takeaway is simple, and fittingly so: things got complicated, but they simplified beautifully.