Genesis

While working on my side project which is an implementation of a famous mmo rpg server, I had the need to edit a small json file containing an array of object of not really uniform data, the sum of unique field was ~200 fields. I had a lot of trouble to see which object were similar and so on.

I tried several tools like json crack or json table/tree visualizer.
Json crack was not adapted to my need: due to the number of different fields, a table was more appropriate to display the information.
Unfortunately tools I found had big performance issues and they were unusable, even on a file of 1.5mb.

Json table editor

To solve my issue I have developed a GUI tool json table editor which gives a table representation of json array, to ease visualisation and edition of large json array with performance as main objective.

It allow to edit large json files > 500mb. It is written in rust and use the amazing egui library

Why it is fast?

Use the right data structure for the problem

I have written my own parser (writting json parser is very easy) to produce a flat json format instead of tree data structure as usual.

This data structure consist of an array of key/value pair, where key is a json pointer and value the value at this pointer.
This data structure perfectly fit to use in a table

Only render what is visible

If we render only visible cells of the table, we can display a table of any size

Memory is scarified in favor of CPU usage

As this tool use an immediate mode GUI library, it is important to not slowdown the event loop. One way to limit the slowdown of the event loop is to cache expensive operation, which consume memory.

In addition, a feature of the tool is to switch between depth of the json file, to be able to change depth immediately, we keep in memory each depth of the json

For example:

{"skills": [{"copyFlags": {"reproduce": true, "plagiarism": false}}]}
/skills -> [{"copyFlags": {"reproduce": true, "plagiarism": false}}] # depth: 1
/skills/0 -> {"copyFlags": {"reproduce": true, "plagiarism": false}} # depth: 2
/skills/0/copyFlags -> {"reproduce": true, "plagiarism": false}  # depth: 3
/skills/0/copyFlags/reproduce -> true # depth: 4
/skills/0/copyFlags/plagiarism -> false # depth: 4

If in the GUI we choose to display json expanded to depth 4, it will render 2 columns: copyFlags/reproduce and copyFlags/plagiarism but in memory we will have the data shown above.

Data are present more than once, which can take lot of memory for big file.
But it has another advantage, the serialization is instant even in large file because when we edit the json we can update the representation at depth 2 of a row, which is way faster than serializing the whole json

Demo

Check out video on github

Why i am sharing this

It keep me motivated to share projects I am working on, and someone might find my tool useful or provide interesting feedback