I am currently looking for a better way to save and retrieve and save data for my favorite side project. I did some research and I thought it might be nice to share what I found out so far.

Requirements

The app is meant to run as pwa as well as electron desktop app and uses rxjs to a wide degree. At the moment there is no backend and if there ever will be one it will probably be optional. As the main focus is the electron app I fortunately don't have to care too much about browser support, though all the libs seem to have support for all modern browsers.

These are the requirements I came up with:

Essential

• Super speed: it should feel as fast as a native desktop app for about 2000 different entities at minimum and up to 10mb of data stored and quick to load combined data and subsets of data
• Relatively easy to implement without too much overhead
• Future proof: should be under active development and supported by a big community
• No UI Blocking should occur even when working with larger datasets
• Framework agnostic

Bonus

• Working typescript support (at least no errors)
• Readable indexddb model => the data should be browseable via chrome dev tools
• Ideally should provide the ability to get a dataset as an observable or it should be easy to implement such thing
• Replication to a backend server: though not yet implemented, it might be nice to have the option
• Bonus bonus: simple file storage replication to google drive, owncloud or dropbox

The competitors

As a starting point I used this list. Please keep in mind that I am by no means an expert for any of the libraries used here, so chances are that I missed something important. If so please let me know in the comments.

These are my picks:

WatermelonDB

Description: Reactive & asynchronous database for powerful React and React Native apps

advantages

• Sync capabilities build in
• Quite fast thanks to LokiJS
• Includes a query language similar to mongodb
• Schema validation
• Database interactions run inside a worker so no risk of ui blocking
• Write sqllite to disk (which might be nice to use with electron)

drawbacks

• Typescript Errors
• Strongly focused on react
• Some issue with fast consecutive writes (but that might have been me using it wrong)
• IndexedDB is just stored as one big string

RxDB

Description: A realtime Database for JavaScript Applications

advantages

• Sync (thanks to PouchDB probably the most advanced)
• Nice Syntax
• Wide framework support and examples
• Schema validation via JSON Schema

drawbacks

• Very slow for larger datasets
• Render blocking occurs when interacting with larger datasets (this might be my fault again though)
• Data is stored separately but is not easy browsable because of all the PouchDB stuff

Dexie

Description: A Minimalistic Wrapper for IndexedDB

advantages

• Schema Validation
• Relatively fast
• Clean Database usage for IndexedDB
• Framework agnostic

drawbacks

• Not clear to me yet if syncing is easy to implement
• No real observable interface (yes there is dexie observable but, but that is just for watching change events rather then getting updated data)

LokiJS

Description: javascript embeddable / in-memory database

advantages

• Fast
• Includes a query language similar to mongodb
• Framework agnostic
• Less overhead than RxDB and WatermelonDB

drawbacks

• No sync capabilities built in
• ? No observable data interface ?

remotestorage

Description: An open protocol for per-user storage on the Web

advantages

• Schema validation
• Own your data: Sync to google drive / dropbox and more

drawbacks

• Out of date dependencies
• No active development in the last couple of months
• Smaller community than the rest

Native IndexedDB

advantages

• no overhead
• relatively fast
• framework agnostic
• typescript should work

drawbacks

• completely custom sync required
• No observable interface
• probably lots of code to write
• not all browsers support all the features

Speed Test

This might be outdated but apart from my own non systematic tests I used this tool to compare the performance.

Premature Conclusion

So far I've only experimented with WatermelonDB, RxDB and a little bit with native IndexedDB. Here are the branches for WatermelonDB and RxDB in case you want to check for yourself:

https://github.com/johannesjo/super-productivity/tree/feat/watermelonDB
https://github.com/johannesjo/super-productivity/tree/feat/rxdb

WatermelonDB put me off because of its strong focus on React and it not working well with typescript atm. It shares one of the "problems" with LokiJS: The data in stored in IndexedDB is just a big string. This might not be a real problem, but it feels wrong to go about persistence like that. I didn't come much further than playing around a little with LokiJS, because I felt that it might be basically WatermelonDB with less issues, but also less of the stuff I need and I was still hoping for something better.

RxDB looked pretty promising and I assume that it really shines when you are using a server. But the bad performance was just not an option for an app that aims at desktop app like performance.

Using native IndexedDB should work fine for the most part. The main disadvantage is that I would have to write a lot of code and I don't want to re-invent the wheel (just quite yet).

Next step is Dexie! The community might not be as big as the ones of it's competitors and the lack of a prebuild performant way to use collections as observables is a little bit disappointing. I am also unsure about the sync stuff, as I didn't find too many good examples and it seemed to as if it was more of a gimmick than something people are using in production, but I hope I am wrong! So far it has been the only library offering a clean IndexedDB to browse while at the same time offering some structure making sure I am not the only going about saving data this way.

I'll update the article once I know more!

Did I get something wrong? Probably! This why I'm very curious to hear your thoughts and experiences on the matter. Please share them in the comments!