**Part 1 of 2** As an backend/infra guy, I've grown a lot of respect for the intricacies of frontend development the last few years. By and large, a lot of the first wave problems of frontend javascript development have been fixed. The language facilities have gotten really nice, browser support has become more consistent, you can find the typescript typings for most things now, a lot of key packages have stabilized and upgrading things is less of a nightmare. As soon as Ecmascript gets python style [ignored-by-default type hints](https://docs.python.org/3/library/typing.htm) (most transpilers currently do this) so it can interop with typescript more easily, javscript, or typescript, really, might become my favorite langauge. But still, frontend development is pretty damn hard! And that makes sense. Web applications can have interfaces as complex as [IDEs](https://github.com/microsoft/vscode) and data exchange layers as complex as [distributed databases](https://github.com/yjs/yjs). A relatively "easy" problem I was dealing with recently in the data interchange layer demonstrates this well. As with most frontend tutorials it starts with the the problem of Todos. Consider the following: ```typescript import React, { useCallback, useState } from "react" interface Todo { id: number title: string done: boolean } type IdType = Todo["id"] const Todo = (props: { todo: Todo; remove: () => void; update: (todoId: IdType, updates: Partial) => void }) => { const { todo, remove, update } = props return (

update(todo.id, { title: e.target.value })} /> update(todo.id, { done: !todo.done })} />

) } const Todos = () => { const [todos, setTodos] = useState([]) const [newTodo, setNewTodo] = useState(null) const createTodo = useCallback((todo: Todo) => setTodos((todos) => [...todos, todo]), [setTodos]) const updateTodo = useCallback( (todoId: IdType, updates: Partial) => setTodos((todos) => todos.map((t) => (t.id !== todoId ? t : { ...t, ...updates }))), [setTodos] ) const removeTodo = useCallback((todoId: IdType) => setTodos((todos) => todos.filter((t) => t.id !== todoId)), [setTodos]) return (

{todos.map((t) => ( removeTodo(t.id)} /> ))}

{newTodo && ( )}

) } ``` Bam in just a few lines of code we've implemented pretty much all the CRUD methods for todos. We can even update the titles and make them done. Really cool. Told ya, React is great. Look how easy it is to implement todos? But it's not saving anywhere. This shouldn't be too difficult either. We whip up our favorite instant-backend in the format de jour (GraphQL obviously, REST for example's sake) and API is ready. Just a few lines to update on the frontend now: ```typescript const [todos, setTodos] = useState([]) // Connect to our backend const fetchData = useCallback(async () => { const resp = await fetch("/todos") setTodos(resp.data) }, [setTodos]) // Fetch our todos on load useEffect(() => { fetchData() }, []) // our createTodos should now use the API methods const createTodo = useCallback((todo: Todo) => { const resp = await post("/todos", todo) // refresh data fetchData() }) const updateTodos = useCallback((todo: Todo) => { const resp = await patch("/todos", todo) // refresh data fetchData() }) ``` We fire it up. Thing seem to mostly work, but the UI is kind-of **glitchy**. You see, our webserver is running locally, so our net latency is as close to zero as we'll get. Our API is responding in 40ms but things still don't feel 'instant', there's a little flash in the UI as todos are added, and we wait for responses. This will only get worse as the network latency goes up when we deploy to production. We also notice when we update the todos we get mad **race conditions**, sometimes the update returns a stale object because responses are out of order. This makes sense our async APIs can respond whenever they want so if they request and responses aren't ordered and we fire them off willy-nilly the new data be out of order. Now we realize we have two **big** data synchronization problems: 1. We need to synchronize our data with the DOM and avoid unnecessary rendering. 2. We need to synchronize our local data with the backend server Turns out both of these problems are pretty difficult. And we've barely addressed any of the **Advanced Rich Webapp Requirements™**: #### Error Catching We need to let the user know when there was an error in the API request. This can happen on any operation and depending on which operation (initial load vs an update) we have to do different things. So we add: ```typescript const [error, setError] = useState(null) const [initialLoadError, setLoadError] = useState(null) useEffect(() => { // For some toast or notification toast.error("Unable to process request") }, [error]) if (initialLoadError) { return

{initialLoadError}

} else { // ... render component } ``` But what does this mean for our local state? How do we rollback the UI if this happened in an update or a delete? #### Load Screens We need to show the user that their initial load/query/etc are still loading and inflight. There are also different forms of loading. When we're loading the initial data we want a full load-spinner overlay on the rendering area but when we're doing updates we want to just a load spinner in the corner. Some more hooks: ```typescript const [loading, setLoading] = useState<"initial" | "partial" | null>("initial") if (initialLoadError) { return

{initialLoadError}

} else if (loading === "initial") { return (

) } else { ;

{loading === "partial" && (

)} // ... render rest of component{" "}

} ``` #### Debouncing Users type fast and we can't send every keystroke as an API request. The natural way to solve this is to add a debounce: ```typescript const updateTodosDebounced = useDebounce(updateTodos, 2000, { trailing: true }, [updateTodos]) ``` Wait do I want trailing or leading? Hmm. We add this and we still see some weird rollback behavior as the user types ( this is due to the request races). Good enough. #### Synthetic local data (optimistic UIs) We decide to solve our flashing problem by having synthetic local state. Basically we temporarily add data to a local **synthetic** array of our existing data from the API and local mutations that have still not been persisted. This one is tricky, because it's hard to figure out which data is fresher (see race conditions mentioned above). Lets try a solution thats good enough: ```typescript const [todos, setTodos] = useState([]) const [deletedTodos, setDeletedTodos] = useState([]) const [localTodos, setLocalTodos] = useState([]) // mergeTodos is left as an (complex) excercise for the reader const syntheticTodos = useMemo(() => mergeTodos(todos, localTodos, deletedTodos), [todos, localTodos, deletedTodos]) ``` Now say we delete something, we add the id to deleted todos and our `mergeTodos` will drop that entry when creating the synthetic results. The function will also merge any mutations into the todos e.g. `todo = {...todo, ...localTodo}` Our synthetic array has reduced the flashing significantly. Everything feels instant now. We're not sure about the logic of the merge function as you can tell its still not race-proof. Also, what if the API operations related to the synthetic updates fail? How do we rollback? #### Working offline, retry and network-down logic: We're on an airplane and we realize that when there is no wifi, the app is behaving poorly. Because of our synthetic data changes we're getting fake mutations that aren't actually persisted. Our favorite apps webapps let us know when there is no connectivity to the backend and either halt new operations or let us work offline for syncing later. We decide on the former (its hacky but quicker): ```typescript const [networkOffline, setNetworkOffline] = useState(navigator.onLine) useEffect(() => { const updateOnlineStatus = () => { setNetworkOffline(navigator.onLine) } window.addEventListener("online", updateOnlineStatus) window.addEventListener("offline", updateOnlineStatus) return () => { window.removeEventListener("online", updateOnlineStatus) window.removeEventListener("offline", updateOnlineStatus) } }, []) ``` We add a bunch of logic switches around the place to avoid updates and changes when things are offline. We realize we need a few UI elements to either let the user see initial load data or block it off completely. #### Undo logic Now we wonder, how the f\*\*\* is `cmd-z` implemented in Figma? This requires full knowledge of local operation order and very very smart synchronization of our backend. Yeah, screw it, users don't need cmd-z right now, we'll figure out how to stich it into all these other things down the road. #### Live reloading and Multiuser collaboration Who uses todo apps without collaboration? When another user modifies a todo it should be relfected locally and update our UI so we don't overwrite their changes. We learn about CRDTs but that feels like overkill. Okay, lets do it the easy way: ```typescript // Update our data every few seconds useEffect(() => { const interval = setInterval(() => { fetchData() }, 5000) return () => { clearInterval(interval) } }, []) ``` Obviously this will create some races and overwrite things but why where our users collaborating on the same todo within 5 seconds to begin with? They shouldn't be doing that. #### Data caching Why not store the last fetch data locally so we can load it while the newer data is loading? Maybe something like: ```typescript const [todos, setTodos] = useState() // Load initial data from local storage useEffect(() => { const res = localStorage.getItem("todo-cache") if (res) { setTodos(JSON.parse(res)) } }, []) // Update our todo cache everytime todos array changes useEffect(() => { localStorage.setItem("todo-cache", JSON.stringify(todos)) }, [todos]) ``` We need to key the cached query based on the query and we still need to expire super old data and on user logout. #### Query reuse and bidrectional data binding. If we use a similar query in a completely different component on the page we should bind the same results/udpates from the earlier query. If a todo is rendered in multiple places or can be edited in multiple places the data should cross sync between the two components in realtime. This requires _lifting_ the state. Let's skip this for now. ## Hook Soup and Off The Shelf Tools At this point, our `Todo.tsx` has something like **40 hooks** and **12 components**. All to implement a half-assed glitchy CRUD on some simple todos. Our dependency arrays are insane and someone recently reported that there is a loose API request that's firing every 10ms. We look at the git blame and see someone added something to the dependency array they shouldn't have (to be fair Eslint blindly warned them to add it). **Surely someone else has solved this...** And we would be correct, depending on which part of the problem we care about. #### Problem 1: Binding data to the DOM/React First we look at solutions to the DOM data binding problem. [There are a ton](https://github.com/olegrjumin/awesome-react-state-management): - **React Hooks**: Great for entry level work, absolute mess when we start introducing all those things. Threading these state variables across the 15 components we have is turning to be a nightmare. - **Redux**: Looks great. The event stream it uses fits very well with some undo/rollback logic we think we'll need. After trying it, we find that the out-of-band side effects spread over a bunch of reducer files are unclear. Access to global state is hard and API requests are weird... wtf is a thunk? - **Mobx**: Whoa this looks easy. We make a class, mark variables we rerender on as observable and things look simple and imperative. Facebook uses it on WhatsApp. None of the event stream stuff from Redux here. We either snapshot the class at points in time, or we roll our own solution. - **XState**: FSMs are cool. We've used them a few times on some backend flows with much success. We whip up an example and realize that the machine became super complex. There are a ton of flows and things like rollback/partial-loading/etc become a bit tough to reason about. Maybe we keep the main logic in the FSM and sub rendering logic independent? After trying a few we land on **mobx**. Theres a bit of magic around the wrapping, but we find that 98% of the time that magic works great. Using `observers` everywhere is annoying, but we read about how it minimizes our rerendering by watching only the fields we used in the component (effectively memoizing every component), and we decide its worth it. #### Problem 2: Binding data to the backend Now that we have a solution to the data binding problem we need a solution to backend synchronization problem. There are a ton of options here too: - **useSWR**: A react data fetching API that handles a lot of components like caching, states (loading/error/results), optimistic UI support, and we have to be very uniform rest. - **Apollo GraphQL Client**: Lots of nice things built into this powerful library. Hard requirement is that we use GraphQL. - **Basic Fetch**: Using the native browser APIs to make requests and manage state ourselves. - **Cloud storage clients like Firebase**: Many cloud APIs come with SDKs and react data bindings like Google Firebase. Our API isn't GraphQL (maybe it should be?) so we land on **useSWR**. This lib only handles _some_ of our Advanced Rich Webapp™ requirements. #### Problem 2.5: Connecting these two pieces: Sadly, the lib we use for fetching data is also highly intertwined with the lib we use to synchronize data. In the case of **useSWR** our hands become forced to adopt their hook based system or we need to create some bindings into our own state management system. So we kinda get frameworked, one way or another. ## Next Steps At this point, hopefully, we'd be content enough with the off-the-shelf tools. We grab some of them, create some of the glue code and proceed to use it. Where we need things like rollback and network state we put some of our ad-hoc logic in there to handle it appropriately. But we're not totally satisfied. `Todos` are just one data model in our application. We'll probably have a 30 more and repeating the same patched-together hooks and methods across all these will suck. Also adding new functionality as we need it will become arduous once we have these half-assed hooks sprinkled everywhere. Our team is big enough and this is a big enough problem. Let's do the unthinkable. Lets _roll out our own solution._ **Next time**: In the next blog post (hopefully the next week), I will cover how to create a frontend transaction log that satisfies a lot of our Advanced Rich Webapp™ requirements. We will implement a log that tries it's best to linearize operations and provide ways to mutate and rollback things while keeping the component logic minimal. We implement the transaction manager as a generic so we can use it for `Todos` and any other types we need. Want to be notified when we drop the post? Follow along on [RSS](https://synth.app/rss.xml), [Twitter](https://twitter.com/pnegahdar), or [signup to our mailing list](https://synth.app).

pnegahdar

Todos: The Hard Parts

Join Our Newsletter. No Spam, Only the good stuff.