A new release of your web service has just rolled out with some awesome new features and countless bug fixes. A few days later and you get a call: Why am I not seeing my what-ch-ma-call-it on my thing-a-ma-gig? After setting up that zoom call it is clear that the browser has cached old code, so you ask the person to hard reload the page with Ctrl-F5. Unless its a Mac in which case you need Command-Shift-R. And with IE you have to click on Refresh with Shift. You need to do this on the other page as well. Meet the browser cache, the bane of web service developers!

In this blog we share how we struggled and finally busted the browser cache for new releases of the Zebrium web service, including design and implementation details. Buckle up, it's a bumpy ride!

What Didn't Work?

At Zebrium we build our front-end using React. We find React to be extremely flexible, making it easy to write and maintain a variety of components from simple deployment drop-down menus to complex log and metric visualizations, all with a distinctive Zebrium dark-mode style.

Our build-test-deploy strategy is based on the create-react-app framework. Like React itself, that framework has served us well, but, like many who adopted it in the last few years, we suffered from one pretty big gotcha. Aggressive browser caching of application resources. So aggressive, that our users were missing out on key feature updates and bug fixes because the UI code they had in their browser cache was outdated. For a start-up with a need to quickly iterate on customer feedback, this was a real pain point.

Our customer-service team identified the issue first and the pattern of the problem was elusive.  Many users would see the upgrades automatically. But some would not.  Zebrium has always been lucky to have dedicated and enthusiastic users who understand our value proposition; luckily no more so than at moments like this. So, while we worked through the issue, customer-service helped affected users to clear their caches manually whenever we deployed a new version. But this was painful for us and the customers.

Before the UI team understood the root of the problem, we stepped through the usual remedies. We had our web server deliver headers with ever stricter cache-control settings. We reduced max-age from weeks to days and so on.  That wasn't ideal because theoretically it meant users would be pulling down code versions their browser had already cached. We were surprised to see that approach did not solve the problem either. And we even threw pragma: no-cache at it, a Hail-Mary that unfortunately had no effect. 

So, we began our investigation into create-react-app to discover why these tried-and-true HTTP client/server mechanisms were failing. After a lot of work, we finally isolated the issue to this: our version of create-react-app employed a service worker to cache content. That explained why some users encountered the problem while others did not. Users who were in the habit of closing their browser often did not see the problem. Users who kept their browser up for days and kept our app open in one or more tabs never saw our updates because the service worker was holding on to an old version of our UI code in cache. Here's a good discussion on create-react-app's Github page that lays out the issue and possible solutions ( https://github.com/facebook/create-react-app/issues/5316 ).  At the time of our investigation, we weren't in a position to take and test a new version of the create-react-app framework or to test some of the workarounds mentioned in that discussion. So, we decided to go old school, exposing versioning in our app path. It has worked very well.

Summary of what we did

In every UI build, we set the software version as a custom environment variable in the .env file prefix with REACT_APP_.  We can then access the current running version by referencing process.env.REACT_APP_MY_SOFTWARE_VERSION defined in .env. The current software version that the browser is running is also embedded in the URL and the software version is persisted throughout all UI route paths. 

Whenever an API call is invoked from any page, it returns the software version currently running on the server.  If the server and UI are in sync, the software versions will be the same. No more work to be done. However, if the API returned software version is different from process.env.REACT_APP_MY_SOFTWARE_VERSION, we throw up a popup dialog displaying a message saying a newer version has been detected. It includes a button the user can click to reload the page with content from the new software version.  The newly loaded software version will then be reflected in the URL.

Now let's run through this in more detail...

Routing

Once we decided to take the version in the URL approach, everything was simple, right? Sort of. Our web pages are served from the same Go application that serves the API. We had the build script generate a bit of Go code to compile the release version into the binary and altered the routing to put the release version into the path for serving the static content of the UI. This handler function takes a http.FileSystem that is initialized to the root UI directory and a string with the release version:

func FileServerNotFoundRedirect(fs http.FileSystem, redirect string) http.Handler {
    fsh := http.FileServer(fs)
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        if strings.HasPrefix(r.URL.Path, redirect) {
            r.URL.Path = r.URL.Path[len(redirect):]
            fd, err := fs.Open(path.Clean(r.URL.Path))
            if os.IsNotExist(err) {
                r.URL.Path = "/"
            }
            if err == nil {
                fd.Close()
            }
            fsh.ServeHTTP(w, r)
        } else {
            uri := r.RequestURI
            comps := strings.Split(uri, "/")
            if len(comps) > 1 {
                uri = uri[1+len(comps[1]):]
            }
            RedirectHTTPS(w, r, redirect+uri)
        }
    })
}

The first condition of the IF statement is fairly straight forward. When you have the release name at the start of the path, remove it and serve the request. Here when the requested file is not found we are serving up the root (index.html) required for routing within the UI. But what if the request comes in with an old release number? In that case we compose a new URL replacing the old version with the new one and then redirect the browser to it.

func RedirectHTTPS(w http.ResponseWriter, r *http.Request, redirect string) {
    url := fmt.Sprintf("%s://%s:%s%s",
        os.Getenv("ZWSD_PROTOCOL"),
        strings.Split(os.Getenv("ZWSD_DOMAINS"), ",")[0],
        os.Getenv("ZWSD_ORIGIN_PORT"),
        redirect)
    http.Redirect(w, r, url, http.StatusMovedPermanently)
}

It is important to note that we need the full browser's view of the URL beginning with the protocol (HTTP or HTTPS) and endpoint it is connecting to. This is the same server name that terminates an HTTPS connection which might be a proxy or load-balancer. Then we use the built-in "http" library to form a redirect response. This gets the new version into the browser's URL.

The last bit of work in the Go server was to return the version string on most every API request. We had already decided to encapsulate every response so adding the version involved adding a new tag to the top level:

{
    "data": [ array of data returned from the API ],
    "error": {
        "code": 200,
        "message": ""
    },
    "op": "read",
    "softwareRelease": "20200506071506"
}

Well, that's it! It was a long journey for us, but since making this change, we haven't been bitten by the browser cache again. And, as further proof that it's been working well, we've been delighted by how many more of our customers have started commenting on the great new what-ch-ma-call-it on my thing-a-ma-gig features we've been releasing 😀 We only wish we had done this sooner.

If you want to see it in action -- take our product for a free test run by visiting www.zebrium.com.

Posted with permission of the authors:
Alan Jones, Roy Selig and Ko Wang @ Zebrium