Making a Dynamic Renderer with Golang from Scratch

Understanding Dynamic Rendering: Why It Matters for Your Website's SEO

When it comes to building a website, the three key components are HTML, CSS, and Javascript. In recent years, client-side rendering (CSR) has become increasingly popular. With this approach, the browser downloads an empty HTML shell and uses Javascript to generate the content on the client side.

[Image credits: web.dev ]

However, while CSR may work well for users, it creates a significant problem for search engines and other bots. Bots rely on the generated HTML content to gather data and index your website. But when a bot visits a client-side rendered page, there is no data available because the Javascript hasn't been executed yet. As a result, the bot only sees an empty website, which can severely impact your website's SEO.

That's where dynamic rendering comes in. Dynamic rendering is the process of rendering a fully formed HTML page on the server side and sending it to bots, while still using client-side rendering for users. This approach allows bots to access the fully rendered HTML content and gather the data they need to index your website, without affecting the user experience.

In this article, we'll explore dynamic rendering in more detail and show you how to build a dynamic renderer using Golang, a powerful and efficient programming language, from scratch. With this knowledge, you'll be able to create high-performance web applications that not only deliver a great user experience but also rank well in search engines.

But do I even need Dynamic Rendering?

Well, that depends.

❌ You don't need dynamic rendering when -

If you are working with meta frameworks like NextJS, NuxtJS, Remix, etc. Because these are server-side rendering applications.
If your codebase is small and you can migrate it to server-side rendering applications

✅ You need dynamic rendering when

When your codebase is huge and you don't have the bandwidth to migrate it to server-side rendering applications. Also, you want to improve your website as soon as possible.

But keep in mind,

Dynamic rendering is a workaround and not a long-term fix for problems with javascript.

Implementing Dynamic Rendering

Let's first discuss the high-level overview, before jumping into the code.

Since we need to render the HTML before sending it to the bot, we need a server.
We will define a middleware, it intercepts the request made to the web app.
We will use the user agent to check if the request was made by a bot or a user.
If the request was made by a user, we send the client-side rendered app.
But if the request was made by a bot, we generate the HTML content (with something like puppeteer) and then send the rendered HTML to the bot

Now that we are done with the high-level overview, let's jump into the implementation

Setting up server

For setting up the server, I will be using the Gin framework.



func main() {
    r := gin.Default()
    r.Use(our_middleware)

    r.Static("/", "./frontend/dist")

    if err := r.Run(":3000"); err != nil {
        log.Fatal(err)
    }
}

This starts our server at port 3000. Our CSR frontend files are present in the folder frontend/dist. Now when someone sends a request for a page, we serve HTML, CSS, and JS from this folder.

Now we need middleware to intercept the traffic. So let's implement it.

Adding a middleware



func dynamicRenderer() gin.HandlerFunc {
    return func(c *gin.Context) {
        // Check if request is from a bot
        isBot := checkforBot()
        if isBot {
          // render page and send the rendered page
            return
        }

        // If not a bot, continue to serve as usual
        c.Next()
    }
}

func main() {
    r := gin.Default()
    r.Use(dynamicRenderer())

    r.Static("/", "./frontend/dist")

    if err := r.Run(":3000"); err != nil {
        log.Fatal(err)
    }
}

Now that the middleware is set up, let's write the code for rendering the HTML file. But before that, we need to run our front end on some other port. Because the puppeteer scrapes data from the website. So let's write code to start the frontend

Serving the actual front end from another port



func main() {
    // code

    cmd := exec.Command("command","to","serve","your","frontend")
    cmd.Dir = "./frontend"
    cmd.Stderr = os.Stderr

    wait_for_files_to_be_served()

    if err := r.Run(":3000"); err != nil {
        log.Fatal(err)
    }
}

Writing the code for the renderer



func dynamicRenderer() gin.HandlerFunc {
    return func(c *gin.Context) {
        isBot := checkForBot()

        if isBot {
            // Connect to Puppeteer
            ctx, cancel := chromedp.NewContext(context.Background())
          // We cancel the connection once the response is sent
            defer cancel()

            // Navigate to the page and wait for it to load
            url := "http://localhost:" + reactPort + c.Request.URL.Path
            var html string

          err := chromedp.Run(ctx,
                chromedp.Navigate(url),
                chromedp.InnerHTML("html", &html, chromedp.NodeVisible, chromedp.ByQuery),
            )
            if err != nil {
                log.Println(err)
                c.AbortWithStatus(http.StatusInternalServerError)
                return
            }

            // Send back the rendered HTML
            c.Data(http.StatusOK, "text/html; charset=utf-8", []byte(html))
          // We are done serving
            c.AbortWithStatus(http.StatusOK)
        }

        // If not a bot, continue to serve the React app as usual
        c.Next()
    }
}

Here's a short explanation of what happening:-

Once we know it's a bot, we connect to the puppeteer.
We then navigate to the URL, using chromedp.Navigate(url) .
We wait for all the children of the HTML tag to load by using chromedp.NodeVisible
Once the javascript has generated all the HTML content we store it in html variable and send it to the bot.

Well, that's the entire implementation !! (sort of). I have left some boring parts out, but if you want you can check this repository.

Now, let's look at the result

Our webpage looks like this

When the user is requesting for a file, we get

But when a bot requests a file, we get

Notice that the HTML is all rendered.

NOTE: When we send a rendered page there is not JS. So it won't function. But for a bot that is not an issuse because it does not use JS in any way

Well that's all. Thanks for reading. If you have any doubt, you can post it it comment. I'll try my best to clear your doubts

Blog