File storage microservice for your Express.js Application

In today's world of technology, microservices have become an essential part of web development. Microservices are small, independent applications that can communicate with each other to create larger and more complex but efficient systems. One common use case for microservices is file storage. In this blog post, we will discuss how to set up Total.js OpenFiles as a file storage microservice for an Express application in 3 steps:

first, we will learn about Total.js OpenFiles, why we need it and how to install it.
Secondly, we will see the first approach (the bad one) of how to connect Total.js OpenFiles as a microservice to an express application.
And finally, we will see the second approach (the good and recommended one) of how to connect it.

This blog post includes both code implementations and system design illustrations for a better understanding of some simple concepts of system microservices architectures. Make sure you read till the end to not miss one experience ahead in microservices development.

Total.js OpenFiles

OpenFiles is a lightweight app for storing files from 3rd party applications. The application is built entirely with the Total.js Framework and its functionality is simple: the Files are stored on HDD in the Total.js FileStorage that we talked about in my previous blog post (Read here). In addition, you can easily browse and filter stored files in the OpenFiles setup interface.

The most interesting thing about that application is its API. Yes, OpenFiles exposes an API that allows third-party applications to store files in it.

Installation and setup

Installing Total.js OpenFiles is easy. I have prepared a tutorial video about how to install it. Installing it on your machine is necessary for the rest of this tutorial. Don't forget to SUBSCRIBE to my channel.

Connect OpenFiles to Express.js (Method 1)

Here is the most interesting part of the tutorial. In the previous section, we followed the youtube video tutorial and we now have an awesome file storage microservice running on our machine or wherever. Now we can try the first approach (the bad one) to make an express.js application store files in that file storage. To have a clear idea of what we do, let us first illustrate the design.

System design

The express.js app implementation

First, we need to initiate a simple express.js application and install dependencies via npm.

# Create folder
$ mkdir expressapp && cd expressapp
# Init application
$ npm init -y
# Install dependencies
$ npm install express total4 multer ejs
# Create endpoint file
$ touch index.js
#create folder for views
$ mkdir views

Then, we create an express.js server app by putting the following in index.js:

require('total4');
const Express = require('express');
const multer = require('multer');
// URL of OpenFile instance + auth token (see video)
const Url = 'http://localhost:8000/upload/photos?token=yourtoken&hostname=1';

const app = Express();
const PORT = 3000;
const storage = multer.diskStorage({ 
    destination: function(req, file, cb) {
        return cb(null, 'temp')
    },
    filename: function(req, file, cb) {
        return cb(null, NOW.format('yyyyMMdd_HHmmss') + '_' + file.originalname)
    }
});

const upload = multer({ storage: storage });

app.set('view engine', 'ejs');
app.set('views', PATH.views());

app.use(Express.json());
app.get('/', function(req, res) {
    res.render('index', { info: '', url: '' });
});

app.post('/upload', upload.single('photo'), async function(req, res) {
   var data =  req.body;
   var file = req.file;
   var response;

   // Send file to The Total.js OpenFiles
   if (file) 
        response  = await RESTBuilder.POST(Url, { }).file('file', PATH.root(file.path)).promise();


    // save data
    data.id = UID();
    data.search = (data.name + ' ' + data.phone).toSearch();
    data.photo = response ? response.url : '';
    data.dtcreated = NOW;
    NOSQL('users').insert(data, true).callback(function(err, resp) {
        res.render('index', { info:response.name, url: response.url});
    });
});
app.listen(PORT);

Now we need to add an index.ejs view a file in the views folder:

<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=10" />
    <meta name="format-detection" content="telephone=no"/>
    <meta name="viewport" content="width=1024, user-scalable=yes" />
    <meta name="robots" content="all,follow" />
    <style type="text/css">
        body { padding: 50px; margin: 0; font:normal 12px Arial; color: gray }
        .paging { font-size: 11px; height: 20px }
        .paging a { float: left; margin-right: 5px; }
        .paging div { float: right; }
        .mb5 { margin-bottom: 5px; }
    </style>
</head>
<body>
<div style="background-color:#F0F0F0;padding:3px 10px">
    Uploaded: <b style="color:black"><%= info %></b>
    Image: <br>
    <% if (url) { %>
        <img src="<%= url %>" width="150" alt="Image preview">
    <% } %>
</div>
<br />
<form method="POST" enctype="multipart/form-data" action="/upload">
    <input type="text" name="name" placeholder="Name" /> <br><br>
    <input type="email" name="email" placeholder="Email" /><br><br>
    <input type="phone" name="phone" placeholder="Phone" /><br><br>
    <input type="file" name="photo" /><br><br><br>
    <button>SUBMIT</button>
</form>
</body>
</html>

Finally, we can start our Express server and make some tests:

$ node index.js

After some testing, you should see a simple UI like this:

The file is successfully uploaded and the user information is stored in the database too.

However, this approach for connecting a file storage microservice has problems:

Using an express server to proxy the file makes it slow because the file is processed twice, and DNS is resolved twice too.
Bringing multer in this project could be avoided to make the express application very lightweight, and also avoid storing files in a temporary folder.
Bringing total4 is a very bad idea, Total.js is an independent framework for web development. Bringing the entire Total.js framework into another framework (Express) is working fine but this is not a good practice.

This first approach is working fine, but it is not very good practice, which brings us to the second approach which is far better.

Connect OpenFiles to Express.js (Method 2)

The first method has some shortcomings, so we will reshape our system design to make better microservice architecture.

System design

This time we are sending the file from the web browser to our Total.js OpenFiles microservice. It will return the URL and we will be able to push the URL and the form data to our express application. Just like the following illustration:

The express.js app implementation

Now that we do not send files to OpenFiles via express application, the express app is simplified and better. The index.js file is simplified as follows:

require('total4');
const Express = require('express');

const app = Express();
const PORT = 3000;

app.set('view engine', 'ejs');
app.set('views', PATH.views());

app.use(Express.json());
app.get('/', function(req, res) {
    res.render('index', { info: 'info', url: '' });
});

app.post('/upload', async function(req, res) {
   var data =  req.body;
   console.log(data);
    // save data
    data.id = UID();
    data.search = (data.name + ' ' + data.phone).toSearch();
    data.dtcreated = NOW;
    NOSQL('users').insert(data, true).callback(function(err, resp) {
        res.json({ success: true, value: data.id });
    });
});


app.listen(PORT);

PS: we still have require('total4') just because of the NOSQL database that we are using, but if you would like to use the Mysql database as described in the system design, then total4 is completely useless. Then you have the following in the views/index.ejs file:

<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=10" />
    <meta name="format-detection" content="telephone=no"/>
    <meta name="viewport" content="width=1024, user-scalable=yes" />
    <meta name="robots" content="all,follow" />
    <script src="//cdn.componentator.com/spa.min@19.js"></script>
    <link rel="stylesheet" href="//cdn.componentator.com/spa.min@19.css" />
    <style type="text/css">
        body { padding: 50px; margin: 0; font:normal 12px Arial; color: gray }
        .paging { font-size: 11px; height: 20px }
        .paging a { float: left; margin-right: 5px; }
        .paging div { float: right; }
        .mb5 { margin-bottom: 5px; }
        .button { border: 0; margin: 15px 0 0; background-color: #F0F0F0; height: 34px; padding: 0 15px; color: #000; cursor: pointer; font-family: Arial; line-height: 34px; vertical-align: middle; outline: 0; font-size: 14px; text-decoration: none; transition: all 0.3s; float: left; width: 100%; }
        .button:disabled { background-color: #7c98d5; }
        .button i { width: 15px; text-align: center; margin-right: 5px; }
    </style>
</head>
<body>
<ui-component name="upload"></ui-component>
<br />


<div class="container">
    <ui-plugin name="Form" path="form" >
        <div class="grid-2">
            <ui-component name="input" path="?.name" config="required:1;">Name</ui-component>
            <ui-component name="input" path="?.email" config="required:1;type:email">Email</ui-component>
            <ui-component name="input" path="?.phone" config="required:1;type:phone">Phone</ui-component>
            <ui-component name="input" path="?.photo" config="required:1;type:url;disabled:1">Photo url</ui-component>
            <span></span>
            <ui-component name="preview" path="preview" config="width:300;height:200;url:[openfiles];icon:folder;preview:value => value.url;">Preview</ui-component>
        </div>
        <ui-component name="submit" path="?" config="url:POST /upload/">
            <button class="button" name="submit" disabled="disabled">SUBMIT FORM</button>
        </ui-component>
    </ui-plugin>
</div>
<script>
    var preview = {};
    var form = {};
    ENV('openfiles', 'http://localhost:8000/base64/photos?token=yourtoken&hostname=1');
    console.log(ENV());
    WATCH('preview', function(path, value) {
        SET('form.photo', value.url);
    });
</script>

</body>
</html>

This time we are importing a CDN at the head of the HTML document: it's the client-side library of Total.js. We try to leverage its powerful data-binding features and free UI components for uploading the image directly from the browser to the OpenFiles microservice. You should see the following result:

In summary, we have installed and learned about Total.js OpenFiles before connecting it to an Express.js application as a microservice, in two different approaches. The first approach is good but may become challenging with time, especially as the application is scaling. The second approach is not perfect but at least solves many problems that the first approach is more likely to accounter. I hope it has been helpful. Feel free to play with the code, connect it to any other backend language you want and if you find out some ideas of how to improve any of those system designs, then feel free to leave some comments. Like, share and subscribe.

Blog

File storage microservice for your Express.js Application

Louis Bertson

Total.js OpenFiles

Connect OpenFiles to Express.js (Method 1)

System design

The express.js app implementation

Connect OpenFiles to Express.js (Method 2)

System design

The express.js app implementation

Join Our Newsletter. No Spam, Only the good stuff.

Related