TL;DR;

Our AI specialist at ScaleDynamics built Tensorflow.js snippets ready to use, so you can get on directly with testing and building TFJS right from the browser.

We started with the most used cases:

• Using video feed: Face detection
• Using microphone feed: Sound spectrogram ; Sound speedup
• Doing inference with Node.js
• Text classification: Toxicity

He also added explanations on the Face Detection and Sound Spectrogram snippets to help you understand them easier

All snippets are available on our free Playground where you can code frontend & backend directly from the browser.

Hope you guys like it!
MadeWithTFJS ❤

Table of contents

• AI in the browser
• Tensorflow.js seems easy, but not that easy!
• A simple way to use Tensoflow.js right in the browser
• TensorFlow's snippets by use cases
• Play and learn more effectively on the Playground

AI in the browser

Artificial intelligence and machine learning have been gaining interest and increasing adoption in many applications over the last 6 years. As you might already know, TensorFlow.js has shown an incredible amount of contribution to the JavaScript community, bringing us closer to AI and machine learning. It helps JavaScript developers to build machine learning models using only JavaScript and use it directly in the browser or in Node.js.

In this article, we would like to introduce a new way to learn and play with TensorFlow.js that promisingly provides you with the easiest way to code Tensorflow.js

In case you want to have an in-depth introduction to TensorFlow.js and its application in machine learning, you can check out Dominique’s Introduction to TensorFlow.js, the AI stack for JavaScript

Dominique d’Inverno is an AI and algorithm development engineer at the ScaleDynamics starship 😎. With his 20 years of experience in embedded electronics design, mobile computing systems architecture and mathematical modeling, he built the TFJS snippets that we are going to present today. 😉

So with no further ado, let’s get started!

TensorFlow.js seems easy, but not that easy!

Since its first version in 2018, TensorFlow.js’ team has been working hard in order to align itself with Keras (Python). So far, you can directly use or retrain its +1,000 pre-trained models or convert a Python TS model to build your own one in JavaScript, in the browser, on the client-side. Basically, you can easily perform complex tasks like visual recognition, generating music or detecting human poses with just a few lines of JavaScript.

However, inference on big data sets in browsers is not really efficient in terms of performance due to model and data loading time as well as computing capabilities.

That’s where Node.js saves the day and increases performance by deploying on a high-performance GPU/CPU engine and in the network under the hood of the dataset.

Source: Opportunities and Challenges for TensorFlow.js and beyond - by Jason Mayes

As you can see in the illustration from TensorFlow.js, to make it work smoothly, you will have to take care of different API layers between your client-side and backend Node.js such as the Core or Ops API that allows you to do linear algebra, the Layers API which is the high-level one that helps you do machine learning easier.

Here comes the hard part, you will face complexity when trying to address distributed processing for the next performance step.

HTTPs and JSON management are quite complicated and redundant for developers, even an experienced ones. Even a single misplaced comma or colon can cause a JSON file to go wrong, and not work. You have to be careful to validate any data you are attempting to use (although computer-generated JSON is less likely to include errors, as long as the generator program is working correctly).

Now, imagine how efficient you can be if you can automate this process?

A simple way to use Tensoflow.js right in the browser

Acknowledging the problem, we have been developing a unique compiler technology used in the ScaleDynamics Platform. It enables easy process distribution with very little development effort, and you can see it in action on our Javascript Full Stack Playground.

To ease coding, instead of creating HTTP endpoints and use HTTP calls to execute a remote inference, you just need to build a client for this inference function, deploy and import it in the main application (via import statement in index.js or via script tag in HTML) and then call it like any JavaScript function.

In action: Tensorflow.js face detection

Take a look at this snippet on our Playground to see how easy it is and to see the demo in action: TensorFlow.js face detection or you can check out the code snippet below:
In the beginning, the blazeface model is loaded in the back-end context to reduce further latencies :
Frontend:

await backend.loadBlazeface();

Backend:

const tf = require('@tensorflow/tfjs'); require('@tensorflow/tfjs-node');
const blazeface = require('@tensorflow-models/blazeface');
let model;
...
exports.loadBlazeface = async function () {
   model = await blazeface.load();
}

Images are captured in a front-end canvas offCanvas (this name because this canvas is not displayed) via its associated context offContext and converted to a blob :
Frontend :

offContext.drawImage(video, 0, 0, w, h);
offCanvas.toBlob(process, 'image/jpeg', 0.9);

Once the conversion is finished, blazeface detection model is invoked within process callBack :

const process = async (blob) => {
const buff = await blob.arrayBuffer();
if (hideProc.value == 'browser'){
// perform faces detection (blazeface inference) on server
const faces = await backend.detect(buff);
// perform hiding/blurring on browser
hideAndDraw(faces);
...

On the back-end server, the detection function simply decodes jpeg, infers blazeface, performs cosmetic size adjustments on detected boxes, and sends back the detection result.

exports.detect = async function (imageBuf) {
//imageBuf : ArrayBuffer of jpeg base64 string
const jpjs = require('jpeg-js');
// fetch & decode image
let image = jpjs.decode(imageBuf);
// detect faces
const faces = await model.estimateFaces(image, false, false, false);
...
return faces;
}

In action: Tensorflow.js sound spectrogram

In a way similar to the face detection snippet, sound chucks and computed spectrum amplitudes are passed as ArrayBuffer arguments.
Audio is captured using a MediaRecorder element stopped and restarted at a TIMEFRAME periodicity.
Frontend:

var recorder = new MediaRecorder(inStream);
let blob;
recorder.start();

recorder.onstart = function (e) {
analyser.fftSize = parseInt(fftPoints.value);
sleep(TIMEFRAME).then (() => recorder.stop());
};

recorder.ondataavailable = function (e) {
blob = e.data;
};

recorder.onstop = async function(e) {
if (processOn.value == 'browser') {
// process spectral analysis on browser
const freqU8 = new Uint8Array(analyser.frequencyBinCount);
analyser.getByteFrequencyData(freqU8);
drawSpect(freqU8);
} else {
// Perform backend-spectral analysis
const soundBuf = await blob.arrayBuffer();
const soundDec = await audioCtx.decodeAudioData(soundBuf);
const chan0 = soundDec.getChannelData(0);
const freqBuf = await backend.soundSpect(chan0.buffer, parseInt(fftPoints.value));
const freqF32 = new Float32Array(freqBuf);
// draw spectrogram result
drawSpect(freqF32);
}
// restart recorder after process and display
chunks = [];
if (! finished) recorder.start();  else console.log('analysis terminated');
};

The back-end function is quite straight forward: a spectral analysis using Tensorflow tf.signal.stft utility, and log scaling.

Backend:

const tf = require('@tensorflow/tfjs-node');

exports.soundSpect = async function(inBuf, fftSize) {
const chan0 = new Float32Array((new Uint8Array(inBuf)).buffer);
const signal = tf.tensor1d(chan0);
const res = tf.signal.stft(signal, fftSize, Math.floor(fftSize*0.8), fftSize)
                 .abs() ;
// add 30 db gain and normalize to max Uint8 value to match
// browser analyzer defaults
const freqs = res.mean(axis=0).mul(1000).log1p().mul(255*Math.log10(Math.E)/Math.log10(fftSize)).dataSync();
tf.dispose([signal, res]);
return freqs.buffer;
}

With these examples, you can see the simplicity of coding with TensorFlow.js on our Playground.

TensorFlow.js provides an easy approach to machine learning and artificial intelligence, we help you do it easier by eliminating the painful part of coding in Full Stack! 😉

TensorFlow's snippets by use cases

You can easily search with the keyword ‘TFJS’ or ‘MadewithTFJS’ on the search bar, there will be a lot of cool stuff for you to explore.

Video feed

Tensorflow.js face detection (shown above)
This faces detection snippet was already shown above but in case you missed it, it streams video from a webcam live stream on the backend, and applies a black or blurred overlay on detected faces regions, either on frontend or backend. Images are passed (in or out) to the back-end as ArrayBuffer function arguments.

Sound feed

Tensorflow.js sound spectrogram (shown above)
This snippet streams audio from the device microphone and displays a live spectrogram, whose frequencies are computed on the frontend side with an audio API analyzer or on the backend side using tensorflow.js utility.

Tensorflow.js sound speedup
This snippet performs sound/speech capture, increases sound speed on a server, using simply fixed time frames cross-fading algorithm, then plays it back on the device.
The code structure is quite similar to spectrogram snipped.

Inference

Tensorflow.js interference
This snippet shows how to use Tensorflow.js on the Node.js back-end side to apply furthermore on AI, through buildup and training of a simple sequential neural network.

Text classification

Tensorflow.js toxicity detection
Another example for applying Tensorflow.js on your AI projects, this time, the author infers a pre-trained TensorFlow model for toxic words detection on a text box, showing how to integrate that as a feature in your projects.

Play and learn more effectively on the Playground

The JavaScript Full Stack Playground allows you to code both frontend and backend Node.js in the same place with integrated live-reload. By automatizing HTTPs/JSON management, it also offers a unique HTTP-less experience which makes learning TensorFlow.js easier than ever.

Having access to a diverse snippets library for TensorFlow.js, React, Angular, Vue.js, Bootstrap, MongoDB… and a place to exchange ideas with the growing community, it will probably give you an extra boost to become Full Stack faster than you think 😉

The JavaScript Full Stack Playground is FREE for the developer community, you can easily create an account, get on it and start playing and testing your project with TensorFlow.js, or check what others are doing!

Get your FREE developer plan in this link
To keep yourself updated, feel free to follow us on Twitter, Facebook, Instagram. I guarantee that it’ll be fun, and you’ll never miss a single new snippet on the Playground 😎✨