How Serverless, Puppeteer, FFMPEG, and AWS Lambda can help you get the GIFs from Twitter

Running Puppeteer on Amazon Web Services' (AWS) serverless computing platform, Lambda, is it a bit of a challenge. Puppeteer and it's bundled version of Chromium easily exceeds the lambda size limitation. There are a few options you opt for instead:

Run or maintain your own farm of Google Chrome / Chromium instances
Use a Google Chrome / Chromium-as-a-Service offering like Browserless
Use an NPM module, npm i @serverless-chrome/lambda to handle this for you

I'm going to be walking you through how I built "Get Me the GIF" (referred to as GMTG hereafter) with Puppeteer, Serverless, Chromium, and Nodejs.

The Idea

The idea behind GMTG is an easy one. When Twitter user's tweet GIFs out, Twitter will show these GIFs as videos instead of GIFs.

I'm somewhat of a GIF collector. You never know when a GIF is more useful in communicating an idea than words are. When I see GIFs on Twitter that are funny, I like to save those for later. Twitter for the web and Android make this impossible. One day, I had more free time than usual and decided to open up Chrome DevTools and explore the code behind how a tweet is displayed in the browser. Long story short, it's relatively easy to capture and download these GIFs (videos). I wrote a little extra code to use FFMPEG to convert the video I capture from Twitter into a GIF I can add to my collection.

I'm not going to highlight all the pieces I used, but just a few pieces of code I thought were useful in helping me reach my end goal, getting all of those GIFs.

The Flow

You give GMTG a URL to a tweet containing the GIF, like:

https://twitter.com/EvanHalley/status/1130891914675445760

It should give you a GIF.

The overall flow:

Open the Tweet
Intercept the network request that rendered the MP4 preview frame
Parse out identifier of the MP4 preview frame
Build the URL to access the MP4 and download it
Use FFMPEG to convert the MP4 to a GIF
Make the GIF available to the user

Getting Chrome to Run in AWS Lambda

Note: For this project, I am using the Serverless Framework to debug and deploy my serverless app to AWS Lambda. I'm not going to dive into Serverless, but checkout this great Serverless tutorial.

As mentioned earlier, getting a Puppeteer + Chrome based Nodejs app running in AWS Lambda is difficult because of the deployment package size limits. In order to get around this limitation, I used a NPM module, serverless-chrome.

Without jumping to far into the details, serverless-chrome handles everything needed to get Chrome up and running in a serverless environment and manages to get around the deployment package size limitations.

The aim of this project is to provide the scaffolding for using Headless Chrome during a serverless function invocation. Serverless Chrome takes care of building and bundling the Chrome binaries and making sure Chrome is running when your serverless function executes. In addition, this project also provides a few example services for common patterns (e.g. taking a screenshot of a page, printing to PDF, some scraping, etc.)

Once you have a Nodejs, Puppeteer, and Serverless project bootstrapped, you can easily add serverless-chrome:

npm install --save @serverless-chrome/lambda

In your source code, connect to a Chrome instance running in a serverless environment:

const launchChrome = require("@serverless-chrome/lambda");

async function getChrome() {
    let chrome = await launchChrome();

    let response = await request
        .get(`${chrome.url}/json/version`)
        .set("Content-Type", "application/json");

    console.log(JSON.stringify(response.body));
    let endpoint = response.body.webSocketDebuggerUrl;

    return {
        endpoint,
        instance: chrome
    };
}

The code snippet above, calls launchChrome() to start a Chrome process. Once it's launched, we can query the Chrome instance to find the URL to the Chrome DevTools Protocol (CDP) socket. Puppeteer uses this URL to connect to Chrome.

Making a GET request to this URL + /json/version returns:

{
   "Browser": "HeadlessChrome/78.0.3904.97",
   "Protocol-Version": "1.3",
   "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/78.0.3904.97 Safari/537.36",
   "V8-Version": "7.8.279.23",
   "WebKit-Version": "537.36 (@021b9028c246d820be17a10e5b393ee90f41375e)",
   "webSocketDebuggerUrl": "ws://127.0.0.1:9222/devtools/browser/0fbe3418-968a-4d57-9b53-0cf20d590eec"
}

We are after webSocketDebuggerUrl.

Once we have a URL to the CDP socket, connecting with Puppeteer is straighforward:

const puppeteer = require("puppeteer");
...
let cdpSocketUrl = 'ws://127.0.0.1:9222/devtools/browser/0fbe3418-968a-4d57-9b53-0cf20d590eec';
let browser = await puppeteer.connect({
    browserWSEndpoint: cdpSocketUrl
});

There is a caveat to know about when using serverless-chrome. It has not been updated in over a year, which means the latest prebuilt version of Chromium (69.0.3497.81) it uses is over a year old. This means it is pretty much only guaranteed to work with older versions Puppeteer.

Intercepting Requests with Puppeteer

Puppeteer has a handy API for intercepting ALL network requests the browser makes when loading a web page. You can intercept these requests and either continue or abort them. There are some really useful cases where this level of control is desired, such as capturing a webpage screenshot, but not processing any of the image or javascript. In my case, I just wanted to identify the MP4 thumbnail network request.

I discovered that the URL to the MP4 thumbnail looks like:

https://pbs.twimg.com/tweet_video_thumb/1234567890.jpg

The link to the MP4 looks like:

https://video.twimg.com/tweet_video/1234567890.mp4

Using Puppeteer I am able to write request interception code that looks for this URL.

const VIDEO_THUMBNAIL_PREFIX = 'https://pbs.twimg.com/tweet_video_thumb/';
let videoUrl = null;
page = await browser.newPage();
await page.setRequestInterception(true);

page.on('request', request => {

    if (request.url().startsWith(VIDEO_THUMBNAIL_PREFIX) && request.url().endsWith('.jpg')) {
        let thumbnailUrl = request.url();
        let assetId = thumbnailUrl.replace(VIDEO_THUMBNAIL_PREFIX, '')
            .replace('.jpg', '');
        videoUrl = VIDEO_URL_PREFIX + assetId + '.mp4';
    }
    request.continue();
});
await page.goto(tweetUrl);

Once I had the URL to the video thumbnail, I'm easily able to build a URL to the video so I can download it later.

Converting the Video

FFMPEG is one of the most popular command line utilities for transcoding a wide area of video, audio, and still images. It's written in C. However, like many things nowadays, you can instrument it with JavaScript. I discovered a GitHub Gist that was tackling a similar problem, converting a video into a GIF.

Using traditional command line FFMPEG, you could execute the operation with:

ffmpeg -i input_video.mp4 output.gif

Using a Nodejs library, fluent-ffmpeg, the same operation looks like:

const ffmpegPath = require('@ffmpeg-installer/ffmpeg').path;
const ffmpeg = require('fluent-ffmpeg');
ffmpeg.setFfmpegPath(ffmpegPath);

ffmpeg('input_video.gif')
    .on('end', () => {
        // done, do something with output.gif
    })
    .on('error', err => {
        // oh noe error!
    })
    .save('output.gif');

The use of fluent-ffmpeg requires the use of another Nodejs libray, @ffmpeg-installer/ffmpeg. It installs a version of ffmpeg compatible with the underlying platform (Linux, Mac, or Windows). You then tell fluent-ffmpeg where it has been installed and it takes over from there.

The End

I've deployed GMTG to AWS Lambda. You can make HTTP GET calls to it. The value of the URL parameter should be a URL to a Tweet containing a GIF (short movies probably work as well).

https://1ehn2dwcfk.execute-api.us-east-1.amazonaws.com/dev/?url=

In this example, we will extract the GIF out of this Tweet

Using the following GMTG API call:

https://1ehn2dwcfk.execute-api.us-east-1.amazonaws.com/dev/?url=https://twitter.com/ThePracticalDev/status/1194435785082187778

The GIF: