This blog post expects the reader to be somewhat familiar with the HLS streaming format, and AWS Elemental.

Intro

Ever wanted to take an existing live stream and put it somewhere? Now you can.
Introducing our first version of HLS-Pull-Push, a node library which creates service with a REST API that can generate fetchers that pull segments from a source HLS stream and then pushes them to an output destination, accompanied with HLS manifests referencing the segments in their new home.

The output destination can be a folder on your local machine, an S3 bucket, or even AWS Elemental MediaPackage. The limits of the output destinations are up to the implementers of the Output adapters (more on that later).

With this post, I would like to share with you an overview of how the library is built and how it works.

Pulling

In this lib, pulling content from a source HLS stream is done through the @eyevinn/hls-recorder library. You can find it here on npmjs 🙌

HLS-Recorder

In short, HLS-Recorder is an open-source library, written in TypeScript by Eyevinn, that will continuously fetch the multivariant and media playlists from an input HLS stream and parse them, extracting segment data from the newest segment additions in the playlists, and storing them internally as a JSON object.
HLS-Recorder can also playback a new HLS stream containing every stored (or recorded) segment, as it also has a restify server built-in. By default, HLS-Recorder will serve an event HLS stream, and will add an endlist tag once the recording session is stopped.

"An event playlist is specified by the EXT-X-PLAYLIST-TYPE tag with a value of EVENT. It doesn't initially have an EXT-X-ENDLIST tag, indicating that new media files will be added to the playlist as they become available." Reference

HLS-Recorder can accept various types of HLS streams as input. Namely, HLS streams that are LIVE, VOD, EVENT, with Audio Tracks, with Subtitle Tracks, using AES-128 encryption, or fragmented MP4.

But what's most interesting is that HLS-Recorder also uses node EventEmitter to emit an event triggered each time a new segment is added to the internal segment storage. On each trigger, the segment storage itself is sent. Allowing the user to do anything they want with the parsed segment objects. Below shows an example of how the emitted data could look like...

// Recorded Segments for Playlist Variant_1 
{
  "mediaSeq": 0,
  "segList": [
    {
      "index": 5,
      "duration": 11,
      "uri": "https://lab-live.cdn.eyevinn.technology/SHORT60SEC/video/39acb45308be9cf94f27a66b6377a324.ts",
      "cue": null,
      "key": null,
      "map": null
    },
    {
      "index": 6,
      "duration": 12,
      "uri": "https://lab-live.cdn.eyevinn.technology/SHORT60SEC/video/88154ae79efd8e0f8b8490d8ae177514.ts",
      "cue": null,
      "key": null,
      "map": null
    } ]
}

This emitted data is what HLS-Pull-Push uses as input for its Pushing operations. To learn more about the HLS-Recorder lib, check it out on github.

Pushing

As mentioned above, the data pushed is based on the emitted JSON data from the pulling operations. Once collected, the pushing operations can start. It will first generate and push a new multivariant playlist manifest to the output destination.
Afterward, it pushes all the newest segments received from the pulling operation. Then lastly, it generates and pushes new media playlist manifests based on the pushed segments from the previous step.

To support a variety of output destinations, the uploading part of the pushing operation is mainly done through adapters that implement an OutputPlugin interface.

export interface IOutputPlugin {
  createOutputDestination(opts: any);
  getPayloadSchema();
}

export type Logger = (logMessage: string) => void;

export interface IOutputPluginDest {
  logger: Logger;
  _fileUploader(opts: any): Promise<boolean>;
  uploadMediaPlaylist(opts: any): Promise<boolean>;
  uploadMediaSegment(opts: any): Promise<boolean>;
}

Currently, only one adapter is included in the library, but more adapters are to come, and you are naturally welcome to implement your own.
What will differ between all adapters are of course their file uploading methods, as different destinations may require different technologies and SDKs.
The adapters' main responsibilities are, of course, downloading and uploading the media files, and m3u8 files to their final destination, with customized file names.

Output Plugin: MediaPackage

The available output plugin in HLS-Pull-Push currently is for AWS Elemental MediaPackage. The plugin will fetch media files and then put them to a given MediaPackage channel ingest URL using the network protocol WebDAV.

Working with WebDAV you will need a set of credentials. Credentials that AWS Elemental MediaPackage will generate automatically when you create a new MediaPackage channel.

Suffice to say, in order for you to use this plugin you will need access to the WebDAV credentials (username & password) from MediaPackage.

Here's a fun fact about pushing to MediaPackage, file naming is key. We figure out that files must be uploaded under the name channel* (e.g. channel.m3u8, channel_1.m3u8, channel_1_50.ts) otherwise MediaPackage will reject it like yesterday's french fries.

Demo

You yourself can try out HLS-Pull-Push. Only requirements are nodejs 12+ and access to your AWS Elemental MediaPackage Channel and Channel credentials.

Step 1: Install

npm install @eyevinn/hls-pull-push

Step 2: Script Set-Up

Paste the following code block in a .js file named pullpush

const { HLSPullPush, MediaPackageOutput } = require("@eyevinn/hls-pull-push");

const pullPushService = new HLSPullPush();
pullPushService.registerPlugin("mediapackage", new MediaPackageOutput());
// Start the Service
pullPushService.listen(8080);

Step 3: Run the Script

Run node pullpush.js.

Step 4: Make a POST

Lastly, make a POST request to the api/v1/fetcher endpoint. For convienience you can use the swagger api at: http://localhost:8080/api/docs


Use this JSON as your POST payload. Just make sure to insert your own MediaPackage details.

{
  "name": "Demo_Push_To_MediaPackage",
  "url": "https://cph-p2p-msl.akamaized.net/hls/live/2000341/test/master.m3u8",
  "output": "mediapackage",
  "payload": {
    "ingestUrls": [
      {
        "url": <insert MediaPackage channel input URL here>,
        "username": <insert MediaPackage channel username here>,
        "password": <insert  MediaPackage channel password here>
      }
    ]
  }
}

The demo source stream used in my example is a 24/7 Live Stream of the short film "Tears of Steel" looped over. I've seen this film many times now. You could almost say it's my favorite... almost say.

Verify by making a GET request, and see if you get any session data in the response. If all went well, then you should be able to see new content on all of your MediaPackage channel output URLs.

Well, that's all there is to it really. Not too complicated, right? Oh yeah, if you need to stop the fetcher just make a DELETE request. Thank you for reading!

Current Limitations & Future Work

As mentioned in the beginning, this is only a first version of HLS-Pull-Push. More work will be done to the library to address the limitations it has.

Output: MediaPackage

When pushing to AWS Elemental MediaPackage one must be aware that, for playback, it works best with segments of similar transcoding.

Furthermore, when it comes to DASH repackaging, there are some extra conditions for a source HLS stream to work with Ad breaks in it. AWS Elemental MediaPackage requires SCTE-35 messaging in the Ad filled source HLS stream to be able to properly repackage the input stream into DASH.

We are planning on extending HLS-Pull-Push to add SCTE-35 messaging into the input stream based on CUE tags found in the source HLS stream. Thus taking this kind of restriction out of the mind of the user.

For HLS repackaging with Ads, all you need to do is to enable Ad markers and set it to 'Passthrough' in the MediaPackage Endpoints settings.

Output: S3 Bucket

We are also planning to implement and add a plugin for pushing to an AWS S3 bucket. Stay tuned.

About Eyevinn Technology

Eyevinn Technology is an independent consultant firm specialized in video and streaming. Independent in a way that we are not commercially tied to any platform or technology vendor.

At Eyevinn, every software developer consultant has a dedicated budget reserved for open source development and contribution to the open source community. This give us room for innovation, team building and personal competence development. And also gives us as a company a way to contribute back to the open source community.

Want to know more about Eyevinn and how it is to work here. Contact us at work@eyevinn.se!