Update: join #RunwayML on Discord!

Despite maybe some minor improvements here and there, machine learning has been and in many ways still is primarily for experienced developers. Not anymore. Runway brings the power of artificial intelligence to your creative projects with an intuitive and simple visual interface. Start exploring new ways of creating today.

A few words

Some of you know that I keep introductions to an absolute minimum and like to dive right into the material but this article is about a product (more specifically a commercial product) and because of that I want to say the following.

As much as I would love to be involved some day, I am not not affiliated, associated or in any way officially connected with Runway. I do not get paid to write this. The only reason for writing this article is my own professional opinion about the quality of this product, my experience with it so far and my personal interest in machine learning in general. I highly recommend it to anyone (developer or not) who shares this interest.

You have probably seen some of the amazing things we can do nowadays with the help of machine learning. If you are a creative person like a musician or designer then you have undoubtedly imagined what else is possible and quite likely have a few ideas of your own. With Runway you can turn these ideas into a working prototype today with a few mouse clicks.

Runway is focused primarily on accessibility to people in creative professions. If you are a developer then you will find that Runway does not fall short.

Installation

Installing Runway is pretty simple. It doesn't require any development environment or stuff. All you need to do is navigate to https://runwayml.com/ and download the installer by clicking the button on the top right corner of the page. The application is not that large, a little over 200MB. Once downloaded, just click 'next' a couple of times. I think it is currently only for Windows but it is a Chromium application so I don't think it will stay that way. Besides, a browser version has been announced.

Getting Started

Once the Runway application is installed, getting started is quite easy. Start the program and the splash screen will look something like the following.

(in my opinion this splash screen is redundant and might as well be removed because the interface contains these menus too)

I will briefly describe what a model is but don't worry, you do not have to understand this in great depth if you are using Runway as a creator.

A machine learning model is a file that has been trained to recognize certain types of patterns. You train a model over a set of data, providing it an algorithm that it can use to reason over and learn from those data. The training data you feed to the machine learning model needs to be represented as numbers in order.

Once you have trained the model, you can use it to reason over data that it hasn't seen before, and make predictions about it. For example, let's say you want to build an application that can recognize a user's emotions based on their facial expressions. You can train a model by providing it with images of faces that are each tagged with a certain emotion, and then you can use that model in an application that can recognize any user's emotion.

That's enough of that. Thanks to Runway, you don't have to deal with the complexity of these systems. Just select the model you need and run it.

Runway offers an abundance of ready to use popular machine learning models such as StyleGAN(2) for image synthesis, GPT-2 for text generation, pose detection in images and videos, style transfer of two images and a variety of other fun and interesting things to play with. This includes StyleGAN models trained by the Runway community on various collections of images and a whole bunch of other cool stuff.

There are many different types of models and I can't possibly talk about all of them here so I will go through a few of these models that are popular and I use a lot myself.

GPT-2

A state of the art text synthesis model that can generate coherent paragraphs of text of any kind, autocomplete your sentences and even answer questions about a given text.

• Paper
• Code for the paper
• Talk to Transformer Demo website using GPT-2
• TextSynth Another online demo with more examples

GPT-2 can be fine-tuned on custom datasets. This means that if you feed it a large number of movie descriptions, it will generate an infinite amount of new unique movie descriptions. Fine-tune it on chat logs and it will generate more dialog. I did many fun and pointless experiments with GPT-2 (spiritual text generator, a Flash episode generator, ...) which are all available as gists or repositories on my GitHub.

At the time of writing this article, only 117M and 345M are available. I've been asking about this but they don't give any indication when the 774M and 1558M model will be available.

StyleGAN(2)

Generative Adversarial Networks, or GANs for short, are effective at generating large high-quality images.

My favorite model is StyleGAN.

The Style Generative Adversarial Network, or StyleGAN for short, is an extension to the GAN architecture that proposes large changes to the generator model, including the use of a mapping network to map points in latent space to an intermediate latent space, the use of the intermediate latent space to control style at each point in the generator model, and the introduction to noise as a source of variation at each point in the generator model.

The resulting model is capable not only of generating impressively photo-realistic high-quality photos of faces, but also offers control over the style of the generated image at different levels of detail through varying the style vectors and noise.

Read more about this

Here is an example of a latent space interpolation video by StyleGAN trained on images from xkcd. This was actually trained by me and is publicly available as a hosted model which I will talk about below.

The Runway application lets you create project workspaces to which you add the model(s) that you need. This of course lets you easily separate/group models for different projects. Although project workspaces are not all that special, the simplicity and intuitiveness of them does show how dedicated they are to making machine learning easy for everybody.

Training or fine-tuning machine models to fit your needs is as simple as selecting a model, uploading your dataset(s) and running it. As you might know, the training of machine learning models is an intensive task that requires a powerful GPU. Thanks to Runway you don't have to worry about the hardware requirements or session timeouts/limits because training happens on powerful machines in the cloud. You don't even need to keep the application running, once training has started you can safely power off your system and you will receive an email notification when it is complete.

Why not use Google Colab or Paperspace?

Personally I have used Google Colab for training a lot and even though I am grateful for the free service, I have always been keeping my eyes open for something better. Google Colab sessions have a time limit of approximately 12 hours and may occasionally timeout at random for no apparent reason. To save any type of progress in Colab, you need to connect to Google Drive or some other host to store your files after training and to import them when you wish to continue. Every time you use Google Colab, everything needs to be installed and configured again. You can't use it in production or rely on it for these reasons. Spinning up a server on Paperspace and doing the training there is not ideal either. It can take quite some time before you finally have the development environment all set up (expensive time) and to deploy anything for real world applications, the system must stay up and running 24/7 which is not affordable.

A number of extensions for third party applications are already available and more are being created almost every day. This includes extensions for Adobe Photoshop and Unity. Utilize style transfer models, turn a picture into an anime character or synthesize images inside Photoshop with the click of a button.

Screenshot of the Adobe Photoshop plugin

https://github.com/runwayml/RunwayML-for-Photoshop

Unity

https://github.com/runwayml/RunwayML-for-Unity

Although there is an abundance of models already available on Runway, every project is unique and it is not unlikely that your project might require some custom solution. Maybe you or a developer in your team created a model that your project needs. You can import your own machine learning models to Runway and this is fairly easy to do. I might write an article about this at some point.

• https://github.com/runwayml/model-sdk
• https://github.com/runwayml/model-template

Of course you can also choose to share your ported models with others, which is something I highly recommend.

Deploy trained models and use them in your web application using a simple intuitive no-bs API.

One major reason if not the biggest reason for my decision to invest time in Runway is the recent introduction of hosted models.

Look. I am developer. I know a thing or two about stuff. So far got things to run locally at home, hosted remotely on a GPU powered VPS or running temporary on Google Colab. Deployment however, has so far been virtually impossible simply because of how resource intensive (expensive) it is. Deploying a trained machine learning model into a running web application requires it to be available all the time. Not just be online 24 hours a day but also able to handle multiple requests at the same time for different users. Several thousands of dollars a month to host a machine like that for projects that are not yet in a development stage where such numbers are realistic is no option for me.

Runway solves this problem for me entirely.

Once a hosted model is queried, it starts up the model remotely if it hasn't started already. (this will take a minute or two) Once it started you can do inference in real-time. When a model has not been used for so many minutes, it will automatically shut itself down and go back to sleep. An active model (one that is constantly being used) will stay awake. Only initial startup causes a small delay (only once) in which it is probably a good idea to show a loading screen. Users that are subscribed to the creative plan can host unlimited amount of models. (!)
In my opinion the costs of hosted models are very reasonable. They kept it at an absolute minimum. Runway does not charge users for the time it takes to start a hosted model or to shut it down. Credits only go to actual usage and are about $0.005 per query or something.

A software development kit (SDK) is a collection of software development tools in one installable package. They ease creation of applications by having compiler, debugger and perhaps a software framework. They are normally specific to a hardware platform and operating system combination.
In this case, the Runway SDK refers to a collection of tools that simplify communication between your application and the Runway service/API.

The ability to not only utilize but also manage highly complicated machine learning models programmatically with such high level of abstraction that you can do it with a few lines of JavaScript code in the browser or your Node application brings a lot of potential.

Pipelines are a form of computer organization in which successive steps of an instruction sequence are executed in turn by a sequence of modules able to operate concurrently, so that another instruction can be begun before the previous one is finished.

You can use the output of a hosted model as the input of the next model, provided it is the right format, then repeat this as long as you need. In development, this we generally call pipelines.

For example:

• Model 1 takes some input vector and produces a base64 image.
• Model 2 takes image input and outputs an image too.
• Model 3 takes image input and outputs text.

The pipeline in this case will send the output of model1 to model2 and from model2 to model3 so that you end up with something that takes a vector as input and outputs text.
I might write an article in the future to provide a few examples on what this looks like in the simplest form of (JavaScript) code, using the Runway SDK.

It is very easy to get in touch with the Runway team. It is my experience that they are friendly and professional. If you think that you found a bug or experience any problems then you can count on it that they will look into it right away and fix the problem if there is one. They are open to feature suggestions (after all, every feature I suggested is now a reality) and they happily involve users in the process of beta testing new stuff.
However, personally I am not overwhelmed with enthusiasm. I do not find them very engaging or responsive on a social level. They don't think with you or participate in a brainstorm session. It is unlikely that they are the ones to initiate a dialog with you. They are friendly but they are not your friend. I have expressed my love and intention to get involved a number of times and suggested many ways I can be useful but received little response. There is absolutely nothing wrong with this. How you handle support as a business is a choice.
A great thing about the Runway team is that they are available on Slack for direct communication. On their Slack channel you can also connect with other members/users of the community, exchange ideas and share your projects. For those (interested in) using Runway I highly recommend introducing yourself there as well.

Runway is still in beta version so the community is relatively small. However it is a very active and engaged community which I think is a lot more important. People are sharing their trained models, their ported models, their creative works and their ideas on the Runway platform itself but also on places like Github and YouTube.
There are numerous courses, showcases and YouTube video's that show just how wonderful and creative people can be in the way they utilize machine learning thanks to Runway.

The creator plan of Runway costs about $10 a month that you can pay with your credit or debit card. Additionally you can purchase credits to train and run your models. Whether you are willing to make this investment is of course entirely your business. All I can say is that I spend this with love because it is a minimal investment for a product/service that fast forwards the development of my projects and my own creative expression by at least a few years.

Runway is constantly adding new features and it impossible to talk about them all here. Personally I am looking forward to the Web Version that has been announced. I have not had the pleasure of using it yet but I expect it to be pretty much the same as the desktop application, only running in the browser. This means that pretty soon you can even use your phone!

I can talk about Runway for hours and I am itching to dive in some code examples but that is outside the scope of this article so I'm going to stop here and conclude.

Runway is the Swiss army knife of machine learning. It lets you rapidly prototype your machine learning projects. Discover an ever-growing collection of features and extensions. It has an API and SDK. Since recently you can even deploy your applications in the cloud using Runway, making it also very interesting for developers like myself that want to use machine learning in their web applications.
As a developer you have certain requirements. As a creator too. Runway offers both. It is a commercial product but very affordable compared to any other current alternative out there. Because it is still in early stage of development, there are some things that I like and some things I really don't. I am looking forward to the future and I will dive into some code real soon.

Runway lives up to its name. It accelerates your creative process and helps you lift your ideas off the ground.