2020 AI & ML Opensource

Cutting edge open-source frameworks, tools, libraries, and models for research exploration to large-scale production deployment.

Frameworks & Tools

1) PyTorch

Github: https://github.com/pytorch/pytorch

PyTorch is an open-source deep learning framework built to be flexible and modular for research, with the stability and support needed for production deployment. It enables fast, flexible experimentation through a tape-based autograd system designed for immediate and python-like execution. With the release of PyTorch 1.0, the framework will also offer graph-based execution, a hybrid front-end allowing seamless switching between modes, distributed training, as well as efficient and performant mobile deployment.

Dynamic neural networks

While static graphs are great for production deployment, the research process involved in developing the next great algorithm is truly dynamic. PyTorch uses a technique called reverse-mode auto-differentiation, which allows developers to modify network behaviour arbitrarily with zero lag or overhead, speeding up research iterations.

The best of both worlds

Bringing together elements of flexibility, stability, and scalability, the next release of PyTorch will include a unique hybrid front end. This means AI / ML researchers and developers no longer need to make compromises when deciding which tools to use. With PyTorch’s hybrid front end, developers can seamlessly switch between imperative, define-by-run execution and graph mode, boosting productivity and bridging the gap between research and production.

Tape based autograd

Get Started

1. Install PyTorch. Multiple installation options are supported, including from source, pip, conda, and pre-built cloud services like AWS.

2. Review documentation and tutorials to familiarize yourself with PyTorch’s tensor library and neural networks.

3. Check out below tools, libraries, pre-trained models, and datasets to support your development needs.

4. Build, train, and evaluate your neural network. Here’s an example of code used to define a simple network

   import torch
   from torch.autograd import Variable
   import torch.nn as nn
   import torch.nn.functional as F

   class Net(nn.Module):

   def __init__(self):
       super(Net, self).__init__()
       # 1 input image channel, 6 output channels, 5x5 square convolution
       # kernel
       self.conv1 = nn.Conv2d(1, 6, 5)
       self.conv2 = nn.Conv2d(6, 16, 5)
       # an affine operation: y = Wx + b
       self.fc1 = nn.Linear(16 * 5 * 5, 120)
       self.fc2 = nn.Linear(120, 84)
       self.fc3 = nn.Linear(84, 10)
   
   def forward(self, x):
       # Max pooling over a (2, 2) window
       x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
       # If the size is a square you can only specify a single number
       x = F.max_pool2d(F.relu(self.conv2(x)), 2)
       x = x.view(-1, self.num_flat_features(x))
       x = F.relu(self.fc1(x))
       x = F.relu(self.fc2(x))
       x = self.fc3(x)
       return x
   
   def num_flat_features(self, x):
       size = x.size()[1:]  # all dimensions except the batch dimension
       num_features = 1
       for s in size:
           num_features *= s
       return num_features
   

   net = Net()
   print(net)

2) ONNX

Github: https://github.com/onnx/onnx

ONNX is an open format for representing deep learning models, allowing AI developers to easily move models between state-of-the-art tools and choose the best combination. ONNX accelerates the process from research to production by enabling interoperability across popular tools including PyTorch, Caffe2, Microsoft Cognitive Toolkit, Apache MXNet, and more.

Framework interoperability

ONNX enables models to be trained in one framework, and then exported and deployed into other frameworks for inference. ONNX models are currently supported in frameworks such as PyTorch, Caffe2, Microsoft Cognitive Toolkit, Apache MXNet and Chainer with additional support for Core ML, TensorFlow, Qualcomm SNPE, Nvidia’s TensorRT and Intel’s nGraph.

Hardware optimizations

Any tool that exports ONNX models can benefit from ONNX-compatible runtimes and libraries designed to maximize performance. ONNX currently supports Qualcomm SNPE, AMD, ARM, Intel and other hardware partners.

Get Started

1. Install ONNX from binaries using pip or conda, or build from source.

2. Review documentation and tutorials to familiarize yourself with ONNX’s functionality and advanced features.

3. Follow the importing and exporting directions for the frameworks you’re using to get started.

4. Explore and try out the community’s models in the ONNX model zoo.

3) Tensor Comprehensions

Research: https://research.fb.com/announcing-tensor-comprehensions/

Github: https://github.com/facebookresearch/TensorComprehensions

Tensor Comprehensions (TC) accelerates development by automatically generating efficient GPU code from high-level mathematical operations. TC is a C++ library and mathematical language that helps bridge the gap between researchers, who communicate in terms of mathematical operations, and engineers who are focused on running large-scale models.

Boost productivity

Tensor Comprehensions (TC) is based on generalized Einstein notation for computing on multi-dimensional arrays. It greatly simplifies the development of new operations by providing a concise and powerful syntax which can be automatically and efficiently translated into high-performance computation CUDA kernels.

Tensor Comprehensions provides a lightweight and seamless integration with PyTorch.

Get Started

1. Set up orinstall Anaconda if you don’t already have it.

1. Install Tensor Comprehensions.

1. Review the tutorial and documentation to familiarize yourself with how to use Tensor Comprehensions.

4) Glow

Github: https://github.com/pytorch/glow

Glow is a machine learning compiler that accelerates the performance of deep learning frameworks on different hardware platforms. It enables the ecosystem of hardware developers and researchers to focus on building next-gen hardware accelerators that can be supported by deep learning frameworks like PyTorch.

Powerful hardware optimizations

Glow accepts a computation graph from deep learning frameworks, such as PyTorch, and generates highly optimized code for machine learning accelerators. It contains many machine learning and hardware optimizations like kernel fusion to accelerate model development.

Glow is currently in active development.

Get Started

Visit GitHub to get started.

5) faiss

Github: https://github.com/facebookresearch/faiss

FAISS (Facebook AI Similarity Search) is a library that allows developers to quickly search for embeddings of multimedia documents that are similar to each other. It solves the limitations of traditional query search engines that are optimized for hash-based searches and provides more scalable similarity search functions.

Efficient similarity search

With FAISS, developers can search multimedia documents in ways that are inefficient or impossible with standard database engines (SQL). It includes nearest-neighbour search implementations for million-to-billion-scale datasets that optimize the memory-speed-accuracy tradeoff. FAISS aims to offer state-of-the-art performance for all operating points.

FAISS contains algorithms that search in sets of vectors of any size, and also contains supporting code for evaluation and parameter tuning. Some of its most useful algorithms are implemented on the GPU. FAISS is implemented in C++, with an optional Python interface and GPU support via CUDA.

Get Started

1. Install FAISS.

2. Review documentation and tutorials to familiarize yourself with how FAISS works and its capabilities.

3. Experiment with building indexes and searching using FAISS.

6) StarSpace

Github: https://github.com/facebookresearch/StarSpace

StarSpace is a general-purpose neural embedding model that can be applied to a number of machine learning tasks including ranking, classification, information retrieval, similarity learning, and recommendations. It’s both highly competitive with existing methods while generalizing well to new use cases.

A multi-purpose learning model

StarSpace learns to represent objects of different types into a common vectorial embedding space in order to compare them against each other. This makes it well suited for a variety of problems, including:

• Learning word, sentence or document level embeddings.

• Information retrieval — the ranking of sets of entities/documents or objects, e.g., ranking web documents.

• Text classification, or any other labelling task.

• Metric/similarity learning, e.g., learning sentence or document similarity.

• Content-based or Collaborative Filtering-based Recommendation, e.g., recommending music or videos.

• Embedding graphs, e.g., multi-relational graphs such as Freebase.

Additional details on StarSpace can be found in the research paper.

7) Visdom

Github: https://github.com/facebookresearch/visdom

Visdom is a visualization tool that generates rich visualizations of live data to help researchers and developers stay on top of their scientific experiments that are run on remote servers. Visualizations in Visdom can be viewed in browsers and easily shared with others.

Rich, live visualizations

visdom provides an interactive visualization tool that supports scientific experimentation. Visualizations of plots, images, and text can be easily broadcast for yourself and collaborators.

The visualization space can be organized through the Visdom UI or programmatically, allowing researchers and developers to inspect experiment results across multiple projects and debug code. Features like windows, environments, states, filters, and views also provide multiple ways to view and organize important experimental data.

8) Adanet

Github: https://github.com/tensorflow/adanet

Fast and flexible AutoML with learning guarantees

AdaNet is a lightweight TensorFlow-based framework for automatically learning high-quality models with minimal expert intervention. It uses the AdaNet algorithm by Cortes et al. 2017 to learn the structure of a neural network as an ensemble of subnetworks while providing learning guarantees. Importantly, AdaNet provides a general framework for not only learning a neural network architecture but also for learning to the ensemble to obtain even better models.

9) AutoML Video On-Device

Github: github.com/google/automl-video-ondevice

Inference using the AutoML Video trained object detection mobile sequence models

The example code that shows how to load the Google Cloud AutoML Video Object Tracking On-Device models and conduct inference on a sequence of images from a video clip. The targeted devices are CPU and Edge TPU.

10) Budou

Github: github.com/google/budou

Automatic line-breaking tool for Chinese, Japanese, and Korean (CJK) languages

Budou automatically translates CJK sentences into organized HTML code with meaningful chunks to provide beautiful typography on the web

How Google uses Budou

Budou is used in some of our websites to provide legible and beautiful typography in CJK (Chinese, Japanese, and Korean). Headings are broken into multiple lines at meaningful positions according to the browser screen width.

• Google Japanese Input

• Google 広告

• Google Play Developer Policy Center

• Womenwill

11) Bullet Physics SDK

Real-time collision detection and multi-physics simulation for VR, games, visual effects, robotics, machine learning etc

The Bullet Physics SDK is a professional open-source collision detection, rigid body and soft body dynamics library written in portable C++. The library is primarily designed for use in games, visual effects and robotic simulation. The library is free for commercial use under the Zlib license.

pybullet is an easy to use Python module for physics simulation, robotics and machine learning. With pybullet you can load articulated bodies from URDF, SDF and other file formats. pybullet provides forward dynamics simulation, inverse dynamics computation, forward and inverse kinematics and collision detection and ray intersection queries. Aside from physics simulation, pybullet supports rendering, with a CPU renderer and OpenGL visualization and support for virtual reality headsets.

How Google uses Bullet Physics SDK

Google uses the Bullet Physics SDK for game development, virtual reality, robotics simulation and machine learning.

12) Category Builder

Github: github.com/google/categorybuilder

Expand sets semantically

Set Expansion

Set Expansion expands a set of items to semantically similar items. This is described through examples below.

The seeds {ford, nixon} get expanded to a large set starting with {nixon, Obama, bush, johnson, Clinton, Ford, Reagan, … }. Similarly, {ford, chevy} expands to {ford, chevy, Toyota, Chevrolet, Honda, BMW, Nissan, … } and {ford, stallone} to {ford, Stallone, khan, Kapoor, Sylvester Stallone, Depp, tom cruise, … }.

Sets of verbs and adjectives can also be expanded.

Analogies

The same technique can solve a different problem, one of the analogies such as ‘What is the Mount Everest of Africa?’ Here, we expand to semantically related items on the one hand and to situationally related on the other.

Note that these are harder than “proportional” analogies such as “hand:glove:🦶?”. People don’t need to be provided with the first term (“hand”) and can answer “What is the glove for a foot?”

Some examples:

• mount Everest of Africa: Kilimanjaro

• mount Everest of Alaska: Denali

• the glove of the foot: shoe

• darwin of physics: einstein

• corolla of honda: honda civic

• Voldemort of Tolkien: Sauron

• Voldemort of star wars: Vader

• Tolkien of Voldemort: rowling

13) CausalImpact

Github: google.github.io/CausalImpact

A statistics library for estimating the causal effect of a designed intervention on a time series

The CausalImpact R package implements an approach to estimating the causal effect of a designed intervention on a time series. For example, how many additional daily clicks were generated by an advertising campaign? Answering a question like this can be difficult when a randomized experiment is not available. The package aims to address this difficulty using a structural Bayesian time-series model to estimate how the response metric might have evolved after the intervention if the intervention had not occurred.

14) DeepMind Lab

github.com/deepmind/lab

A customizable 3D platform for agent-based AI research

DeepMind Lab is a first-person 3D game platform designed for research and development of general artificial intelligence and machine learning systems. It provides a suite of challenging navigation and puzzle-solving tasks that are especially useful for deep reinforcement learning. Its simple and flexible API enables creative task-designs and novel AI-designs to be explored and quickly iterated upon.

How Google uses DeepMind Lab

DeepMind Lab is used internally at DeepMind to research and train learning agents.

15) dialogflow-integrations

Integrate various messaging platforms with Google Dialogflow

Dialogflow integrations provide a set of connectors to integrate various messaging platforms (KIK, Spark, Skype, Twilio, Twitter, etc.) with Google Dialogflow.

To learn more about Dialogflow, see Dialogflow | Google Cloud.

16) Dopamine

github.com/google/dopamine

A research framework for fast prototyping of reinforcement learning algorithms

Dopamine is a TensorFlow-based research framework for fast prototyping of reinforcement learning algorithms. It aims to fill the need for a small, easily grokked codebase in which users can freely experiment with wild ideas (speculative research).

Our design principles are:

• Easy experimentation: Make it easy for new users to run benchmark experiments.

• Flexible development: Make it easy for new users to try out research ideas.

• Compact and reliable: Provide implementations for a few, battle-tested algorithms.

• Reproducible: Facilitate reproducibility in results.

17) Gene Expression Programming in Go

github.com/gmlewis/gep

Gene Expression Programming (GEP) is a machine-learning algorithm created by Dr. Cândida Ferreira while attempting to implement a classical genetic programming framework. She discovered some key insights that dramatically improved performance compared to previous implementations. One of her key insights was the representation of the gene that guarantees correctness, thereby completely eliminating the need for the costly gene validation step. Dr. Ferreira’s book was very easy to read and comprehend and GEP is an experiment to code up her algorithm in Go.

18) Google Cloud Datalab

An easy to use interactive tool for data exploration, analysis, visualization and machine learning

Cloud Datalab is a powerful interactive tool created to explore, analyze, transform and visualize data and build machine learning models on Google Cloud Platform. It runs on Google Compute Engine and connects to multiple cloud services easily so you can focus on your data science tasks.

19) Kubeflow

The Machine Learning Toolkit for Kubernetes

The Kubeflow project is dedicated to making deployments of machine learning (ML) workflows on Kubernetes simple, portable, and scalable. Our goal is not to recreate other services, but to provide a straightforward way to deploy best-of-breed OS systems for ML to diverse infrastructures. Anywhere you are running Kubernetes, you should be able to run Kubeflow.

20) Magenta

Magenta: Music and Art Generation with Machine Intelligence

Magenta is a research project exploring the role of machine learning in the process of creating art and music. Primarily this involves developing new deep learning and reinforcement learning algorithms for generating songs, images, drawings, and other materials. But it’s also an exploration in building smart tools and interfaces that allow artists and musicians to extend (not replace!) their processes using these models.

Magenta was started by some researchers and engineers from the Google Brain team but many others have contributed significantly to the project. We use TensorFlow and release our models and tools in open source on GitHub.

21) NVIDIA Libs Test

github.com/google/nvidia_libs_test

Tests and Benchmarks for cuDNN

The repository contains a set of convolution tests and benchmarks for NVIDIA’s cuDNN library.

22) Open Images Dataset

storage.googleapis.com/openimages/web/download.html

A dataset of ~9 million images that have been annotated with image-level labels and object bounding boxes

The images are very diverse and often contain complex scenes with several objects (8.4 per image on average) and the dataset is annotated with image-level labels spanning thousands of classes.

23) Tensor Flow

TensorFlow

TensorFlow is an end-to-end open-source platform for machine learning

TensorFlow is an end-to-end open-source platform for machine learning. It has a comprehensive, flexible ecosystem of tools, libraries, and community resources that lets researchers push the state-of-the-art in ML, and gives developers the ability to easily build and deploy ML-powered applications.

TensorFlow provides a collection of workflows with intuitive, high-level APIs for both beginners and experts to create machine learning models in numerous languages. Developers have the option to deploy models on a number of platforms such as on servers, in the cloud, on mobile and edge devices, in browsers, and on many other JavaScript platforms. This enables developers to go from model building and training to deployment much more easily.

Easy model building

Build and train ML models easily using intuitive high-level APIs like Keras with eager execution, which makes for immediate model iteration and easy debugging.

Robust ML production anywhere

Easily train and deploy models in the cloud, on-prem, in the browser, or on-device no matter what language you use.

Powerful experimentation for research

A simple and flexible architecture to take new ideas from concept to code, to state-of-the-art models, and to publication faster.

24) TensorFlow Playground

playground.tensorflow.org

An interactive in-browser visualization of neural networks

TensorFlow Playground is an interactive visualization of neural networks, written in typescript using d3.js. It contains a tiny neural network library that meets the demands of this educational visualization. You can simulate, in real time, in your browser, small neural networks and see the results.

25) Voice Builder

github.com/google/voice-builder

Voice Builder is an opensource text-to-speech (TTS) voice building tool

Voice Builder is an opensource text-to-speech (TTS) voice building tool that focuses on simplicity, flexibility, and collaboration. Our tool allows anyone with basic computer skills to run voice training experiments and listen to the resulting synthesized voice.

26) Turkish Morphology

github.com/google-research/turkish-morphology

A two-level morphological analyzer for Turkish

This finite-state-tranducer (FST) based Turkish morphological analyzer is composed of three layers:

• Lexicons:

Wide-coverage Turkish lexicons which are manually annotated and validated for part-of-speech and morphophonemic irregularities. They are intended to be used in building Turkish natural language processing tools, such as morphological analyzers. The set of base lexicons that we provide includes annotated lexical items for 47,202 words.

• Morphotactics:

A set of FST definitions which are implemented in a custom format which is similar to AT&T FSM format (only difference being that we can use strings as state names and input/output labels for each transition instead of integers). With each of these FSTs we define the suffixation patterns and the morpheme inventories together with their corresponding output morphological feature category-value pairs for a given part-of-speech.

• Morphophonemics:

A set of Thrax grammars, where each implements a standalone morphophonemic process (such as vowel harmony, vowel drop, consonant voicing and consonant drop and so on).

27) Tensorflow and deep learning

codelabs.developers.google.com/codelabs/cloud-tensorflow-mnist

Sample code for the presentation and code lab

From this codelab, you will learn how to build and train a neural network that recognizes handwritten digits. Along the way, as you enhance your neural network to achieve 99% accuracy, you will also discover the tools of the trade that deep learning professionals use to train their models efficiently.

Part 1

This lab takes 2.5 hours and takes you through the design and optimization of a neural network for recognizing handwritten digits, from the simplest possible solution all the way to a recognition accuracy above 99%. It covers dense and convolutional networks, as well as techniques such as learning rate decay and dropout.

Part 2

The second part of the “Tensorflow and deep learning — without a PhD” presentation covers recurrent neural networks. This code trains a language model on the complete works of William Shakespeare to see if a neural network can do as well as the great poet.

28) SLING

github.com/google/sling

SLING — A natural language frame semantics parser

SLING is a parser for annotating text with frame semantic annotations. It is trained on an annotated corpus using Tensorflow and Dragnn. The parser is a general transition-based frame semantic parser using bi-directional LSTMs for input encoding and a Transition Based Recurrent Unit (TBRU) for output decoding. It is a jointly trained model using only the text tokens as input and the transition system has been designed to output frame graphs directly without any intervening symbolic representation. The SLING framework includes an efficient and scalable frame store implementation as well as a neural network JIT compiler for fast parsing at runtime.

29) Pixel Visual Core Camera

github.com/google/pixelvisualcorecamera

Pixel Visual Core Camera will help Android developers get started with Pixel Visual Core

The Pixel Visual Core Camera Application was designed to provide Android developers with a simple example on how to enable Pixel Visual Core in their camera applications to accelerate HDR+ processing using Camera API 1 and 2.

Device Requirements

Pixel Visual Core is available in Google Pixel 2 and Pixel 3 phones.

Software Requirements

Applications should target API Level 26 (or greater) to get access to Pixel Visual Core functionality. Pixel Visual Core has been available to developers since Android Oreo build OPM1.171019.011, as a developer option. Pixel Visual Core is officially enabled by default starting from Android Oreo build OPM1.171019.019.

How to enable HDR+ with Pixel Visual Core in the code

Camera API 1

• takePicture() uses Pixel Visual Core when the following settings are applied:

• Effect mode set to EFFECT_NONE.

• Flash mode set to FLASH_MODE_OFF.

• While balance set to WHITE_BALANCE_AUTO.

• No exposure compensation.

Camera API 2

• Pixel Visual Core is enabled only for TEMPLATE_STILL_CAPTURE requests.

• CONTROL_ENABLE_ZSL shall be set to true (default).

• Pixel Visual Core is enabled for capture requests that only include JPEG and YUV outputs.

Pixel Visual Core Camera Features

• Pixel Visual Core HDR+ processing with Camera API 1 and 2.

• Double shot, take two shots back to back:

• one shot is processed as HDR+ with Pixel Visual Core.

• one shot is processed with default processing.

• Zoom control.

• Front and back camera support.

30) Quick, Draw! Dataset

github.com/googlecreativelab/quickdraw-dataset

A unique dataset of doodles!

The Quick Draw Dataset is a collection of 50 million drawings across 345 categories, contributed by players of the game Quick, Draw!.

They can be used to help developers train new neural networks, help researchers see patterns in how people around the world draw, and help artists create things we haven’t begun to think of.

The drawings were captured as timestamped vectors, tagged with metadata including what the player was asked to draw and in which country the player was located. You can browse the recognized drawings on https://quickdraw.withgoogle.com/data.