Training machine learning models, you may use pytorch or tensorflow.

But how do you deploy them? Below are some selected tools to serve your machine learning model with introducing much overhead. Go online in no time!

Tensorflow Serving

GitHub

Pro:

• Tensorflow serving is a good choice to serve your model if you've already used tensorflow to train your models.

Con:

• If you are not using tensorflow, oops.
• If you follow the documentations, err, it takes time to setup and debug.
• If your prediction goes complicated, the overhead of writing its handlers, configurations and debug can take a lot of time.

Torchserver

GitHub

Pro:

• Similar as tensorflow, if you are using pytorch, it's a good choice. Backend uses python, which is your main language during training you're familiar with.

Con:

• Same, if you're not using pytorch, then, not a good choice. also if your prediction goes complicated, the overhead is high.

Triton

GitHub

Pro:

• Written in cpp, so it's fast.
• Support multiple machine learning framework.

Cons:

• Written in cpp, inference takes time, and web server in cpp cannot help much, and it become harder to read and debug.
• Really poor documentation. The documentation is more likely for the developers not for the users. And also, a lot of configurations.

Pinferencia

GitHub

Pro:

• Support any machine learning framework.
• Pure in Python and just a simple library. Overhead is very low.
• Implementation is intuitive and can integrate with you current codes, which means no matter how complicated your prediction workflow is, you don't need to make many changes. Actually, almost no changes, so you save your time.
• User friendly documentation.

Con:

• Relative young project. But it has a full 100% test coverage which beats all the other candidates, that is being young, also being mature.
• Also about documentations, more example are adding in the progress. You need to find out if you already have the example of your framework: Pinferencia (underneathall.app)

That is the model serving tools for today, next we will introduce how to deploy your models into kubernetes.