Python Flask: Interacting with Docker Containers

Python3 Flask: Interacts with Docker Containers

Last Friday, I was talking with my co-worker at my work. Just about work, you know. He told me he had to implement a server for serving his features. Although I didn't understand what he was going to do exactly, it sounded really interesting and I wanted to try. I got really busy at my company a couple of weeks ago, I've had to write a bunch of HTML. It wasn't technically interesting(I love my work but the work was almost the same in this release). I kind of got tired of that. I want to do something new. While I was feeling like this, I had a small talk with the coworker.

He was going to do

Make an API server that gets an image file
Execute a docker container and pass the image file to the container and the container will create the text file that is generated from the image file
Execute another docker container and pass the text
The API Server knows that the container has done with its work and notify to another server(The backend that servers the data to users).

It sounds interesting, ha?, I decided to implement the system on the weekend. I thought it's going to be fun. But it's been really a long time since I used Python and Docker. So, I first had to learn these. During the weekends, I couldn't learn all the things, so that, I would use the minimum skills for implementing the system. If you're a python developer or working with the docker, it might be not in your mind at some points. Please, comment your opinion comment below. I'd love that.

Anyways, Let's get started!

Prerequisite

pipenv

Pipenv is a tool that aims to bring the best of all packaging worlds (bundler, composer, npm, cargo, yarn, etc.) to the Python world. Windows is a first-class citizen, in our world.
When I first use python a few years ago, I used conda or venv for managing packages. When I searched for package management, I saw this. It seems like npm, I didn't know there is like this tool. I thought it must be worth it to try.

docker

Docker is a platform designed to help developers build, share, and run modern applications. We handle the tedious setup, so you can focus on the code.
As I introduced at first, this is the main in this post.

Flask

Flask is a web application framework written in Python. It was developed by Armin Ronacher, who led a team of international Python enthusiasts called Poocco.
To implement a simple API web server, I used Flask. there were other libraries like Fast API. Since I used to Flask a little, I went with Flask For reducing my hours.

This is the process that I'm going to implement.

The API server receives a file from the user
Container A processes A file then saves it to the B File
Container B processes the B File then saves it to the C File
Users can see the result through API

Python Apps and Docker Images

I made two images and uploaded to my them repository in docker-hub.

A tokenizer and word-counting app.

[Tokenizer]



import sys, json, os, requests
from textblob import TextBlob


def extract_nouns(text):
    blob = TextBlob(text)
    filtered_tags = list(filter(lambda tag: tag[1] == "NN", blob.tags))
    nouns = list(map(lambda tag: tag[0], filtered_tags))
    return nouns


def read_file(path):
    with open(path) as f:
        contents = f.read()
    return contents


def save_data(path, data):
    with open(path, "w") as f:
        json.dump(data, f)


def get_filename_from_path(path):
    return os.path.splitext(os.path.basename(path))[0]


def notify_done(url, file_name):
    requests.get(f"{url}/docker/tokenizer_done?file_name={file_name}")


if __name__ == "__main__":
    if len(sys.argv) < 4:
        print("You must pass file path as an argument")
        print("python3 main.py [file path to read] [dir to save] [notification api]")
        print("Example) python3 main.py ./test.txt ./ http://host.docker.internal:20000")
        sys.exit()

    api_url = sys.argv[3]
    file_path = sys.argv[1]
    file_name = get_filename_from_path(file_path)
    target_path = os.path.join(sys.argv[2], file_name + ".json") 

    text = read_file(file_path)
    nouns = extract_nouns(text)

    save_data(target_path, {"nouns": nouns})
    notify_done(api_url, file_name)

    print("Done")

[word-counting]



import sys, json, os, requests


def count_word(nouns_list):
    count_dict = dict()

    for noun in nouns_list:
        if noun in count_dict:
            count_dict[noun] += 1
        else:
            count_dict[noun] = 1

    return count_dict


def load_data(path):
    with open(path) as f:
        json_data = json.load(f)
    return json_data


def save_data(path, data):
    with open(path, "w") as f:
        json.dump(data, f)


def get_filename_from_path(path):
    return os.path.splitext(os.path.basename(path))[0]


def notify_done(url, file_name):
    requests.get(f"{url}/docker/word_count_done?file_name={file_name}")


if __name__ == "__main__":
    if len(sys.argv) < 4:
        print("You must pass file path as an argument")
        print("python3 main.py [file path to read] [dir to save] [notification api]")
        print("Example) python3 main.py ./test.txt ./ http://host.docker.internal:20000")
        sys.exit()

    api_url = sys.argv[3]
    file_path = sys.argv[1]
    file_name = get_filename_from_path(file_path)
    target_path = os.path.join(sys.argv[2], file_name + ".json") 

    json_data = load_data(file_path)
    count_dict = count_word(json_data["nouns"])

    save_data(target_path, {"result": count_dict})
    notify_done(api_url, file_name)
    print("Done")

For running the apps from the API server, I built both python files with below Dockerfiles.

[Tokenizer]



FROM python:3.9

WORKDIR /app
COPY . .

RUN pip install pipenv
RUN pipenv install
RUN pipenv run python3 -m textblob.download_corpora

ENTRYPOINT ["pipenv", "run", "python3", "./main.py"]

[word-counting]



FROM python:3.9

WORKDIR /app
COPY . .

RUN pip install pipenv
RUN pipenv install

ENTRYPOINT ["pipenv", "run", "python3", "./main.py"]

API Server

This is the main code and it's kind of simple.



from flask import Flask
from dotenv import load_dotenv

load_dotenv()

app = Flask(__name__)

import routes

Routes

[routes/docker.py]



import os
from flask import jsonify, request
from server import app
from lib import docker, json


result = []


@app.route('/docker/tokenizer_done')
def get_tokenizer_done():
    file_name = request.args.get("file_name")
    docker.run_word_count_container(file_name)
    return "run a word_count container"


@app.route('/docker/word_count_done')
def get_word_count_done():
    file_name = request.args.get("file_name")

    json_data = json.load_data(
        os.path.join(os.getenv("SHARED_VOLUME_PATH"),
        "word_count_output",
        f"{file_name}.json"
    ))
    result.append(json_data)

    return "all works done"


@app.route('/docker/result')
def get_result():
    file_name = request.args.get("file_name")
    return jsonify({
        "result": result
    })

[routes/upload.py]



import os
from flask import jsonify, request
from werkzeug.utils import secure_filename
from server import app
from lib import docker


@app.route("/upload", methods=["POST"])
def upload_file():
    f = request.files["file"]

    file_name = secure_filename(f.filename)
    f.save(os.path.join(os.getenv("SHARED_VOLUME_PATH"), "input", file_name))

    docker.run_tokenizer_container(file_name)

    return "succeed to upload"

[routes/__init__.py]



from routes import docker, upload

[lib/docker.py]



import os

API_URL = os.getenv("API_URL")
VOLUME_ROOT_PATH = os.getenv("SHARED_VOLUME_PATH")
RUN_TOKENIZER_CONTAINER = 'docker run -it --add-host=host.docker.internal:host-gateway -v "' + VOLUME_ROOT_PATH + ':/shared_volume" hskcoder/tokenizer:0.2 /shared_volume/input/{FILE_NAME_WITH_EXTENSION} /shared_volume/tokenizer_output ' + API_URL
RUN_WORD_COUNT_CONTAINER = 'docker run -it --add-host=host.docker.internal:host-gateway -v "' + VOLUME_ROOT_PATH + ':/shared_volume" hskcoder/word_count:0.2 /shared_volume/tokenizer_output/{FILE_NAME_WITHOUT_EXTENSION}.json /shared_volume/word_count_output ' + API_URL


def run_tokenizer_container(file_name):
    print(RUN_TOKENIZER_CONTAINER.format(
        FILE_NAME_WITH_EXTENSION = file_name
    ))
    os.popen(RUN_TOKENIZER_CONTAINER.format(
        FILE_NAME_WITH_EXTENSION = file_name
    ))



def run_word_count_container(file_name):
    os.popen(RUN_WORD_COUNT_CONTAINER.format(
        FILE_NAME_WITHOUT_EXTENSION = file_name
    ))

[iib/json.py]



import json


def load_data(path):
    with open(path) as f:
        json_data = json.load(f)
    return json_data

This app read environment variables from .env file, so you need to set up like this.
The below variables just fit my system.



API_URL=http://host.docker.internal:20000
ROOT_PATH=C:\Users\hskco\OneDrive\바탕 화면\stuff\docker\api
SHARED_VOLUME_PATH=C:\Users\hskco\OneDrive\바탕 화면\stuff\docker\api\shared_volume

You can run the server with this script



python3 -m pipenv run flask run -h 0.0.0.0 --port 20000

Before running this command, you need to set an environment variable FLASK_APP.
Since I was developing in Windows, I ran this command in api dir.



$env:FLASK_APP = './server.py'

If you enter http://127.0.0.1/docker/result you will see this page.

Let's send a file to the API server and see the result.

[1]

[2]

[3]

Conclusion

It was really fun. I've learned a lot of things.
Regardless of your position, I think It would be really good to try anything you're interested in.

This example is a really beginner level though, I mean.
It should've considered like

Authorization and Security (In this example, the API server exposes all routes to the public)
Communication between containers (I adopted RestfulAPI for interacting between containers, however there must be better ways)
Managing containers (When containers that have done, they need to be deleted. And, load balancing must be also needed. There are tools for this like Kubernetes)
Deployment for production

there are many things that I didn't mention and you need to consider. (I respect for backend developers) I was just focusing on implementing the system, If I tried to make it perfect, I couldn't write this article. Haha,, I have to go back to work tomorrow...

Anyways, it was worth it, it's definitely true :)
I hope it'll be helpful for someone.