Table Of Contents

    * Introduction to Visual Microservices

    * Environment Setup

    * Executing Algonia project

    * Project description

    * Conclusion

Introduction to Visual Microservices

Modern world challenges

The requirements and expectations of the modern world drive us to the point where specialists from different fields must collaborate and work in tandem.

Software engineers are focused on developing backends and UI engineers on designing applications. Data scientists and analysts are gathering and analyzing data. System integrators are implementing solutions in end customer environments.

The specialists mostly use different and incompatible technologies as there is no one that rules them all.

Python is the most sought-after among data professionals because it provides an environment for easily setting up foundations for Machine and Deep learning. Software engineers may prefer C# which has great support for developing multithreaded applications. On the other hand, C/C++ and Rust are very good at solving performance bottlenecks.

Algonia vision

Algonia's vision is to overcome those technology barriers and create tool for smooth collaboration between different technical profiles.

The first step onward to accomplish this goal is ensuring interoperability between programming languages.

Data scientists must be able to seamlessly share their algorithms and models with Software Engineers. On the other hand Software Engineers must provide data sources to data scientists.
UI engineers must be able to flawlessly integrate developed designs with backends, without much of concern about technology powering the backend.

Another problem is that complex systems are hard to maintain. Software is a living creature, constantly developed and driven by new requirements that can easily break existing codebase. Sometimes new features just cannot be added due to current codebase foundations.

Lets see how Algonia deals with those problems.

Language interoperability

Everyone who has ever worked on a system that involves a mix of different languages and technologies can witness that project can easily turn into a nightmare. Calling functions from one language to another can easily lead to improper memory releasing causing hard-to-find bugs, memory leaks and crashes.
Algonia’s main advantage is a solution where engineers can easily pack their code into drag'n droppable visual boxes. Once packed, they become a part of the existing ecosystem. They can be connected to other algorithms in any order, regardless of programming language.
Many programming languages are supported and new ones are adding constantly.

Easy maintenance

Visual programming languages offer easy maintenance and there is no exception here. Visual boxes can be re-connected, deleted from and added to visual workflows.

Performance

If you used some of the visual programming tools already, then you know that performance was sacrificed for that benefit.
Algonia is lightning fast and low memory usage on-premise environment that executes visual workflows. It is by far the most powerful visual programming language ever created, with more raw processing horsepower than anything available at any price.

For non developers

System integrators, end users and other non-developers can create new functionality by purchasing visual boxes from the marketplace and connecting them into existing workflows.

Setting up Algonia Environment

An introductory example will be an application for displaying charts of the historical stock prices.

Architecture, demonstrated on practical Stock Prices project could serve for other projects, even larger ones. Of course you can adopt it also to your needs.

C# is going to be used for fetching data from api, Python for creating charts and JS/HTML for designing user interface.

Prerequisites

Make sure you have met following requirements:

• Windows or MacOS
• Git
• VS Code
• Python 36, 37 or 38
• NetCore 3.1
• Alpha Vantage api key. It's for free. We’ll use this API for getting Stock prices.

Environment setup

Clone StockPrices project

Open command prompt and type command
git clone https://github.com/tvinko/algonia.git
Go to the repository folder after cloning.

Install Python Packages

We’ll use virtualenv to install Python packages. There are many benefits of using virtualenv for package management. It really shines in an environment with many projects that don’t share the same Python installation. This way we solve conflicting requirements.

• Run virtualenv --version command to check if you have installed virtualenv.
  If you don’t have it then install it with command pip install virtualenv

• Next step is to create a virtual environment with the name venv_38. Run the python --version command to check your python default version.

Then run the command according to your default Python version
virtualenv venv_38 --python=python3.8
virtualenv venv_37 --python=python3.7
virtualenv venv_38 --python=python3.6

This command will create a new folder in your project root:

• Activate virtual environment with .\venv_38\Scripts\activate
  (Windows) or
  source venv_38/bin/activate (macos) command

• Install packages from requirements: pip install -r requirements.txt

Now the packages are installed in your virtual, isolated python environment.

Download Algonia Bundle

Download Algonia Bundle from github repo

Windows
Use the latest release and extract algonia-bundle-win64.zip to your repository folder.
Your folder structure should now look like this:

MacOS
Download algonia-bundle-macos.zip, extract it and drag algonia-designer to your Applications folder.

Project structure description

First and foremost, the weapon of choice here is VSCode because of its versatility. It is a source code editor that can be used with a variety of programming languages and supports multiple platforms such as Linux, Windows, and macOS.

algonia-bundle-win64 / algonia-bundle-macos

Bundle contains a development tool (IDE) for gluing your algorithms (Algonia Visual Designer) and an environment for executing visual scripts (Algonia Engine).
Algonia Visual Designer contains (among other things) bundled .NET Core SDK, so it doesn't have to be installed on production environment. However, it must be installed on development machine.

When you ship your project to customers, you’ll create a package. This is a lightweight executing environment containing only Algonia Engine, without Algonia Designer.

Using your own .NET Core version and creating Algonia Engine packages will be explained in the following tutorials.

algos

Your team’s projects. Stock Prices contains three projects. They are located in algos directory of repository:

• Python project for graphs imaging algos/py/graphs/plot_graph.py
• NetCore project for fetching data from api algos/netcore/StockPrices/StockPrices.csproj
• HTML/Js for user interface algos/html/assets

Those projects will be glued together in Algonia Visual Designer and executed by Algonia Engine.

templates

The Algonia Visual Designer projects are called visual templates - drawing canvases for connecting your projects. Templates folder contains those Algonia Visual Designer projects.

venv_36/venv_37/venv_38

Python packages, required for the Stock Prices project.

requirements.txt

Python requirements file

sync.bat / sync.sh

Scripts for synchronizing your teams and 3rd party projects with Algonia Designer. Use sync.bat on Windows and sync.sh on MacOS systems.
Algonia Designer "glues" your project results (.dll, *.so., *.py, *.js,.html and similar files).
When you compile code in your preferred IDE (VisualStudio Code, Visual Studio...) the library is normally created in the bin folder of your project which is out of Algonia Designer reach.
Those scripts synchronize your libraries with Algonia Designer.
The rule of thumb is: run sync script when you make changes outside of Algonia Designer (in VS Code or your preferred IDE)

Executing Algonia project

Compile StockPrices project (C#)

Open algonia.code-workspace in VSCode.
Click on CTRL+Shift+B (Win) or Shift+Command+B (MacOS) to get a list of build tasks and select StockPricesTest project.

Sync libraries and Algonia Designer

Synchronize .Net Core library with Algonia Designer by executing sync.bat/sync.sh script. Script will also synchronize our Python and HTML files.

Set python virtual environment packages

You need to set the location to virtualenv packages so that Algonia Engine will add it to it’s packages path.

Open algonia-designer.
On Windows navigate to the algonia-bundle-win64 folder and click on algonia-designer executable to launch Algonia Designer.

On MacOS go to Applications and open algonia-designer. On the first run you’ll get following warning:

Click on Cancel, navigate to Security & Privacy -> General and click Open Anyway

Click on the Project menu tab and select Open. Navigate to the templates/StockPrices folder of your cloned repository and select StockPrices.algo to open the Algonia project.

Packages are located in the site-packages directory.
On Mac this is normally in your cloned repository under StockPrices/venv_38/lib/python3.8/site-packages and on Windows venv_38\Lib\site-packages

Click on the Settings tab, select Open and enter full path to the packages:

Set Alpha Vantage api key

Alpha Vantage API is used to fetch Stock prices.

Select PRICES_OBJECT visual box:

Click on the Add args in the properties window:

Enter your API key in the value field of API_KEY property

Click on the Save button

Click Project and Save to save the project. You can also use shortcut CTRL+S.

Set Python version if needed

Project is currently set for the Python 3.8 environment. If you’re using Python 3.6 or 3.7 then the version must be set on Python visual boxes.
Select GUI visual box and change Python version:

Select SAVE_GRAPH visual box and repeat procedure.
Click on the Project menu item and click on Save to save the project.

Run project

Press F5 to run the application and open main application window

Click on the Draw Graph button

You can change the value from MSFT to let's say ETH and see the result.

Project description

In the main project window you’ll see green boxes, some of them connected and some of them just floating around:

Those boxes are wrappers around our C# and Python projects.

For example, SET_SYMBOL visual box calls C# function ChangeSymbol from StockPrices project (algos\netcore\StockPrices):

And the GET_PRICES visual box calls function GetPrices from the same solution.

SAVE_GRAPH visual box for instance calls plotGraph method from Python project (algos\py\graphs\plot_graph.py):

Confused? How can “visual box” just call the function from our, or someone else’s project? Create an object instance?
Don’t worry, it’s going to be explained in the next sections.

But I hope that you briefly got some clue at this point: projects created with different technologies and programming languages can be glued together inside Algonia Designer. You can connect them in any meaningful order you want.

Now, let’s rewind to the beginning and take a look at the first row of the boxes:

Start box

Start box is the entry point to our application. Applications can have multiple entry points running in parallel, but there is no need for this in our Stock Prices project.

PRICES_OBJECT

Next, PRICES_OBJECT box, creates an object from the StockPrices class defined in algos/netcore/StockPrices/StockPrices.csproj project.

Select the box and click on the Add code button in it’s properties window:

This is ordinal C# code, with some extensions:

• In header section we reference C# dlls and write using directives
• Code is written in the code section. Tags containing attribute run with value true are entry points to visual boxes

Rest of the code should look familiar to you. Except for the two functions, get_arg and set_result.
Those two functions are part of the Algonia SDK and their purpose is to control visual boxes and workflows. There are few more functions for that purpose and will be explained later.

get_arg
this function receives the argument key and returns it’s value. We can set those arguments programmatically or through Algonia Designer user interface. In fact you already set argument on Algonia Designer UI when you set your API key:

set_result
this function takes an object and saves it into the current Visual Box. Each Visual Box can store some result that is accessible to other Visual Boxes.

GUI Box

This is a Python visual box containing a user interface. It uses the Python Eel package for making Electron-like offline HTML/JS GUI apps. I’m not going into Eel details as this would be out of scope for our tutorial, but I strongly recommend that you check it out here

The wrapper code inside Visual Box is:

Let’s just go through a few interesting points.
def registerGui(self)
Entry point to Visual Box is registerGui function, set in it’s properties window:

This function starts gui with main.html. This is Eel’s way of starting UI:

def getPricesRequest(symbol)

Algonia SDK functions here are:
algonia_helpers.set_element_arg
sets the value of the visual box’s argument. First parameter is the name of the visual box that argument is set and the second parameter is it’s value (besides storing results, visual boxes can also store arguments).

This function is exposed to Javascript with eel.expose attribute.
It receives a Stock symbol parameter from html input and it’s called by jQuery (we’ll get to HTML and jQuery part later).
The call to set_element_arg stores this symbol as an argument.

algonia_helpers.exec executes Visual Box and all chained boxes connected into the same workflow. The parameter of this function is the visual box name.

Lets just recap what we just did:

• We set the SET_SYMBOL argument to the current Stock symbol that we got from the user interface and then executed it.
• After SET_SYMBOL finishes, GET_PRICES is automatically executed (because those two boxes are visually connected).

def registerExecNodes(self)
There is another Algonia system function, or more precise system callback:

All Visual Boxes names, that are executed dynamically with the algonia_helpers.exec method, must be registered and returned as string.

def saveGraphReuest()
This function returns graph image, encoded as base64 string and will be explained in following steps

Lets go to the next row of Visual Boxes

SET_SYMBOL box

There is a new algonia SDK function here that gets the result of Visual Box - get_result_raw.
It takes the name of the box as a parameter.

Remember, Visual boxes can see each other’s results. StockPrices object is stored in a PRICES_OBJECT box.
Now we are going to use this object and call the ChangeSymbol function.

get_element_arg gets the box’s current argument. We set it in previously described eel exposed Python function:

GET_PRICES box

This box calls the GetPrices method. There is nothing new here, as we already described all Algonia SDK functions contained in this box:

Lets recap what we did:

• getPricesRequest Python function is called from Javascript
• This function set the symbol name of the SET_SYMBOL box argument and executed it
• SET_SYMBOL called our C# StockPrices ChangeSymbol method.
• After the SET_SYMBOL, GET_PRICES is executed, because those two boxes are directly connected with virtual wire. GET_PRICES call our GetPrices method defined in StockPrices object

SAVE_GRAPH box

Let's move to the last Visual Box. If you open the code, you can see following Python script:

Visual Box reads result from GET_PRICES (prices in CSV format):

Interesting part is call to function plotGraph, which is defined in our plot_graph Python project and returns an image in base64 string format:

The image is then shown on UI.

User Interface

All six Visual Boxes from Algonia Designer canvas were described, but there is one topic left - User Interface.

It is located in libs\algonia\html\assets\main.html file.
The Javascript part that communicates with Visual Boxes is:

STEP 1

Draw Graph button on the user interface executes eel.getPricesRequest function and passes currently selected symbol from html input (ETH, MSFT...)
This call is handled by Eel in Python GUI box that executes SET_SYMBOL visual box:

STEP 2

Python code executes callback to javascript onGetPricesCompleted function:

This function then calls saveGraphRequest in GUI box:

Here we execute the SAVE_GRAPH box to get a base64 encoded image string and call the javascript function onSaveGraphCompleted.

STEP 3

onSaveGraphCompleted function shows a graph image in divGraph div

Conclusion

We just went through all steps of our Stock Prices example. In real life we should handle exceptions and errors more conveniently. We could also simplify workflow by reconnecting Visual Boxes differently. But the purpose of the introduction tutorial was to show different ways of Visual Boxes execution.

Main Algonia purpose is technology interoperability, collaboration foundations between different specialists. In our example we used just C# and Python, but of course there are many more choices.

Algonia also forces loosely coupled and modular architecture from the start. You can easily rearrange visual boxes, adding new and deleting existing ones.
Full blown code including all object oriented goodies can be written right inside Visual Boxes. But the biggest strength of Visual Boxes is that they act like wrappers to your and 3rd party libraries. Libraries are developed independently but their functions are wrapped inside boxes.
Visual Boxes hold their object state - there are not just fire’n’forget function executions.

And last but not least, system integrators and non-developers can maintain systems on their own by adding or removing Visual Boxes purchased from Algonia Marketplace.

We are just entering the next era of programming - Visual Microservices.