As microservices get popular, there are and will be more and more applications developed based on microservices framework. To accomplish data processing tasks within microservices conveniently and efficiently thus becomes an inevitable issue. For a conventional monolithic application, we can implement computations in the database using the specialized, set-oriented SQL. The database’s powerful computing abilities enable quite easy data processing. In the microservices frameworks, a database is mainly used to realize data persistence and data is processed at the application (server) side.

Hardcoding in a high-level language (mainly in Java) is the main approach used to process data at the server side. The native Java code can be seamlessly integrated into the microservice-based application developed in Java. Moreover, Java supports procedural coding that implements computations step by step and the Stream introduced in Java8 have much improved the language for computations.

Yet, Java is still not nearly so convenient and efficient as SQL in achieving data processing goals. It is rather simple to achieve a grouping and aggregation operation in SQL, but it takes dozens of lines of Java code to finish the same calculation. The gap between the two languages is even bigger in dealing with complex computations. The root of Java’s inefficiencies is that the high-level language is not a specialized set-oriented language and lacks special class libraries for structured data computations, resulting in long and complicated code as well as "macro" microservices. What’s more, Java, as a compiled language, is hard to achieve hot swap, and it needs a heavy mechanism like Docker, which is unfriendly to the volatile microservices. But as the phrase goes, “one cannot have the cake and eat it”, you cannot embrace microservices and refuse Java’s shortcomings.

However, an ideal microservices’ data processing component should have certain features.

It should be easy to be integrated into microservices, so smooth that it can be embedded seamlessly in the latter. It should have powerful yet simple to use computational capabilities to accomplish data processing tasks conveniently. It should support hot swap (and hot deployment) to provide services without shutting down the system for maintenance. And it should have an open computing system that can directly access diverse data sources.

esProc SPL is the ideal one that can end all issues.

The open-source data processing engine not only can be seamlessly integrated into a Java application (which is based on microservices framework) but offers all-around computational abilities that enable far easier SPL programming than Java, even SQL, programming. Its support of hot swap and diverse data sources introduce new ideas for processing data within microservices.

Efficient SPL programming

The specialized and versatile SPL data processing engine is able to deal with any structured data computations within microservices.

And it handles them simply and straightforwardly. The agile SPL syntax enables much more convenient programming than Java. The following example shows the SPL conciseness.

According to the stock table, we are trying to find stocks whose prices rise for at least five days in a row and count their rising days (treat same price as rising).

A

1

=db.query@x("select * from stock_record order by ddate")

2

=A1.group(code)

3

=A2.new(code,~.group@i(price

Count days for each stock that rises in a row

4

=A3.select(maxrisedays>=5)

Get eligible stock records

The source data is retrieved from the database and computed in SPL. To achieve the computing goal in this example, even SQL needs a triple-layer nested query, let alone Java (including Stream and Kotlin). SPL uses stepwise coding to do this, which is more convenient than SQL’s descriptive programming style. The language’s syntax combines merits of both Java and SQL.

SPL offers a wealth of class libraries to further simplify complex computations.

SPL can connect to and directly access diverse data sources, such as RDBs, NoSQL, CSV, Excel, HDFS, Restful/Webservice and Kafka, and perform mixed computations between them, significantly facilitating the handling of heterogeneous data sources tasks in microservices and further increasing programming efficiency.

SPL enables much shorter code for implementing service logic, achieving "micro" computational implementation for microservices in addition to its light architecture.

Seamless integration ability

The Java-written SPL supports jar files for embedding. Microservices can import the SPL engine in the form of jar files to be integrated by the application. SPL also encapsulates the standard JDBC driver, through which the application can invoke an SPL script within the microservices.

Programmers just need to import the corresponding jar files and deploy the configuration file (raqsoftConfig.xml) to enable the JDBC driver. Then they can invoke an SPL script with a short piece of code.

Hot swap that enables resource-effective microservices

The interpreted execution SPL intrinsically supports hot swap, making it more adaptable to the volatile scenarios in microservices. Any change to a service will take effect immediately without restarting the server.

The data processing logic is stored in a SPL file (with the extension .splx). Any modification to it will come into effect in real time. This is really convenient and efficient compared with the restarting server design in a compiled language, such as Java.

SPL’s hot swap capability helps reduce microservices’ resource consumption. In order to separately operate and maintain each service instance of microservices where Java is used to process data, often they are deployed individually on VM or Docker. Though services are physically isolated from each other, the Docker will use more resources – that is, it is resource-inefficient. SPL is capable of being independently operated and maintained (hot-swap ability), giving the same capability to each of service instances deployed together and thus using resources efficiently. Sometimes services need to be separately deployed in order to reduce influence on each other, and in these cases, even SPL cannot achieve resource efficiency. Yet, the computing engine offers a large variety of options in order to slim down microservices in terms of both architecture and resource usage.

High-performance computing

SPL can also run as a stand-alone computing server, which can be invoked by an SPL script within microservices to achieve high-performance computations.

The SPL server supports distributed computing through scale-out across a cluster, load-balancing and fault tolerance. It also supplies a lot of high-performance algorithms to achieve high-performance computing.

In a nutshell, SPL can achieve fast and efficient data processing for microservices through its seamless integration ability, hot swap ability, easy programming and high-performance computing while offering architectural conveniences to effectively facilitate microservices development, making comprehensively "micro" microservices.

SPL Resource

SPL Official Website

SPL Source Code

Download esProc SPL