Improving throughput and latency using Java Virtual Threads in Spring
Vignesh Muthukumaran
Posted on June 30, 2024
One of the major changes that Java 21 brought about is Virtual threads. There is so much hype around it, but let's see a real-world example with some metrics. Spring introduced the ability to create GraalVM native images that use Spring Boot and Java 21's virtual threads(Project Loom).
We will look at virtual threads in detail and the many features that come with them in another article. We will focus on a simple project where we can enable the features of Java Virtual Threads and see the difference in performance gains.
Why virtual threads are faster than normal threads?
Normal threads in Java are tied to OS threads. So, there is a limitation to the actual number of threads it can create. Also, time is lost waiting for blocking I/O Calls. Normal Threads are called Platform Threads. They are an expensive resource that needs to be managed well.
Now, when it comes to virtual threads, they are lightweight constructs. Multiple virtual threads can be tied to a single platform thread which in turn gets tied to an OS thread.
The simple idea behind Virtual Threads
The Spring Project used for benchmarking
The project used in this article can be found here. It has a simple fetch API that returns the Thirukkural based on the ID sent.
Now, by default the http server in Tomcat can run many threads in parallel and this will not let us test out this feature easily. So let's throttle it to 10 max threads by adding the below property in the application.properties file. This will allow Tomcat to only use 10 threads at max.
server.tomcat.threads.max=10
We will mimic a blocking IO call using sleep. We will also log the current thread it's using.
Thread.sleep(1000);
log.info("Running on " + Thread.currentThread());
Benchmarking using hey
We will benchmark this using hey tool.
hey -n 200 -c 30 http://localhost:8080/thirukural/1
Summary:
Total: 36.1759 secs
Slowest: 8.0463 secs
Fastest: 2.0080 secs
Average: 5.6840 secs
Requests/sec: 4.9757
Response time histogram:
2.008 [1] |
2.612 [10] |■■■
3.216 [0] |
3.819 [0] |
4.423 [11] |■■■
5.027 [0] |
5.631 [0] |
6.235 [156] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
6.839 [0] |
7.442 [0] |
8.046 [2] |■
Latency distribution:
10% in 4.0305 secs
25% in 6.0212 secs
50% in 6.0263 secs
75% in 6.0340 secs
90% in 6.0376 secs
95% in 6.0387 secs
99% in 8.0463 secs
Details (average, fastest, slowest):
DNS+dialup: 0.0007 secs, 2.0080 secs, 8.0463 secs
DNS-lookup: 0.0005 secs, 0.0000 secs, 0.0041 secs
req write: 0.0000 secs, 0.0000 secs, 0.0003 secs
resp wait: 5.6832 secs, 2.0079 secs, 8.0419 secs
resp read: 0.0001 secs, 0.0000 secs, 0.0005 secs
Status code distribution:
[200] 180 responses
A few log entries are listed below to see that these requests are using Platform Threads.
2024-06-30T22:07:29.659+05:30 INFO 2120 --- [thriukural] [nio-8080-exec-7] p.v.thriukural.web.ThirukuralController : Running on Thread[#45,http-nio-8080-exec-7,5,main]
2024-06-30T22:07:29.659+05:30 INFO 2120 --- [thriukural] [io-8080-exec-10] p.v.thriukural.web.ThirukuralController : Running on Thread[#48,http-nio-8080-exec-10,5,main]
2024-06-30T22:07:29.659+05:30 INFO 2120 --- [thriukural] [nio-8080-exec-2] p.v.thriukural.web.ThirukuralController : Running on Thread[#40,http-nio-8080-exec-2,5,main]
Now, to enable the application to use virtual threads, we just add the below property in application.properties.
spring.threads.virtual.enabled=true
Let's run and see the same benchmark results again.
hey -n 200 -c 30 http://localhost:8080/thirukural/1
Summary:
Total: 12.2478 secs
Slowest: 2.1747 secs
Fastest: 2.0086 secs
Average: 2.0410 secs
Requests/sec: 14.6965
Response time histogram:
2.009 [1] |
2.025 [149] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
2.042 [0] |
2.058 [0] |
2.075 [0] |
2.092 [0] |
2.108 [0] |
2.125 [0] |
2.141 [0] |
2.158 [0] |
2.175 [30] |■■■■■■■■
Latency distribution:
10% in 2.0115 secs
25% in 2.0129 secs
50% in 2.0158 secs
75% in 2.0188 secs
90% in 2.1724 secs
95% in 2.1739 secs
99% in 2.1747 secs
Details (average, fastest, slowest):
DNS+dialup: 0.0015 secs, 2.0086 secs, 2.1747 secs
DNS-lookup: 0.0014 secs, 0.0000 secs, 0.0093 secs
req write: 0.0000 secs, 0.0000 secs, 0.0002 secs
resp wait: 2.0392 secs, 2.0085 secs, 2.1656 secs
resp read: 0.0003 secs, 0.0000 secs, 0.0026 secs
Status code distribution:
[200] 180 responses
A few log entries to see them running in virtual threads.
2024-06-30T22:12:38.485+05:30 INFO 17700 --- [thriukural] [mcat-handler-48] p.v.thriukural.web.ThirukuralController : Running on VirtualThread[#103,tomcat-handler-48]/runnable@ForkJoinPool-1-worker-6
2024-06-30T22:12:38.485+05:30 INFO 17700 --- [thriukural] [mcat-handler-66] p.v.thriukural.web.ThirukuralController : Running on VirtualThread[#121,tomcat-handler-66]/runnable@ForkJoinPool-1-worker-12
2024-06-30T22:12:38.485+05:30 INFO 17700 --- [thriukural] [mcat-handler-69] p.v.thriukural.web.ThirukuralController : Running on VirtualThread[#124,tomcat-handler-69]/runnable@ForkJoinPool-1-worker-7
Benchmarking results
As evident from the results, we see an increased throughput and better latency as well once virtual threads are enabled.
| Parameter | Without Virtual Threads | With Virtual Threads |
|---|---|---|
| Requests/sec | 4.9757 | 14.6965 |
| 99% Latency | 8.0463 sec | 2.1747 sec |
Summary
A simple parameter change has enabled us with a lot of improved performance and throughput. We will further explore in future articles some of the considerations that need to be taken before using virtual threads.
Originally published at vignesh.page.
Please let me know if any improvements you have experienced using Java Virtual Threads
Posted on June 30, 2024
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.
Related
September 11, 2023