NEW: 2024 comparison out HERE.

Last month, I published a comparison of the Compute VM performance & price among 10 Cloud providers. While I covered both on-demand and reserved prices, I left Spot VM pricing for a future addendum, which is what this post is about. I'll explore the substantial savings that Spot instances can offer for many scenarios, even though they are not appropriate for all workloads.

Jump directly to results

What are Spot / Preemptible VMs?

Several Cloud providers allow customers to use their excess/unused compute capacity at deep discounts. The drawback is that the heavily discounted VMs can be preempted (terminated) at any time when the capacity is needed to cover standard VM usage. This makes Spot VMs a good match for fault-tolerant workloads.

The exact way Spot instances work in regard to pricing, preemption warning, uptime etc varies depending on the provider, so we will take a closer look at the specifics for each.

At SpareRoom, we use GCP Spot VMs for running our perl test suite, part of our CI pipeline. The machine types we are using would cost $500-$700/month per instance with on-demand prices - we currently get an 80-90% discount over that ($70-$80/month)! We also use Spot instances to launch full test environments for development branches, this allows us to use production-class instance types to evaluate performance, without paying dearly for the privilege.

Any stateless workload where you can handle an instance dying (normally with a prior warning of 30 secs or more) can be a good use-case for a Spot VM, especially applications that require a high level of scalability. You could even work with workloads that would not seemingly be fault tolerant. E.g. if you have web servers behind a load balancer, you will already be handling session/state separate from the VM instance (e.g. we use Memorystore), so you can remove a Spot instance that is about to terminate from the load balancer, replacing it with a new one. Depending on the specifics you might want to have a mix of Spot and standard instances and perhaps even to route specific types of requests to the latter, but such a solution would still provide significant savings, or a significant increase in capacity for a specific budget.

Note that in general the discounts apply to compute resources (CPUs, RAM, GPU), but not storage, bandwidth, licenses etc.

Providers offering Spot VMs

6 out of the 10 providers I tested are providing Spot VM pricing. Spot pricing can vary with time, the listed prices were sampled from June 10th to 20th, for the region I used in the original comparison (usually US East):

Amazon Elastic Compute Cloud (EC2)

Amazon EC2 Spot Instances advertise discounts of up to 90%. In reality, the discounts vary significantly per region, type and even instance size and end up being much more modest if you require a specific instance type/size. They can also change quickly (their pricing tool is updated every 5 minutes) and they allow you to set a maximum price you are willing to pay (with your VM terminated if market price exceeds that). In all, the discounts I obtained for the tested instances were 50% or less.

Even though the discounts were not as spectacular as other providers, Amazon offers a very generous 2 minute preemption warning, which should allow most tasks to gracefully end (and perhaps even time for a replacement instance to spin up). In addition, AWS makes it easy to set up their Auto Scaling with Spot instances, requiring less work to include Spot instances in your setup.

Google Compute Engine (GCE)

GCP Spot VMs offer some of the most generous discount for compute resources among providers. What's more, the prices are not very different from region to region and are stable enough (can only change up to once per month) for them to be on the main price list - no need for special pricing tools. Also, the discounts are only different per type, regardless of instance size. All this allow you to plan ahead and easily figure out what you are paying.

From the table you can see some amazing savings, with the C3 being a big standout. That's the latest Sapphire Rapids type, so it would seem that Google's customers have not yet started using it en-masse and there is a lot of spare capacity. At full price, we saw that the C3 was not great value, but at a 90% discount this should be a whole different discussion!

The GCP Spot VMs we are using seem to have a lifetime measured often in hours and sometimes in days, depending on the type and region. However, the preemption warning Google offers is 30 seconds, which is less generous than AWS.

Microsoft Azure

The Azure Spot Virtual Machines behave like on AWS, with prices that can vary in time and preemption when market price is above your set maximum. You can use the "spot advisor" to find the best discounts. However, the actual discounts are much deeper than AWS for most types - I'd say the pattern is similar to GCP. Preemption warning is also like GCP at 30 seconds.

As you'd expect from Microsoft, Windows VMs have similar discounts.

Oracle Compute VM

The Oracle Cloud Infrastructure's Preemptible Instances follow the provider's simpler pricing scheme and they are a fixed 50% discount over their on-demand pricing. The instance preemption notification comes as a 2 minute warning, which is as good as AWS.

The Standard.E2 I tested previously seems to be getting discontinued, so I removed it (no Preemptible price available). But the Standard.A1 which was the revelation of the comparison should still be competitive.

Alibaba Elastic Compute Service (ECS)

The ECS Preemptible Instances are a bit similar to Azure in that they offer deep discounts, but they vary with time. There is a bidding process that's similar to AWS/Azure, basically if spot prices go higher than what you are willing to pay, the instance is released.

The Alibaba preemptible instances have a couple of great and unique features:

• You get a very generous 5 minute warning before they are released/terminated.
• You can launch them with a 1 hour minimum uptime guaranteed if you wish. You lose an extra 10% discount in that case (the discount is included in the table above).

That's the good, the bad is that there's no simple pricing tool, you have be logged in to search through the options in the pricing calculator and then when you actually try to order you might find some options missing. e.g. I could price an instance with a lower cost "Ultra Disk", but could only order it with the more expensive Enhanced SSD type.

Tencent Cloud Virtual Machine (CVM)

Spot Instances are a new feature of CVM and that shows, as there's still some contradictory info in the FAQ. Overall, you have to log in and start the buying process to see prices, and prices vary with time similar to Alibaba etc. The discounts are usually better than 50% and the preemption notification comes at 2 minutes.

No Spot VM Providers

I am listing the rest of the providers included in my performance tests that have to "compete" without any Spot VM discounts, only OVH had price changes since last month:

DigitalOcean Droplets

Akamai (Linode)

OVHcloud

The on-demand prices seem to have gone quite a bit higher for all but the D2. It is a bit confusing as OVH has a different site for each region, but I am looking at the global site price list for the table above.

IBM Cloud Compute

Performance/price comparison

I'll use the performance evaluation from my original comparison, so look there for details on testing and caveats.

Single-thread Performance/Price (Perl)

Google and Azure's Spot VMs provide some tremendous value. You get up to FOUR TIMES the performance for price compared to anything AWS (or even Linode/DigitalOcean) have to offer. I mentioned the GCP c3 instances above - use them now for your Spot instances while they're so cheap! Remember they are available only for 4 vCPU minimum, if you want 2 vCPU you'd have to go to n2d (and t2d for 1 vCPU). If you're on the Azure cloud, the Milan or Altra instances are almost as good.

Alibaba's compute also offers great value, coming just behind the two leading providers, along with Oracle - which is still a good value with their Standard.A1. Tencent beats AWS, but lags behind the 4 leading providers.

Multi-thread Performance/Price (Perl)

The non-SMT (ARM, GCP t2d) and shared CPU types get a boost in the multi-thread comparison, so Azure's Ampere Altra type leads with Alibaba's shared n4, GCP's c3, c2 and t2d and Azure's Milan following closely.

The Oracle Standard.A1 is closer to the top, while the rest of the field is not very different than the single-thread comparison.

Geekbench 5 Performance/Price

Geekbench 5 pretty much confirms what we saw with the Perl benchmark suite - Azure's Altra and Milan instances along with GCP's Sapphire Rapids and Milan offer the best value for dedicated CPU instances (the Alibaba n4 shared is up there with them), with Oracle's Altra following closely.
AWS only competes well with the providers that are not offering Spot VMs.

Conclusion

Overall, Spot instances can provide some amazing savings. If you can take advantage of Spot instances, I would rate the cloud providers in this order with value in mind:

• Google Cloud: They have the clearest and stables pricing structure, along with some of the deepest discounts. I gave the example of our own testing pipeline - the c3-highcpu-22 instances we are currently using would set us back $692/instance-month (excluding storage), and we are getting them for just $63! If these new instances become more popular with customers and their Spot pricing increases, t2d, c2, n2d instances are all offering near top performance/price. The only disadvantage is the 30 second preemption notification.

• Microsoft Azure: Similar discounts and 30 second notification as GCP, but with a more complex and less fixed pricing structure. Their ARM offering is the top multi-threaded performance/price and of course they are the obvious choice for Windows.

• Oracle Cloud Infrastructure: If you are looking for a mixed standard + Spot cloud and can use ARM VMs, the Standard.A1 will probably offer the best value, depending on the standard vs Spot instance ratio, as it was by far the best on-demand instance in the original comparison.

• Alibaba ECS: Some great discounts, best one on the shared CPU n4 type, but of course a few caveats, including limited regions/availability in the US/Europe, complicated spot pricing etc.

While I found Amazon's Spot pricing quite disappointing in comparison to Azure/GCP, it does offer significant savings. Significant enough to compete in price with budget providers like Akamai/Linode or DigitalOcean (while trouncing most OVH solutions), and perhaps even easier to take advantage of given the longer notification time and integration with their own Auto Scaler. I wouldn't move to AWS just for their Spot instances, but I'd definitely take advantage of them if I were using the provider, either as my main cloud, or as part of a hybrid solution.