We continue our overview of the Well-Architected Framework with our take on the AWS Cost Optimization Tool. Serverless has changed how IT is funded from large enterprise investments written off over several years to pay as you go incremental payments on a weekly or monthly basis. In other words Opex v Capex.

AWS Cost Optimization Pillar which is fifth in the series of talks on the Well Architected Framework

Dave Anderson

We continue our journey through the well-architected pillars and our quest to find our favourite pillar. There’s no order, but we’re on the fifth pillar which is AWS cost optimization. I think this is one of my favourite ones because basically no one understands it or knows what it is. I think it’s got much better now with the modern cloud. Like the other pillars, there are 10 questions and 5 subsections:

1. Practice Cloud Financial Management
2. Expenditure Awareness
3. Cost Effective Resources
4. Matching Supply and Demand
5. Optimizing Over Time

Practice Cloud Financial Management

The first one is Practice Cloud Financial Management. How do you implement cloud financial management? Most people say: ‘we don’t know what that is!’. I always thought it was great fun to ask a development team: ‘how much did that cost? Most of the time, you get back a blank stare!

Mark McCann

It’s a maturity step for teams to be able to respond and know how much their stack costs. If you get blank stares, then you know you need to dig a little deeper into their operational excellence, observability, and general engineering practices. Good engineers have an awareness of this but they need to be pointed in the right direction. Cloud financial management doesn’t need to mean ‘big, scary, or loads of spreadsheets’, it can be as simple as knowing your monthly or weekly bill.

Mike O’Reilly

There has been a big shift in architecture since we’ve got more visibility into cost in the last number of years. When you were working on the enterprise mainframes, you were dealing with capacity. And maybe you did get into licensing and availability of licensing but you never talked in terms of cost. Moving out from mainframes and into the cloud, you go expansive with extrapolated architecture to fit whatever scale you’re working to. You definitely have to factor cloud financial management into your decisions with regard to architecture.

Developers understanding costs

Dave Anderson

Back in the enterprise days, I used to see some of those things. The cost question was: ‘Is it five figures, six figures, or seven figures?’. You sometimes saw many hundreds of 1000s? Or millions? It’s a great question because as Mark says, you can assess if a developer knows what they’re talking about. The first question you can ask is: ‘How much did that cost?’. And if they turn around and respond with ‘$16’. You know that’s a good sign.

Mark McCann

In the serverless and microservice world, your costs are can fluctuate rapidly up and down, if you’re not aware of it. In the past, that would have been pre-bought five years ago, using tax-efficient methods, written off, and paid down over multiple years. What you designed and implemented would not have had a big impact. Now, the success or failure of your organization (depending on the scaling) could come down to how well you manage your cloud costs.

Dave Anderson

Here’s a good tip. If you are interviewing a developer, and they tell you about their fantastic system, just ask: ‘How much does that cost per day?’. That’ll unpack a lot of stuff. Sorry, if that’s unpopular advice. Sorry to the developers out there.

Mark McCann

Even though it’s one question there’s a huge amount behind it. You can go really deep into savings plans, tagging and being savvy and skilled with your cloud financial management. There’s an emerging cloud economics role.

The partnership between CFO and CTO

Dave Anderson

What’s really interesting is that cloud providers are cleverly targeting CFOs. So a partnership forms between the CFO and the CTO. The CFO team has fin ops people. This is a great excuse for a savvy architect to talk to the Finance department to figure out how billing, costs, and budgets link with your architecture. That’s a great way to drive improvement instead of just wanting to refactor because it’s cool. Instead, you can refactor to save half a million dollars because Finance has told you this.

Mike O’Reilly

Looking at expenditure awareness for serverless teams there are two types of financial plans: OpEx v’s CapEx with AWS. CapEx, capital expenditure is when you plan upfront, and OpEx, operational expenditure is not planned for ie. it’s pay-as-you-go. Your dynamic workloads typically fall into OpEx and a lot of organizations struggle with the transition from one to the other.

Dave Anderson

And that’s not just serverless, OpEx v’s CapEx in AWS applies to the cloud in general. The fact is that you don’t know what your bill is going to be for that month. With data centers you pay three years in advance, and you can offset tax.

CapEx and OpEx expectations

Mark McCann

Setting expectations around OpEx v’s CapEx with the AWS Cost Optimization pillar is critically important, especially with business partners, who maybe aren’t aware of this and will wonder why my bill went from £50 to £3,000 this week. And the explanation might be running a load test for a new feature. So you need to be very upfront and very good about setting expectations OpEx v’s CapEx AWS costs will fluctuate up and down give reasons why. It relates back to ‘clarity of purpose’, understanding what you’re doing, and being able to articulate that in a business way that links up business and IT together.

Dave Anderson

Or an even worse example: ‘Wee Jimmy ran a load test and then went off on holidays and forgot to turn it off’.

Mike O’Reilly

We’ve definitely seen a few of those incidents happen in the past!

Dave Anderson

And just to let you know that ‘Wee Jimmy’ is made up.

Mike O’Reilly

In relation to Wee Jimmy and what Wee Jimmy would have learned about cost. I’ve seen a few teams do this quite well. A lot of cloud providers show how your cost is calculated. You can replicate those algorithms in your dashboards. Through your throughput, you can try to predict costs in your own dashboards.

Dave Anderson

How many times has a developer come to your desk in the morning with the blood drained from their face saying ‘I think I have just spent $20,000!’? And you think okay, this is a good way to get some focus on well-architected!

Photo by Sharon McCutcheon on Unsplash.com

Expenditure Awareness

Mark McCann

That brings us to the expenditure awareness section of the question set. To grow that awareness, education is critical. You need to make teams aware that these things are available to you. You can go and look at these. The revamp of the console and having cost on the first screen means that cost awareness is growing more and more, especially with the stuff we talked about. Your cost is an operational expenditure critical to the profitability of your business. It’s not going to be a written-off item from 10 years ago.

Dave Anderson

Expenditure awareness is simple stuff, but for some developers, it is very new. So how do you govern usage? How do you monitor usage and costs? And how do you decommission/how do you switch things off? That’s not what developers of yesteryear, had to worry about. Back then it was a ‘sysadmin’ thing. The fact that people can potentially get alerts or emails if they leave things running over the weekend will drive the right behavior. Because there is real money being spent.

Mike O’Reilly

Tagging and tagging resources are levels of discipline that you need to get into.

Dave Anderson

That’s monitoring. Because if something’s not tagged properly, then it’s not monitored, but you still have to pay for it.

Mike O’Reilly

It’s definitely worthwhile to think about your tagging.

Cost breakdown

Dave Anderson

If you’re in a leadership position, you should be able to see a breakdown. If you have five applications in your portfolio you should be able to know the cost breakdown from those five, at the very least. That’s not that hard to do.

Mark McCann

The things to consider around governing usage, are guardrails, how you set up your organizations, and how you set up your service control policies. If you’re going to be a serverless first shop, you can turn off the non-serverless capabilities that are very expensive if they’re left running. There are ways to establish good guardrails that give you the best cost optimization.

Dave Anderson

The popular allow list? Or the allow lists that only allow serverless services to be popular Mark?

Mark McCann

It’s serverless first, not serverless only but if you come up with a well-articulated excuse or reason why the serverless capability doesn’t work then we will alter the policy!

Cost Effective Resources

Dave Anderson

An exception for EC2! The next one is Cost-Effective Resources. And that’s done quite nicely. There are lots of ways to skin a cat when you’re building something. Developers, always want to pick the fastest and wildest thing. But is it more cost-effective? Sometimes you don’t need the fastest thing and a moderate speed will do the job. This point to sustainability as well. It’s not how fast can you get it. It’s how fast do you need to provide adequate service?

Mark McCann

The total cost of ownership comes to the fore here. It’s not what is for right now. It’s the long-term operational burden and cost. You can choose a technology that’s super low cost, but the cognitive burden is massive because it’s a new technology that’s outside your team right now. Then, what’s the learning cost for that team to learn the tech stack or language that you chose for cost-efficiency reasons? You have got to take a bigger view. It’s not just for what you’re doing right now.

Mike O’Reilly

I know there’s a tonne of new stuff out there at the minute. I’ve never really gotten into it in a lot of sort of depth. But how do you plan for data transfer charges? That’s definitely one to look at if you have a large data footprint.

Dave Anderson

When you look at the operational business processes, it can have backups, DRs, or whatever. You need to be careful that you don’t get charged for those.

Mark McCann

Ingress and egress, especially if you’re going multi-region, can be a massive cost. You need to be aware that there are costs for data moving around regions.

Matching Supply and Demand and Optimising Over Time

Dave Anderson

The last two are Matching Supply and Demand and Optimising Over Time. I’ve lumped these together. There’s a piece around keeping up with the latest and greatest in AWS and tweaking your design so that you’re continuing efficiently. There’s also a bit about selecting new services as you build new stuff and selecting wisely for a decent cost impact.

Mark McCann

This is where we see the serverless advantage kicking in. Matching Supply and Demand is taken care of for you as it scales with the load. If you’re on traditional architectures or EC2, you might need to pre-provision some other stuff and have it at hand and ready to go. That’s a hard calculation to get right. You’re going to have a lot of wastage with an excess capacity just waiting to go whenever the demand comes in. So it’s a lot easier when you’re serverless.

Mike O’Reilly

Being able to extrapolate your costs based on various dimensions is important.

Dave Anderson

So that’s the craic. That’s the AWS cost optimization pillar and OpEx v CapEx in AWS. We’ll hit the Sustainability Pillar next. So please give us a like or a follow on YouTube or the Podcast. Our blog is TheServerlessEdge.com. And follow us on @ServerlessEdge on Twitter. Thank you very much.

Transcribed by https://otter.ai