Introduction to the GOLD Stack
prozz
Posted on June 17, 2020
Few days ago I stumbled upon this tweet.
Santiago Martínez Q.@smartinezqu
Introducing the GoLD Stack: GoLang + Lambda + DynamoDB.22:54 PM - 07 Jun 2020
It made me smile as it puts a nice label on the tools I work with on daily basis for almost a two years now. I think that this combination of tools is outstanding and deserves more recognition (and the GOLD label obviously).
⚡️ Serverless
At OLX Group we build products for people to use. We need them delivered fast and with minimal overhead. If your goal is to optimize costs and time to market metric, then going serverless seems like a natural move. For more arguments please read the article by @szymanskilukasz.
So you made the decision to serverless all the things... but how? There is no single recipe for that as the landscape is ever changing. Few things are definitely battle tested by us and rock solid:
- Go, the language
- Lambda, your compute power
- DynamoDB, the database that really scales
Go
A pragmatic language that took over the industry in recent years (or at least is climbing the TIOBE index hard). It's very good for serverless usage:
- it is a compiled language (single binary FTW!),
- has almost everything baked in its robust and feature rich standard library, thus can be easily used across the whole stack: in your CI/CD pipelines, CLI tools and Lambda functions,
- Lambda cold starts are minimal, it's one of the fastest languages on the platform,
- offers a nice concurrency model, that allows you to optimize your code if needed (and effectively cut your bills).
On top of those, Go is simple and relatively easy to learn. Try Go by Example or Learn Go with tests. If you don't dig the language, please read my other post.
There lies a hidden beauty in Go's simplicity and many of the language limitations are in fact its hidden features.
Lambda
Lambda is where you execute your business logic. To build an app, you can compose many little single-purpose functions together or serve a full web application from a single function. Frequently this is a point where confusion starts. I'd advise to start with the latter and as your confidence and ability grow find the right balance that works for you.
Having many Lambda functions in your project doesn't mean we write spaghetti code that is hard to maintain. Quite the opposite. By using Hexagonal Architecture and DDD ideas, we treat Lambda functions as an entry points to our system. The same code can be fired from a CLI tool, migration scripts or CI/CD pipeline. It's also easily testable.
It's a very powerful tool to be able to treat your Lambda functions like LEGO bricks and move those around while planning how the data flows thru your backend. Underneath it's still a well-defined and concise domain code, the same you would write when using your own Kubernetes cluster or EC2 instances.
And servers are the key point here, as with Lambdas:
- there is no need to maintain any servers,
- it's cheaper than you think it is,
- magically SCALES up and down out of the box, in similar fashion as our mighty database...
DynamoDB
DynamoDB is a different beast. It's not your usual database. If you are coming from SQL world, you need to forget everything you already know about modelling data and relearn from scratch. It's a very versatile tool, but needs to be approached without any assumptions. My one sentence DynamoDB description is 'a key-value store with two killer features: sort keys and streams'.
Sort keys make it possible, unlike typical key-value stores, to query for more than one record at a time. Combine them with key conditions (begins_with works wonders for hierarchical data structures) and indexing capabilities and you can do a whole lot with them. Experts even claim that you can fit any app in a single table!
As crazy as it may seem, so called single table design, is a good thing to aim for. In early days of a new product requirements may change and new access patterns may be discovered. From my experience evolving your app is easier if you use a table per domain or bounded context, not necessarily one per whole application. It's especially true when you use streaming feature a lot, to minimize Lambda invocations.
Sort keys are nice, but kind of boring. The real killer feature of DynamoDB are streams. They allow you to react to data changes, decouple your code and enable building elegantly composed data flows. Applying event-driven approach was never easier in practice.
In one of our projects we had to maintain various counters for some domain entities. First, very naive implementation was to just modify existing code with increment and decrement calls. We quickly realised this approach was not scaling well, keeping in mind all the dependencies for where to call those little functions caused a lot of mental overhead. The solution was to sync counters based on a stream output. Feature was built in a non invasive way, totally decoupled from main functionality. No increment/decrement call was ever missed since then.
To start with DynamoDB you need:
- proper mindset to unlearn old habits, to get you started ALWAYS name attributes for your primary and sort keys as
PKandSK, - DynamoDB Guide
Deployment time!
Choosing your infrastructure as a code tool may be a headache. It's a complicated topic and there is a lot happening in that area. CloudFormation config files (assembly language for infrastructure) can get really big fast and the whole experience of writing them is at best daunting.
In practice there are three tools worth mentioning:
- SAM (Serverless Application Model from AWS)
- CDK (Cloud Development Kit from AWS)
- Serverless Framework
I have very little experience with SAM, but from my limited exposure, it reminded me CloudFormation a bit too much. I'll definitely keep checking it back, as progress done on many AWS tools is really impressive.
If you are starting, consider using Serverless Framework as it smoothers learning curve, simplifies the ugly stuff and makes you focus on your app.
My next choice, for mature and experienced team, would be CDK. Having autocompletion in your IDE, as well as an ability to craft reusable LEGO-like blocks out of an infra bits, is a really nice experience.
If you are keen to learn either SAM or CDK, and are not afraid of JavaScript or TypeScript (like I am), here are great learning courses put together by my teammate @tlakomy:
Summary
Resigning from maintaining your own servers require a mindset switch for sure. Along the way you will learn a lot of new stuff like S3, CloudWatch, SNS, SQS or HTTP Gateway. The road is bumpy and learning curve seem steep.
However, if you are an application developer, it's a step in a right direction. I'm convinced that going ⚡️ Serverless is a correct move. Using GOLD stack for that was a correct move for us.
I witnessed some Twitter war recently. You may have your reservations and be sceptical:
Crying at this typical Serverless infrastructure. wtf happened to the "simple" functions-as-a-service idea?18:09 PM - 23 May 2020
Yet it's very hard to beat out of the box scalability and overall developer happiness that GOLD stack gives you. At the end of the day what really matters are:
- your productivity and working on solving business problems (as opposite to infrastructure problems)
- ability to deliver new features fast
- happy customers!
Chris Munns@chrismunnsHaters gonna hate. I see a full app architecture where i don’t manage a single server or OS or container, have metrics/monitoring/loggging built in, and with deny by default/fine grained security throughout. Oh and it costs me pennies when not in use and scales automatically. 👍🏻 twitter.com/paulbiggar/sta…18:48 PM - 24 May 2020Paul Biggar @paulbiggarCrying at this typical Serverless infrastructure. wtf happened to the "simple" functions-as-a-service idea? https://t.co/w9bHipFcKD
Let me know if you have any question or want me to dive deep on any related topic. Happy to write a blog post or two!
Posted on June 17, 2020
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.