TECH SCHOOL
Posted on October 26, 2020
Hello and welcome back to the backend master class.
So far we have learned a lot about working with database in Go. Now it’s time to learn how to implement some RESTful HTTP APIs that will allow frontend clients to interact with our banking service backend.
Here's:
- Link to the full series playlist on Youtube
- And its Github repository
Go web frameworks and HTTP routers
Although we can use the standard net/http package to implement those APIs, It will be much easier to just take advantage of some existing web frameworks.
Here are some of the most popular golang web frameworks sorted by their number of Github stars:
They offer a wide range number of features such as routing, parameter binding, validation, middleware, and some of them even have a built-in ORM.
If you prefer a lightweight package with only routing feature, then here are some of the most popular HTTP routers for golang:
For this tutorial, I’m gonna use the most popular framework: Gin
Install Gin
Let’s open the browser and search for golang gin, then open its Github page. Scroll down a bit and select Installation.
Let’s copy this go get command, and run it in the terminal to install the package:
❯ go get -u github.com/gin-gonic/gin
After this, in the go.mod file of our simple bank project, we can see that gin is added as a new dependency together with some other packages that it uses.
Define server struct
Now I’m gonna create a new folder called api. Then create a new file server.go inside it. This is where we implement our HTTP API server.
First let’s define a new Server struct. This Server will serves all HTTP requests for our banking service. It will have 2 fields:
- The first one is a
db.Storethat we have implemented in previous lectures. It will allow us to interact with the database when processing API requests from clients. - The second field is a router of type
gin.Engine. This router will help us send each API request to the correct handler for processing.
type Server struct {
store *db.Store
router *gin.Engine
}
Now let’s add a function NewServer, which takes a db.Store as input, and return a Server. This function will create a new Server instance, and setup all HTTP API routes for our service on that server.
First, we create a new Server object with the input store. Then we create a new router by calling gin.Default(). We will add routes to this router in a moment. After this step, we will assign the router object to server.router and return the server.
func NewServer(store *db.Store) *Server {
server := &Server{store: store}
router := gin.Default()
// TODO: add routes to router
server.router = router
return server
}
Now let’s add the first API route to create a new account. It’s gonna use POST method, so we call router.POST.
We must pass in a path for the route, which is /accounts in this case, and then one or multiple handler functions. If you pass in multiple functions, then the last one should be the real handler, and all other functions should be middlewares.
func NewServer(store *db.Store) *Server {
server := &Server{store: store}
router := gin.Default()
router.POST("/accounts", server.createAccount)
server.router = router
return server
}
For now, we don’t have any middlewares, so I just pass in 1 handler: server.createAccount. This is a method of the Server struct that we need to implement. The reason it needs to be a method of the Server struct is because we have to get access to the store object in order to save new account to the database.
Implement create account API
I’m gonna implement server.createAccount method in a new file account.go inside the api folder. Here we declare a function with a server pointer receiver. Its name is createAccount, and it should take a gin.Context object as input.
func (server *Server) createAccount(ctx *gin.Context) {
...
}
Why does it have this function signature? Let’s look at this router.POST function of Gin:
Here we can see that the HandlerFunc is declared as a function with a Context input. Basically, when using Gin, everything we do inside a handler will involve this context object. It provides a lot of convenient methods to read input parameters and write out responses.
Alright, now let’s declare a new struct to store the create account request. It will have several fields, similar to the createAccountParams from account.sql.go that we used in the database in previous lecture:
type CreateAccountParams struct {
Owner string `json:"owner"`
Balance int64 `json:"balance"`
Currency string `json:"currency"`
}
So I’m gonna copy these fields and paste them to our createAccountRequest struct. When a new account is created, its initial balance should always be 0, so we can remove the balance field. We only allow clients to specify the owner’s name and the currency of the account. We’re gonna get these input parameters from the body of the HTTP request, Which is a JSON object, so I’m gonna keep the JSON tags.
type createAccountRequest struct {
Owner string `json:"owner"`
Currency string `json:"currency"`
}
func (server *Server) createAccount(ctx *gin.Context) {
...
}
Now whenever we get input data from clients, it’s always a good idea to validate them, because who knows, clients might send some invalid data that we don’t want to store in our database.
Lucky for us, Gin uses a validator package internally to perform data validation automatically under the hood. For example, we can use a binding tag to tell Gin that the field is required. And later, we call the ShouldBindJSON function to parse the input data from HTTP request body, and Gin will validate the output object to make sure it satisfy the conditions we specified in the binding tag.
I’m gonna add a binding required tag to both the owner and the currency field. Moreover, let’s say our bank only supports 2 types of currency for now: USD and EUR. So how can we tell gin to check that for us? Well, we can use the oneof condition for this purpose:
type createAccountRequest struct {
Owner string `json:"owner" binding:"required"`
Currency string `json:"currency" binding:"required,oneof=USD EUR"`
}
We use a comma to separate multiple conditions, and a space to separate the possible values for the oneof condition.
Alright, now in the createAccount function, we declare a new req variable of type createAccountRequest. Then we call ctx.ShouldBindJSON() function, and pass in this req object. This function will return an error.
If the error is not nil, then it means that the client has provided invalid data. So we should send a 400 Bad Request response to the client. To do that, we just call ctx.JSON() function to send a JSON response.
The first argument is an HTTP status code, which in this case should be http.StatusBadRequest. The second argument is the JSON object that we want to send to the client. Here we just want to send the error, so we will need a function to convert this error into a key-value object so that Gin can serialize it to JSON before returning to the client.
func (server *Server) createAccount(ctx *gin.Context) {
var req createAccountRequest
if err := ctx.ShouldBindJSON(&req); err != nil {
ctx.JSON(http.StatusBadRequest, errorResponse(err))
return
}
...
}
We’re gonna use this errorResponse() function a lot in our code later, and it can be used for other handlers as well, not just for account handlers, so I will implement it in the server.go file.
This function will take an error as input, and it will return a gin.H object, which is in fact just a shortcut for map[string]interface{}. So we can store whatever key-value data that we want in it.
For now let’s just return gin.H with only 1 key: error, and its value is the error message. Later we might check the error type and convert it to a better format if we want.
func errorResponse(err error) gin.H {
return gin.H{"error": err.Error()}
}
Now let’s go back to the createAccount handler. In case the input data is valid, there will be no errors. So we just go ahead to insert a new account into the database.
First we declare a CreateAccountParams object, where Owner is req.Owner, Currency is req.Currency, and Balance is 0. Then we call server.store.CreateAccount(), pass in the input context, and the argument. This function will return the created account and an error.
func (server *Server) createAccount(ctx *gin.Context) {
var req createAccountRequest
if err := ctx.ShouldBindJSON(&req); err != nil {
ctx.JSON(http.StatusBadRequest, errorResponse(err))
return
}
arg := db.CreateAccountParams{
Owner: req.Owner,
Currency: req.Currency,
Balance: 0,
}
account, err := server.store.CreateAccount(ctx, arg)
if err != nil {
ctx.JSON(http.StatusInternalServerError, errorResponse(err))
return
}
ctx.JSON(http.StatusOK, account)
}
If the error is not nil, then there must be some internal issue when trying to insert to the database. Thus, we will return a 500 Internal Server Error status code to the client. We also reuse the errorResponse() function to send the error to the client, then return immediately.
If no errors occur, then the account is successfully created. We just send a 200 OK status code, and the created account object to the client. And that’s it! The createAccount handler is done.
Start HTTP server
Next, we have to add some more code to run the HTTP server. I’m gonna add a new Start() function to our Server struct. This function will take an address as input and return an error. Its role is to run the HTTP server on the input address to start listening for API requests.
func (server *Server) Start(address string) error {
return server.router.Run(address)
}
Gin already provided a function in the router to perform this action, so all we need to do is calling server.router.Run(), and pass in the server address.
Note that the server.router field is private, so it cannot be accessed from outside of this api package. That’s one of the reasons we have this public Start() function. For now, it has just 1 single command, but later, we might want to add some graceful shutdown logics in this function as well.
OK, now let’s create an entry point for our server in the main.go file at the root of this repository. The package name should be main, and it should have a main() function.
In order to create a Server, we need to connect to the database and create a Store first. It’s gonna be very similar to the code that we’ve written before in the main_test.go file.
So I’m gonna copy these constants for the dbDriver and dbSource, paste them to the top of our main.go file. Then also copy the block of code that establishes connections to the database and paste it inside the main function.
With this connection, we can create a new store using db.NewStore() function. Then we create a new server by calling api.NewServer() and pass in the store.
const (
dbDriver = "postgres"
dbSource = "postgresql://root:secret@localhost:5432/simple_bank?sslmode=disable"
)
func main() {
conn, err := sql.Open(dbDriver, dbSource)
if err != nil {
log.Fatal("cannot connect to db:", err)
}
store := db.NewStore(conn)
server := api.NewServer(store)
...
}
To start the server, we just need to call server.Start() and pass in the server address. For now, I’m just gonna declare it as a constant: localhost, port 8080. In the future, we will refactor the code to load all of these configurations from environment variables or a setting file. In case some error occurs when starting the server, we just write a fatal log, saying cannot start server.
One last but very important thing we must do is to add a blank import for lib/pq driver. Without this, our code would not be able to talk to the database.
package main
import (
"database/sql"
"log"
_ "github.com/lib/pq"
"github.com/techschool/simplebank/api"
db "github.com/techschool/simplebank/db/sqlc"
)
const (
dbDriver = "postgres"
dbSource = "postgresql://root:secret@localhost:5432/simple_bank?sslmode=disable"
serverAddress = "0.0.0.0:8080"
)
func main() {
conn, err := sql.Open(dbDriver, dbSource)
if err != nil {
log.Fatal("cannot connect to db:", err)
}
store := db.NewStore(conn)
server := api.NewServer(store)
err = server.Start(serverAddress)
if err != nil {
log.Fatal("cannot start server:", err)
}
}
Alright, so now the main entry for our server is completed. Let’s add a new make command to the Makefile to run it.
I’m gonna call it make server. And it should execute this go run main.go command. Let’s add server to the phony list.
...
server:
go run main.go
.PHONY: postgres createdb dropdb migrateup migratedown sqlc test server
Then open the terminal and run:
make server
Voila, the server is up and running. It’s listening and serving HTTP requests on port 8080.
Test create account API with Postman
Now I’m gonna use Postman to test the create account API.
Let’s add a new request, select the POST method, fill in the URL, which is http://localhost:8080/accounts.
The parameters should be sent via a JSON body, so let’s select the Body tab, choose Raw, and select JSON format. We have to add 2 input fields: the owner’s name, I will use my name here, and a currency, let’s say USD.
{
"owner": "Quang Pham",
"currency": "USD"
}
OK, then click Send.
Yee, it’s successful. We’ve got a 200 OK status code, and the created account object. It has ID = 1, balance = 0, with the correct owner’s name and currency.
Now let’s try to send some invalid data to see what will happen. I’m gonna set both fields to empty string, and click Send.
{
"owner": "",
"currency": ""
}
This time, we’ve got 400 Bad Request, and an error saying the fields are required. This error message looks quite hard to read because it combines 2 validation errors of the 2 fields together. This is something we might want to improve in the future.
Next I’m gonna try to use an invalid currency code, such as xyz.
{
"owner": "Quang Pham",
"currency": "xyz"
}
This time, we also get 400 Bad Request status code, but the error message is different. It say the validation failed on the oneof tag, which is exactly what we want, because in the code we only allow 2 possible values for the currency: USD and EUR.
It’s really great how Gin has handled all the input binding and validation for us with just a few lines of code. It also prints out a nice form of request logs, which is very easy to read for human eyes.
Implement get account API
Alright, next we’re gonna add an API to get a specific account by ID. It would be very similar to the create account API, so I will duplicate this routing statement:
func NewServer(store *db.Store) *Server {
...
router.POST("/accounts", server.createAccount)
router.GET("/accounts/:id", server.getAccount)
...
}
Here instead of POST, we will use GET method. And this path should include the id of the account we want to get /accounts/:id. Note that we have a colon before the id. It’s how we tell Gin that id is a URI parameter.
Then we have to implement a new getAccount handler on the Server struct. Let’s move to the account.go file to do so. Similar as before, we declare a new struct called getAccountRequest to store the input parameters. It will have an ID field of type int64.
Now, since ID is a URI parameter, we cannot get it from the request body as before. Instead, we use the uri tag to tell Gin the name of the URI parameter:
type getAccountRequest struct {
ID int64 `uri:"id" binding:"required,min=1"`
}
We add a binding condition that this ID is a required field. Also, we don’t want client to send an invalid ID, such as a negative number. To tell Gin about this, we can use the min condition. In this case, let’s set min = 1, because it’s the smallest possible value of account ID.
OK, now in the server.getAccount handler, we will do similar as before. First we declare a new req variable of type getAccountRequest. Then here instead of ShouldBindJSON, we should call ShouldBindUri.
If there’s an error, we just return a 400 Bad Request status code. Otherwise, we call server.store.GetAccount() to get the account with ID equals to req.ID. This function will return an account and an error.
func (server *Server) getAccount(ctx *gin.Context) {
var req getAccountRequest
if err := ctx.ShouldBindUri(&req); err != nil {
ctx.JSON(http.StatusBadRequest, errorResponse(err))
return
}
account, err := server.store.GetAccount(ctx, req.ID)
if err != nil {
if err == sql.ErrNoRows {
ctx.JSON(http.StatusNotFound, errorResponse(err))
return
}
ctx.JSON(http.StatusInternalServerError, errorResponse(err))
return
}
ctx.JSON(http.StatusOK, account)
}
If error is not nil, then there are 2 possible scenarios.
- The first scenario is some internal error when querying data from the database. In this case, we just return
500 Internal Server Errorstatus code to the client. - The second scenario is when the account with that specific input ID doesn’t exist. In that case, the error we got should be a
sql.ErrNoRows. So we just check it here, and if it’s really the case, we simply send a404 Not Foundstatus code to the client, and return.
If everything goes well and there’s no error, we just return a 200 OK status code and the account to the client. And that’s it! Our getAccount API is completed.
Test get account API with Postman
Let’s restart the server and open Postman to test it.
Let’s add a new request with method GET, and the URL is http://localhost:8080/accounts/1. We add a /1 at the end because we want to get the account with ID = 1. Now click send:
The request is successful, and we’ve got a 200 OK status code together with the found account. This is exactly the account that we’ve created before.
Now let’s try to get an account that doesn’t exist. I’m gonna change the ID to 100: http://localhost:8080/accounts/100, and click send again.
This time we’ve got a 404 Not Found status code, and an error: sql no rows in result set. Exactly what we expected.
Let’s try one more time with a negative ID: http://localhost:8080/accounts/-1
Now we got a 400 Bad Request status code with an error message about the failed validation.
Alright, so our getAccount API is working well.
Implement list account API
Next step, I’m gonna show you how to implement a list account API with pagination.
The number of accounts stored in our database can grow to a very big number over time. Therefore, we should not query and return all of them in a single API call. The idea of pagination is to divide the records into multiple pages of small size, so that the client can retrieve only 1 page per API request.
This API is a bit different because we will not get input parameters from request body or URI, but we will get them from query string instead. Here’s an example of the request:
We have a page_id param, which is the index number of the page we want to get, starting from page 1. And a page_size param, which is the maximum number of records can be returned in 1 page.
As you can see, the page_id and page_size are added to the request URL after a question mark: http://localhost:8080/accounts?page_id=1&page_size=5. That’s why they’re called query parameters, and not URI parameter like the account ID in the get account request.
OK, now let’s go back to our code. I’m gonna add a new route with the same GET method. But this time, the path should be /accounts only, since we’re gonna get the parameters from the query. The handler’s name should be listAccount.
func NewServer(store *db.Store) *Server {
server := &Server{store: store}
router := gin.Default()
router.POST("/accounts", server.createAccount)
router.GET("/accounts/:id", server.getAccount)
router.GET("/accounts", server.listAccount)
server.router = router
return server
}
Alright, let’s open the account.go file to implement this server.listAccount function. It’s very similar to the server.getAccount handler, so I’m gonna duplicate it. Then change the struct name to listAccountRequest.
This struct should store 2 parameters, PageID and PageSize. Now note that we’re not getting these parameters from uri, but from query string instead, so we cannot use the uri tag. Instead, we should use form tag.
type listAccountRequest struct {
PageID int32 `form:"page_id" binding:"required,min=1"`
PageSize int32 `form:"page_size" binding:"required,min=5,max=10"`
}
Both parameters are required and the minimum PageID should be 1. For the PageSize, let’s say we don’t want it to be too big or too small, so I set its minimum constraint to be 5 records, and maximum constraint to be 10 records.
OK, now the server.listAccount handler function should be implemented like this:
func (server *Server) listAccount(ctx *gin.Context) {
var req listAccountRequest
if err := ctx.ShouldBindQuery(&req); err != nil {
ctx.JSON(http.StatusBadRequest, errorResponse(err))
return
}
arg := db.ListAccountsParams{
Limit: req.PageSize,
Offset: (req.PageID - 1) * req.PageSize,
}
accounts, err := server.store.ListAccounts(ctx, arg)
if err != nil {
ctx.JSON(http.StatusInternalServerError, errorResponse(err))
return
}
ctx.JSON(http.StatusOK, accounts)
}
The req variable’s type should be listAccountRequest. Then we use another binding function: ShouldBindQuery to tell Gin to get data from query string.
If an error occurs, we just return a 400 Bad Request status. Else, we call server.store.ListAccounts() to query a page of account records from the database. This function requires a ListAccountsParams as input, where we have to provide values for 2 fields: Limit and Offset.
Limit is simply the req.PageSize. While Offset is the number of records that the database should skip, wo we have to calculate it from the page id and page size using this formula: (req.PageID - 1) * req.PageSize
The ListAccounts function returns a list of accounts and an error. If an error occurs, then we just need to return 500 Internal Server Error to the client. Otherwise, we send a 200 OK status code with the output accounts list.
And that’s it, the ListAccount API is done.
Test list account API with Postman
Let’s restart the server then open Postman to test this request.
It’s successful, but we’ve got only 1 account on the list. That’s because our database is quite empty at the moment. We just created only 1 single account. Let’s run the database tests that we’ve written in previous lectures to have more random data.
❯ make test
OK, now we should have a lot of accounts in our database. Let’s resend this API request.
Voila, now the returned list has exactly 5 accounts as expected. The account with ID 5 is not here because I think it’s got deleted in the test. We’ve got the account with ID 6 here instead.
Let’s try to get the second page.
Cool, now we get the next 5 accounts with ID from 7 to 11. So it’s working very well.
I’m gonna try one more time to get a page that doesn’t exist, let’s say page 100.
OK, so now we’ve got a null response body. Although it’s technically correct, I think it would be better if the server returns an empty list in this case. So let’s do that!
Return empty list instead of null
Here in the account.sql.go file that sqlc has generated for us:
func (q *Queries) ListAccounts(ctx context.Context, arg ListAccountsParams) ([]Account, error) {
rows, err := q.db.QueryContext(ctx, listAccounts, arg.Limit, arg.Offset)
if err != nil {
return nil, err
}
defer rows.Close()
var items []Account
for rows.Next() {
var i Account
if err := rows.Scan(
&i.ID,
&i.Owner,
&i.Balance,
&i.Currency,
&i.CreatedAt,
); err != nil {
return nil, err
}
items = append(items, i)
}
if err := rows.Close(); err != nil {
return nil, err
}
if err := rows.Err(); err != nil {
return nil, err
}
return items, nil
}
We can see that the Account items variable is declared without being initialized: var items []Account. That’s why it will remain null if no records are added.
Lucky for us, in the latest released of sqlc, which is version 1.5.0, we have a new setting that will instruct sqlc to create an empty slice instead of null.
The setting is called emit_empty_slices, and its default value is false. If we set this value to true, then the result returned by a many query will be an empty slice.
OK, so now let’s add this new setting to our sqlc.yaml file:
version: "1"
packages:
- name: "db"
path: "./db/sqlc"
queries: "./db/query/"
schema: "./db/migration/"
engine: "postgresql"
emit_json_tags: true
emit_prepared_queries: false
emit_interface: false
emit_exact_table_names: false
emit_empty_slices: true
Save it, and open the terminal to upgrade sqlc to the latest version. If you’re on a Mac and using Homebrew, just run:
❯ brew upgrade sqlc
You can check your current version by running:
❯ sqlc version
v1.5.0
For me, it’s already the latest version: 1.5.0, so now I’m gonna regenerate the codes:
❯ make sqlc
And back to visual studio code. Now in the account.sql.go file, we can see that the items variable is now initialized as an empty slice:
func (q *Queries) ListAccounts(ctx context.Context, arg ListAccountsParams) ([]Account, error) {
...
items := []Account{}
...
}
Cool! Let’s restart the server and test it on Postman. Now when I send this request, we’ve got an empty list as expected.
So it works!
Now I’m gonna try some invalid parameters. For example, let’s change page_size to 20, which is bigger than the maximum constraint of 10.
This time we’ve got 400 Bad Request status code, and an error saying the validation of page_size failed on the max tag.
Let’s try one more time with page_id = 0.
Now we still got 400 Bad Request status, but the error is because page_id validation failed on the required tag. What happens here is, in the validator package, any zero-value will be recognized as missing. It’s acceptable in this case because we don’t want to have the 0 page, anyway.
However, if your API has a zero value parameter, then you need to pay attention to this. I recommend you to read the documentation of validator package to learn more about it.
Alright, so today we have learned how easy it is to implement RESTful HTTP APIs in Go using Gin. You can based on this tutorial to try implementing some more routes to update or delete accounts on your own. I leave that as a practice exercise for you.
Thanks a lot for reading this article. Happy coding and I will see you soon in the next lecture!
If you like the article, please subscribe to our Youtube channel and follow us on Twitter for more tutorials in the future.
If you want to join me on my current amazing team at Voodoo, check out our job openings here. Remote or onsite in Paris/Amsterdam/London/Berlin/Barcelona with visa sponsorship.
Posted on October 26, 2020
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.
Related
