Golang's migration package for better database state management

Introduction
Create a migrator package
Prepare AWS infrastructure
Create working example of DDB migration
Launch migration
Conclusion

Introduction

A few months ago, my team and I were rewriting PHP Laravel app with traditional relational database (MySQL) on full serverless application supported by AWS and Go Lang.
During our work, we encountered many challenges. One of them was a case of migration. When it comes to classic server applications supported by Laravel all management of migration is very simple. There is a list of commands which help us create a migration script, populate changes (or revert) on a database, clear a current database state and many others.
When we faced the new technical stack, we have had to change our approach. A couple of questions came up. First how we will write single migration script, next how we could manage our scripts, how we will know which script was launched earlier and many, many other questions around this topic.

In this post, we'll provide answers to many above questions. I'm going to show you how to write a simple migrator package and a simple guide on how to use it. Additionally, we'll spend some time on AWS configuration for our migration set-up. In the end, we'll launch our example migration and check, if everything works fine.

For a less patient reader, I have shared a link to repository with a ready migrator package and DynamoDB example (RDS example is in progress). On the master branch, you can see a full version with some extra improvements. However, if you want to check the same code as in the article, please use a link to this branch.
Please select what you need:

Create a migrator package

In the beginning, we should collect all assumptions about our expectations for the migrator package. This should be something like migration scripts manager which help manage our migrations.

Migrator package assumptions:

Ability to handle one or many new migration scripts by one migrator execution,
Ability to launch only new migrations and all previous ones are omitted,
Ability to run on many environments like dev, prod, staging, etc.,
Ability to keep data in simple and easy to manage storage/database.

Having considered the last point, we have picked DynamoDB as we have been using AWS infrastructure.
Based on our assumptions, we need to write a function responsible for:

Getting information about previous migrations from database,
Comparing a list of migrations with the database state,
Launching only newly migrations,
Updating the information about done migrations.

We have our basic assumptions, so let's start coding…tests :) OK, I'm kidding now. We don't have enough space for TDD. If you haven’t tried, I encourage you to do it.

Let's start creating a migration domain with necessary structs and interfaces.

// src/migrator/migrator.go

// API needed to fulfill the contract, we could use *dynamodb.Client instead, but if you want to generate
// mocks for tests create own interface will be better choice.
type API interface {
    PutItem(context.Context, *dynamodb.PutItemInput, ...func(*dynamodb.Options)) (*dynamodb.PutItemOutput, error)
    Query(context.Context, *dynamodb.QueryInput, ...func(*dynamodb.Options)) (*dynamodb.QueryOutput, error)
}

// Migrator allows firing migration definitions.
type Migrator struct {
    db        API
    tableName string
}

// Definition keeps shape of single definition
type Definition struct {
    Name string
    Func func() error
}

// Summary keeps details about launched migration
type Summary struct {
    StartingVersion int
    CurrentVersion  int
    Executions      []Execution
}

// Execution keeps single migration execution details.
type Execution struct {
    Name    string
    FiredAt time.Time
    Elapsed time.Duration
}

// version keeps data about single done migration
type version struct {
    MigrationSet string    `dynamodbav:"migration_set"`
    VersionNo    int       `dynamodbav:"version_number"`
    Name         string    `dynamodbav:"name"`
    FiredAt      time.Time `dynamodbav:"firedAt"`
    Elapsed      int64     `dynamodbav:"elapsed"`
}

Two things are the most significant from the above code. First, the version’struct’ describes the structure of our DynamoDB table.
Next, the API interface represents an implementation of methods, which help us communicate with the database.

Now, we can create methods responsible for putting information about launched migrations in the database and getting the last launched migration.

// src/migrator/store.go

// put creates new record in migration DDB table
func (m *Migrator) put(ctx context.Context, v version) error {
    item, err := attributevalue.MarshalMap(v)
    if err != nil {
        return err
    }

    expr, err := expression.NewBuilder().WithCondition(
        expression.And(
            expression.AttributeNotExists(expression.Name("migration_set")),
            expression.AttributeNotExists(expression.Name("version_number")))).Build()
    if err != nil {
        return err
    }

    _, err = m.db.PutItem(ctx, &dynamodb.PutItemInput{
        ConditionExpression:      expr.Condition(),
        ExpressionAttributeNames: expr.Names(),
        Item:                     item,
        TableName:                aws.String(m.tableName),
    })
    if err != nil {
        return err
    }

    return nil
}

func (m *Migrator) get(ctx context.Context, migrationSet string) (version, error) {
    expr, err := expression.NewBuilder().WithKeyCondition(
        expression.KeyEqual(expression.Key("migration_set"), expression.Value(migrationSet))).Build()
    if err != nil {
        return version{}, err
    }

    out, err := m.db.Query(ctx, &dynamodb.QueryInput{
        ExpressionAttributeNames:  expr.Names(),
        ExpressionAttributeValues: expr.Values(),
        KeyConditionExpression:    expr.KeyCondition(),
        // we need to only one (the last) record from database
        Limit: aws.Int32(int32(1)),
        // descending order
        ScanIndexForward: aws.Bool(false),
        TableName:        aws.String(m.tableName),
    })
    if err != nil {
        return version{}, err
    }

    if len(out.Items) == 0 {
        return version{}, nil
    }

    var v []version
    err = attributevalue.UnmarshalListOfMaps(out.Items, &v)
    if err != nil {
        return version{}, err
    }

    return v[0], nil
}

Ok, we have methods responsible for communication with the database, it's time to use them. Let's continue writing a code in store.go file.

// src/migrator/store.go

// New creates an instance of Migrator.
// Table name should point to a valid migration table, example one defined in testdata/serverless.yml
func New(db API, tableName string) *Migrator {
    return &Migrator{db: db, tableName: tableName}
}

// Run launch migration definitions for specific migrationSet
func (m *Migrator) Run(ctx context.Context, migrationSet string, defs []Definition) (Summary, error) {
    v, err := m.get(ctx, migrationSet)
    if err != nil {
        return Summary{}, err
    }

    if len(defs) == 0 {
        return Summary{}, nil
    }

    currentVersion := v.VersionNo
    if len(defs) < currentVersion {
        return Summary{}, ErrMigrationHole
    }

    var executions []Execution
    // iterate through all definitions and try to run function from a single definition
    for i := len(defs) - currentVersion - 1; i >= 0; i-- {
        def := defs[i]
        firedAt := time.Now()
        var elapsed time.Duration
        err := def.Func()
        if err != nil {
            return Summary{
                StartingVersion: v.VersionNo,
                CurrentVersion:  currentVersion,
                Executions:      executions,
            }, fmt.Errorf("migration '%s' failure: %w", def.Name, err)
        }
        elapsed = time.Since(firedAt)

        executions = append(executions, Execution{
            Name:    def.Name,
            FiredAt: firedAt,
            Elapsed: elapsed,
        })

        err = m.put(ctx, version{
            MigrationSet: migrationSet,
            VersionNo:    currentVersion + 1,
            Name:         def.Name,
            FiredAt:      firedAt,
            Elapsed:      elapsed.Nanoseconds(),
        })
        if err != nil {
            return Summary{
                StartingVersion: v.VersionNo,
                CurrentVersion:  currentVersion,
                Executions:      executions,
            }, err
        }
        // If everything is fine, we raise a version
        currentVersion++
    }

    return Summary{
        StartingVersion: v.VersionNo,
        CurrentVersion:  currentVersion,
        Executions:      executions,
    }, nil

}

If you want to launch migration, you need to call Run function with the wanted environment as migrationSet argument and with an array of your migration definitions. As you remember, our Definition struct contains two properties, Name and Function. Next, this function iterate through all definitions set and execute your code for every, single definition. Information about all previous migration are saved in DynamoDB database. When you try to run the migration again, only new definitions will be fired.

Let's skip to the next chapter in order to prepare an infrastructure to test Run function.

Prepare AWS infrastructure

We are going to prepare AWS infrastructure for our migration case. I assume you have at least a basic understanding of AWS. If not, please use links in the text below to configure a basic environment. Don't be scared about that, it doesn't take rocket science to figure out that.

We need to create two databases. First, to keep migration records (represent by version struct). The second is only for running examples. For testing purposes, we need to use a user profile with access to created DynamoDB tables. In case profile doesn't have access to manage created DynamoDB table, please create such a new role and assume role or extend permissions of your current role. If you don't know how to create DynamoDB tables, please follow up with the documentation.
In my examples, I named my tables example-migration-table for database with launched migration records and example-records-table as database on which I'm going to do changes with migrations.

If you have ready the AWS environment, we can go ahead to the next part.

Create working example of DDB migration

In this chapter we're going to write a working example that will add records to example-records-table. If you launch this example again, without new definitions of migration, it shouldn't execute previous functions.

At first, we need to create a method to add a new record to the example-records-table.

// src/examples/ddb/scripts/create_record.go

// ExampleRecord represents structure of single record in DDB
type ExampleRecord struct {
    ID        string    `dynamodbav:"id"`
    CreatedAt time.Time `dynamodbav:"created_at"`
}

// CreateRecord creates a single record of ExampleRecord in DDB
func CreateRecord(ctx context.Context, db *dynamodb.Client, tableName string) error {
    r := ExampleRecord{
        ID:        uuid.NewString(),
        CreatedAt: time.Now(),
    }

    attrs, err := attributevalue.MarshalMap(r)
    if err != nil {
        return err
    }

    _, err = db.PutItem(ctx, &dynamodb.PutItemInput{
        Item:      attrs,
        TableName: aws.String(tableName),
    })

    if err != nil {
        return err
    }

    return nil
}

Next, we can create a set of migration definitions:

// src/examples/ddb/defs_example.go

// DefsExample contains definitions of our migrations
// db - necessary to use DDB table
func DefsExample(ctx context.Context, db *dynamodb.Client) []migrator.Definition {
    return []migrator.Definition{
        // The First migration definition
        {
            Name: "#1 example migration",
            Func: func() error {
                // Insert your table name in place of mine
                err := scripts.CreateRecord(ctx, db, "example-records-table")
                return err
            },
        },
    }
}

Now, we already have all the pieces of the puzzle. Let's use them in an entry point of this example.

// src/examples/ddb/migration/main.go

// main - entry point of example
func main() {
    var set string
    flag.StringVar(&set, "set", "", "migration set name")

    var table string
    flag.StringVar(&table, "table", "", "migration history table name")

    flag.Parse()

    if set == "" {
        log.Fatal("empty set name")
    }
    if table == "" {
        log.Fatal("empty table name")
    }

    // make sure apex logger marshals and outputs JSON
    apexlog.SetHandler(json.New(os.Stderr))

    ctx := context.Background()
    // Needed to create aws sdk configuration
    conf, err := config.LoadDefaultConfig(ctx)
    if err != nil {
        log.Fatal(err)
    }
    // NewFromConfig returns a new DDB client necessary to connection with database 
    db := dynamodb.NewFromConfig(conf)

    o := migrator.DefaultDDBProviderOptions{
        MigrationSet: set,
        DB:           db,
    }

    var defs []migrator.Definition
    switch set {
    case "example":
        defs, err = ddb.DefsExample(ctx, o.DB), nil
    default:
        defs, err = nil, fmt.Errorf("unknown migration set: %s", o.MigrationSet)
    }
    if err != nil {
        log.Fatal(err)
    }

    // Here we create a new instance of our migrator 
    m := migrator.New(db, table)
    summary, err := m.Run(ctx, set, defs)
    if err != nil {
        log.Fatal(err)
    }

    log.Print(summary)
}

Launch migration

In one of my favorite songs, Phill Collins sings the following words: "This is the time, this is the place". So, I guess, this is the best moment to test our example.
The best place to run such scripts is a pipeline job. Yet this time let's do this as simple as possible from local terminal.
Please follow me if your AWS profile doesn't have access to operations on DynamoDB tables, and you decided to create a separate role to do that. If that doesn't apply to you, please ignore the next steps as related to AWS configuration.

Please write the below command to attach your DynamoDB access role to your AWS profile.

aws --profile YOUR_PROFILE_NAME sts assume-role --role-arn ARN_OF_YOUR_ROLE --role-session-name emil --profile YOUR_PROFILE_NAME > assume-output.txt

Now it's time to export some credentials from assume-output.txt

- export AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY_ID
- export AWS_SECRET_ACCESS_KEY=YOUR_AWS_SECRET_ACCESS_KEY
- export AWS_SESSION_TOKEN=YOUR_AWS_SESSION_TOKEN

We are ready to launch the first migration! First, please build example code:
go build -o build examples/ddb/migration/main.go
In the end, you need to execute the below command. Please remember to replace my variable, if you are using your own.
./build --table example-migration-table --set example
If everything is fine, you should see on terminal:

Let's check the database state. Primo, we look at the table with migrations:

Take a look at records in the table:

To continue the migration script test, please run build script again. You should see output on your terminal below:

Still, we have only one fired migration. This time, executions didn't happen.

Let's add some new migration definitions to def_example file for better functionality tested.

// src/examples/ddb/defs_example.go

// DefsExample contains definitions of our migrations
// db - necessary to use DDB table
func DefsExample(ctx context.Context, db *dynamodb.Client) []migrator.Definition {
    return []migrator.Definition{
        {
            Name: "#3 example migration",
            Func: func() error {
                // Insert your table name in place of mine
                err := scripts.CreateRecord(ctx, db, "example-records-table")
                return err
            },
        },
        {
            Name: "#2 example migration",
            Func: func() error {
                // Insert your table name in place of mine
                err := scripts.CreateRecord(ctx, db, "example-records-table")
                return err
            },
        },
        // The First migration definition
        {
            Name: "#1 example migration",
            Func: func() error {
                // Insert your table name in place of mine
                err := scripts.CreateRecord(ctx, db, "example-records-table")
                return err
            },
        },
    }
}

Then please build and run migration ones again. You should see output below:

Let's take a look at our database table with migrations state:

The database with example records should be looking like this:

As you can see everything is fine. I hope your screen looks similar.

Conclusion

Our journey is coming to the end. The article covered how we can run and manage migrations with AWS and Golang.
We went through a simple code example. I work with a very similar setup almost every day, and I can assure you that this approach is very efficient and reliable.
The best thing is that you can use it in a relational database, like MySQL or PostgreSQL.
To do that, adjust the code base by changing a function that is passed to migration definitions. Then provide a suitable database client to that function. In the nearest future, I'll try to put an example with the RDS and the MySQL to show how easy it can be.

Blog

Golang's migration package for better database state management

Emil Buszyło

Table of Contents