The world of application development is continually changing. Since the early 2000s, agile development processes introduced controlled change (not chaos) into the development process. Instead of long-run development projects that are difficult to change, agile projects make incremental changes in short sprints, from a few days to a few weeks. Database changes are a part of the agile process, and the migration process from one version to the next must be carefully documented along the way. Database migration processes and tools are critical ingredients in the agile methodology.

What is Database Migration?

Any change in a relational database requires a migration to the new design. This process includes any changes or modifications to the current database table. Keeping track of the database changes is an essential step in development and testing. Developers can write migration code in the programming language or use a migration tool to manage the process and write notes (aka artifacts) in the code to track those changes.

There are two types of database migration processes. The state-based migration process starts with a blank database and tracks the changes until the new database features are coded. Change-based migration begins with the current database table and adds the incremental stages on top of that table. The advantage of using change-based migration is the speed of development. The drawback is the need to look at each incremental step to fully view the database evolution.

Benefits of Database Migration

As applications are incrementally changed, databases need to be changed too. The database migration process keeps track of these changes to aid in the development and testing process. Migration tools create clear documentation of this process helps developers and users keep track of new versions for quality control purposes.

Database migrations are made for many reasons. One reason may be to improve application performance. Saving money is another reason for migrating to a new database structure. An application may have an old database structure, and upgrading to a new database may lower operating costs. Technology improvements are another way to migrate databases. A typical business case includes moving an on-premise application to a cloud-based application. The new cloud technology may have different data needs, and there needs to be a process to track and control these changes.

Cyber threats often target database tables. Security concerns are another reason to initiate database migrations. If the database structure is left unchanged, it may make it easier for hackers to make future attacks. Updating the data structure on a regular basis will help prevent future attacks.

Database Migration Challenges

Complex applications with multiple databases make the migration process difficult. It can be challenging to identify all of the databases that need updating. If the data isn’t correctly identified, schema mapping can prevent problems and extend the time and cost involved with the migration process.

According to Simform, “Enterprise database migration projects can cost up to $875K and experience more than $250K in cost overruns.” A significant reason for these overruns is the lack of an integrated strategy and project plan. Without proper advanced planning, change orders can add considerable time and costs.

What Happens During Database Migration?

Database applications need to be upgraded and migrated regularly. For instance, the database might need to add new properties or objects. Perhaps the business logic changed, and the dataset needs to be adjusted. These types of upgrades happen regularly for SaaS applications that use databases to power their server-side applications. Well organized migrations will create a cleaner database and even improve the response and productivity of the application.

Why use Database Migrations?

Database migrations can be used in the development stage of an application and the testing and deployment stage. Migration tools generate documentation threads from an empty state to an initial-level schema. During the testing and development stage, migration tools keep track of the test results and database changes until the application is deemed secure and ready for deployment.

How do you Perform A Database Migration?

Most developers use migration tools to create the scripts and artifacts necessary to track the database state changes. Many tools are available in the development platform you are using to write the program and SQL code. These tools tend to be popular since they integrate with the platform making it convenient for the developer.

Beyond the tools, database migration projects require planning and coordination between the entire development and testing team. If the team uses an agile strategy, daily stand-up calls and one to two-week sprints split the migration steps into manageable steps. The migration tool will create migration code in the platform’s language—the migration code is written in the same development language you are using. Artifacts should be clearly written, and you should avoid tools that create difficult to understand artifacts.

There are several features and factors to consider before investing in a database migration tool. The development platform you are using may include migration software. These tools are often a good choice because they integrate with the other development tools. The maturity of the tool is another consideration. The tool should have a good track record and a lot of positive reviews. It should also come with adequate documentation and technical support in case you run into technical snags. Another consideration is the artifacts the migration tool creates.

Database Migration Best Practices

Best practices develop based on the trials and errors experienced by fellow developers. For starters, migration scripts should be kept separate from the production code. Complex projects need pre-planning, which will make the process more efficient and less costly. Keeping your data clean will pay off with a smoother migration process. Conduct your cleansing and normalizing operations before you start your database migration project. You should test each migration step separately so you don’t confuse test results. It will be easier to run your migration project if you use a strategic plan and include data migration as part of your agile development process.

Conclusion

Database structures and schemas evolve as the applications they support change. Using an agile development process requires database migrations to happen on a continual basis to keep up with application updates. Providing a path of well-defined artifacts will keep you from missing critical steps during the development, test, and deployment stages. To improve the process, you can use migration tools integrated with popular development platforms. There are third-party tools that can be downloaded or purchased separately. You should keep your migration code separate from your development code and maintain accurate records to keep pace with the rapid development process.