Data migration projects - why and how
Data migration reasons
Sooner or later, if you are a database developer or a data architect, you will be faced with a data migration project. Data migrations are executed by organizations due to many different factors. Organizations try to avoid data migrations if they can because in many cases it is a costly and complicated process. Optimally, a data-driven application should work for many years without major changes, aside from only patching and updating its data store. Indeed, even upgrading a database version to a newer one can become quite a complicated project, well exceeding the backup-restore approach, so desired by the management and the development teams.
Here are a few of the many reasons why the migrations take place.
- regulatory requirements
- technology aging or going out of support
- performance issues
- changes in the data flow
- new reporting requirements
- optimization of the TCO
The advent of cloud platforms added an opportunity to migrate data stores owned by an organization to a serviced platform. Organizations decide to move their databases to the cloud platforms, either maintaining the core database technology or migrating to a completely different stack.
What are some of the considerations the data architects, developers, and implementation engineers have to make when migrating an organization's data?
Preparing for a data migration project
How does a team prepare for a data migration project?
Some of the questions include
- documentation of the source data schema and transformations
- documentation of the target data architecture
- transformation of the data models between source and target platforms
- modification of upper levels of the application
- testing
- deployment of the new model
- migration of the actual data
- data quality assurance
- operations support
Just a small elaboration on some of the above questions.
Documentation of database systems usually is obsolete, and this is if it exists at all. Database systems evolve, new features are added, and database schemas change, but these changes are often not reflected in the documentation. The first step that the team must do is to document the existing state of the art, and make sure it doesn't change while they work on the migration project!
In some cases, when the target technology is different, the architects have to answer questions about not only entity mapping but also data type mapping. While most technologies have compatible base types, there are situations where there are differences. Sometimes the data types in source and target technology require the data model change - for example, an enum data type in MySQL does not have a corresponding type in SQL Server, and while it can be simulated, it may make sense to change the entity model to optimize the data flows. If the migration is implemented the other way, it may be beneficial to replace SQL Server constructs with ones more adapted to MySQL.
I will be writing in the future about some of the problems data architects encounter in data migration projects.
Please leave your comment under the post if you find the topic interesting.
Stay tuned!
Leave a comment
Please note, comments must be approved before they are published