Introducing Prom-migrator

A universal, open-source Prometheus data migration tool
We've built a brand-new, 100% free, open-source tool that makes it easy to migrate your Prometheus metrics data to and from various long-term storage systems. Learn how it works and get started.

Prometheus is a metric collection, alerting, and storage system. It is architected to store data locally in its own time-series database (TSDB) and/or to ship data to other long-term storage systems via its remote_write/remote_read APIs. Over time, many long-term storage systems have emerged, and, as a result, users can have data stored in a variety of systems. For example, we recently shipped Promscale, an analytical platform and long-term store for Prometheus, built on top of TimescaleDB. But switching between these storage systems has not been possible because, until now, there has been no universal tool to migrate data between them.

Prom-migrator solves this problem. It is a universal, open-source Prometheus data migration tool that is community-driven and free to use. Prom-migrator migrates data from one remote-storage system to another remote-storage system, leveraging Prometheus’ remote storage API. This means that this tool can be used to migrate data from any storage system that supports the Prometheus remote_read protocol. Similarly, it can migrate data to any storage system that supports the Prometheus remote_write protocol.

Why did we build Prom-migrator?

Prometheus has many remote-storage systems and this has been growing over time. However, there was no universal tool for data migration, leaving users with few good options if they wanted to switch between different remote-storage systems:

• They could throw away the old data they had in their previous system. This led to gaps in historical knowledge about a system.

• They could continue to run both the old and the new system and try to redirect relevant queries to either system. This led to operational and data management headaches.

• They could continue using the previous system and not switch. This is classic vendor lock-in.

Prom-migrator features

Prom-migrator offers several new features:

• Data migration from and to any storage system. This tool is designed to work with any remote storage system, so that users can migrate data across any system in a wide range of scenarios.
• Informative outputs during runtime, allowing users to track progress. Prom-migrator keeps users informed about the migration progress, so that users can plan their time accordingly.
• Ability to resume migration(s) in case of any unintended shutdowns. Keep a record of migration progress and, in the case failure or interruptions, automatically resume the migration where you left off.
• Stateless working model. Easy deployment due to the fact that the migrator does not need to keep state. No need to worry about mounting volumes or attaching persistent storage.

Find out more

Please see our blog post where we share why we built Prom-migrator, including the problems it solves, which systems it is compatible with, and how it works. We'll also take you through two short examples to show how you'd use Prom-migrator in two very different scenarios – and, in the process, hopefully inspire you to give it a try yourself.

Or got right to the GitHub repo to get started.