Data Recovery via Snapshot in Nebula Graph

In production, Nebula Graph handles massive data and high frequency business requests, therefore, faults caused by human, hardware or processing are inevitable.

Some fatal faults even lead to abnormal operation or data failure in the cluster. When such situation occurs, rebuilding cluster and reimporting data becomes rather time-consuming.

As a solution to this problem, Nebula Graph supports creating snapshot for the clusters. You first create a snapshot then use it to to restore the cluster to an available state when catastrophic failures take place.

Terms You Should Know

• Storage engine

Nebula Graph’s smallest physical storage unit, currently supports RocksDB and Hbase, this document is only for RocksDB.

• Partition

Nebula Graph’s smallest logical storage unit. A StorageEngine contains multiple partitions. A partition is divided into a leader and multiple followers, and Raft protocol is used to ensure data consistency between the leader and the followers.

• Graph space

Each graph space is an isolated graph unit that has its own tags and edges. A Nebula Graph cluster contains many graph spaces.

• Checkpoint

Checkpoints can be used as a point in time snapshot for the storage engine. Checkpoint can be used for full backup. Checkpoint file is a hard link for the sst file.

• Snapshot

The snapshot in this document refers to a snapshot that captures a point-in-time view of Nebula Graph cluster, i.e. a collection of the checkpoints for all the storage engines in the cluster. A cluster can be restored to the state when a certain snapshot is created via the snapshot.

• wal

Write-ahead Log (wal) is used by Raft to ensure the consistency between leaders and followers.

Architecture

Architecture Overview

Storage System Structure

Storage Sytem File Structure

Logic Analysis Processing

The CREATE SNAPSHOT is triggered with the client api or the console. The graph server parses the AST of the CREATE SNAPSHOT and sends the creation request to the meta server via the meta client. After receiving the request, the meta server first obtains all the active hosts and creates requests required by the adminClient. The creation requests are sent to each storage engine through the adminClient. After receiving the requests, the storage engine traverses all the storage engines of the specified spaces and creates checkpoint, then hard links the wals of all the partitions in storage engine. When creating checkpoint and the wal hard links, the database is read-only because the write blocking requests have been sent to all the leader partitions in advance.

Because the snapshot names are generated automatically with the system timestamp, you do not need to worry about renaming the snapshots. If you created unnecessary snapshots, you can delete them with the DROP SNAPSHOT command.

Create Snapshot

Create Checkpoint

Key Code Implementation

Create Snapshot

Create Checkpoint

User Guide

Create Snapshot

The CREATE SNAPSHOT command creates a snapshot at the current point in time for the whole cluster. The snapshot name is composed of the timestamp of the meta server.

If snapshot creation fails in the current version, you must use the DROP SNAPSHOT to clear the invalid snapshots. The current version does not support creating snapshot for the specified graph spaces, and executing CREATE SNAPSHOTcreates a snapshot for all graph spaces in the cluster. For example:

Show Snapshots

The command SHOW SNAPSHOT looks at the states (VALID or INVALID), names and the IP addresses of all storage servers when the snapshots are created in the cluster. For example:

Delete Snapshot

The DROP SNAPSHOT command deletes a snapshot with the specified name, the syntax is:

You can get the snapshot names with the command SHOW SNAPSHOTS. DROP SNAPSHOT can delete both valid snapshots and invalid snapshots that failed during creation. For example:

Now the deletes snapshot is not in the show snapshots list.

Tips

• When the system structure changes, it is better to create a snapshot immediately. For example, when you add host, drop host, create space, drop space or balance.
• The current version does not support automatic garbage collection for the failed snapshots in creation. We will develop cluster checker in meta server to check the cluster state via asynchronous threads and automatically collect the garbage files in failure snapshot creation.
• The current version does not support customized snapshot directory. The snapshots are created in the data_path/nebula directory by default.
• The current version does not support snapshot restore. Users need to write a shell script based on their actual productions to restore snapshots. The implementation logic is rather simple, you copy the snapshots of the engine servers to the specified folder, set this folder to data_path/, then start the cluster.