Migrate between Redis instances and clusters easily. Get Redis Migrator on Github here!

Introduction

I recently have been both working with go and Redis a lot. When I came across the need to copy all the keys in a cluster to a single host (for use with development and testing), I set out to find a simple key migrator utility that supported clusters. Turns out, there aren’t any I could find! Clustering just came to Redis in version 3.0 and is still pretty new.

I used this opportunity to write some GO. GO had a package that supported Redis clustering and is fast enough to work with even large deployments… So I set to work. The result is go-redis-migrator. It’s pretty fast at 6,367 keys copied per second in my test!

Setup

First, get the dependencies: go get -v

Then, make a build: go build

Finally, run the binary to see the help: ./go-redis-migrator

Examples

./go-redis-migrator -copyData -sourceHosts=172.31.37.164:6379,172.31.37.162:6379,172.31.37.168:6379,172.31.37.170:6379,172.31.37.169:6379 -destinationHosts=172.31.34.231:6379,172.31.34.228:6379,172.31.34.227:6379,172.31.34.230:6379,172.31.34.229:6379,172.31.34.226:6379 -keyFile=keyList.txt 
Migrated 57 keys.

Fetch a filtered key list from a host:

./go-redis-migrator -getKeys -keyFilter=sessions:\* -sourceHosts=127.0.0.1:6379

Fetch a filtered key list from a cluster:

./go-redis-migrator -getKeys -keyFilter=sessions:\* -sourceHosts=127.0.0.1:6379,127.0.0.1:6380

From a cluster to a single host:

./go-redis-migrator -copyData -sourceHosts=127.0.0.1:6379,127.0.0.1:6380 -destinationHosts=192.168.0.1:6379

Copying keys specified in a file from a host to a host:

./go-redis-migrator -copydata=true -sourceHosts=127.0.0.1:6379 -destinationHosts=192.168.0.1:6379 -keyFile=./onlyMoveTheseKeys.txt

Simply run the binary to get the following help:

- Redis Key Migrator - 
https://github.com/integrii/go-redis-migrator

Migrates all or some of the keys from a Redis host or cluster to a specified host or cluster.  Supports migrating TTLs.
go-redis-migrator can also be used to list the keys in a cluster.  Run with the -getKey and -sourceHosts= flags to do so.

You must run this program with an operation before it will do anything.

Flags:
  -getKeys=false: Fetches and prints keys from the source host.
  -copyData=false: Copies all keys in a list specified by -keyFile= to the destination cluster from the source cluster.
  -keyFile="": The file path which contains the list of keys to migrate.  One per line.
  -keyFilter="*": The filter for which keys to migrate.  Does not work when -keyFile is also specified.
  -destinationHosts="": A list of source cluster host:port servers seperated by spaces. Use a single host without a ',' if there is no cluster. EX) 127.0.0.1:6379,127.0.0.1:6380
  -sourceHosts="": A list of source cluster host:port servers seperated by commas. Use a single host without a ',' if there is no cluster. EX) 127.0.0.1:6379,127.0.0.1:6380

Contribution Wanted

• It would be nice to have pre-compiled binaries for people to download for their OS and architecture.
• Moving to SCAN to migrate keys would support super large deployments better.
• Multi-threading the copy process for super large migrations.

• Share with