(go-redis-migrator) Migrate keys between Redis hosts or clusters using GO
I recently have been both working with GO and Redis a lot. When I came across the need to copy all the keys in a cluster to a single host (for use with development and testing), I set out to find a simple key migrator utility that supported clusters. Turns out, there aren’t any I could find! Clustering just came to Redis in version 3.0 and is still pretty new.
I used this opportunity to write some GO. GO had a package that supported Redis clustering and is fast enough to work with even large deployments… So I set to work. The result is go-redis-migrator. It’s pretty fast at 6,367 keys copied per second in my test!
First, get the dependencies:
go get -v
Then, make a build:
Finally, run the binary to see the help:
./go-redis-migrator -copyData -sourceHosts=172.31.37.164:6379,172.31.37.162:6379,172.31.37.168:6379,172.31.37.170:6379,172.31.37.169:6379 -destinationHosts=172.31.34.231:6379,172.31.34.228:6379,172.31.34.227:6379,172.31.34.230:6379,172.31.34.229:6379,172.31.34.226:6379 -keyFile=keyList.txt Migrated 57 keys.
Fetch a filtered key list from a host:
./go-redis-migrator -getKeys -keyFilter=sessions:\* -sourceHosts=127.0.0.1:6379
Fetch a filtered key list from a cluster:
./go-redis-migrator -getKeys -keyFilter=sessions:\* -sourceHosts=127.0.0.1:6379,127.0.0.1:6380
From a cluster to a single host:
./go-redis-migrator -copyData -sourceHosts=127.0.0.1:6379,127.0.0.1:6380 -destinationHosts=192.168.0.1:6379
Copying keys specified in a file from a host to a host:
./go-redis-migrator -copydata=true -sourceHosts=127.0.0.1:6379 -destinationHosts=192.168.0.1:6379 -keyFile=./onlyMoveTheseKeys.txt
Simply run the binary to get the following help:
- Redis Key Migrator - https://github.com/integrii/go-redis-migrator Migrates all or some of the keys from a Redis host or cluster to a specified host or cluster. Supports migrating TTLs. go-redis-migrator can also be used to list the keys in a cluster. Run with the -getKey and -sourceHosts= flags to do so. You must run this program with an operation before it will do anything. Flags: -getKeys=false: Fetches and prints keys from the source host. -copyData=false: Copies all keys in a list specified by -keyFile= to the destination cluster from the source cluster. -keyFile="": The file path which contains the list of keys to migrate. One per line. -keyFilter="*": The filter for which keys to migrate. Does not work when -keyFile is also specified. -destinationHosts="": A list of source cluster host:port servers seperated by spaces. Use a single host without a ',' if there is no cluster. EX) 127.0.0.1:6379,127.0.0.1:6380 -sourceHosts="": A list of source cluster host:port servers seperated by commas. Use a single host without a ',' if there is no cluster. EX) 127.0.0.1:6379,127.0.0.1:6380
- It would be nice to have pre-compiled binaries for people to download for their OS and architecture.
- Moving to SCAN to migrate keys would support super large deployments better.
- Multi-threading the copy process for super large migrations.
In case you missed it. Get it on Github here!