Based on Python script, realize Redis data migration (self-built Redis migrate to Tencent Cloud brings Redis itself)
1. Reference URL:https://blog.csdn.net/zhanghan18333611647/article/details/81434786. Implemented through the program (Python version 2.x ). Create a new migrate.py file.
#!/usr/bin/env python
# -*- coding:utf-8 -*-
# 把redis里面一个数据库的东西，迁移到另外一个数据库里面
# 建立两个redis连接
# 获取到所有的key .keys()
# 判断key的类型 string\hash\list\set
import redis
#redis_from源数据，redis_to目标数据 redis_from——>redis_to
redis_from = redis.StrictRedis(host='ip',port=6379,password='',db=10)
redis_to = redis.StrictRedis(host='ip',port=6379,password='',db=0)
if __name__ == '__main__':
cnt = 0
scnt = 0
lcnt = 0
setcnt = 0
hcnt = 0
for k in redis_from.keys():
# 循环keys里面每一个key
data_type = redis_from.type(k)
# 判断key的类型 string\hash\list\set
if data_type == 'string':
v = redis_from.get(k)
t = redis_from.ttl(k)
redis_to.set(k, v)
if int(t) > 0:
redis_to.expire(k,t)
scnt = scnt + 1
elif data_type == 'list':
values = redis_from.lrange(k, 0, -1)
t = redis_from.ttl(k)
redis_to.lpush(k, values)
if int(t) > 0:
redis_to.expire(k,t)
lcnt = lcnt + 1
elif data_type == 'set':
values = redis_from.smembers(k)
t = redis_from.ttl(k)
redis_to.sadd(k, values)
if int(t) > 0:
redis_to.expire(k,t)
setcnt = setcnt + 1
elif data_type == 'hash':
hcnt = hcnt + 1
keys = redis_from.hkeys(k)
for key in keys:
value = redis_from.hget(k, key)
t = redis_from.ttl(k)
redis_to.hset(k, key, value)
if int(t) > 0:
redis_to.expire(k,t)
else:
print 'not known type'
print data_type
file_object = open('/data/thefile.txt','a+')
file_object.write(data_type)
file_object.write('\n')
file_object.close( )
cnt = cnt + 1
print 'total', cnt
print 'string', scnt
print 'list', lcnt
print 'set', setcnt
print 'hash', hcnt
2. On the self-built Redis machine, connect the self-built Redis and Tencent Cloud come with Redis. Tencent Cloud’s own redis database 0 is empty. It is planned to use this database as the migration target. as shown in Figure 1
Figure 1
3. Check the Python version, Python 2.7.5. Version meets the requirements. as shown in Figure 2
Figure 2
[root@iz2zeeh3rrqalw11oajd6zz ~]# python
Python 2.7.5 (default, Nov 16 2020, 22:23:17)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
4. Connect to self-built Redis, check the total number of health under database 0: 40.
[root@iz2zeeh3rrqalw11oajd6zz ~]# redis-cli -h 127.0.0.1 -p 6379 -a 2021Chinamcloud
127.0.0.1:6379> select 0
OK
127.0.0.1:6379> dbsize
(integer) 40
127.0.0.1:6379>
5. Run python migrate.py. Import error: ImportError: There is no module named redis. as shown in Figure 3
Figure 3
[root@iz2zeeh3rrqalw11oajd6zz ~]# python migrate.py
Traceback (most recent call last):
File "migrate.py", line 9, in
import redis
ImportError: No module named redis
[root@iz2zeeh3rrqalw11oajd6zz ~]#
6. After installing redis-py. Keep running python migrate.py.
[root@iz2zeeh3rrqalw11oajd6zz ~]# python migrate.py
Traceback (most recent call last):
File "migrate.py", line 39, in
redis_to.lpush(k, values)
File "/usr/lib/python2.7/site-packages/redis/client.py", line 1961, in lpush
return self.execute_command('LPUSH', name, *values)
File "/usr/lib/python2.7/site-packages/redis/client.py", line 900, in execute_command
conn.send_command(*args)
File "/usr/lib/python2.7/site-packages/redis/connection.py", line 725, in send_command
self.send_packed_command(self.pack_command(*args),
File "/usr/lib/python2.7/site-packages/redis/connection.py", line 775, in pack_command
for arg in imap(self.encoder.encode, args):
File "/usr/lib/python2.7/site-packages/redis/connection.py", line 120, in encode
"bytes, string, int or float first." % typename)
redis.exceptions.DataError: Invalid input of type: 'list'. Convert to a bytes, string, int or float first.
[root@iz2zeeh3rrqalw11oajd6zz ~]#
7. Check the Redis version separately. The version number of self-built Redis: 3.2.12. Tencent Cloud comes with the version number of Redis: 2.8.23. Version numbers are inconsistent. Figure 4, Figure 5
Figure 4
Figure 5
[root@iz2zeeh3rrqalw11oajd6zz ~]# redis-cli -h 127.0.0.1 -p 6379 -a 2021Chinamcloud
127.0.0.1:6379> select 0
OK
127.0.0.1:6379> info
# Server
redis_version:3.2.12
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:7897e7d0e13773f
redis_mode:standalone
os:Linux 3.10.0-1160.11.1.el7.x86_64 x86_64
RDM Redis Console
连接中...
已连接。
腾讯云(公司开发环境):0>info
"# Server
redis_version:2.8.23
run_id:0a99b4d4dac9744a76075aa77a978a2dfcdf1ba6
uptime_in_seconds:74995115
uptime_in_days:867
hz:10
lru_clock:8808554
8. Python to view the Redis installation version. Create a new file python_redis.py and see the version 3.5.3. as shown in Figure 6
Figure 6
import redis
print redis.VERSION
9. Execute the command: pip install redis==2.10.6. Fallback version to 2.10.6. as shown in Figure 7
Figure 7
10. After rolling back to 2.10.6, continue to run python migrate.py. Data migration was successful. The reason why the total number of health under database 0 is migrated: 53, because the data in the self-built Redis is caused by the change. as shown in Figure 8
8
[root@iz2zeeh3rrqalw11oajd6zz ~]# python python_redis.py
(2, 10, 6)
[root@iz2zeeh3rrqalw11oajd6zz ~]# python migrate.py
total 53
string 25
list 3
set 0
hash 25
11. Check the data under database 0 in Tencent Cloud’s own redis. Total: 53. Confirm the successful migration. When Tencent Cloud comes with redis 4.0 version, it can also be successfully migrated. as shown in Figure 9
Figure 9
