MySQL 大数据操作注意事项

时间:2022-05-03
本文章向大家介绍MySQL 大数据操作注意事项，主要内容包括MySQL 大数据操作注意事项、1. 关于 delete、2. 关于 update、3. 关于创建索引、4. 关于 OPTIMIZE、5. 关于切换引擎、6. 确保SELECT不被受阻、基本概念、基础应用、原理机制和需要注意的事项等，并结合实例形式分析了其使用技巧，希望通过本文能帮助到大家理解应用这部分内容。

MySQL 大数据操作注意事项

http://netkiller.github.io/journal/mysql.parallel.html

摘要

目录

  • 1. 关于 delete
  • 2. 关于 update
  • 3. 关于创建索引
  • 4. 关于 OPTIMIZE
  • 5. 关于切换引擎
  • 6. 确保SELECT不被受阻

1. 关于 delete

delete from mytable 必死无疑，你需要分批删除，尽量缩小每个批次删除的记录数，delete 是可以并行执行的，你可以同时运行多个删除操作

		mysql> show processlist;
+--------+-----------------+---------------------+-----------+---------+-------+-----------------------------+--------------------------------------------------------+
| Id     | User            | Host                | db        | Command | Time  | State                       | Info                                                   |
+--------+-----------------+---------------------+-----------+---------+-------+-----------------------------+--------------------------------------------------------+
|      1 | event_scheduler | localhost           | NULL      | Daemon  |    52 | Waiting for next activation | NULL                                                   |
| 115986 | dba             | localhost           | example   | Query   |     0 | NULL                        | show processlist                                       |
| 117446 | dba             | localhost           | example   | Query   |    20 | updating                    | delete from mytable where OPEN_TIME like '2011.11.28%' |
| 117525 | dba             | localhost           | example   | Query   |     2 | updating                    | delete from mytable where OPEN_TIME like '2011.12.02%' |
| 117526 | dba             | localhost           | example   | Query   |    49 | updating                    | delete from mytable where OPEN_TIME like '2011.12.12%' |
| 117527 | dba             | localhost           | example   | Query   |     6 | updating                    | delete from mytable where OPEN_TIME like '2011.12.21%' |
| 117528 | dba             | localhost           | example   | Query   |    64 | updating                    | delete from mytable where OPEN_TIME like '2011.12.30%' |
| 117546 | dba             | localhost           | example   | Query   |    33 | updating                    | delete from mytable where OPEN_TIME like '2011.11.10%' |
+--------+-----------------+---------------------+-----------+---------+-------+-----------------------------+--------------------------------------------------------+
23 rows in set (0.00 sec)

2. 关于 update

在电商领域常常遇到一个问题“调价”，经常需要调整一批商品的价格, 程序猿一条语句搞定有没有？

update goods set price=price+10 where category_id = xxx

在开发，测试环境是可以通过测试的，一旦部署到生产环境，必死无疑

3. 关于创建索引

大表创建索引需要很久的时间，通常要经历 manage keys 与 copy to tmp table 的过程

mysql> show processlist;
+--------+-----------------+---------------------+----------+---------+-------+-----------------------------+------------------------------------------------------------------+
| Id     | User            | Host                | db       | Command | Time  | State                       | Info                                                             |
+--------+-----------------+---------------------+----------+---------+-------+-----------------------------+------------------------------------------------------------------+
|      1 | event_scheduler | localhost           | NULL     | Daemon  |    47 | Waiting for next activation | NULL                                                             |
| 115986 | dba             | localhost           | example  | Query   |     0 | NULL                        | show processlist                                                 |
| 118814 | dba             | 192.168.6.20:50459  | example  | Query   |     8 | copy to tmp table           | ALTER TABLE `mytable` ADD INDEX `modifiy_time` (`MODIFY_TIME`)   |
+--------+-----------------+---------------------+----------+---------+-------+-----------------------------+------------------------------------------------------------------+
17 rows in set (0.00 sec)

删除索引，也需要经理 copy to tmp table 过程，漫长的等待

		mysql> show processlist;
+--------+-----------------+---------------------+--------------+---------+-------+-----------------------------+-------------------------------------------------+
| Id     | User            | Host                | db           | Command | Time  | State                       | Info                                            |
+--------+-----------------+---------------------+--------------+---------+-------+-----------------------------+-------------------------------------------------+
|      1 | event_scheduler | localhost           | NULL         | Daemon  |    11 | Waiting for next activation | NULL                                            |
| 115986 | dba             | localhost           | example      | Query   |     0 | NULL                        | show processlist                                |
| 118814 | dba             | 192.168.6.20:50459  | example      | Query   |     4 | copy to tmp table           | ALTER TABLE `mytable`	DROP INDEX `modifiy_time` |
+--------+-----------------+---------------------+--------------+---------+-------+-----------------------------+-------------------------------------------------+
17 rows in set (0.00 sec)

所以数据设计要深思熟虑，做到提前未雨绸缪，不要亡羊补牢

4. 关于 OPTIMIZE

OPTIMIZE 的操作是将当前表复制到临时表操作后再删除当前表，最后将临时表改名

		mysql> show processlist;
+--------+-----------------+---------------------+---------------------------+---------+-------+-----------------------------+--------------------------+
| Id     | User            | Host                | db                        | Command | Time  | State                       | Info                     |
+--------+-----------------+---------------------+---------------------------+---------+-------+-----------------------------+--------------------------+
|      1 | event_scheduler | localhost           | NULL                      | Daemon  |    14 | Waiting for next activation | NULL                     |
| 115835 | dba             | 192.168.6.20:49664  | example                   | Query   |     9 | copy to tmp table           | OPTIMIZE TABLE `mytable` |
| 115986 | dba             | localhost           | example                   | Query   |     0 | NULL                        | show processlist         |
+--------+-----------------+---------------------+---------------------------+---------+-------+-----------------------------+--------------------------+
17 rows in set (0.00 sec)

5. 关于切换引擎

转换ENGINE从MyISAM到InnoDB会经历creating table然后copy to tmp table在修改表名几个阶段，过程非常缓慢

		mysql> show processlist;
+------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+
| Id   | User            | Host                | db      | Command | Time  | State                       | Info                                     |
+------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+
|    1 | event_scheduler | localhost           | NULL    | Daemon  |    10 | Waiting for next activation | NULL                                     |
| 3167 | dba             | 192.168.6.20:56723  | example | Query   |     2 | creating table              | ALTER TABLE `mytable`	ENGINE=InnoDB   |
| 3172 | dba             | localhost           | example | Query   |     0 | NULL                        | show processlist                         |
+------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+
18 rows in set (0.00 sec)

copy to tmp table 过程

		mysql> show processlist;
+------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+
| Id   | User            | Host                | db      | Command | Time  | State                       | Info                                     |
+------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+
|    1 | event_scheduler | localhost           | NULL    | Daemon  |    21 | Waiting for next activation | NULL                                     |
| 3167 | dba             | 192.168.6.20:56723  | example | Query   |    13 | copy to tmp table           | ALTER TABLE `mytable`	ENGINE=InnoDB   |
| 3172 | dba             | localhost           | example | Query   |     0 | NULL                        | show processlist                         |
+------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+
18 rows in set (0.00 sec)

此时我们查看mysql data目录会看到临时表文件

# ll /var/lib/mysql/hx9999_real_history/
		
-rw-rw---- 1 mysql mysql      9522 May 16 17:17 #sql-c2f_c5f.frm
-rw-rw---- 1 mysql mysql        48 May 16 17:17 #sql-c2f_c5f.par
-rw-rw---- 1 mysql mysql 637534208 May 16 17:29 #sql-c2f_c5f#P#p0.ibd
-rw-rw---- 1 mysql mysql    180224 May 16 17:17 #sql-c2f_c5f#P#p1.ibd
-rw-rw---- 1 mysql mysql    180224 May 16 17:17 #sql-c2f_c5f#P#p2.ibd
-rw-rw---- 1 mysql mysql    180224 May 16 17:17 #sql-c2f_c5f#P#p3.ibd
-rw-rw---- 1 mysql mysql    180224 May 16 17:17 #sql-c2f_c5f#P#p4.ibd
-rw-rw---- 1 mysql mysql    180224 May 16 17:17 #sql-c2f_c5f#P#p5.ibd
-rw-rw---- 1 mysql mysql    180224 May 16 17:17 #sql-c2f_c5f#P#p6.ibd
-rw-rw---- 1 mysql mysql    180224 May 16 17:17 #sql-c2f_c5f#P#p7.ibd

6. 确保SELECT不被受阻

使用各种手段保证select操作不被受阻，只要select一直可以查询网站前端就能提供80%的功能，一旦select受阻一切都是浮云。

保证 select 操作优先于其他操作

UPDATE [LOW_PRIORITY] [IGNORE] tbl_name  
SET col_name1=expr1 [, col_name2=expr2 ...]  
[WHERE where_definition]  
[ORDER BY ...]  
[LIMIT row_count]

update的时候增加 LOW_PRIORITY 参数，可以降低更新语句的优先级。

my.cnf

[mysqld]		
low_priority_updates=1

或者启动是添加--low-priority-updates参数

全局开启

SET @@global.low_priority_updates = 1;

适用于本次会话连接

SET @@session.low_priority_updates = 1;

随机文章
mysql知识点必读
最近更新