High iowait on Amazon EC2 MySQL instance with EBS volume
Solution 1
It would help if you posted your my.cnf and whether you are using InnoDB or MyISAM tables and whether you are a read-heavy or write-heavy workload. Otherwise, we are just making guesses. Here are mine:
First, I would look and make sure that your queries are properly indexed. High I/O on MySQL databases is either caused by extremely high concurrency, a poorly tuned server, or by poorly performing queries that have to do full table or index scans. Some hints on how to find the poorly performing queries can be found in my post on Ideeli's technical blog.
Check your my.cnf. If you are using InnoDB, make sure innodb_buffer_pool_size and innodb_log_file_size are sufficiently large. Because EBS has such variable latency, maxing out innodb_log_file_size can have substantial performance benefits. If you are using MyISAM (and you shouldn't), make sure your key_buffer size is large enough.
If you are confident that your queries are well optimized, and your server is well-tuned, we can move on to the next item. ext3 is less than ideal for databases. One of the main reasons for this is that ext3 only allows a single thread to update an inode at a time (trying to find documentation for this). If you aren't running with innodb-file-per-table, this means that there is a ton of filesystem contention on the ibdata file. xfs does not have this limitation and has been shown to perform much better (need source) for database workloads.
If you can't change to xfs, make sure that you are using innodb-file-per-table and at a minimum, make sure that you have noatime,nodiratime on the mount.
Next, on to your instance size. A c1.medium is not an ideal instance size for most databases unless the data set is tiny. MySQL ordinarily will benefit from memory over compute power. c1.medium only has 1.7GB of RAM! How big is your data set? In general, a m1.large (with 7.5GB of RAM) will outperform a c1.medium except in very rare cases. It is also twice as expensive, at $0.34/hr.
Now on to RAID of EBS volumes. Yes, RAID will greatly increase your IOPS. (As will increasing your instance size). Do not RAID0...If you care about your data, at least. I have explained this in many places, including on my blog, as a speaker at Percona Live NYC in 2011, and here on serverfault. The short version is that EBS volumes fail in atypical ways and being able to remove a volume from the set has proven to be valuable on serveral occasions, most notably during the great EBS outage of 2011 where some sites were offline for several days...We were offline for 45 minutes at 4AM despite having dozens of instances affected by the EBS issue.
Here are some benchmarks for RAIDed EBS volumes using MySQL.
Finally, Percona Server has a huge number of scalability optimizations. Here is a white paper about my company's experience when switching from MySQL to Percona Server. We were experiencing database stalls and outages every day. Simply switching to Percona Server from MySQL resolved that issue literally overnight due to a number of scalability improvements.
So, in summary...
- Tune your queries
- Tune your server
- Get yourself better "hardware"
- Use xfs, not ext3
- RAID10, not RAID0
- Switch from MySQL to Percona Server
As for MySQL Cluster, it's a completely different animal than MySQL and generally not suitable for most OLTP applications. Galera/Percona XtraDB Cluster are new and interesting clustering products, as well. You have a lot of options before you get to any of this, however. We served 24k qps at peak from a single m2.4xlarge with RAID10 in EC2.
Good luck!
Solution 2
Since 500gps is a fairly mild load on the sql server i suggest looking at the percentage of temp tables created on disk and start optimizing your queries and MySQL Server Settings.
1, Do not do the Raid0 approach, it will eventually fail and you will regret it.
2, No, at this low number of queries per second you do not need MySQL Cluster.
3, Yes, it surely does affect App Performance, to measure how much you could enable the slow log and see for yourself.
How much memory is mysql using currently, is there any headroom left?
If not you should consider switching to a bigger instance and start optimizing settings with sundry's mysql tuning script:
http://www.day32.com/MySQL/tuning-primer.sh
Solution 3
This is a problem many, many companies have run into, and solutions to it are fairly well-discussed on various online forums.
Typically to increase the potential iops, two or more EBS volumes are joined together in a RAID0 array. This doesn't come without risk, though. As you know, with RAID0, all it takes is for one of the member EBS volumes to have an issue and your data is toast. As such, you may consider using a more resilient RAID level, say RAID 10 perhaps.
Solution 4
3- Do you believe such scenario is heavily impacting our apps? Would they perform much better in case we move to a RAID 0 and/or cluster solution?
Since you are running an SQL server, it would make more sense to take a look at the SQL server metrics instead to know if queries are served quickly. Looking at your single-digit average request wait times (await), I do not think I/O would be much of a concern yet.
Also, as what you mostly see is read load, you could reduce it by having a larger cache / increasing the amount of RAM and tuning the cache parameters of your MySQL instance. I would expect this to have a significantly larger performance impact than having your storage changed to handle more I/Os.
Luis Fernando Alen
Updated on September 18, 2022Comments
-
Luis Fernando Alen almost 2 years
We have a MySQL server running on an Amazon EC2 c1.medium instance relying on a single EBS volume using the ext3 filesystem for storage.
This MySQL server is queried ~500/ps by several applications running on some web servers, which are also at the Amazon EC2.
As you can see below, server's load avg and processor idle time seems fine, but there's something disturbing and worrying me right now, which is the high iowait it's been experiencing.
Another number that worried me a lot is the number of iostat's transfers per second (tps), which stays above 450 most of the time. After doing some research on this topic, I saw some people saying that this is ask too much of a EBS volume: https://forums.aws.amazon.com/thread.jspa?threadID=30769
By the way, the command outputs you'll see below were not captured in a peak time. That's the way the server behaves/performs most of the time.
Well, all said, here go my questions:
1- Is it time to consider moving for a RAID architecture (I'd say RAID 0)?
2- Should I spent time on a clustering solution such as MySQL Cluster?
3- Do you believe such scenario is heavily impacting our apps? Would they perform much better in case we move to a RAID 0 and/or cluster solution? (Seems that the apps are happy so far, but would they be happier?)
Please let me know if you need any further information.
################################################## my.cnf~ # uptime 12:34:14 up 2 days, 4:06, 1 user, load average: 2.24, 1.90, **1.84** ######################################################## ~ # vmstat 5 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id **wa** st 0 1 52 11168 16420 1498728 0 0 4586 231 11 81 6 3 52 39 0 2 1 52 10460 16320 1499588 0 0 11631 397 3194 4319 10 4 47 39 0 4 1 52 11448 16064 1499156 0 0 12231 592 2301 3331 9 5 50 36 0 4 0 52 10328 16068 1500176 0 0 8578 392 2131 2745 8 6 49 37 0 0 1 52 11164 15732 1499928 0 0 9604 578 2609 3510 7 4 49 40 0 0 1 52 10824 15768 1499836 0 0 5038 634 1912 2509 8 3 47 42 0 3 1 52 12040 15888 1498096 0 0 5068 204 1927 2531 10 8 45 37 0 8 2 52 11252 15784 1499272 0 0 8521 390 2437 3100 14 15 39 31 0 1 2 52 11436 15724 1499748 0 0 8287 401 2159 3113 11 10 42 36 1 0 1 52 12016 15704 1498752 0 0 11576 499 3324 3984 16 17 31 36 0 1 1 52 10536 15664 1500508 0 0 8430 718 2686 3265 15 14 37 34 0 1 1 52 10300 15676 1500744 0 0 10186 720 2488 3488 16 5 45 34 0 ######################################################## ~ # iostat -dm 5 /dev/sdf Linux 2.6.21.7-2.fc8xen (database-new) 01/20/12 Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sdf 464.81 8.84 0.33 1658860 61390 Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sdf 402.20 7.39 0.43 36 2 Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sdf 431.40 7.74 0.32 38 1 Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sdf 461.40 8.26 0.39 41 1 Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sdf 475.65 9.20 0.29 46 1 Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sdf 534.80 9.82 0.52 49 2 Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sdf 526.60 9.97 0.52 49 2 ######################################################## ~ # iostat -mdx 5 /dev/sdf Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sdf 22.21 46.28 427.47 37.54 8.84 0.33 40.38 1.78 3.82 1.72 79.87 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sdf 22.36 80.04 450.30 60.48 9.29 0.55 39.44 1.45 2.85 1.58 80.48 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sdf 23.40 43.60 370.60 47.00 7.75 0.35 39.76 1.45 3.47 1.97 82.08 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sdf 20.20 33.20 382.60 29.60 8.02 0.25 41.05 1.31 3.17 2.11 87.12 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sdf 28.80 35.20 422.40 33.40 9.04 0.27 41.80 1.45 3.19 1.95 88.96 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sdf 14.20 45.00 291.80 51.40 5.97 0.38 37.86 1.45 4.22 2.50 85.68 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sdf 19.16 56.89 535.33 41.32 11.44 0.38 42.00 1.49 2.59 1.53 88.46 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sdf 20.40 81.40 233.00 64.40 4.86 0.57 37.39 1.74 5.84 3.18 94.72
################################################## sundry's tuning script output[mysqld] datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock user=mysql long_query_time=1 key_buffer = 64M thread_cache_size = 30 table_cache = 1024 table_definition_cache = 512 query_cache_type = 1 query_cache_size = 64M tmp_table_size = 64M max_heap_table_size = 64M innodb_buffer_pool_size = 512M old_passwords=1 max_connections=400 wait_timeout=30 [mysqld_safe] log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid [ndbd] connect-string="nodeid=2;host=localhost:1186" [ndb_mgm] connect-string="host=localhost:1186"
~ # ./tuning-primer.sh -- MYSQL PERFORMANCE TUNING PRIMER -- - By: Matthew Montgomery - MySQL Version 5.1.52 i686 Uptime = 0 days 1 hrs 1 min 1 sec Avg. qps = 517 Total Questions = 1894942 Threads Connected = 94 Warning: Server has not been running for at least 48hrs. It may not be safe to use these recommendations To find out more information on how each of these runtime variables effects performance visit: http://dev.mysql.com/doc/refman/5.1/en/server-system-variables.html Visit http://www.mysql.com/products/enterprise/advisors.html for info about MySQL's Enterprise Monitoring and Advisory Service SLOW QUERIES The slow query log is NOT enabled. Current long_query_time = 1.000000 sec. You have 207 out of 1894981 that take longer than 1.000000 sec. to complete Your long_query_time seems to be fine BINARY UPDATE LOG The binary update log is NOT enabled. You will not be able to do point in time recovery See http://dev.mysql.com/doc/refman/5.1/en/point-in-time-recovery.html WORKER THREADS Current thread_cache_size = 30 Current threads_cached = 8 Current threads_per_sec = 0 Historic threads_per_sec = 0 Your thread_cache_size is fine MAX CONNECTIONS Current max_connections = 400 Current threads_connected = 93 Historic max_used_connections = 195 The number of used connections is 48% of the configured maximum. Your max_connections variable seems to be fine. INNODB STATUS Current InnoDB index space = 1.33 G Current InnoDB data space = 5.04 G Current InnoDB buffer pool free = 0 % Current innodb_buffer_pool_size = 512 M Depending on how much space your innodb indexes take up it may be safe to increase this value to up to 2 / 3 of total system memory MEMORY USAGE Max Memory Ever Allocated : 1.13 G Configured Max Per-thread Buffers : 1.04 G Configured Max Global Buffers : 642 M Configured Max Memory Limit : 1.67 G Physical Memory : 1.70 G Max memory limit exceeds 90% of physical memory KEY BUFFER Current MyISAM index space = 379 M Current key_buffer_size = 64 M Key cache miss rate is 1 : 162 Key buffer free ratio = 80 % Your key_buffer_size seems to be fine QUERY CACHE Query cache is enabled Current query_cache_size = 64 M Current query_cache_used = 43 M Current query_cache_limit = 1 M Current Query cache Memory fill ratio = 67.44 % Current query_cache_min_res_unit = 4 K MySQL won't cache query results that are larger than query_cache_limit in size SORT OPERATIONS Current sort_buffer_size = 2 M Current read_rnd_buffer_size = 256 K Sort buffer seems to be fine JOINS Current join_buffer_size = 132.00 K You have had 4013 queries where a join could not use an index properly You should enable "log-queries-not-using-indexes" Then look for non indexed joins in the slow query log. If you are unable to optimize your queries you may want to increase your join_buffer_size to accommodate larger joins in one pass. Note! This script will still suggest raising the join_buffer_size when ANY joins not using indexes are found. OPEN FILES LIMIT Current open_files_limit = 2458 files The open_files_limit should typically be set to at least 2x-3x that of table_cache if you have heavy MyISAM usage. Your open_files_limit value seems to be fine TABLE CACHE Current table_open_cache = 1024 tables Current table_definition_cache = 512 tables You have a total of 45237 tables You have 1024 open tables. Current table_cache hit rate is 0% , while 100% of your table cache is in use You should probably increase your table_cache You should probably increase your table_definition_cache value. TEMP TABLES Current max_heap_table_size = 64 M Current tmp_table_size = 64 M Of 38723 temp tables, 44% were created on disk Perhaps you should increase your tmp_table_size and/or max_heap_table_size to reduce the number of disk-based temporary tables Note! BLOB and TEXT columns are not allow in memory tables. If you are using these columns raising these values might not impact your ratio of on disk temp tables. TABLE SCANS Current read_buffer_size = 128 K Current table scan ratio = 537 : 1 read_buffer_size seems to be fine TABLE LOCKING Current Lock Wait ratio = 1 : 954 You may benefit from selective use of InnoDB. If you have long running SELECT's against MyISAM tables and perform frequent updates consider setting 'low_priority_updates=1' If you have a high concurrency of inserts on Dynamic row-length tables consider setting 'concurrent_insert=2'.
-
Luis Fernando Alen over 12 yearsThanks for you reply, ErikA. I know that many companies run into this problem and RAID0 is by no means a safe approach, although it's quite simple to configure and performs very well. I also know that in order to solve this issue a more sophisticated solution such as RAID and clustering is required. However, what I'm really asking here is: Is it time to put my efforts on that? Should I be worrying that much? I ask that because I see the processor idle half of the time and the server's load is also fine. Such scenario is somehow different from the ones I see all over the web.
-
EEAA over 12 yearsIf it's causing performance problems with your application, then yes, it's probably good to start thinking about it. If not, well then don't worry about it.
-
Luis Fernando Alen over 12 yearsHello, syneticon-dj. We already have some parameters set on MySQL for caching such as query_cache_size and table_cache but I'll dig deeper into MySQL tuning options in order to make better use of the available RAM and unburden the EBS volume. Thanks for the tip.
-
the-wabbit over 12 years@LuisFernandoAlen table_cache is not doing what you think it is. If you are using MyISAM tables, there is no explicit data cache beyond the OS filesystem cache. An index cache is maintained and controlled by the key_buffer_size global variable. But with your instance having just 1.7 GB of RAM, migrating to a "Large" EC2 instance would probably have the best visible effect without an awful lot of tuning.
-
Admin over 12 yearsWrite performance under Percona XtraDB Cluster / Galera actually doesn't degrade as you might expect; it can actually improve, especially on slow I/O like EBS, because writing to a remote server's RAM is faster than writing to local "disk". Henrik Ingo has some benchmarks on this. We should benchmark and write it up more, too. This might be off-topic for this discussion, though.
-
Admin over 12 yearsThe more EBS volumes you have, the more likely you are to have performance problems. You run into the weakest-link problem where the slowest one hurts the whole system. And the more volumes you have, the more likely that one of them will misbehave.
-
Admin over 12 yearsThe tuning script referenced is dangerous and I don't recommend it. I've seen more DE-optimized servers with such tuning scripts than optimized ones.
-
Luis Fernando Alen over 12 yearsFirst of all I'd like to thank you for such a complete answer. Second, making things clearer, most of our tables use the InnoDB engine and the server has a read-heavy workload. I also edited my question and added our my.cnf and the output of sundry's mysql tuning script. I tried some tuning parameters you suggested, specially the innodb_buffer_pool_size but the server is still experiencing a high iowait. Nevertheless, since I'm not a DBA I'm sure I'm missing several things here and will need to work harder :) Regarding the filesystem, I'll follow your advice and move to xfs as soon as I can.
-
Luis Fernando Alen over 12 yearsNiko, thanks for the script. Dangerous or not, I can say it provided some important information and some points I need to work on :)