MySQL extremely slow on very simple SELECT queries

Question

We have a simple web application running on a virtual machine that saves its data in a MySQL 5.5 database with the InnoDB engine. Everything worked fine for around three years, but suddenly it became extremely slow.

For example, I have a very simple table holding addresses:

CREATE TABLE `addresses` (
  `address_id` int(11) NOT NULL AUTO_INCREMENT,
  `name` varchar(64) CHARACTER SET latin1 NOT NULL,
  `firstname` varchar(64) CHARACTER SET latin1 NOT NULL,
  `street` varchar(64) CHARACTER SET latin1 NOT NULL,
  `housenumber` varchar(16) CHARACTER SET latin1 NOT NULL,
  `zip` varchar(5) CHARACTER SET latin1 NOT NULL,
  `city` varchar(64) CHARACTER SET latin1 NOT NULL,
  `email` varchar(64) CHARACTER SET latin1 NOT NULL,
  `phone` varchar(16) CHARACTER SET latin1 NOT NULL,
  `birthdate` date NOT NULL,
  PRIMARY KEY (`address_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin

This table hold around 800 entries which is really not much. But running the query

SELECT * FROM addresses

for test purposes, it seems to never finish. I checked this with the mysql CLI on the server itself: It outputs some rows of the table and then waits very long until it outputs the next rows.

So maybe it is a problem in the data sending phase, but I am not sure.

The VM has 2GB of RAM and only 320MB are used. The CPU also runs at very low 1 to 2%. mytop does not show any other queries that are blocking the server. The IT admin said that they didn't change anything at the hardware side.

I already tried some thing like restarting the database server, restarting the virtual machine. Nothing helped.

edit:

EXPLAIN SELECT * FROM addresses

gives me this result:

+----+-------------+-----------+------+---------------+------+---------+------+------+-------+
| id | select_type | table     | type | possible_keys | key  | key_len | ref  | rows | Extra |
+----+-------------+-----------+------+---------------+------+---------+------+------+-------+
|  1 | SIMPLE      | addresses | ALL  | NULL          | NULL | NULL    | NULL |  793 |       |
+----+-------------+-----------+------+---------------+------+---------+------+------+-------+
1 row in set (0.00 sec)

Solution 1

If the cpu load is low then this indicates that there are no problems with missing indexes, if that was the case the query would just need take more cpu and disk access. Also you said it worked fine for 3 years.

Did you check the general disk access speed (specifically on the partition where the database is located)? E.g. using dd like here. What you're describing sounds like a dead disk or half-dead raid. Got backups i hope?

Solution 2

You can trying a couple things,

1. Do you have indexes setup?

Indexing makes it possible to quickly find records without doing a full table scan first, cuts execution times dramatically.

CREATE INDEX idx_name ON addresses(name);

2. Before running the query use the EXPLAIN keyword first,

When used in front of a SELECT query, it will describe how MySQL intends to execute the query and the number of rows it will need to process before it finishes.

3. Make some changes to your mysql.ini, if its a VM increase the RAM and configure your mysql.ini to see if the performance increases.

There are many MySQL optimizers out there that can guide you.

Help this helps

Comments

[Admin]

Virtual Box, Xen or something else? How is the host disk set up? are there other VM's on the same box running slow also? Answering those questions might reveal the answer

[Admin]

I am not the owner of the hypervisor, so I cannot really answer this. I already asked the admin of the hypervisor if they know of any recent changes, but he said no.

[Admin]

if you can, shut down mysql and run some disk and memory benchmark. Doesn't have to be complicated. The idea is to just isolate if it's an issue with mysql, perhaps index problem as indicated in the answer below, or more widespread.

[Admin]

Good point, I experienced it to probably not being an mysql problem. Running mysql -u username -ppassword mydb -e 'SELECT * FROM addresses is outputting slow, but appending ` > test.txt`, it runs very quick. Now this would probably be a different question!? How could I investigate on this?

[Admin]

Contact the owner of the hypervisor and ask them to check the logs for any errors. Specifically disk errors. Tell him your symptoms. Backup now.

[tabb]

Okay, but creating an index on a quite small table (<800 rows) seems to not be as helpful as I would need. It takes over a minute to finish the query quoted above. A full table scan of such a small table shouldn't take so long.

[Anthony Fornito]

So... did you add the indexes? If you are not sure what the problem is I would start basic and work my way down. Adding indexes will only increase performance if done correctly.

[tabb]

Yes, I added an index on the name column. But my above query doesn't even have any restricting WHERE clause, so it will have to read the whole table and print it out. Added the index didn't help.

[tabb]

I added the result of the EXPLAIN query above. This looks fine to me, but the performance is still very low. I cannot increase the RAM as I am not the admin of the virtualization manager. But htop shows that only 307MB out of 2050MB RAM are used.

[Anthony Fornito]

If you have that much RAM available edit you my.ini restart the service see if makes a difference.

[tabb]

please see my comment above, it seems to not only be a MySQL-related problem ...

[Anthony Fornito]

Ok i read the comment " But htop shows that only 307MB out of 2050MB RAM are used" Is that what mysql us configured to use? Or is that what the system has available to it?

[hookenz]

That's a good point. It could be something as simple as a disk error on the host.

[Tero Kilkanen]

@AnthonyFornito , the test query was SELECT *, so no indices help there, so the problem is elsewhere.