Will indexing improve varchar(max) query performance, and how to create index

Question

Firstly, I should point out I don't have much knowledge on SQL Server indexes.

My situation is that I have an SQL Server 2008 database table that has a varchar(max) column usually filled with a lot of text.

My ASP.NET web application has a search facility which queries this column for keyword searches, and depending on the number of keywords searched for their may be one or many LIKE '%keyword%' statements in the SQL query to do the search.

My web application also allows searching by various other columns in this table as well, not just that one column. There is also a few joins from other tables too.

My question is, is it worthwhile creating an index on this column to improve performance of these search queries? And if so, what type of index, and will just indexing the one column be enough or do I need to include other columns such as the primary key and other searchable columns?

Solution 1

It's not worthwhile creating a regular index if you're doing LIKE '%keyword%' searches. The reason is that indexing works like searching a dictionary, where you start in the middle then split the difference until you find the word. That wildcard query is like asking you to lookup a word that contains the text "to" or something-- the only way to find matches is to scan the whole dictionary.

You might consider a full-text search, however, which is meant for this kind of scenario (see here).

Solution 2

The best analogy I've ever seen for why an index won't help '%wildcard%' searches:

Take two people. Hand each one the same phone book. Say to the person on your left:

Tell me how many people are in this phone book with the last name "Smith."

Now say to the person on your right:

Tell me how many people are in this phone book with the first name "Simon."

An index is like a phone book. Very easy to seek for the thing that is at the beginning. Very difficult to scan for the thing that is in the middle or at the end.

Every time I've repeated this in a session, I see light bulbs go on, so I thought it might be useful to share here.

Solution 3

you cannot create an index on a varchar(max) field. The maximum amount of bytes on a index is 900. If the column is bigger than 900 bytes, you can create the index but any insert with more then 900 bytes will fail.

I suggest you to read about fulltext search. It should suits you in this case

Comments

[johna]

Diego, my understanding is I could index on a varchar(max) column as long as it is a non-clustered, offline index and that the var(max) column is INCLUDEed, eg. CREATE NONCLUSTERED INDEX IndexName ON TableName(SomeColumn) INCLUDE(VarcharMaxColumn) WITH (ONLINE=OFF)

[Diego]

the varchar(max) can only be included on a index, it means only leaf levels will participate on the index, meaning it will hardly be used on a search query

[Diego]

WITH (ONLINE=OFF) is not a property of the index. It indicates that the index will be set offline during creation. If you set it to ON, the index will be created on the tempDB and the table will be accessible during the index creation process. It should be used for huge tables where creating an index takes a long time and the table cannot be unaccessible during creation

[Mitch Wheat]

This is the analogy I always use.