What are co-occurence matrixes and how are they used in NLP?

Question

The pypi docs for a google ngram downloader say that "sometimes you need an aggregate data over the dataset. For example to build a co-occurrence matrix."

The wikipedia for co-occurence matrix has to do with image processing and googling the term seems to bring up some sort of SEO trick.

So what are co-occurrence matrixes (in computational linguistics/NLP)? How are they used in NLP?

Solution 1

What is a co-occurrence matrix ?

Generally speaking, a co-occurrence matrix will have specific entities in rows (ER) and columns (EC). The purpose of this matrix is to present the number of times each ER appears in the same context as each EC. As a consequence, in order to use a co-occurrence matrix, you have to define your entites and the context in which they co-occur.

In NLP, the most classic approach is to define each entity (ie, lines and columns) as a word present in a text, and the context as a sentence.

Consider the following text :

Roses are red. Sky is blue.

With the classic approach described before, we'll have the following matrix :

      |  Roses | are | red | Sky | is | blue
Roses |    1   |  1  |  1  |  0  |  0 |   0
are   |    1   |  1  |  1  |  0  |  0 |   0
red   |    1   |  1  |  1  |  0  |  0 |   0
Sky   |    0   |  0  |  0  |  1  |  1 |   1
is    |    0   |  0  |  0  |  1  |  1 |   1
Blue  |    0   |  0  |  0  |  1  |  1 |   1

Here, each cell indicates wether the two items co-occur or not. You may replace it with the number of times it appears, or with a more sophisticated approach. You may also change the entities themselves, by putting nouns in columns and adjective in lines instead of every word.

What are they used for in NLP ?

The most evident use of these matrix is their ability to provide links between notions. Let's suppose you're working on products reviews. Let's also suppose for simplicity that each review is only composed of short sentences. You'll have something like that :

ProductX is amazing.

I hate productY.

Representing these reviews as one co-occurrence matrix will enable you associate products with appreciations.

Solution 2

The co-occurrence matrix indicates how many times the row word (e.g. 'digital') is surrounded (in a sentence, or in the ±4 word window - depends on the application) by the column word (e.g. 'pie').

The entry '5' in the following table, for example, means that we had 5 sentences in our text where 'digital' was surrounded by 'pie'.

These sentences could have been:

• I love a digital pie.
• What's digital is often a pie.
• May I have some digital pie?
• Digital world necessitates pie-eating.
• There's something digital about this pie.

---

Note that the co-occurrence matrix is always symmetric - the entry with the row word 'pie' and the column word 'digital' will be 5 as well (as these words co-occur in the very same sentences!).