Counting words in a single document from corpus in R and putting it in dataframe

Question

I have got text documents, in each document I have text featuring tv series spoilers. Each of the documents is a different series. I want to compare the most used words of each series, I was thinking I could plot them using ggplot, and have 'Series 1 Terms that occur at least x times' on one axis and ' 'Series 2 Terms that occur at least x times' on another. I expect what I need is a dataframe with 3 columns 'Terms', 'Series x', 'Series Y'. With series x and y having the number of times that word occurs.

I have tried multiple ways to do this but failed. The closest I have got is I can read the corpus and create a dataframe with all the terms in one column like so:

library("tm")

corpus <-Corpus(DirSource("series"))
corpus.p <-tm_map(corpus, removeWords, stopwords("english"))  #removes stopwords
corpus.p <-tm_map(corpus.p, stripWhitespace)  #removes stopwords
corpus.p <-tm_map(corpus.p, tolower)  
corpus.p <-tm_map(corpus.p, removeNumbers)
corpus.p <-tm_map(corpus.p, removePunctuation)
dtm <-DocumentTermMatrix(corpus.p)
docTermMatrix <- inspect(dtm)
termCountFrame <- data.frame(Term = colnames(docTermMatrix))

I then know I could add a column adding up the words like this:

termCountFrame$seriesX <- colSums(docTermMatrix)

but that would add occurrences from both of the documents, when I only want one.

So my questions are:

1) Is it possible to use colSums on a single doc, if not is there another way to turn the doctermmatrix into a dataframe with term counts for each document

2) Does anybody know how I can limit this so I get the most used terms in each document

Solution

If your data are in a Document Term Matrix, you'd use tm::findFreqTerms to get the most used terms in a document. Here's a reproducible example:

require(tm)
data(crude)
dtm <- DocumentTermMatrix(crude)
dtm
A document-term matrix (20 documents, 1266 terms)

Non-/sparse entries: 2255/23065
Sparsity           : 91%
Maximal term length: 17 
Weighting          : term frequency (tf)

# find most frequent terms in all 20 docs
findFreqTerms(dtm, 2, 100)

# find the doc names
dtm$dimnames$Docs
 [1] "127" "144" "191" "194" "211" "236" "237" "242" "246" "248" "273" "349" "352" "353" "368" "489" "502"
[18] "543" "704" "708"

# do freq words on one doc
findFreqTerms(dtm[dtm$dimnames$Docs == "127"], 2, 100)
 [1] "crude"     "cut"       "diamond"   "dlrs"      "for"       "its"       "oil"       "price"    
 [9] "prices"    "reduction" "said."     "that"      "the"       "today"     "weak"

Here's how you'd find the most frequent words for each doc in the dtm, one document at a time:

# find freq words for each doc, one by one
list_freqs <- lapply(dtm$dimnames$Docs, 
              function(i) findFreqTerms(dtm[dtm$dimnames$Docs == i], 2, 100))


list_freqs
[[1]]
 [1] "crude"     "cut"       "diamond"   "dlrs"      "for"       "its"       "oil"       "price"    
 [9] "prices"    "reduction" "said."     "that"      "the"       "today"     "weak"     

[[2]]
 [2] "\"opec"       "\"the"        "15.8"         "ability"      "above"        "address"      "agreement"   
 [8] "analysts"     "and"          "before"       "bpd"          "but"          "buyers"       "current"     
[15] "demand"       "emergency"    "energy"       "for"          "has"          "have"         "higher"      
[22] "hold"         "industry"     "its"          "keep"         "market"       "may"          "meet"        
[29] "meeting"      "mizrahi"      "mln"          "must"         "next"         "not"          "now"         
[36] "oil"          "opec"         "organization" "prices"       "problem"      "production"   "said"        
[43] "said."        "set"          "that"         "the"          "their"        "they"         "this"        
[50] "through"      "will"        

[[3]]
[3] "canada"   "canadian" "crude"    "for"      "oil"      "price"    "texaco"   "the"     

[[4]]
[4] "bbl."    "crude"   "dlrs"    "for"     "price"   "reduced" "texas"   "the"     "west"   

[[5]]
 [5] "and"        "discounted" "estimates"  "for"        "mln"        "net"        "pct"        "present"   
 [9] "reserves"   "revenues"   "said"       "study"      "that"       "the"        "trust"      "value"     

[[6]]
 [6] "ability"       "above"         "ali"           "and"           "are"           "barrel."      
 [7] "because"       "below"         "bpd"           "bpd."          "but"           "daily"        
[13] "difficulties"  "dlrs"          "dollars"       "expected"      "for"           "had"          
[19] "has"           "international" "its"           "kuwait"        "last"          "local"        
[25] "march"         "markets"       "meeting"       "minister"      "mln"           "month"        
[31] "official"      "oil"           "opec"          "opec\"s"       "prices"        "producing"    
[37] "pumping"       "qatar,"        "quota"         "referring"     "said"          "said."        
[43] "sheikh"        "such"          "than"          "that"          "the"           "their"        
[49] "they"          "this"          "was"           "were"          "which"         "will"         

[[7]]
 [7] "\"this"        "and"           "appears"       "are"           "areas"         "bank"         
 [7] "bankers"       "been"          "but"           "crossroads"    "crucial"       "economic"     
[13] "economy"       "embassy"       "fall"          "for"           "general"       "government"   
[19] "growth"        "has"           "have"          "indonesia\"s"  "indonesia,"    "international"
[25] "its"           "last"          "measures"      "nearing"       "new"           "oil"          
[31] "over"          "rate"          "reduced"       "report"        "say"           "says"         
[37] "says."         "sector"        "since"         "the"           "u.s."          "was"          
[43] "which"         "with"          "world"        

[[8]]
 [8] "after"      "and"        "deposits"   "had"        "oil"        "opec"       "pct"        "quotes"    
 [9] "riyal"      "said"       "the"        "were"       "yesterday."

[[9]]
 [9] "1985/86"     "1986/87"     "1987/88"     "abdul-aziz"  "about"       "and"         "been"       
 [8] "billion"     "budget"      "deficit"     "expenditure" "fiscal"      "for"         "government" 
[15] "had"         "its"         "last"        "limit"       "oil"         "projected"   "public"     
[22] "qatar,"      "revenue"     "riyals"      "riyals."     "said"        "sheikh"      "shortfall"  
[29] "that"        "the"         "was"         "would"       "year"        "year's"     

[[10]]
 [10] "15.8"      "about"     "above"     "accord"    "agency"    "ali"       "among"     "and"      
 [9] "arabia"    "are"       "dlrs"      "for"       "free"      "its"       "kuwait"    "market"   
[17] "market,"   "minister," "mln"       "nazer"     "oil"       "opec"      "prices"    "producing"
[25] "quoted"    "recent"    "said"      "said."     "saudi"     "sheikh"    "spa"       "stick"    
[33] "that"      "the"       "they"      "under"     "was"       "which"     "with"     

[[11]]
 [11] "1.2"        "and"        "appeared"   "arabia's"   "average"    "barrel."    "because"    "below"     
 [9] "bpd"        "but"        "corp"       "crude"      "december"   "dlrs"       "export"     "exports"   
[17] "february"   "fell"       "for"        "four"       "from"       "gulf"       "january"    "january,"  
[25] "last"       "mln"        "month"      "month,"     "neutral"    "official"   "oil"        "opec"      
[33] "output"     "prices"     "production" "refinery"   "said"       "said."      "saudi"      "sell"      
[41] "sources"    "than"       "the"        "they"       "throughput" "week"       "yanbu"      "zone"      

[[12]]
 [12] "and"       "arab"      "crude"     "emirates"  "gulf"      "ministers" "official"  "oil"      
 [9] "states"    "the"       "wam"      

[[13]]
 [13] "accord" "agency" "and"    "arabia" "its"    "nazer"  "oil"    "opec"   "prices" "saudi"  "the"   
[12] "under" 

[[14]]
 [14] "crude"   "daily"   "for"     "its"     "oil"     "opec"    "pumping" "that"    "the"     "was"    

[[15]]
 [15] "after"   "closed"  "new"     "nuclear" "oil"     "plant"   "port"    "power"   "said"    "ship"   
[11] "the"     "was"     "when"   

[[16]]
 [16] "about"       "and"         "development" "exploration" "for"         "from"        "help"       
 [8] "its"         "mln"         "oil"         "one"         "present"     "prices"      "research"   
[15] "reserve"     "said"        "strategic"   "the"         "u.s."        "with"        "would"      

[[17]]
 [17] "about"       "and"         "benefits"    "development" "exploration" "for"         "from"       
 [8] "group"       "help"        "its"         "mln"         "oil"         "one"         "policy"     
[15] "present"     "prices"      "protect"     "research"    "reserve"     "said"        "strategic"  
[22] "study"       "such"        "the"         "u.s."        "with"        "would"      

[[18]]
 [18] "1.50"    "company" "crude"   "dlrs"    "for"     "its"     "lowered" "oil"     "posted"  "prices" 
[11] "said"    "said."   "the"     "union"   "west"   

[[19]]
 [19] "according"    "and"          "april"        "before"       "can"          "change"       "efp"         
 [8] "energy"       "entering"     "exchange"     "for"          "futures"      "has"          "hold"        
[15] "increase"     "into"         "mckiernan"    "new"          "not"          "nymex"        "oil"         
[22] "one"          "position"     "prices"       "rule"         "said"         "spokeswoman." "that"        
[29] "the"          "traders"      "transaction"  "when"         "will"        

[[20]]
 [20] "1986,"        "1987"         "billion"      "cubic"        "fiscales"     "january"      "mln"         
 [8] "pct"          "petroliferos" "yacimientos"  

If you want this output in a dataframe, you can do this:

# from here http://stackoverflow.com/a/7196565/1036500
L <- list_freqs
cfun <- function(L) {
  pad.na <- function(x,len) {
    c(x,rep(NA,len-length(x)))
  }
  maxlen <- max(sapply(L,length))
  do.call(data.frame,lapply(L,pad.na,len=maxlen))
}
# make dataframe of words (but probably you want words as rownames and cells with counts?)
tab_freqa <- cfun(L)

But if you want to plot 'doc 1 high freq terms vs doc 2 high freq terms', then we'll need a different approach...

# convert dtm to matrix
mat <- as.matrix(dtm)

# make data frame similar to "3 columns 'Terms', 
# 'Series x', 'Series Y'. With series x and y 
# having the number of times that word occurs"
cb <- data.frame(doc1 = mat['127',], doc2 = mat['144',])

# keep only words that are in at least one doc
cb <- cb[rowSums(cb)  > 0, ]

# plot
require(ggplot2)
ggplot(cb, aes(doc1, doc2)) +
  geom_text(label = rownames(cb), 
           position=position_jitter())

Or perhaps slightly more efficiently, we can make one big dataframe of all the docs and make plots from that:

# this is the typical method to turn a 
# dtm into a df...
df <- as.data.frame(as.matrix(dtm))
# and transpose for plotting
df <- data.frame(t(df))
# plot
require(ggplot2)
ggplot(df, aes(X127, X144)) +
  geom_text(label = rownames(df), 
           position=position_jitter())

After you remove stopwords this will look better, but this is a good proof of concept. Is that what you were after?