R – How to find term frequency within a DTM in R

rtext-mining

I've been using the tm package to create a DocumentTerm Matrix as follows:

library(tm)
library(RWeka)
library(SnowballC)
src <- DataframeSource(data.frame(data3$JobTitle))

# create a corpus and transform data
# Sets the default number of threads to use
options(mc.cores=1)
c_copy <- c <- Corpus(src)
c <- tm_map(c, content_transformer(tolower), mc.cores=1)
c <- tm_map(c,content_transformer(removeNumbers), mc.cores=1)
c <- tm_map(c,removeWords, stopwords("english"), mc.cores=1)
c <- tm_map(c,content_transformer(stripWhitespace), mc.cores=1)

#make DTM
dtm <- DocumentTermMatrix(c, control = list(tokenize = BigramTokenizer))

Now, the DTM comes out fine – what I want to do is get the frequencies of the frequent terms within the DTM. Obviously, I can use findFreqTerms to get the terms themselves, but not the actual frequencies. termFreq only works on TextDocument, not a DTM or TDM – any ideas?

Output from str – the frequent terms are in $ Terms:

> str(dtm)
List of 6
 $ i       : int [1:190] 1 2 3 4 5 6 7 8 9 10 ...
 $ j       : int [1:190] 1 2 3 4 5 6 7 8 9 10 ...
 $ v       : num [1:190] 1 1 1 1 1 1 1 1 1 1 ...
 $ nrow    : int 119
 $ ncol    : int 146
 $ dimnames:List of 2
  ..$ Docs : chr [1:119] "1" "2" "3" "4" ...
  ..$ Terms: chr [1:146] "account administrator" "account assistant" "account director" "account executive" ...
 - attr(*, "class")= chr [1:2] "DocumentTermMatrix" "simple_triplet_matrix"
 - attr(*, "weighting")= chr [1:2] "term frequency" "tf"

Best Answer

Thanks to NicE for the advice - it works well. Adding in the weighting argument allows me to get out the term frequencies when I inspect the DTM. Simple matter then of summing up per column.

dtm <- DocumentTermMatrix(c, control = list(tokenize = BigramTokenizer, weighting=weightTf))
freqs <- as.data.frame(inspect(dtm))
colSums(freqs)

Related Solutions

R – How to sort a dataframe by multiple column(s)

You can use the order() function directly without resorting to add-on tools -- see this simpler answer which uses a trick right from the top of the example(order) code:

R> dd[with(dd, order(-z, b)), ]
    b x y z
4 Low C 9 2
2 Med D 3 1
1  Hi A 8 1
3  Hi A 9 1

Edit some 2+ years later: It was just asked how to do this by column index. The answer is to simply pass the desired sorting column(s) to the order() function:

R> dd[order(-dd[,4], dd[,1]), ]
    b x y z
4 Low C 9 2
2 Med D 3 1
1  Hi A 8 1
3  Hi A 9 1
R>

rather than using the name of the column (and with() for easier/more direct access).

R – How to join (merge) data frames (inner, outer, left, right)

By using the merge function and its optional parameters:

Inner join: merge(df1, df2) will work for these examples because R automatically joins the frames by common variable names, but you would most likely want to specify merge(df1, df2, by = "CustomerId") to make sure that you were matching on only the fields you desired. You can also use the by.x and by.y parameters if the matching variables have different names in the different data frames.

Outer join: merge(x = df1, y = df2, by = "CustomerId", all = TRUE)

Left outer: merge(x = df1, y = df2, by = "CustomerId", all.x = TRUE)

Right outer: merge(x = df1, y = df2, by = "CustomerId", all.y = TRUE)

Cross join: merge(x = df1, y = df2, by = NULL)

~~Just as with the inner join, you would probably want to explicitly pass "CustomerId" to R as the matching variable.~~ I think it's almost always best to explicitly state the identifiers on which you want to merge; it's safer if the input data.frames change unexpectedly and easier to read later on.

You can merge on multiple columns by giving by a vector, e.g., by = c("CustomerId", "OrderId").

If the column names to merge on are not the same, you can specify, e.g., by.x = "CustomerId_in_df1", by.y = "CustomerId_in_df2" where CustomerId_in_df1 is the name of the column in the first data frame and CustomerId_in_df2 is the name of the column in the second data frame. (These can also be vectors if you need to merge on multiple columns.)

Best Answer

Related Solutions

R – How to sort a dataframe by multiple column(s)

R – How to join (merge) data frames (inner, outer, left, right)

Related Topic