Drop unused factor levels in a subsetted data frame
Solution 1
All you should have to do is to apply factor() to your variable again after subsetting:
> subdf$letters
[1] a b c
Levels: a b c d e
subdf$letters <- factor(subdf$letters)
> subdf$letters
[1] a b c
Levels: a b c
EDIT
From the factor page example:
factor(ff) # drops the levels that do not occur
For dropping levels from all factor columns in a dataframe, you can use:
subdf <- subset(df, numbers <= 3)
subdf[] <- lapply(subdf, function(x) if(is.factor(x)) factor(x) else x)
Solution 2
Since R version 2.12, there's a droplevels()
function.
levels(droplevels(subdf$letters))
Solution 3
If you don't want this behaviour, don't use factors, use character vectors instead. I think this makes more sense than patching things up afterwards. Try the following before loading your data with read.table
or read.csv
:
options(stringsAsFactors = FALSE)
The disadvantage is that you're restricted to alphabetical ordering. (reorder is your friend for plots)
Solution 4
It is a known issue, and one possible remedy is provided by drop.levels()
in the gdata package where your example becomes
> drop.levels(subdf)
letters numbers
1 a 1
2 b 2
3 c 3
> levels(drop.levels(subdf)$letters)
[1] "a" "b" "c"
There is also the dropUnusedLevels
function in the Hmisc package. However, it only works by altering the subset operator [
and is not applicable here.
As a corollary, a direct approach on a per-column basis is a simple as.factor(as.character(data))
:
> levels(subdf$letters)
[1] "a" "b" "c" "d" "e"
> subdf$letters <- as.factor(as.character(subdf$letters))
> levels(subdf$letters)
[1] "a" "b" "c"
Solution 5
Another way of doing the same but with dplyr
library(dplyr)
subdf <- df %>% filter(numbers <= 3) %>% droplevels()
str(subdf)
Edit:
Also Works ! Thanks to agenis
subdf <- df %>% filter(numbers <= 3) %>% droplevels
levels(subdf$letters)
Comments
-
medriscoll almost 2 years
I have a data frame containing a
factor
. When I create a subset of this dataframe usingsubset
or another indexing function, a new data frame is created. However, thefactor
variable retains all of its original levels, even when/if they do not exist in the new dataframe.This causes problems when doing faceted plotting or using functions that rely on factor levels.
What is the most succinct way to remove levels from a factor in the new dataframe?
Here's an example:
df <- data.frame(letters=letters[1:5], numbers=seq(1:5)) levels(df$letters) ## [1] "a" "b" "c" "d" "e" subdf <- subset(df, numbers <= 3) ## letters numbers ## 1 a 1 ## 2 b 2 ## 3 c 3 # all levels are still there! levels(subdf$letters) ## [1] "a" "b" "c" "d" "e"
-
Dirk Eddelbuettel almost 15 yearsThat's fine for a one-off, but in a data.frame with a large number of columns, you get to do that on every column that is a factor ... leading to the need for a function such as drop.levels() from gdata.
-
hatmatrix almost 15 yearsI see... but from a user-perspective it's quick to write something like subdf[] <- lapply(subdf,function(x) if(is.factor(x)) factor(x) else x) ...Is drop.levels() much more efficient computationally or better with large data sets? (One would have to rewrite the line above in a for-loop for a huge data frame, I suppose.)
-
medriscoll almost 15 yearsThanks Stephen & Dirk - I'm giving this one the thumbs up for the caes of one factor, but hopefully folks will read these comments for your suggestions on cleaning up an entire data frame of factors.
-
daroczig over 13 yearsThe
reorder
parameter of thedrop.levels
function is worth mentioning: if you have to preserve the original order of your factors, use it withFALSE
value. -
Johan about 10 yearsAs a side-effect the function converts the data frame to a list, so the
mydf <- droplevels(mydf)
solution suggested by Roman Luštrik and Tommy O'Dell below is preferable. -
Matt Bannert about 10 yearsWhat also might be noteworthy:
rlm
really goes wrong when your data.frame contains factors which contain levels with no data. You'll get an error: singular fits are not implemented in 'rlm' . Most of the time your matrix is not singular, it's just exactly this problem. -
Mars over 8 yearsAn advantage of this method over using
factor()
is that it's not necessary to modify the original dataframe or create a new persistent dataframe. I can wrapdroplevels
around a subsetted dataframe and use it as the data argument to a lattice function, and groups will be handled correctly. -
David Arenburg over 8 yearsI think the
data.table
way would be something likefor (j in names(DT)[sapply(DT, is.factor)]) set(DT, j = j, value = factor(DT[[j]]))
-
webelo almost 8 yearsAlso: this method does preserve the ordering of the variable.
-
Meep almost 8 yearsI've noticed that if I have an NA level in my factor (a genuine NA level), it is dropped by dropped levels, even if the NAs are present.
-
jangorecki over 7 years@DavidArenburg it doesn't change much here as we call
[.data.table
only once -
Vrokipal about 6 yearsUsing gdata for just drop.levels yields "gdata: read.xls support for 'XLS' (Excel 97-2004) files ENABLED." "gdata: Unable to load perl libaries needed by read.xls()" "gdata: to support 'XLSX' (Excel 2007+) files." "gdata: Run the function 'installXLSXsupport()'" "gdata: to automatically download and install the perl". Use droplevels from baseR (stackoverflow.com/a/17218028/9295807)
-
Dirk Eddelbuettel about 6 yearsStuff happens over time. You are commenting on an answer I wrote nine years ago. So let's take this as a hint to generally prefer base R solutions as those are the ones using functionality that is still going to be around N years from now.
-
David Arenburg over 5 yearsHa, after all these years I didn't know there is a
`[.factor`
method that has adrop
argument and you've posted this in 2009... -
David Arenburg over 5 yearsThis is a dupe of this answer that was posted 5 years earlier.
-
Gregor Thomas over 5 yearsI mean,
factor(as.chracter(...))
works, but just less efficiently and succinctly thanfactor(...)
. Seems strictly worse than the other answers.