R - Warning message: "In cor(...): the standard deviation is zero"

Question

I have a single vector of flow data (29 data) and a 3D matrix data(360*180*29)

i want to find the correlation between single vector and 3D vector. The correlation matrix will have a size of 360*180.

> str(ScottsCk_flow_1981_2010_JJA)
 num [1:29] 0.151 0.644 0.996 0.658 1.702 ...
> str(ssta_winter)
 num [1:360, 1:180, 1:29] NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN ...
> summary(ssta_winter)
    Min.  1st Qu.   Median     Mean  3rd Qu.     Max.     NA's 
    -2.8     -0.2      0.1      0.2      0.6      6.0 596849.0 

This above is the structure of the vector and 3D matrix. 3D matrix has many values as Null.

> for (i in 1:360) {
+   for(j in 1:180){
+       cor_ScottsCk_SF_SST_JJA[i,j] = cor(ScottsCk_flow_1981_2010_JJA,ssta_winter[i,j,]) 
+    }
+ }
There were 50 or more warnings (use warnings() to see the first 50)

This part of code above is the code to find correlation. But it gives waring as

> warnings()
Warning messages:
1: In cor(ScottsCk_flow_1981_2010_JJA, ssta_winter[i, j,  ... :
  the standard deviation is zero
2: In cor(ScottsCk_flow_1981_2010_JJA, ssta_winter[i, j,  ... :
  the standard deviation is zero
3: In cor(ScottsCk_flow_1981_2010_JJA, ssta_winter[i, j,  ... :
  the standard deviation is zero
4: In cor(ScottsCk_flow_1981_2010_JJA, ssta_winter[i, j,  ... :
  the standard deviation is zero
5: In cor(ScottsCk_flow_1981_2010_JJA, ssta_winter[i, j,  ... :
  the standard deviation is zero

also, the result of the correlation matrix is all NULL. how did this happen?

> str(cor_ScottsCk_SF_SST_JJA)
 num [1:360, 1:180] NA NA NA NA NA NA NA NA NA NA ...

I have used exact same code bfr with 350 flow vector and 360*180*350 matrix. This code works perfectly.

Solution 1

A few thoughts.

First, by using apply(), you can replace that nested loop with something like this:

cor_ScottsCk_SF_SST_JJA <- 
    apply(ssta_winter, MARGIN = 1:2, FUN = cor, ScottsCk_flow_1981_2010_JJA)

Second, it appears that >31% (596849/(360*180*29)) of the points in ssta_winter are NaN or (possibly) NA_real_. Given the return value of a correlation calculated on vectors that contain even a single NaN,

cor(c(1:3, NaN), c(1:4))
# [1] NA

isn't it likely that all those NaNs are causing cor_ScottsCk_SF_SST_JJA to be filled with NAs?

Third, as the warning messages plainly tell you, some of the vectors you are passing to cor() have zero variance. They have nothing to do with the NaNs: as the following shows, R doesn't complain about standard deviations of 0 when NaN are involved. (Quite sensibly too, since you can't calculate standard deviations for undefined numbers):

cor(c(NaN, NaN, NaN, NaN), c(1,1,1,1))
# [1] NA

cor(c(1,1,1,1), c(1,2,3,4))
# [1] NA
# Warning message:
# In cor(c(1, 1, 1, 1), c(1, 2, 3, 4)) : the standard deviation is zero

Solution 2

This error might also be shown if a column has the same values for all observations. So, you might want to remove those rows.

Solution 3

The following uses library("psych")

partial.r(sd,c("GPA","SAT"),"GRADE1",use = "complete.obs")
Warning Message:
 In cor(data, use = use, method = method) : the standard deviation is zero

sd contains NA for SAT.

partial.r(subset,c("GPA","SAT"),"GRADE1", use = "complete.obs")
no warnings

subset has NA's removed

Comments

[Yu Deng]

For loop, all loop counter are not hard coded, hard coding here is just to simplify question.

[Yu Deng]

But why it works previously, the same ssta data, a lot of NAs.

[Josh O'Brien]

I'm curious about that as well, but have no way of knowing (or learning anything) without the data in front of me. Best of luck!

[Yu Deng]

sorry, the problem is the ssta_winter array, the loop doesn't pass down, the data are saved corresponding to the last loop counter. Thank you for your help.