ValueError: Data must be positive (boxcox scipy)

Question

I'm trying to transform my dataset to a normal distribution.

0      8.298511e-03
1      3.055319e-01
2      6.938647e-02
3      2.904091e-02
4      7.422441e-02
5      6.074046e-02
6      9.265747e-04
7      7.521846e-02
8      5.960521e-02
9      7.405019e-04
10     3.086551e-02
11     5.444835e-02
12     2.259236e-02
13     4.691038e-02
14     6.463911e-02
15     2.172805e-02
16     8.210005e-02
17     2.301189e-02
18     4.073898e-07
19     4.639910e-02
20     1.662777e-02
21     8.662539e-02
22     4.436425e-02
23     4.557591e-02
24     3.499897e-02
25     2.788340e-02
26     1.707958e-02
27     1.506404e-02
28     3.207647e-02
29     2.147011e-03
30     2.972746e-02
31     1.028140e-01
32     2.183737e-02
33     9.063370e-03
34     3.070437e-02
35     1.477440e-02
36     1.036309e-02
37     2.000609e-01
38     3.366233e-02
39     1.479767e-03
40     1.137169e-02
41     1.957088e-02
42     4.921303e-03
43     4.279257e-02
44     4.363429e-02
45     1.040123e-01
46     2.930958e-02
47     1.935434e-03
48     1.954418e-02
49     2.980253e-02
50     3.643772e-02
51     3.411437e-02
52     4.976063e-02
53     3.704608e-02
54     7.044161e-02
55     8.101365e-03
56     9.310477e-03
57     7.626637e-02
58     8.149728e-03
59     4.157399e-01
60     8.200258e-02
61     2.844295e-02
62     1.046601e-01
63     6.565680e-02
64     9.825436e-04
65     9.353639e-02
66     6.535298e-02
67     6.979044e-04
68     2.772859e-02
69     4.378422e-02
70     2.020185e-02
71     4.774493e-02
72     6.346146e-02
73     2.466264e-02
74     6.636585e-02
75     2.548934e-02
76     1.113937e-06
77     5.723409e-02
78     1.533288e-02
79     1.027341e-01
80     4.294570e-02
81     4.844853e-02
82     5.579620e-02
83     2.531824e-02
84     1.661426e-02
85     1.430836e-02
86     3.157232e-02
87     2.241722e-03
88     2.946256e-02
89     1.038383e-01
90     1.868837e-02
91     8.854596e-03
92     2.391759e-02
93     1.612714e-02
94     1.007823e-02
95     1.975513e-01
96     3.581289e-02
97     1.199747e-03
98     1.263381e-02
99     1.966746e-02
100    4.040786e-03
101    4.497264e-02
102    4.030524e-02
103    8.627087e-02
104    3.248317e-02
105    5.727582e-03
106    1.781355e-02
107    2.377991e-02
108    4.299568e-02
109    3.664353e-02
110    5.167902e-02
111    4.006848e-02
112    7.072990e-02
113    6.744938e-03
114    1.064900e-02
115    9.823497e-02
116    8.992714e-03
117    1.792453e-01
118    6.817763e-02
119    2.588843e-02
120    1.048027e-01
121    6.468491e-02
122    1.035536e-03
123    8.800684e-02
124    5.975065e-02
125    7.365861e-04
126    4.209485e-02
127    4.232421e-02
128    2.371866e-02
129    5.894714e-02
130    7.177195e-02
131    2.116566e-02
132    7.579219e-02
133    3.174744e-02
134    0.000000e+00
135    5.786439e-02
136    1.458493e-02
137    9.820156e-02
138    4.373873e-02
139    4.271649e-02
140    5.532575e-02
141    2.311324e-02
142    1.644508e-02
143    1.328273e-02
144    3.908473e-02
145    2.355468e-03
146    2.519321e-02
147    1.131868e-01
148    1.708967e-02
149    1.027661e-02
150    2.439899e-02
151    1.604058e-02
152    1.134323e-02
153    2.247722e-01
154    3.408590e-02
155    2.222239e-03
156    1.659830e-02
157    2.284733e-02
158    4.618550e-03
159    3.674162e-02
160    4.131283e-02
161    8.846273e-02
162    2.504404e-02
163    6.004396e-03
164    1.986309e-02
165    2.347111e-02
166    3.865636e-02
167    3.672307e-02
168    6.658419e-02
169    3.726879e-02
170    7.600138e-02
171    7.184871e-03
172    1.142840e-02
173    9.741311e-02
174    8.165448e-03
175    1.529210e-01
176    6.648081e-02
177    2.617601e-02
178    9.547816e-02
179    6.857775e-02
180    8.129399e-04
181    7.107914e-02
182    5.884794e-02
183    8.398721e-04
184    6.972981e-02
185    4.461767e-02
186    2.264404e-02
187    5.566633e-02
188    6.595136e-02
189    2.301914e-02
190    7.488919e-02
191    3.108619e-02
192    4.989364e-07
193    4.834949e-02
194    1.422578e-02
195    9.398186e-02
196    4.870391e-02
197    3.841369e-02
198    6.406801e-02
199    2.603315e-02
200    1.692629e-02
201    1.409982e-02
202    4.099215e-02
203    2.093724e-03
204    2.640732e-02
205    1.032129e-01
206    1.581881e-02
207    8.977325e-03
208    1.941141e-02
209    1.502126e-02
210    9.923589e-03
211    2.757357e-01
212    3.096234e-02
213    4.388900e-03
214    1.784778e-02
215    2.179550e-02
216    3.944159e-03
217    3.703552e-02
218    4.033897e-02
219    1.157076e-01
220    2.400446e-02
221    5.761179e-03
222    1.899621e-02
223    2.401468e-02
224    4.458745e-02
225    3.357898e-02
226    5.331003e-02
227    3.488753e-02
228    7.466599e-02
229    6.075236e-03
230    9.815318e-03
231    9.598735e-02
232    7.103607e-03
233    1.100602e-01
234    5.677641e-02
235    2.420500e-02
236    9.213369e-02
237    4.024043e-02
238    6.987694e-04
239    8.612055e-02
240    5.663353e-02
241    4.871693e-04
242    4.533811e-02
243    3.593244e-02
244    1.982537e-02
245    5.490786e-02
246    5.603109e-02
247    1.671653e-02
248    6.522711e-02
249    3.341356e-02
250    2.378629e-06
251    4.299939e-02
252    1.223163e-02
253    8.392798e-02
254    4.272826e-02
255    3.183946e-02
256    4.431299e-02
257    2.661024e-02
258    1.686707e-02
259    4.070924e-03
260    3.325947e-02
261    2.023611e-03
262    2.402284e-02
263    8.369778e-02
264    1.375093e-02
265    8.899898e-03
266    2.148740e-02
267    1.301483e-02
268    8.355791e-03
269    2.549934e-01
270    2.792516e-02
271    4.652563e-03
272    1.556313e-02
273    1.936942e-02
274    3.547794e-03
275    3.412516e-02
276    3.932606e-02
277    5.305868e-02
278    2.354438e-02
279    5.379380e-03
280    1.904203e-02
281    2.045495e-02
282    3.275855e-02
283    3.007389e-02
284    8.227664e-02
285    2.479949e-02
286    6.573835e-02
287    5.165842e-03
288    7.599650e-03
289    9.613557e-02
290    6.690175e-03
291    1.779880e-01
292    5.076263e-02
293    3.117607e-02
294    7.495692e-02
295    3.707768e-02
296    7.086975e-04
297    8.935981e-02
298    5.624249e-02
299    7.105331e-04
300    3.339868e-02
301    3.354603e-02
302    2.041988e-02
303    3.862522e-02
304    5.977081e-02
305    1.730081e-02
306    6.909621e-02
307    3.729478e-02
308    3.940647e-07
309    4.385336e-02
310    1.391891e-02
311    8.898305e-02
312    3.840141e-02
313    3.214408e-02
314    4.284080e-02
315    1.841022e-02
316    1.528207e-02
317    3.106559e-03
318    3.945481e-02
319    2.085094e-03
320    2.464190e-02
321    7.844914e-02
322    1.526590e-02
323    9.922147e-03
324    1.649218e-02
325    1.341602e-02
326    8.124446e-03
327    2.867380e-01
328    2.663867e-02
329    5.342012e-03
330    1.752612e-02
331    2.010863e-02
332    3.581845e-03
333    3.652284e-02
334    4.484362e-02
335    4.600939e-02
336    2.213280e-02
337    5.494917e-03
338    2.016594e-02
339    2.118010e-02
340    2.964000e-02
341    3.405549e-02
342    1.014185e-01
343    2.451624e-02
344    7.966998e-02
345    5.301538e-03
346    8.198895e-03
347    8.789368e-02
348    7.222417e-03
349    1.448276e-01
350    5.676056e-02
351    2.987054e-02
352    6.851434e-02
353    4.193034e-02
354    7.025054e-03
355    8.557358e-02
356    5.812736e-02
357    2.263676e-02
358    2.922588e-02
359    3.363161e-02
360    1.495056e-02
361    5.871619e-02
362    6.235094e-02
363    1.691340e-02
364    5.361939e-02
365    3.722318e-02
366    9.828477e-03
367    4.155345e-02
368    1.327760e-02
369    7.205372e-02
370    4.151130e-02
371    3.265365e-02
372    2.879418e-02
373    2.314340e-02
374    1.653692e-02
375    1.077611e-02
376    3.481427e-02
377    1.815487e-03
378    2.232305e-02
379    1.005192e-01
380    1.491262e-02
381    3.752658e-02
382    1.271613e-02
383    1.223707e-02
384    8.088923e-03
385    2.572550e-01
386    2.300194e-02
387    2.847960e-02
388    1.782098e-02
389    1.900759e-02
390    3.647629e-03
391    3.723368e-02
392    4.079514e-02
393    5.510332e-02
394    3.072313e-02
395    4.183566e-03
396    1.891549e-02
397    1.870293e-02
398    3.182769e-02
399    4.167840e-02
400    1.343152e-01
401    2.451973e-02
402    7.567017e-02
403    4.837843e-03
404    6.477297e-03
405    7.664675e-02
Name: value, dtype: float64

This is the code I used for transforming dataset:

from scipy import stats
x,_ = stats.boxcox(df)

I get this error:

            if any(x <= 0):
-> 1031         raise ValueError("Data must be positive.")
   1032 
   1033     if lmbda is not None:  # single transformation

ValueError: Data must be positive

Is it because my values are too small that it's producing an error? Not sure what I'm doing wrong. New to using boxcox, could be using it incorrectly in this example. Open to suggestions and alternatives. Thanks!

Solution 1

Your data contains the value 0 (at index 134). When boxcox says the data must be positive, it means strictly positive.

What is the meaning of your data? Does 0 make sense? Is that 0 actually a very small number that was rounded down to 0?

You could simply discard that 0. Alternatively, you could do something like the following. (This amounts to temporarily discarding the 0, and then using -1/λ for the transformed value of 0, where λ is the Box-Cox transformation parameter.)

First, create some data that contains one 0 (all other values are positive):

In [13]: np.random.seed(8675309)

In [14]: data = np.random.gamma(1, 1, size=405)

In [15]: data[100] = 0

(In your code, you would replace that with, say, data = df.values.)

Copy the strictly positive data to posdata:

In [16]: posdata = data[data > 0]

Find the optimal Box-Cox transformation, and verify that λ is positive. This work-around doesn't work if λ ≤ 0.

In [17]: bcdata, lam = boxcox(posdata)

In [18]: lam
Out[18]: 0.244049919975582

Make a new array to hold that result, along with the limiting value of the transform of 0 (which is -1/λ):

In [19]: x = np.empty_like(data)

In [20]: x[data > 0] = bcdata

In [21]: x[data == 0] = -1/lam

The following plot shows the histograms of data and x.

Solution 2

Rather than normal boxcox, you can use boxcox1p. It adds 1 to x so there won't be any "0" record

from scipy.special import boxcox1p
scipy.special.boxcox1p(x, lmbda)

For more info check out the docs at https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.boxcox1p.html

Solution 3

Is your data that you are sending to boxcox 1-dimensional ndarray?

Second way could be adding shift parameter by summing shift (see details from the link) to all of the ndarray elements before sending it to boxcox and subtracting shift from the resulting array elements (if I have understood boxcox algorithm correctly, that could be solution in your case, too).

https://docs.scipy.org/doc/scipy-0.16.1/reference/generated/scipy.stats.boxcox.html