"OverflowError: Python int too large to convert to C long" on windows but not mac

Question

I am running the exact same code on both windows and mac, with python 3.5 64 bit.

On windows, it looks like this:

>>> import numpy as np
>>> preds = np.zeros((1, 3), dtype=int)
>>> p = [6802256107, 5017549029, 3745804973]
>>> preds[0] = p
Traceback (most recent call last):
  File "<pyshell#13>", line 1, in <module>
    preds[0] = p
OverflowError: Python int too large to convert to C long

However, this code works fine on my mac. Could anyone help explain why or give a solution for the code on windows? Thanks so much!

Solution 1

You'll get that error once your numbers are greater than sys.maxsize:

>>> p = [sys.maxsize]
>>> preds[0] = p
>>> p = [sys.maxsize+1]
>>> preds[0] = p
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: Python int too large to convert to C long

You can confirm this by checking:

>>> import sys
>>> sys.maxsize
2147483647

To take numbers with larger precision, don't pass an int type which uses a bounded C integer behind the scenes. Use the default float:

>>> preds = np.zeros((1, 3))

Solution 2

You can use dtype=np.int64 instead of dtype=int

Solution 3

Could anyone help explain why

Numpy arrays normally* have fixed size elements, including integers of various sizes, single or double precision floating point numbers, fixed length byte and Unicode strings and structures built up from the aforementioned types.

In Python 2 a python "int" was equivalent to a C long. In Python 3 an "int" is an arbitrary precision type but numpy still uses "int" it to represent the C type "long" when creating arrays.

The size of a C long is platform dependent. On windows it is always 32-bit. On unix-like systems it is normally 32 bit on 32 bit systems and 64 bit on 64 bit systems.

or give a solution for the code on windows? Thanks so much!

Choose a data type whose size is not platform dependent. You can find the list at https://docs.scipy.org/doc/numpy/reference/arrays.scalars.html#arrays-scalars-built-in the most sensible choice would probably be np.int64

* Numpy does allow arrays of python objects, but I don't think they are widely used.

Solution 4

Convert to float:

import pandas as pd

df = pd.DataFrame()
l_var_l = [8258255190131389999999000003296, 50661]
df['temp'] = l_var_l
df['temp'] = df['temp'].astype(int)

Above fails with error:

OverflowError: Python int too large to convert to C long.

Now try with float conversion:

df['temp'] = df['temp'].astype(float)

View more solutions

Comments

[Tim]

You're sure both are 64 bit? can you test on linux?

[user2357112]

Even if both systems are on 64-bit Python, are they both on 64-bit NumPy?

[VladimirM]

Another stackoverflow question explains 'why'. On Windows long is 32bit and on Unux-like long is 64bit. Please see the question stackoverflow.com/questions/384502/…

[Eryk Sun]

Use dtype='int64' or dtype=np.int64. The int type uses a C long, which is always 32-bit on Windows.

[packybear]

to Tim: Yes, both are 64bit. I do not have a linux machine, sorry. to user2357112: Yes, both are 64bit python and numpy. to VladimirM: Thanks! I think that question answers mine! to eryksun: Thanks! It works!

[jtlz2]

How would you do this without numpy?

[Veronica Cheng]

if you do get a number larger than this, how to tackle?

[Moses Koledoye]

@VeronicaWenqianCheng Don't pass an int dtype, use the default float.

[fireball.1]

what if it needs to passed as an index which then needs to be int ?

[Moses Koledoye]

I don't understand your question clearly. The index or the value itself? In the case of the value, use a float. You can easily convert to int in plain Python if you need the value as an int.

[jtlz2]

MosesKoledoye I think @fireball means what if a non-float is required as an index argument and hence cannot be a float (which you say is required to circumvent this problem)? Should one do int(float(x)) - surely not?

[jtlz2]

e.g. TypeError: integer argument expected, got float

[Moses Koledoye]

@jtlz2 Maybe show some code. I can't see why a float is being passed when an int is required.

[plugwash]

At least according to the documentation sys.maxsize is the maximum value of py_ssize_t (essentially ssize_t), not long. In particular win64 has a 64-bit size_t, but a 32-bit long.

[Axel Puig]

Thanks, I just had to use the unsigned type np.uint64 (to store hashes).

[Cliff AB]

With enormous ints, they are very likely to be id's of sorts. Doesn't converting them to floats mean that the digits will be truncated, thus breaking the uniqueness of the ids?

[information_interchange]

I tried using both np.int64 and np.uint64 to store 109323892912381287389218291378123872293293923929392929289283‌​928 Neither work

[plugwash]

If you need to store insanely large numbers exactly then numpy probablly isn't the tool for you. If you can tolerate loss of precision then you can use the float type, alternatively it's possible to have a numpy array of python objects ( stackoverflow.com/questions/6141853/… ) but at that point some would question why you are using a numpy array at all.

[User1010]

Consider using code formatting in your answer.