IndexError: too many indices for array

Question

I know there is a ton of these threads but all of them are for very simple cases like 3x3 matrices and things of that sort and the solutions do not even begin to apply to my situation. So I'm trying to graph G versus l1 (that's not an eleven, but an L1). The data is in the file that I loaded from an excel file. The excel file is 14x250 so there are 14 arguments, each with 250 data points. I had another user (shout out to Hugh Bothwell!) help me with an error in my code, but now another error has surfaced.

So here is the code in question:

# format for CSV file:
header = ['l1', 'l2', 'l3', 'l4', 'l5', 'EI',
      'S', 'P_right', 'P1_0', 'P3_0',
      'w_left', 'w_right', 'G_left', 'G_right']

def loadfile(filename, skip=None, *args):
    skip = set(skip or [])
    with open(filename, *args) as f:
        cr = csv.reader(f, quoting=csv.QUOTE_NONNUMERIC)
        return np.array(row for i,row in enumerate(cr) if i not in skip)
#plot data
outputs_l1 = [loadfile('C:\\Users\\Chris\\Desktop\\Work\\Python Stuff\\BPCROOM - Shingles analysis\\ERR analysis\\l_1 analysis//BS(1) ERR analysis - l_1 - P_3 = {}.csv'.format(p)) for p in p3_arr]

col = {name:i for i,name in enumerate(header)}

fig = plt.figure()
for data,color in zip(outputs_l1, colors):
    xs  = data[:, col["l1"     ]]
    gl = data[:, col["G_left" ]] * 1000.0    # column 12
    gr = data[:, col["G_right"]] * 1000.0    # column 13
    plt.plot(xs, gl, color + "-", gr, color + "--")
for output, col in zip(outputs_l1, colors):
    plt.plot(output[:,0], output[:,11]*1E3, col+'--')
plt.ticklabel_format(axis='both', style='plain', scilimits=(-1,1))
plt.xlabel('$l1 (m)$')
plt.ylabel('G $(J / m^2) * 10^{-3}$')
plt.xlim(xmin=.2)
plt.ylim(ymax=2, ymin=0)

plt.subplots_adjust(top=0.8, bottom=0.15, right=0.7)

After running the entire program, I recieve the error message:

Traceback (most recent call last):
  File "C:/Users/Chris/Desktop/Work/Python Stuff/New Stuff from Brenday 8 26 2014/CD_ssa_plot(2).py", line 115, in <module>
    xs  = data[:, col["l1"     ]]
IndexError: too many indices for array

and before I ran into that problem, I had another involving the line a few below the one the above error message refers to:

Traceback (most recent call last): File "FILE", line 119, in <module> 
gl = data[:, col["G_left" ]] * 1000.0 # column 12 
IndexError: index 12 is out of bounds for axis 1 with size 12

I understand the first error, but am just having problems fixing it. The second error is confusing for me though. My boss is really breathing down my neck so any help would be GREATLY appreciated!

Solution 1

I think the problem is given in the error message, although it is not very easy to spot:

IndexError: too many indices for array
xs  = data[:, col["l1"     ]]

'Too many indices' means you've given too many index values. You've given 2 values as you're expecting data to be a 2D array. Numpy is complaining because data is not 2D (it's either 1D or None).

This is a bit of a guess - I wonder if one of the filenames you pass to loadfile() points to an empty file, or a badly formatted one? If so, you might get an array returned that is either 1D, or even empty (np.array(None) does not throw an Error, so you would never know...). If you want to guard against this failure, you can insert some error checking into your loadfile function.

I highly recommend in your for loop inserting:

print(data)

This will work in Python 2.x or 3.x and might reveal the source of the issue. You might well find it is only one value of your outputs_l1 list (i.e. one file) that is giving the issue.

Solution 2

The message that you are getting is not for the default Exception of Python:

For a fresh python list, IndexError is thrown only on index not being in range (even docs say so).

>>> l = []
>>> l[1]
IndexError: list index out of range

If we try passing multiple items to list, or some other value, we get the TypeError:

>>> l[1, 2]
TypeError: list indices must be integers, not tuple

>>> l[float('NaN')]
TypeError: list indices must be integers, not float

However, here, you seem to be using matplotlib that internally uses numpy for handling arrays. On digging deeper through the codebase for numpy, we see:

static NPY_INLINE npy_intp
unpack_tuple(PyTupleObject *index, PyObject **result, npy_intp result_n)
{
    npy_intp n, i;
    n = PyTuple_GET_SIZE(index);
    if (n > result_n) {
        PyErr_SetString(PyExc_IndexError,
                        "too many indices for array");
        return -1;
    }
    for (i = 0; i < n; i++) {
        result[i] = PyTuple_GET_ITEM(index, i);
        Py_INCREF(result[i]);
    }
    return n;
}

where, the unpack method will throw an error if it the size of the index is greater than that of the results.

So, Unlike Python which raises a TypeError on incorrect Indexes, Numpy raises the IndexError because it supports multidimensional arrays.