ValueError: Item wrong length 907 instead of 2000

Question

I have a csv file, that has 1000 columns. I need to read only the first 100 columns. I wrote this program for that:

import pandas as pd

list = []
for i in range (1, 100):
    list.append(i)
df = pd.read_csv('piwik_37_2016-07-08.csv',dtype = "unicode")
df = df[df.columns.isin(list)]
df.to_csv('abc.csv', index = False)

But I get error: ValueError: Item wrong length 907 instead of 2000. Can't figure out where I went wrong

Solution 1

There are a lot of things strange about your code. For example, there is no reason to iterate over the range object and update a list just to get a list of numbers. Just use list(range(1,100)).

However, if you just need the first 100 columns in the csv, there is built-in functionality for what you are trying to do:

df = pd.read_csv('piwik_37_2016-07-08.csv',dtype = "unicode", usecols = list(range(100)))

Solution 2

Though its down voted, still answering as its a small correction in the code.

import pandas as pd

l = list(range(0,100))
df = pd.read_csv('piwik_37_2016-07-08.csv',dtype = "unicode")
df = df.loc[:,df.columns.isin(l)]
df.to_csv('abc.csv', index = False)

Comments

[Andy Hayden]

Wow, I didn't realise that this would short circuit (skip to the next line) even if the csv was malformed. Nice!

[Diganta Bharali]

Completely new to python. I tried your code. It says TypeError: 'list' object is not callable

[Andy Hayden]

@DigantaBharali yes, because you overrode list. Don't do that.

[juanpa.arrivillaga]

@DigantaBharali Because you shadowed the built-in list function with an actual list object. Don't do that.

[Diganta Bharali]

Sorry for asking a dumb question, I still don't see the remedy. I did import pandas as pd df = pd.read_csv('piwik_37_2016-07-08.csv',dtype = "unicode", usecols = list(range(1,100))) df.to_csv('abc.csv', index = False). I recieve the list obect not callable error

[juanpa.arrivillaga]

@DigantaBharali Are you in the same interpreter session? You need to restart the interpreter. Also, if you use list(range(1,100)) it will give you columns 2,3,...100. if you want columns 1,2,3...100 then you want to use list(range(100)) because it accepts a list of the indices.

[Diganta Bharali]

Working. I was running it in the same session. Can you guide me to some resources where I can get started with Python.

[Diganta Bharali]

@junapa.arrivillaga, And if it is not too much to ask, could you guide me as to where my code went wrong?

[Merlin]

@DigantaBharali --too many places the code isnt pythonic... Need a good tutorial-- gave you one.

[Toodoo]

Can you explain your answer please ?

[Shahidur]

Hi @Toodoo, the difference is in using loc convention from pandas. Selecting columns using boolean. df = df.loc[:,df.columns.isin(l)]