Python: skip comment lines marked with # in csv.DictReader

Question

Processing CSV files with csv.DictReader is great - but I have CSV files with comment lines (indicated by a hash at the start of a line), for example:

# step size=1.61853
val0,val1,val2,hybridisation,temp,smattr
0.206895,0.797923,0.202077,0.631199,0.368801,0.311052,0.688948,0.597237,0.402763
-169.32,1,1.61853,2.04069e-92,1,0.000906546,0.999093,0.241356,0.758644,0.202382
# adaptation finished

The csv module doesn't include any way to skip such lines.

I could easily do something hacky, but I imagine there's a nice way to wrap a csv.DictReader around some other iterator object, which preprocesses to discard the lines.

Solution 1

Actually this works nicely with filter:

import csv
fp = open('samples.csv')
rdr = csv.DictReader(filter(lambda row: row[0]!='#', fp))
for row in rdr:
    print(row)
fp.close()

Solution 2

Good question. Python's CSV library lacks basic support for comments (not uncommon at the top of CSV files). While Dan Stowell's solution works for the specific case of the OP, it is limited in that # must appear as the first symbol. A more generic solution would be:

def decomment(csvfile):
    for row in csvfile:
        raw = row.split('#')[0].strip()
        if raw: yield raw

with open('dummy.csv') as csvfile:
    reader = csv.reader(decomment(csvfile))
    for row in reader:
        print(row)

As an example, the following dummy.csv file:

# comment
 # comment
a,b,c # comment
1,2,3
10,20,30
# comment

returns

['a', 'b', 'c']
['1', '2', '3']
['10', '20', '30']

Of course, this works just as well with csv.DictReader().

Solution 3

Another way to read a CSV file is using pandas

Here's a sample code:

df = pd.read_csv('test.csv',
                 sep=',',     # field separator
                 comment='#', # comment
                 index_col=0, # number or label of index column
                 skipinitialspace=True,
                 skip_blank_lines=True,
                 error_bad_lines=False,
                 warn_bad_lines=True
                 ).sort_index()
print(df)
df.fillna('no value', inplace=True) # replace NaN with 'no value'
print(df)

For this csv file:

a,b,c,d,e
1,,16,,55#,,65##77
8,77,77,,16#86,18#
#This is a comment
13,19,25,28,82

we will get this output:

       b   c     d   e
a                     
1    NaN  16   NaN  55
8   77.0  77   NaN  16
13  19.0  25  28.0  82
           b   c         d   e
a                             
1   no value  16  no value  55
8         77  77  no value  16
13        19  25        28  82

Comments

[Duncan]

That will read the whole file into memory. If it isn't too large then no problem, otherwise you might want to use a generator expression or itertools.ifilter().

[Andy Mikhaylenko]

...or a generator expression: csv.DictReader(row for row in fp if not row.startswith('#'))

[The Aelfinn]

@Duncan no need for itertools in Python3.6, as filter() will return an iterator by default, therefore the file will not be loaded into memory.

[Lacek]

pandas is indeed a powerful library, yet it is a dependency that require setup and learning to use. Moreover, the author had already stated in the question that he simply wanted to use the built-in csv.DictReader module and relevant answers were provided years ago already. I don't understand why you add this solution as an alternative.

[Granny Aching]

The author of the question might not need pandas. But the purpose of this forum is more than just help each question's author with their specific problem.

[Thibault Reuille]

I believe you meant "yield row" not "yield raw" in the decomment() function. A CSV file can contain # characters in a string and it is perfectly valid.

[sigvaldm]

@ThibaultReuille: It is true that many CSV files can contain # in strings, although the CSV format is not well standardized. I meant yield raw. My suggestion would not deal with # in strings in any case.

[sigvaldm]

@ThibaultReuille: What you're pointing at is exactly why it is inadvisable to manually type a lot of code for something a library can do for you; you probably won't get all the details right the first time (for instance, you could also have newlines in strings), and it will take away time from the task you're actually solving. I consider my solution a quick fix for something that ought to have been in csv. If it would need considerable expansion to work for you, perhaps you should consider another csv library, for instance the one in pandas. Hope that helps.

[sigvaldm]

This will not work when comments follow at the end of rows, e.g., a,b,c # comment.

[Micheal J. Roberts]

@GrannyAching What exactly does .sort_index() achieve here? :)

[Ulf Gjerdingen]

pretty sure @Andy Mikhaylenko's generator expression worked really well but it doesn't any more. what up? (Python 3.7.5)