Python algorithm of counting occurrence of specific word in csv
Solution 1
Basic example, with using csv
and collections.Counter
(Python 2.7+) from standard Python libraly:
import csv
import collections
grades = collections.Counter()
with open('file.csv') as input_file:
for row in csv.reader(input_file, delimiter=';'):
grades[row[1]] += 1
print 'Number of A grades: %s' % grades['A']
print grades.most_common()
Output (for small dataset):
Number of A grades: 2055
[('A', 2055), ('B', 2034), ('D', 1995), ('E', 1977), ('C', 1939)]
Solution 2
You should of course read all the grades, which in this case also means reading the entire file. You can use the csv
module to easily read comma separated value files:
import csv
my_reader = csv.reader(open('my_file.csv'))
ctr = 0
for record in my_reader:
if record[1] == 'A':
ctr += 1
print(ctr)
This is pretty fast, and I couldn't do better with the Counter
method:
from collections import Counter
grades = [rec[1] for rec in my_reader] # generator expression was actually slower
result = Counter(grades)
print(result)
Last but not least, lists have a count
method:
from collections import Counter
grades = [rec[1] for rec in my_reader]
result = grades.count('A')
print(result)
laotanzhurou
Updated on June 29, 2022Comments
-
laotanzhurou almost 2 years
I've just started to learn python. I'm curious about what are the efficient ways to count the occurrence of a specific word in a CSV file, other than simply use for loop to go through line by line and read.
To be more specific, let's say I have a CSV file contain two columns, "Name" and "Grade", with millions of records.
How would one count the occurrence of "A" under "Grade"?
Python code samples would be greatly appreciated!
-
Admin over 12 yearsYou have to read the whole file, otherwise your algorithm can be proven to be incorrect. Reading it linearly, line by line, is not a bad approach.
-
agf over 12 years
import csv; count = sum(1 for row in csv.dictreader(open(filename)) if row['Grade'] == 'A')
-
steabert over 12 years@agf: nice, but when I tried this it was a factor of 6-8 slower than the other answers
-
agf over 12 years@steabert That speed factor almost certainly doesn't matter.
-
-
steabert over 12 yearsOK, but you could apply
Counter
on a generator expression for the first element of the lines in the file -
laotanzhurou over 12 yearsThanks! I accepted your answer. But I was wondering comparing this with using dictionary, with grade as key and occurrence as value, which way will be more efficient?
-
reclosedev over 12 years@laotanzhurou,
Counter
is adict
subclass, but it's little slower. If you really need speedupcollections.defaultdict(int)
orif ... count += 1
probably will be faster. But you always can benchmark it by yourself with timeit, see Johnsyweb's answer