how to merge 200 csv files in Python

Question

Guys, I here have 200 separate csv files named from SH (1) to SH (200). I want to merge them into a single csv file. How can I do it?

Solution 1

As ghostdog74 said, but this time with headers:

fout=open("out.csv","a")
# first file:
for line in open("sh1.csv"):
    fout.write(line)
# now the rest:    
for num in range(2,201):
    f = open("sh"+str(num)+".csv")
    f.next() # skip the header
    for line in f:
         fout.write(line)
    f.close() # not really needed
fout.close()

Solution 2

Why can't you just sed 1d sh*.csv > merged.csv?

Sometimes you don't even have to use python!

Solution 3

Use accepted StackOverflow answer to create a list of csv files that you want to append and then run this code:

import pandas as pd
combined_csv = pd.concat( [ pd.read_csv(f) for f in filenames ] )

And if you want to export it to a single csv file, use this:

combined_csv.to_csv( "combined_csv.csv", index=False )

Solution 4

fout=open("out.csv","a")
for num in range(1,201):
    for line in open("sh"+str(num)+".csv"):
         fout.write(line)    
fout.close()

Solution 5

I'm just going to throw another code example into the basket:

from glob import glob

with open('singleDataFile.csv', 'a') as singleFile:
    for csvFile in glob('*.csv'):
        for line in open(csvFile, 'r'):
            singleFile.write(line)

View more solutions