Read csv into a list in clojure

Question

I know there are a lot of related questions, I have read them but still have not gained some fundamental understanding of how to read-process-write. Take the following function for example which uses clojure-csv library to parse a line

(defn take-csv
  "Takes file name and reads data."
  [fname]
  (with-open [file (reader fname)]
    (doseq [line (line-seq file)]
      (let [record (parse-csv line)]))))

What I would like to obtain is data read into some collection as a result of (def data (take-csv "file.csv")) and later to process it. So basically my question is how do I return record or rather a list of records.

Solution 1

"doseq" is often used for operations with side effect. In your case to create collection of records you can use "map":

(defn take-csv
  "Takes file name and reads data."
  [fname]
  (with-open [file (reader fname)]
    (doall (map (comp first csv/parse-csv) (line-seq file)))))

Better parse the whole file at ones to reduce code:

(defn take-csv
  "Takes file name and reads data."
  [fname]
  (with-open [file (reader fname)]
    (csv/parse-csv (slurp file))))

You also can use clojure.data.csv instead of clojure-csv.core. Only should rename parse-csv to take-csv in previous function.

(defn put-csv [fname table]
  (with-open [file (writer fname)]
    (csv/write-csv file table)))

Solution 2

With all the things you can do with .csv files, I suggest using clojure-csv or clojure.data.csv. I mostly use clojure-csv to read in a .csv file.

Here are some code snippets from a utility library I use with most of my Clojure programs.

from util.core

    (ns util.core
      ^{:author "Charles M. Norton",
        :doc "util is a Clojure utilities directory"}

      (:require [clojure.string :as cstr])
      (:import java.util.Date)
      (:import java.io.File)
      (:use clojure-csv.core))

(defn open-file
"Attempts to open a file and complains if the file is not present."

[file-name]
(let [file-data (try 
               (slurp file-name)
               (catch Exception e (println (.getMessage e))))]
  file-data))

(defn ret-csv-data
"Returns a lazy sequence generated by parse-csv.
 Uses open-file which will return a nil, if
 there is an exception in opening fnam.

 parse-csv called on non-nil file, and that
 data is returned."

[fnam]
(let [csv-file (open-file fnam)
      inter-csv-data (if-not (nil? csv-file)
                       (parse-csv csv-file)
                        nil)

      csv-data 
        (vec (filter #(and pos? (count %) 
           (not (nil? (rest %)))) inter-csv-data))]

    (if-not (empty? csv-data)
      (pop csv-data)
       nil)))

(defn fetch-csv-data
    "This function accepts a csv file name, and returns parsed csv data,
     or returns nil if file is not present."

    [csv-file]
        (let [csv-data (ret-csv-data csv-file)]
            csv-data))

Once you've read in a .csv file, then what you do with its contents is another matter. Usually, I am taking .csv "reports" from one financial system, like property assessments, and formatting the data to be uploaded into a database of another financial system, like billing.

I will often either zipmap each .csv row so I can extract data by column name (having read in the column names), or even make a sequence of zipmap'ped .csv rows.

Solution 3

Just to add this good answers, here is a full example

First, add clojure-csv into your dependencies

(ns scripts.csvreader
  (:require [clojure-csv.core :as csv]
            [clojure.java.io :as io]))

(defn take-csv
 "Takes file name and reads data."
 [fname]
 (with-open [file (io/reader fname)]
  (-> file
      (slurp)
      (csv/parse-csv))))

usage (take-csv "/path/youfile.csv")