How do I read line by line a text file in ruby (hosting it on s3)?

ruby-on-rails ruby text import heroku

71,445

Solution 1

File.open("my/file/path", "r").each_line do |line|
  # name: "Angela"    job: "Writer"    ...
  data = line.split(/\t/)
  name, job = data.map{|d| d.split(": ")[1] }.flatten
end

Solution 2

You want IO.foreach:

IO.foreach('foo.txt') do |line|
  # process the line of text here
end

Alternatively, if it really is tab-delimited, you might want to use the CSV library:

File.open('foo.txt') do |f|
  CSV.foreach(f, col_sep:"\t") do |csv_row|
    # All parsed for you
  end
end

Solution 3

  IO.foreach("input.txt") do |line| 
    out.puts line
    # You might be able to use split or something to get attributes
    atts = line.split
  end

Solution 4

You can use OpenURI to read remote or local files.

Assuming that your model has an attachment named file:

# If object is stored in amazon S3, access it through url
file_path = record.file.respond_to?(:s3_object) ? record.file.url : record.file.path
open(file_path) do |file|
  file.each_line do |line|
    # In your case, you can split items using tabs
    line.split("\t").each do |item|
      # Process item
    end
  end
end

Solution 5

Have you tried using OpenURI (http://ruby-doc.org/stdlib-2.1.2/libdoc/open-uri/rdoc/OpenURI.html)? You would have to make your files accessible from S3.

Or try using de aws-sdk gem (http://aws.amazon.com/sdk-for-ruby).

View more solutions

71,445

Author by

Satchel

Former electrical engineer but learning ruby from scratch thanks to stack overflow. Still so much to learn.

Updated on January 01, 2020

Comments

Satchel over 4 years

I know I've done this before and found a simple set of code, but I cannot remember or find it :(.

I have a text file of records I want to import into my Rails 3 application.

Each line represents a record. Potentially it may be tab delimited for the attributes, but am fine with just a single value as well.

How do I do this?
Patm almost 11 years

This doesn't address the problem of the file being on s3
fl00r almost 11 years

@Patm, oh I see :) But at the moment all three answerers were answering the topic wasn't about S3. It was just How do I read line by line a text file in ruby
Satchel over 9 years

Is there a way to do so when the file sits in s3?
Satchel over 9 years

the files would have permission , but I guess it still needs to be able to use the answer at the top using the File class to go through it and I wasn't clear it could.
Tejas Manohar over 9 years

@Angela File.open("insert url here")
Joshua Pinter almost 9 years

Does this close the file properly?
Zach Smith over 6 years

Does IO.foreach provide an iterator?
hagello over 6 years

@fl00r What makes you think that this code properly closes the file? See: a = File.open('path', 'r'); a.each_line do |line| puts line end; puts a.closed? prints false!
hagello over 6 years

@fl00r According to the link you are quoting, an explicit close is necessary.