fread (data.table in R) with specification of encoding

Question

Could not find proper answer in previous questions and answers to my problem: 1. I have a 2.3 GB csv file which contains 2.4 million rows of Hebrew Text, currently coded in ASCII. Since we are talking about big file, fread would be preferable but what about the encoding? Any idea how to read csv file coded in ASCII to avoid the famous "embedded nul in string" error?

Thank you

Solution

As of August 25th the case linked by David Arenburg is closed, and the functionality is included in the currently available version of data.table. The encoding parameter can now be used when calling fread:

text <- fread(file, encoding = 'UTF-8')

ASCII is not an explicit encoding option, but ASCII is valid UTF-8, so you can specify UTF-8 when you want to read your Hebrew text.

Comments

[Jeff]

I am using data.table 1.9.7 (confirmed with sessionInfo()) and I get this error: Error in fread("data.csv", encoding = "UTF-8") : unused argument (encoding = "UTF-8")