"Wrong FS... expected: file:///" when trying to read file from HDFS in Java
25,445
Try this:
Configuration configuration = new Configuration();
FileSystem fs = FileSystem.get(new URI(<url:port>), configuration);
Path filePath = new Path(<path/to/file>);
FSDataInputStream fsDataInputStream = fs.open(filePath);
BufferedReader br = new BufferedReader(new InputStreamReader(fsDataInputStream));
Please refer to http://techidiocy.com/java-lang-illegalargumentexception-wrong-fs-expected-file/
A similar problem is addressed.
Author by
jds
Updated on July 09, 2022Comments
-
jds almost 2 years
I am unable to read a file from HDFS using Java:
String hdfsUrl = "hdfs://<ip>:<port>"; Configuration configuration = new Configuration(); configuration.set("fs.defaultFS", hdfsUrl); FileSystem fs = FileSystem.get(configuration); Path filePath = new Path(hdfsUrl + "/projects/harmonizome/data/achilles/attribute_list_entries.txt.gz"); FSDataInputStream fsDataInputStream = fs.open(filePath); SEVERE: Servlet.service() for servlet [edu.mssm.pharm.maayanlab.Harmonizome.api.DownloadAPI] in context with path [/Harmonizome] threw exception java.lang.IllegalArgumentException: Wrong FS: hdfs://146.203.54.165:8020/projects/harmonizome/data/achilles/attribute_list_entries.txt.gz, expected: file:/// at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:310) at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:47) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:357) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125) at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:356) at edu.mssm.pharm.maayanlab.Harmonizome.api.DownloadAPI.readLines(DownloadAPI.java:37) at edu.mssm.pharm.maayanlab.Harmonizome.api.DownloadAPI.doGet(DownloadAPI.java:27) at javax.servlet.http.HttpServlet.service(HttpServlet.java:622) ...
I didn't setup our HDFS, so I don't know what I don't know. Any help is appreciated.
-
scoa over 8 yearsWhile this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes.
-
jds over 8 yearsI agree with @scoa. I was able to resolve my problem using this link and have edited your answer with the working code.
-
java_bee over 8 years@scoa thanks for suggestion. I will follow this in future.
-
Worse_Username almost 7 yearsI don't get it. What was the root cause?
-
akki almost 3 yearsNow the link is invalid & I can't know the root cause anymore 😅
-
theannouncer almost 3 yearsArchived version: web.archive.org/web/20150805043209/http://techidiocy.com/…
-
theannouncer almost 3 yearsFrom link, for posterity: "Your Configuration object is not initialized properly because of that it is trying to find this file in your local file system where it doesn’t exists because file [you are accessing] exists on HDFS."