Can hadoop take input from multiple directories and files
11,100
You can take input from multiple directories and files by using the ***** operator. Most likely it's because the "arg[0]" argument isn't correct and therefore it's not finding the files.
As an alternative, you can also use InputFormat.addInputPath or if you need separate formats or mappers the MultipleInputs class can be used.
Example of basic adding a path
FileInputFormat.addInputPath(job, myInputPath);
Here is an example of MultipleInputs
MultipleInputs.addInputPath(job, inputPath1, TextInputFormat.class, MyMapper.class);
MultipleInputs.addInputPath(job, inputPath2, TextInputFormat.class, MyOtherMapper.class);
This other question is also very similar and has good answers, Hadoop to reduce from multiple input formats.
Author by
JudyJiang
Updated on June 17, 2022Comments
-
JudyJiang almost 2 years
As I set the fileinputFormat as hadoop input. The
arg[0]+"/*/*/*"
said match no files.what I want to is to read from multiple files as:
Directory1 ---Directory11 ---Directory111 --f1.txt --f2.txt ---Directory12 Directory2 ---Directory21
is it possible in Hadoop? Thanks!