hdfs copy multiple files to same target directory
Try to use the following syntax:
hadoop fs -cp /path1/file1 /path2/file2 path3/file3 /pathx/target
R. Sluiter
I'm a Hadoop developer with advanced skills in hive/sqoop/oozie
Updated on June 08, 2022Comments
-
R. Sluiter almost 2 years
I learned that if you want to copy multiple files from one hadoop folder to another hadoop folder you can better create one big 'hdfs dfs -cp' statement with lots of components, instead of creating several hdfs dfs -cp statements. With 'better' I mean that it will improve the overal time it takes to copy files: one command is quicker than several separate -cp commands run after each other.
When I do this and my target directory is the same for all files that I want to copy I get a warning.
I'm executing the following command:
hdfs dfs -cp -f /path1/file1 /pathx/target /path2/file2 /pathx/target /path3/file3 /pathx/target
After executing it I get the following warning returned:
cp: `/pathx/target' to `/pathx/target/target': is a subdirectory of itself
Although I get this weird warning the copy itself succeeds like it should. Is this a bug or am I missing something?
-
R. Sluiter over 7 yearsughh that was is. Thanks! I should have check the documentation first... hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/… so it's
hdfs dfs -cp source1 source2 source3 target