How can i make a new directory in hdfs with java?

14,718

For your issue ,you have to add commons-configuration-1.6.jar jar..

i listed out necessary jars below

{ 
   Configuration config = new Configuration();
   config.addResource(new Path("/etc/hadoop/conf/core-site.xml"));
   config.addResource(new Path("/etc/hadoop/conf/hdfs-site.xml"));

   config.set("fs.hdfs.impl", 
            org.apache.hadoop.hdfs.DistributedFileSystem.class.getName()
        );
       config.set("fs.file.impl",
            org.apache.hadoop.fs.LocalFileSystem.class.getName()
        );
  FileSystem dfs = FileSystem.get(config);
  String dirName = "TestDirectory";
  System.out.println(dfs.getWorkingDirectory() +" this is from /n/n");
  Path src = new Path(dfs.getWorkingDirectory()+"/"+dirName);

   dfs.mkdirs(src); 

} }

You have to add below list of jars in your build path.

commons-cli-1.2.jar

commons-collections-3.2.1.jar

commons-configuration-1.6.jar

commons-lang-2.5.jar

commons-logging-1.1.1.jar

guava-11.0.2.jar

hadoop-auth.jar

hadoop-common.jar

protobuf-java-2.4.0a.jar

slf4j-api-1.6.1.jar

log4j-1.2.17.jar

hadoop-hdfs.jar

These all jars available in hadoop/lib folder if it is cloudera.

Share:
14,718
Ahmed Shalash
Author by

Ahmed Shalash

java and python Developer in faculty of computer and informatics i called my self (javatar) as i love javaaaa&hadoop so much

Updated on June 04, 2022

Comments

  • Ahmed Shalash
    Ahmed Shalash almost 2 years
    public static void main(String[] args) throws IOException, URISyntaxException 
    

    { Configuration config = new Configuration();

      config.set("fs.default.name","hdfs://127.0.0.1:50070/dfshealth.jsp");
    
      FileSystem dfs = FileSystem.get(config);
      String dirName = "TestDirectory";
    
      Path src = new Path(dfs.getWorkingDirectory()+"/"+dirName);
    
      dfs.mkdirs(src); 
    

    } }

    Thier was an exception

    Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration
        at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:37)
        at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:34)
        at org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51)
        at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:217)
        at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:185)
        at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:237)
        at org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:79)
        at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:210)
        at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:185)
        at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:237)
        at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:482)
        at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:468)
        at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1519)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1420)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123)
        at com.TestConnection.main(TestConnection.java:21)
    Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration.Configuration
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        ... 17 more
    

    Configration is true what is the problem ? any help !!!