HDFS from Java - Specifying the User

Question

I'm happily connecting to HDFS and listing my home directory:

Configuration conf = new Configuration();
conf.set("fs.defaultFS", "hdfs://hadoop:8020");
conf.set("fs.hdfs.impl", "org.apache.hadoop.hdfs.DistributedFileSystem");
FileSystem fs = FileSystem.get(conf);
RemoteIterator<LocatedFileStatus> ri = fs.listFiles(fs.getHomeDirectory(), false);
while (ri.hasNext()) {
    LocatedFileStatus lfs = ri.next();
    log.debug(lfs.getPath().toString());
}
fs.close();

What I'm wanting to do now though is connect as a specific user (not the whois user). Does anyone know how you specify which user you connect as?

Solution

As soon as I see this is done through UserGroupInformation class and PrivilegedAction or PrivilegedExceptionAction. Here is sample code to connect to remote HDFS 'like' different user ('hbase' in this case). Hope this will solve your task. In case you need full scheme with authentication you need to improve user handling. But for SIMPLE authentication scheme (actually no authentication) it works just fine.

package org.myorg;

import java.security.PrivilegedExceptionAction;

import org.apache.hadoop.conf.*;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.FileStatus;

public class HdfsTest {

    public static void main(String args[]) {

        try {
            UserGroupInformation ugi
                = UserGroupInformation.createRemoteUser("hbase");

            ugi.doAs(new PrivilegedExceptionAction<Void>() {

                public Void run() throws Exception {

                    Configuration conf = new Configuration();
                    conf.set("fs.defaultFS", "hdfs://1.2.3.4:8020/user/hbase");
                    conf.set("hadoop.job.ugi", "hbase");

                    FileSystem fs = FileSystem.get(conf);

                    fs.createNewFile(new Path("/user/hbase/test"));

                    FileStatus[] status = fs.listStatus(new Path("/user/hbase"));
                    for(int i=0;i<status.length;i++){
                        System.out.println(status[i].getPath());
                    }
                    return null;
                }
            });
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Comments

[Kong]

I'm actually wanting to browse the entire HDFS file system, like what Hue does. i.e. be the hdfs user.

[Gibbs]

I searched for hadoop configuration parameter settings and i don't get any useful explanations. Does hbase means user in conf.set. Where can i get all hadoop class explanations not only avail classes. Any links please?

[Roman Nikitchenko]

org.apache.hadoop.security.UserGroupInformation is key class here and it allows to specify 'remote user' through createRemoteUser(). So Hadoop cluster sees you as user you have specified nidependent of your local user. Of course this works only if your Hadoop cluster trusts you (SIMPLE authentication, actually none). With Kerberous you additionally will need to provide proofs.

[AbtPst]

i tried this but i get java.lang.UnsupportedOperationException: Not implemented by the DistributedFileSystem FileSystem implementation

[Roman Nikitchenko]

@AbtPst As you can see people actively use this and I have only 2 guesses for you: 1) Check exception stacktraace to isolate root cause. 2) Check your addresses / ports / paths. Maybe you just use wrong user name.

[AbtPst]

how would i get a list of correct usernames? please forgive me if this is a basic question but i am quite new to hadoop and unix

[Roman Nikitchenko]

In face there is no single way: hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hd‌​fs/…