Checking if directory in HDFS already exists or not

Question

I am having following directory structure in HDFS,

/analysis/alertData/logs/YEAR/MONTH/DATE/HOURS

That is data is coming on houly basis and stored in format of year/month/day/hour.

I have written a shell script in which i am passing path till

"/analysis/alertData/logs"   ( this will vary depending on what product of data i am handling)

then shell script go through the year/month/date/hour folders and return the most latest path.

For example:

 Directories present in HDFS has following structure: 

 /analysis/alertData/logs/2014/10/22/01
 /analysis/alertData/logs/2013/5/14/04

 shell script is given path till :   " /analysis/alertData/logs "

 it outputs most recent directory :    /analysis/alertData/logs/2014/10/22/01

My question is here is how can i validate whether HDFS directory path pass to shell script is valid or not. Lets say i pass a wrong path as input or path which does not exist so how to handle that in shell script.

Sample wrong path can be:

  wrong path   :  /analysis/alertData ( correct path :  /analysis/alertData/logs/ )
  wrong path   :  /abc/xyz/  ( path does not exit in HDFS )

I tried using Hadoop dfs -test -z/-d/-e options did not worked for me. Any suggestion for this.

NOTE : Not posting my original code here, as solution to my problem does not depend on it.

Thanks in advance.

Solution 1

Try w/o test command []:

if $(hadoop fs -test -d $yourdir) ; then echo "ok";else echo "not ok"; fi

Solution 2

Since

hdfs dfs -test -d $yourdir

return 0 if exists, then

if [ $? == 0 ]; then
    echo "exists"
else
    echo "dir does not exists"
fi

Solution 3

Hadoop fs is deprecated Usage: hdfs dfs -test -[ezd] URI

Options: The -e option will check to see if the file exists, returning 0 if true. The -z option will check to see if the file is zero length, returning 0 if true. The -d option will check to see if the path is directory, returning 0 if true. Example: hdfs dfs -test -d $yourdir

Please check the following for more info: https://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-common/FileSystemShell.html Regards

Solution 4

Hi I have used following script to test the HDFS directory exists or not. I have seen in your question that you tried this test command and not worked. Could you please provide any trace on why this not working..

 hadoop fs -test -d $dirpath
    if [ $? != 0 ]
            then
                hadoop fs -mkdir $dirpath
                else
                    echo "Directory already present in HDFS"
    fi

View more solutions