Hive - How to print the classpath of a Hive service

Question

I need to check the classpath of the Hive service to see the location of the jars it loads while running the hive queries.

I want to update the parquet jars for hive to latest parquet jars to read new parquet format data.

I have updated the jars in hive lib location(/usr/hdp/2.5.XX/hive/lib/) but it is still using the old jars from some other location.

I tried below command to list jars but no output.

hive>list jars;

I have tried adding the new jars using

add jar <'jar file>

but it is still picking the old jars.

Is there any way to find out the classpath or jars used for the hive service?

Solution 1

Run below command to get the hive command location

which hive

Open 'hive' file under /usr/bin/(Your hive location)

vi /usr/bin/hive

You should see something like below. Take a backup of the hive file and add an echo command for the HADOOP_CLASSPATH at the end before exec as below.

#!/bin/bash

if [ -d "/usr/hdp/2.5.0.0-1245/atlas/hook/hive" ]; then
 if [ -z "${HADOOP_CLASSPATH}" ]; then
  export HADOOP_CLASSPATH=/usr/hdp/2.5.0.0-1245/atlas/hook/hive/*
 else
  export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/usr/hdp/2.5.0.0-1245/atlas/hook/hive/*
 fi
fi

...

if [ -z "${HADOOP_CLASSPATH}" ]; then
 export HADOOP_CLASSPATH=${HCATALOG_JAR_PATH}
else
 export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:${HCATALOG_JAR_PATH}
fi

####### Prints hadoop classpath

echo "Classpath=$HADOOP_CLASSPATH"

exec "${HIVE_HOME}/bin/hive.distro" "$@"

Run hive command to display the classpath.

The parquet issue got resolved by adding the new parquet jar location to the environment variable 'HADOOP_CLASSPATH'

Solution 2

To list down jar path use command list jars more details ;

and Add jar to hive.aux.jars.path location in hive-site.xml (add auxiliary jar)

sample of hive-site.xml

<property>
<name>hive.aux.jars.path</name>
<value>file://localpath/yourjar.jar</value>
</property>

Update

check hive hive.aux.jars.path property and look for the physical path mention.

Add <jar> full path.

Solution 3

If you are sure that it picks up an older version of Parquet, then it must be present on the machine, so you can just look for all parquet jars in the filesystem: find / -name 'parquet-*.jar'

If you want to check which specific jar it uses from the ones available on the computer, you may try to use lsof for this purpose. I would start with lsof | grep parquet | grep jar and further fine-tune the filtering if needed.

Comments

[Munesh]

I have tried searching for the parquet jar but couldn't find it. Looks like the jar is not named as parquet*.jar

[Munesh]

I tried adding the jar path in aux path but it didn't pick up the new one. Tried list jars; but no jars listed as it displays only the jars added througj 'add jar' command