How to fix Hadoop WARNING: An illegal reflective access operation has occurred error on Ubuntu

24,842

Solution 1

This has been fixed in Hadoop 3.3 and 3.2.2 via. https://issues.apache.org/jira/browse/HADOOP-10848

Solution 2

There is nothing you can do about these warnings, this is related to jigsaw project and strong(er) encapsulation.

Basically there is some class called sun.security.krb5.Config that is part of some "module" called java.security.jgss. This module "defines" what it exports (what others can use out of it) and to whom. This also means in plain english that this is not for the public usage - don't touch it; well hadoop did, it's part of their effort to fix this. You can report this or try to upgrade hadoop, may be this is already fixed.

Solution 3

this problem is due to latest java upgrades.please configure it with java 8.its working swiftly.

Share:
24,842
Amalendu Kar
Author by

Amalendu Kar

Updated on July 05, 2022

Comments

  • Amalendu Kar
    Amalendu Kar almost 2 years

    I have installed Java openjdk version "10.0.2"and Hadoop 2.9.0 successfully. All processes are running well

    hadoopusr@amalendu:~$ jps
    19888 NameNode
    20388 DataNode
    20898 NodeManager
    20343 SecondaryNameNode
    20539 ResourceManager
    21118 Jps
    

    But when ever i am trying to execute any command like hdfs dfs -ls / getting this warnings

    hadoopusr@amalendu:~$ hdfs dfs -ls /
    WARNING: An illegal reflective access operation has occurred
    WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/usr/local/hadoop/share/hadoop/common/lib/hadoop-auth-2.9.0.jar) to method sun.security.krb5.Config.getInstance()
    WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
    WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
    WARNING: All illegal access operations will be denied in a future release
    18/09/04 00:29:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    

    Please help how to fix this. This is my ~/.bashrc file configuration

    export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
    export HADOOP_HOME=/usr/local/hadoop
    export PATH=$PATH:$HADOOP_HOME/bin
    export PATH=$PATH:$HADOOP_HOME/sbin
    export HADOOP_MAPRED_HOME=$HADOOP_HOME
    export HADOOP_COMMON_HOME=$HADOOP_HOME
    export HADOOP_HDFS_HOME=$HADOOP_HOME
    export YARN_HOME=$HADOOP_HOME
    export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
    export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
    
  • Amalendu Kar
    Amalendu Kar almost 6 years
    Thanks, But is there any way to hide this warnings ?
  • Eugene
    Eugene almost 6 years
    @AmalenduKar no, these are issued by the compiler
  • Amalendu Kar
    Amalendu Kar almost 6 years
    Sir i removed java 11 and installed java 8, All are gone but "WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable " this occour.
  • Eugene
    Eugene almost 6 years
    @AmalenduKar this is entirely different and you should post a different question about it most probably
  • John Hanley
    John Hanley almost 6 years
    @AmalenduKar The message means that the hadoop native libraries are compiled 32-bit and you are running on a 64-bit setup. You can ignore that message for a small Hadoop setup.
  • Alan Bateman
    Alan Bateman almost 6 years
    You can use the --add-opens option to temporarily open the sun.security.krb5 package. That will eliminate the warning but it is important to make sure that the Hadoop maintainers are aware of the issue. Once the JDK internals are fully encapsulated then hacks like this will break.
  • Eugene
    Eugene almost 6 years
    @AlanBateman that is one of the reasons I did not want to post that... thank you!