Package org.apache.hadoop.conf does not exist
23,108
Solution 1
You can run it like following:
javac WordCount.java -cp $(hadoop classpath)
If you get following error for this:
zsh: command not found: hadoop
javac: -cp requires an argument
You can try to find exact location of hadoop, as following worked for me, as I had installed in Mac using brew(It will be different in Linux):
javac WordCount.java -cp $(/usr/local/Cellar/hadoop/2.8.0/bin/hadoop classpath)
The hadoop classpath provides the compiler with all the paths it needs to compile correctly and you should see a resulting WordCount.class appear in the directory.
Solution 2
I too have Hadoop 2.7.1 and I am able to compile WordCount.
All your missing packages are from hadoop-common and hadoop-core jars.
You need to have following Maven dependencies included in your pom.xml (I just have following dependencies and it compiles for me):
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.7.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.7.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>1.2.1</version>
</dependency>
</dependencies>
Author by
g_hat
Updated on July 10, 2022Comments
-
g_hat almost 2 years
I am new to compiling Hadoop code and know this question has been answered multiple times. But, I still find it hard figuring out the correct classpath to make the WordCount.java code to compile. I am running a CentOS version 7.
My Hadoop Version is:
Hadoop 2.7.1 Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 15ecc87ccf4a0228f35af08fc56de536e6ce657a Compiled by jenkins on 2015-06-29T06:04Z Compiled with protoc 2.5.0 From source with checksum fc0a1a23fc1868e4d5ee7fa2b28a58a This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-2.7.1.jarWhen I did echo $PATH, I got the following:
/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/usr/lib/jvm/jre-1.7.0-openjdk/:/usr/local/hadoop/bin:/usr/local/hadoop/sbin:/home/hduser/.local/bin:/home/hduser/bin:/usr/lib/jvm/jre-1.7.0-openjdk/:/usr/local/hadoop/bin:/usr/local/hadoop/sbin
The Java Version I am running is:
java version "1.7.0_91" OpenJDK Runtime Environment (rhel-2.6.2.1.el7_1-x86_64 u91-b00) OpenJDK 64-Bit Server VM (build 24.91-b01, mixed mode)I did set all the environmental variables for Java and Hadoop in the vi .bashrc as shown below and jps passes. Java also compiles with other codes, but I think I am getting the classpath for Hadoop wrong.
Java Variables
export JAVA_HOME=/usr/lib/jvm/jre-1.7.0-openjdk/ export PATH=$PATH:$JAVA_HOMEHadoop variables
export HADOOP_INSTALL=/usr/local/hadoop export PATH=$PATH:$HADOOP_INSTALL/bin export PATH=$PATH:$HADOOP_INSTALL/sbin export HADOOP_MAPRED_HOME=$HADOOP_INSTALL export HADOOP_COMMON_HOME=$HADOOP_INSTALL export HADOOP_HDFS_HOME=$HADOOP_INSTALL export YARN_HOME=$HADOOP_INSTALL export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/nativeI get these errors when i try to compile the WordCount.java in the working directory:
WordCount.java:5: error: package org.apache.hadoop.conf does not exist import org.apache.hadoop.conf.*; ^ WordCount.java:6: error: package org.apache.hadoop.fs does not exist import org.apache.hadoop.fs.*; ^ WordCount.java:7: error: package org.apache.hadoop.conf does not exist import org.apache.hadoop.conf.*; ^ WordCount.java:8: error: package org.apache.hadoop.io does not exist import org.apache.hadoop.io.*; ^ WordCount.java:9: error: package org.apache.hadoop.mapreduce does not exist import org.apache.hadoop.mapreduce.*; ^ WordCount.java:10: error: package org.apache.hadoop.mapreduce.lib.input does not exist import org.apache.hadoop.mapreduce.lib.input.*; ^ WordCount.java:11: error: package org.apache.hadoop.mapreduce.lib.output does not exist import org.apache.hadoop.mapreduce.lib.output.*; ^ WordCount.java:12: error: package org.apache.hadoop.util does not exist import org.apache.hadoop.util.*; ^ WordCount.java:14: error: cannot find symbol public class WordCount extends Configured implements Tool { ^ symbol: class Configured WordCount.java:14: error: cannot find symbol public class WordCount extends Configured implements Tool { ^ symbol: class Tool WordCount.java:47: error: cannot find symbol public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> { ^ symbol: class Mapper location: class WordCount WordCount.java:47: error: cannot find symbol public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> { ^ symbol: class LongWritable location: class WordCount WordCount.java:47: error: cannot find symbol public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> { ^ symbol: class Text location: class WordCount WordCount.java:47: error: cannot find symbol public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> { ^ symbol: class Text location: class WordCount WordCount.java:47: error: cannot find symbol public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> { ^ symbol: class IntWritable location: class WordCount WordCount.java:48: error: cannot find symbol private final static IntWritable one = new IntWritable(1); ^ symbol: class IntWritable location: class Map WordCount.java:49: error: cannot find symbol private Text word = new Text(); ^ symbol: class Text location: class Map WordCount.java:52: error: cannot find symbol public void map(LongWritable key, Text value, ^ symbol: class LongWritable location: class Map WordCount.java:52: error: cannot find symbol public void map(LongWritable key, Text value, ^ symbol: class Text location: class Map WordCount.java:53: error: package Mapper does not exist Mapper.Context context) throws IOException, InterruptedException { ^ WordCount.java:63: error: cannot find symbol public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> { ^ symbol: class Reducer location: class WordCount WordCount.java:63: error: cannot find symbol public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> { ^ symbol: class Text location: class WordCount WordCount.java:63: error: cannot find symbol public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> { ^ symbol: class IntWritable location: class WordCount WordCount.java:63: error: cannot find symbol public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> { ^ symbol: class Text location: class WordCount WordCount.java:63: error: cannot find symbol public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> { ^ symbol: class IntWritable location: class WordCount WordCount.java:66: error: cannot find symbol public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { ^ symbol: class Text location: class Reduce WordCount.java:66: error: cannot find symbol public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { ^ symbol: class IntWritable location: class Reduce WordCount.java:66: error: cannot find symbol public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { ^ symbol: class Context location: class Reduce WordCount.java:17: error: cannot find symbol int res = ToolRunner.run(new WordCount(), args); ^ symbol: variable ToolRunner location: class WordCount WordCount.java:22: error: cannot find symbol Path inputPath = new Path(args[0]); ^ symbol: class Path location: class WordCount WordCount.java:22: error: cannot find symbol Path inputPath = new Path(args[0]); ^ symbol: class Path location: class WordCount WordCount.java:23: error: cannot find symbol Path outputPath = new Path(args[1]); ^ symbol: class Path location: class WordCount WordCount.java:23: error: cannot find symbol Path outputPath = new Path(args[1]); ^ symbol: class Path location: class WordCount WordCount.java:25: error: cannot find symbol Configuration conf = getConf(); ^ symbol: class Configuration location: class WordCount WordCount.java:25: error: cannot find symbol Configuration conf = getConf(); ^ symbol: method getConf() location: class WordCount WordCount.java:26: error: cannot find symbol Job job = new Job(conf, this.getClass().toString()); ^ symbol: class Job location: class WordCount WordCount.java:26: error: cannot find symbol Job job = new Job(conf, this.getClass().toString()); ^ symbol: class Job location: class WordCount WordCount.java:26: error: cannot find symbol Job job = new Job(conf, this.getClass().toString()); ^ symbol: method getClass() WordCount.java:28: error: cannot find symbol FileInputFormat.setInputPaths(job, inputPath); ^ symbol: variable FileInputFormat location: class WordCount WordCount.java:29: error: cannot find symbol FileOutputFormat.setOutputPath(job, outputPath); ^ symbol: variable FileOutputFormat location: class WordCount WordCount.java:33: error: cannot find symbol job.setInputFormatClass(TextInputFormat.class); ^ symbol: class TextInputFormat location: class WordCount WordCount.java:34: error: cannot find symbol job.setOutputFormatClass(TextOutputFormat.class); ^ symbol: class TextOutputFormat location: class WordCount WordCount.java:35: error: cannot find symbol job.setMapOutputKeyClass(Text.class); ^ symbol: class Text location: class WordCount WordCount.java:36: error: cannot find symbol job.setMapOutputValueClass(IntWritable.class); ^ symbol: class IntWritable location: class WordCount WordCount.java:37: error: cannot find symbol job.setOutputKeyClass(Text.class); ^ symbol: class Text location: class WordCount WordCount.java:38: error: cannot find symbol job.setOutputValueClass(IntWritable.class); ^ symbol: class IntWritable location: class WordCount WordCount.java:48: error: cannot find symbol private final static IntWritable one = new IntWritable(1); ^ symbol: class IntWritable location: class Map WordCount.java:49: error: cannot find symbol private Text word = new Text(); ^ symbol: class Text location: class Map WordCount.java:51: error: method does not override or implement a method from a supertype @Override ^ WordCount.java:68: error: cannot find symbol for (IntWritable value : values) { ^ symbol: class IntWritable location: class Reduce WordCount.java:72: error: cannot find symbol context.write(key, new IntWritable(sum)); ^ symbol: class IntWritable location: class Reduce WordCount.java:65: error: method does not override or implement a method from a supertype @Override ^ 52 errorsCan someone point me in the right direction? Thank you.