Eclipse plugin for Hadoop 2.6.0

Question

I want to write MapReduce jobs in Java. For that I have installed Hadoop 2.6.0 in Ubuntu 14.04 LTS VM. The installation directory is /usr/local/hadoop.

Now according to many tutorials, they find an Eclipse plugin in /contrib directory and paste it in /Eclipse/plugins folder.

The problem is that there is no src/contrib folder nor any Eclipse plugin bundeled with Hadoop 2.6.0. So how do I configure Eclipse Europa to run Hadoop MapReduce jobs? If that's not possible, what are the alternatives of writing MapReduce jobs?

Solution 1

Integrating Hadoop-2.6.0 with eclipse

1. User “hdfs” is created where all Hadoop processes are running.
2. Hadoop is installed to the directory “/opt/hadoop“.
3. Eclipse is installed to the directory “/opt/eclipse“.

Step 1: Download the hadoop-eclipse-plugin 2.6.0 jar

Step 2: Copy the Map-Reduce plugin for eclipse in the the plugins directory of your eclipse folder – sudo cp /home/hdfs/Downloads/hadoop-eclipse-plugin-2.6.0.jar /opt/eclipse/plugins/ Restart the eclipse using the command – /opt/eclipse/eclipse -vm /usr/local/jdk1.8.0_05/bin/java -vmargs -Xmx1024m If elcipse is not coming up because of the X11 forwarding issue, try using “sux” instead of “su” while switching to the “hdfs“. Step 3: Start the eclipse  1. $ECLIPSE_HOME/eclipse

step 4: In Eclipse menu click,  Window --> Open Perspective --> Others -->  MapReduce

step 5: In bottom MapReduce icon click to Add new Hadoop location

step 6: Enter MapReduce & HDFS running port For recall, MapReduce port (9001) specified in $HADOOP_HOME/conf/mapred-site.xml  For recall, HDFS port (9000) specified in $HADOOP_HOME/conf/core-site.xml Enter the Hadoop user name

step 7 : Once Hadoop location added, DFS Locations will be seen/displayed in Eclipse Project Explorer window, (Windows-->Show View-->Project Explorer)

step 8: Once Hadoop added, DFS Locations will be seen/displayed in Project Explorer window,

step 9: Right click DFS location and click to Connect

step 10 : Once connected successfully, it will display all the DFS Folder.

Step 11: You can create Directory, Upload files to HDFS location, Download files to local by right click any of the listed Directory.

Solution 2

Download and build this project https://github.com/winghc/hadoop2x-eclipse-plugin

after download follow these steps:

$ cd src/contrib/eclipse-plugin

Assume hadoop installation directory is /usr/share/hadoop

$ ant jar -Dversion=2.4.1 -Dhadoop.version=2.4.1 -Declipse.home=/opt/eclipse -Dhadoop.home=/usr/share/hadoop

final jar will be generated at directory

${hadoop2x-eclipse-plugin}/build/contrib/eclipse-plugin/hadoop-eclipse-plugin-2.6.0.jar

Comments

[user3202144]

and if I want to write my MapReduce Jobs in my Windows host instead of linus VM, can you explain the steps to do it on Windows?

[gsamaras]

Download the hadoop-eclipse-plugin 2.6.0 jar --> from where?