Custom Map Reduce Program on Hive, what's the Rule? How about input and output?

Question

I got stuck for a few days because I want to create a custom map reduce program based on my query on hive, I found not many examples after googling and I'm still confused about the rule.

What is the rule to create my custom mapreduce program, how about the mapper and reducer class?

Can anyone provide any solution?

I want to develop this program in Java, but I'm still stuck ,and then when formatting output in collector, how do I format the result in mapper and reducer class?

Does anybody want to give me some example and explanation about this kind of stuff?

Solution

There are basically 2 ways to add custom mappers/reducers to hive queries.

1. using transform

SELECT TRANSFORM(stuff1, stuff2) FROM table1 USING 'script' AS thing1, thing2

where stuff1, stuff2 are the fields in table1 and script is any executable which accepts the format i describe later. thing1, thing2 are the outputs from script

1. using map and reduce

FROM (
    FROM table
    MAP table.f1 table.f2
    USING 'map_script'
    AS mp1, mp2
    CLUSTER BY mp1) map_output
  INSERT OVERWRITE TABLE someothertable
    REDUCE map_output.mp1, map_output.mp2
    USING 'reduce_script'
    AS reducef1, reducef2;

This is slightly more complicated but gives more control. There are 2 parts to this. In the first part the mapper script will receive data from table and map it to fields mp1 and mp2. these are then passed on to reduce_script, this script will receive sorted output on the key, which we have specified in CLUSTER BY mp1. mind you, more than one key will be handled by one reducer. The output of the reduce script will go to table someothertable

Now all these scripts follow a simple pattern. they will read line by line from stdin. The fields will be \t separated and they will write back to stdout, in the same manner ( fields separated by '\t' )

Check out this blog, there are some nice examples.

http://dev.bizo.com/2009/07/custom-map-scripts-and-hive.html

http://dev.bizo.com/2009/10/reduce-scripts-in-hive.html

Comments

[arsenal]

Thanks bronzebeard for the great explanation. It was really helpful to me. Can you please also take a look into a different problem that I have posted http://stackoverflow.com/questions/11572800/what-will-be-the‌​-rank-udf-for-this-s‌​cenario. It will be of great help to me.

[user152468]

When I try your first query, I get the following error: FAILED: ParseException line 1:33 cannot recognize input near 'FROM' 'table1' 'USING' in record writer specification. I am using hive-0.11. Any ideas?

[user152468]

Seems like the syntax has changed. SELECT TRANSFORM(line) USING '/bin/cat' AS transformed FROM posts; The using keyword now goes before the FROM key word. See stackoverflow.com/questions/16320185/…

[Sohaib]

In case of using Map Reduce can the input table be directly from an S3 source (I mean to say without a create table preceding this)? What I mean is can something like LOAD TABLE be done using MAPPER.