Converting IntWritatble to int

Question

I have the following code and I didn't understand why get() method has been used in the highlighted line. If I remove that get() method it throws me an error.

What I can take out from it is: get() method returns the int value of the IntWritable. Correct me if I am wrong.

public void reduce(IntWritable key, Iterator<IntWritable> values, OutputCollector<IntWritable, IntWritable> output, Reporter reporter) throws IOException {
    int sum = 0;
    while (values.hasNext()) {
        sum += values.next().get(); //error when removing the get()    
    }
}

Solution 1

Your understanding is correct. RamPrasad G's answer is also correct (+1), but, just to make it clearer:

sum is int. values is an Iterator to IntWritable elements, so values.next() is an IntWritable. Now, IntWritable, a data type of hadoop, is not equivalent to the primitive type int of java, even if they are used for the same thing: to store integer values.

Thus, you cannot add an IntWritable to an int (which is what you do if you remove the get() method). They are not of the same type. You have to convert the IntWritable to an int, which is done by the get() method.

Solution 2

In order to handle the Objects in Hadoop way, Hadoop uses Writable classes. For example, Hadoop uses Text instead of java's String. similarly, The IntWritable class in Hadoop is similar to a java int, however, IntWritable implements interfaces like Comparable, Writable and WritableComparable.

These interfaces are all necessary for MapReduce; the Comparable interface is used for comparing when the reducer sorts the keys, and Writable can write the result to the local disk. It does not use the java Serializable because java Serializable is too big or too heavy for hadoop, Writable can serializable the hadoop Object in a very light way.

Writable interface is described as

A serializable object which implements a simple, efficient, serialization protocol, based on DataInput and DataOutput

Your values.next() is IntWritable class you have to use get() method to get primitive type of it.

Solution 3

Unlikeint and Integer auto boxing/un-boxing, same does not work for int and IntWritable.

The source code IntWritable provides more details for you to understanding the concept better.

/** A WritableComparable for ints. */
public class IntWritable implements WritableComparable {
  private int value;

  public IntWritable() {}

  public IntWritable(int value) { set(value); }

  /** Set the value of this IntWritable. */
  public void set(int value) { this.value = value; }

  /** Return the value of this IntWritable. */
  public int get() { return value; }

IntWritable is constructed from the int and the only way to get back int is with usage of get() API

Comments

[Ravindra babu]

You are welcome. If this answer is useful, you can upvote.