Hadoop Beginner's Guide

上QQ阅读APP看书，第一时间看更新

Time for action – using the Writable wrapper classes

Let's write a class to show some of these wrapper classes in action:

Create the following as WritablesTest.java:

import org.apache.hadoop.io.* ;
import java.util.* ;

public class WritablesTest
{
    public static class IntArrayWritable extends ArrayWritable
    {
        public IntArrayWritable()
        {
            super(IntWritable.class) ;
        }
    }
    
    public static void main(String[] args)
    {
System.out.println("*** Primitive Writables ***") ;
        BooleanWritable bool1 = new BooleanWritable(true) ;
        ByteWritable byte1 = new ByteWritable( (byte)3) ;
        System.out.printf("Boolean:%s Byte:%d\n", bool1, byte1.get()) ;
        
        IntWritable i1 = new IntWritable(5) ;
        IntWritable i2 = new IntWritable( 17) ;
        System.out.printf("I1:%d I2:%d\n", i1.get(), i2.get()) ;
        i1.set(i2.get()) ;
        System.out.printf("I1:%d I2:%d\n", i1.get(), i2.get()) ;
        Integer i3 = new Integer( 23) ;
        i1.set( i3) ;
        System.out.printf("I1:%d I2:%d\n", i1.get(), i2.get()) ;
 
System.out.println("*** Array Writables ***") ;       
        ArrayWritable a = new ArrayWritable( IntWritable.class) ;
        a.set( new IntWritable[]{ new IntWritable(1), new IntWritable(3), new IntWritable(5)}) ;
        
        IntWritable[] values = (IntWritable[])a.get() ;
        
        for (IntWritable i: values)
        System.out.println(i) ;
        
        IntArrayWritable ia = new IntArrayWritable() ;
        ia.set( new IntWritable[]{ new IntWritable(1), new IntWritable(3), new IntWritable(5)}) ;
        
        IntWritable[] ivalues = (IntWritable[])ia.get() ;
        
        ia.set(new LongWritable[]{new LongWritable(1000l)}) ;
 
System.out.println("*** Map Writables ***") ;       
        MapWritable m = new MapWritable() ;
        IntWritable key1 = new IntWritable(5) ;
        NullWritable value1 = NullWritable.get() ;
        m.put(key1, value1) ;
        System.out.println(m.containsKey(key1)) ;
        System.out.println(m.get(key1)) ;
        m.put(new LongWritable(1000000000), key1) ;
        Set<Writable> keys = m.keySet() ;
        
        for(Writable w: keys)
        System.out.println(w.getClass()) ;
    }
}

Compile and run the class, and you should get the following output:

*** Primitive Writables ***
Boolean:true Byte:3
I1:5 I2:17
I1:17 I2:17
I1:23 I2:17
*** Array Writables ***
1
3
5
*** Map Writables ***
true
(null)
class org.apache.hadoop.io.LongWritable
class org.apache.hadoop.io.IntWritable

What just happened?

This output should be largely self-explanatory. We create various Writable wrapper objects and show their general usage. There are several key points:

As mentioned, there is no type-safety beyond Writable itself. So it is possible to have an array or map that holds multiple types, as shown previously.
We can use autounboxing, for example, by supplying an Integer object to methods on IntWritable that expect an int variable.
The inner class demonstrates what is needed if an ArrayWritable class is to be used as an input to a reduce function; a subclass with such a default constructor must be defined.

Other wrapper classes

CompressedWritable: This is a base class to allow for large objects that should remain compressed until their attributes are explicitly accessed
ObjectWritable: This is a general-purpose generic object wrapper
NullWritable: This is a singleton object representation of a null value
VersionedWritable: This is a base implementation to allow writable classes to track versions over time

Have a go hero – playing with Writables

Write a class that exercises the NullWritable and ObjectWritable classes in the same way as it does in the previous examples.

Making your own

As you have seen from the Writable and Comparable interfaces, the required methods are pretty straightforward; don't be afraid of adding this functionality if you want to use your own custom classes as keys or values within a MapReduce job.