FAQ
Hello to all,

first of all many thanks for this great piece of software you are all
contributing to :)

I am actually creating a program in Hadoop MapReduce, but I intend to
massively use JUnit Tests to validate my code.

To test a Mapper, I mock a Mapper.Context object and run the Mapper.map
function with it, but inside my Mapper.map function, I am using Counters
to maintain internal statistics. The update of those counters make the
test fails with a null pointer (JavaNullPointerException).
There is an issue "Counters" of variable visibility from my test ...

Do you have any idea how to mock/validate/overcome this issue ?



Thanks a lots
Best Regards
Olivier Varene


: Code :
--------

private static enum Counters {NBLINES};
private static IntWritable one = new IntWritable(1);
private static LongWritable oneL = new LongWritable(1);


// Mapper
/**
* outputs 1 for each line encountered
*/
public static class LCMapper extends
Mapper<LongWritable, Text, IntWritable, LongWritable>
{
private final static IntWritable one = new IntWritable(1);
private final static LongWritable oneL = new LongWritable(1);
/**
* for each line encountered outputs 1 for the key 1
*/
public void map(LongWritable p_key, Text p_inputs, Context p_context)
throws IOException, InterruptedException
{
p_context.getCounter(Counters.NBLINES).increment(1);
p_context.write(one,oneL);
} // end map
} // end Mapper

: Test :
--------

@Test
public void testLCMapper() throws IOException, InterruptedException
{
LCMapper mapper = new LCMapper();
Text value = new Text("ligneA\nlignB\n");
// mock the map job execution context
Context context = mock(Context.class);
try {
mapper.map(null, value, context);
} catch (NullPointerException e)
{
// exception raised by context.getCounter(...) null pointer
// HOW TO MOCK ???
// Counter is a static enum from the englobing class ...
// issue with variable visibility (from map function) in this test
}
// verify that the Mapper issued (one,oneL) pair
verify(context).write(one,oneL);

} // end testLCMapper

Search Discussions

  • Varene Olivier at May 27, 2010 at 4:07 pm
    FYI
    I am on Hadoop 0.20.2, JUnit 4


    Varene Olivier a écrit :
    Hello to all,

    first of all many thanks for this great piece of software you are all
    contributing to :)

    I am actually creating a program in Hadoop MapReduce, but I intend to
    massively use JUnit Tests to validate my code.

    To test a Mapper, I mock a Mapper.Context object and run the Mapper.map
    function with it, but inside my Mapper.map function, I am using Counters
    to maintain internal statistics. The update of those counters make the
    test fails with a null pointer (JavaNullPointerException).
    There is an issue "Counters" of variable visibility from my test ...

    Do you have any idea how to mock/validate/overcome this issue ?



    Thanks a lots
    Best Regards
    Olivier Varene


    : Code :
    --------

    private static enum Counters {NBLINES};
    private static IntWritable one = new IntWritable(1);
    private static LongWritable oneL = new LongWritable(1);


    // Mapper
    /**
    * outputs 1 for each line encountered
    */
    public static class LCMapper extends
    Mapper<LongWritable, Text, IntWritable, LongWritable>
    {
    private final static IntWritable one = new IntWritable(1);
    private final static LongWritable oneL = new LongWritable(1);
    /**
    * for each line encountered outputs 1 for the key 1
    */
    public void map(LongWritable p_key, Text p_inputs, Context p_context)
    throws IOException, InterruptedException
    {
    p_context.getCounter(Counters.NBLINES).increment(1);
    p_context.write(one,oneL);
    } // end map
    } // end Mapper

    : Test :
    --------

    @Test
    public void testLCMapper() throws IOException, InterruptedException
    {
    LCMapper mapper = new LCMapper();
    Text value = new Text("ligneA\nlignB\n");
    // mock the map job execution context
    Context context = mock(Context.class);
    try {
    mapper.map(null, value, context);
    } catch (NullPointerException e)
    {
    // exception raised by context.getCounter(...) null pointer
    // HOW TO MOCK ???
    // Counter is a static enum from the englobing class ...
    // issue with variable visibility (from map function) in this test
    }
    // verify that the Mapper issued (one,oneL) pair
    verify(context).write(one,oneL);

    } // end testLCMapper
  • Aaron Kimball at May 28, 2010 at 12:13 am
    Varene,

    You might want to check out MRUnit. It's a unit test harness that contains
    mock objects for the context & other associated classes, and works with
    JUnit.

    It's included in the (unreleased) Hadoop 0.21, as well as Cloudera's
    Distribution for Hadoop. See
    http://archive.cloudera.com/docs/mrunit/index.html for MRUnit documentation.

    Cheers,
    - Aaron
    On Thu, May 27, 2010 at 9:13 AM, Varene Olivier wrote:

    FYI
    I am on Hadoop 0.20.2, JUnit 4


    Varene Olivier a écrit :

    Hello to all,
    first of all many thanks for this great piece of software you are all
    contributing to :)

    I am actually creating a program in Hadoop MapReduce, but I intend to
    massively use JUnit Tests to validate my code.

    To test a Mapper, I mock a Mapper.Context object and run the Mapper.map
    function with it, but inside my Mapper.map function, I am using Counters
    to maintain internal statistics. The update of those counters make the test
    fails with a null pointer (JavaNullPointerException).
    There is an issue "Counters" of variable visibility from my test ...

    Do you have any idea how to mock/validate/overcome this issue ?



    Thanks a lots
    Best Regards
    Olivier Varene


    : Code :
    --------

    private static enum Counters {NBLINES};
    private static IntWritable one = new IntWritable(1);
    private static LongWritable oneL = new LongWritable(1);


    // Mapper
    /**
    * outputs 1 for each line encountered
    */
    public static class LCMapper extends
    Mapper<LongWritable, Text, IntWritable, LongWritable>
    {
    private final static IntWritable one = new IntWritable(1);
    private final static LongWritable oneL = new LongWritable(1);
    /**
    * for each line encountered outputs 1 for the key 1
    */
    public void map(LongWritable p_key, Text p_inputs, Context p_context)
    throws IOException, InterruptedException
    {
    p_context.getCounter(Counters.NBLINES).increment(1);
    p_context.write(one,oneL); } // end map
    } // end Mapper

    : Test :
    --------

    @Test
    public void testLCMapper() throws IOException, InterruptedException
    { LCMapper mapper = new LCMapper();
    Text value = new Text("ligneA\nlignB\n");
    // mock the map job execution context
    Context context = mock(Context.class);
    try {
    mapper.map(null, value, context);
    } catch (NullPointerException e)
    {
    // exception raised by context.getCounter(...) null pointer
    // HOW TO MOCK ???
    // Counter is a static enum from the englobing class ...
    // issue with variable visibility (from map function) in this test
    }
    // verify that the Mapper issued (one,oneL) pair
    verify(context).write(one,oneL);
    } // end testLCMapper

  • Varene Olivier at May 28, 2010 at 6:32 am
    Thanks a lot Aaron,

    I was looking into it, I just hope, it will be compliant with 0.20.2 ...

    Otherwise, I am stuck :(

    Cheers
    Olivier

    Aaron Kimball a écrit :
    Varene,

    You might want to check out MRUnit. It's a unit test harness that contains
    mock objects for the context & other associated classes, and works with
    JUnit.

    It's included in the (unreleased) Hadoop 0.21, as well as Cloudera's
    Distribution for Hadoop. See
    http://archive.cloudera.com/docs/mrunit/index.html for MRUnit documentation.

    Cheers,
    - Aaron
    On Thu, May 27, 2010 at 9:13 AM, Varene Olivier wrote:

    FYI
    I am on Hadoop 0.20.2, JUnit 4


    Varene Olivier a écrit :

    Hello to all,
    first of all many thanks for this great piece of software you are all
    contributing to :)

    I am actually creating a program in Hadoop MapReduce, but I intend to
    massively use JUnit Tests to validate my code.

    To test a Mapper, I mock a Mapper.Context object and run the Mapper.map
    function with it, but inside my Mapper.map function, I am using Counters
    to maintain internal statistics. The update of those counters make the test
    fails with a null pointer (JavaNullPointerException).
    There is an issue "Counters" of variable visibility from my test ...

    Do you have any idea how to mock/validate/overcome this issue ?



    Thanks a lots
    Best Regards
    Olivier Varene


    : Code :
    --------

    private static enum Counters {NBLINES};
    private static IntWritable one = new IntWritable(1);
    private static LongWritable oneL = new LongWritable(1);


    // Mapper
    /**
    * outputs 1 for each line encountered
    */
    public static class LCMapper extends
    Mapper<LongWritable, Text, IntWritable, LongWritable>
    {
    private final static IntWritable one = new IntWritable(1);
    private final static LongWritable oneL = new LongWritable(1);
    /**
    * for each line encountered outputs 1 for the key 1
    */
    public void map(LongWritable p_key, Text p_inputs, Context p_context)
    throws IOException, InterruptedException
    {
    p_context.getCounter(Counters.NBLINES).increment(1);
    p_context.write(one,oneL); } // end map
    } // end Mapper

    : Test :
    --------

    @Test
    public void testLCMapper() throws IOException, InterruptedException
    { LCMapper mapper = new LCMapper();
    Text value = new Text("ligneA\nlignB\n");
    // mock the map job execution context
    Context context = mock(Context.class);
    try {
    mapper.map(null, value, context);
    } catch (NullPointerException e)
    {
    // exception raised by context.getCounter(...) null pointer
    // HOW TO MOCK ???
    // Counter is a static enum from the englobing class ...
    // issue with variable visibility (from map function) in this test
    }
    // verify that the Mapper issued (one,oneL) pair
    verify(context).write(one,oneL);
    } // end testLCMapper

  • Varene Olivier at May 28, 2010 at 12:48 pm
    FYI

    MRUnit is the way to go and it is compatible (as far as I am aware of)
    with the 0.20.2

    Cheers
    Olivier

    Varene Olivier a écrit :
    Thanks a lot Aaron,

    I was looking into it, I just hope, it will be compliant with 0.20.2 ...

    Otherwise, I am stuck :(

    Cheers
    Olivier

    Aaron Kimball a écrit :
    Varene,

    You might want to check out MRUnit. It's a unit test harness that
    contains
    mock objects for the context & other associated classes, and works with
    JUnit.

    It's included in the (unreleased) Hadoop 0.21, as well as Cloudera's
    Distribution for Hadoop. See
    http://archive.cloudera.com/docs/mrunit/index.html for MRUnit
    documentation.

    Cheers,
    - Aaron
    On Thu, May 27, 2010 at 9:13 AM, Varene Olivier wrote:

    FYI
    I am on Hadoop 0.20.2, JUnit 4


    Varene Olivier a écrit :

    Hello to all,
    first of all many thanks for this great piece of software you are all
    contributing to :)

    I am actually creating a program in Hadoop MapReduce, but I intend to
    massively use JUnit Tests to validate my code.

    To test a Mapper, I mock a Mapper.Context object and run the Mapper.map
    function with it, but inside my Mapper.map function, I am using
    Counters
    to maintain internal statistics. The update of those counters make
    the test
    fails with a null pointer (JavaNullPointerException).
    There is an issue "Counters" of variable visibility from my test ...

    Do you have any idea how to mock/validate/overcome this issue ?



    Thanks a lots
    Best Regards
    Olivier Varene


    : Code :
    --------

    private static enum Counters {NBLINES};
    private static IntWritable one = new IntWritable(1);
    private static LongWritable oneL = new LongWritable(1);


    // Mapper
    /**
    * outputs 1 for each line encountered
    */
    public static class LCMapper extends
    Mapper<LongWritable, Text, IntWritable, LongWritable>
    {
    private final static IntWritable one = new IntWritable(1);
    private final static LongWritable oneL = new LongWritable(1);
    /**
    * for each line encountered outputs 1 for the key 1
    */
    public void map(LongWritable p_key, Text p_inputs, Context p_context)
    throws IOException, InterruptedException
    {
    p_context.getCounter(Counters.NBLINES).increment(1);
    p_context.write(one,oneL); } // end map
    } // end Mapper

    : Test :
    --------

    @Test
    public void testLCMapper() throws IOException, InterruptedException
    { LCMapper mapper = new LCMapper();
    Text value = new Text("ligneA\nlignB\n");
    // mock the map job execution context
    Context context = mock(Context.class);
    try {
    mapper.map(null, value, context);
    } catch (NullPointerException e)
    {
    // exception raised by context.getCounter(...) null pointer
    // HOW TO MOCK ???
    // Counter is a static enum from the englobing class ...
    // issue with variable visibility (from map function) in this test
    }
    // verify that the Mapper issued (one,oneL) pair
    verify(context).write(one,oneL);
    } // end testLCMapper

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgeneral @
categorieshadoop
postedMay 27, '10 at 8:34a
activeMay 28, '10 at 12:48p
posts5
users2
websitehadoop.apache.org
irc#hadoop

2 users in discussion

Varene Olivier: 4 posts Aaron Kimball: 1 post

People

Translate

site design / logo © 2022 Grokbase