FAQ
Hello there, we would like to make some tests with hadoop.

For the tests we would like to have a hadoop filesystem up and configured, so
using stubs and some mocks of core interfaces we can test the overall storage
functionality we're about to develop (which would be a part of map/reduce jobs
later).

As far as we learned from documentation/tutorials, to run hadoop we need to
use it's own startup scripts, which looks like an overhead and a bit
cumbersome to integrate in maven testing.

Could you please advice?

--
Eugene N Dzhurinsky

Search Discussions

  • Owen O'Malley at Nov 21, 2007 at 6:56 pm

    On Nov 21, 2007, at 7:21 AM, Eugeny N Dzhurinsky wrote:

    For the tests we would like to have a hadoop filesystem up and
    configured, so
    using stubs and some mocks of core interfaces we can test the
    overall storage
    functionality we're about to develop (which would be a part of map/
    reduce jobs
    later).
    The unit tests currently use the mini-dfs cluster and mini-mapreduce
    cluster to get this functionality. Look in the src/test subdirectory
    for MiniDFSCluster and MiniMRCluster. Each class creates a virtual
    cluster within the same process.

    -- Owen
  • Adrian Woodhead at Nov 22, 2007 at 7:03 pm
    We have done something like this where we want our unit tests to run
    against a 1-machine "cluster". As a starting point we took
    HadoopTestCase and have written our own modified version of this which
    we configure and tell it whether to run in "local fs" or "mini
    mapreduce" mode. The former is slightly faster but doesn't catch all
    bugs so we have a continuous integration machine setup to perform tests
    in both modes. We have also made some changes so that it only starts and
    stops the cluster when the class is loaded instead of between tests
    which also improves the speed.

    Anyway, I would say that having a look at the source code for
    HadoopTestCase is probably your best starting point and hopefully you
    can take it from there!

    Regards,

    Adrian

    Eugeny N Dzhurinsky wrote:
    Hello there, we would like to make some tests with hadoop.

    For the tests we would like to have a hadoop filesystem up and configured, so
    using stubs and some mocks of core interfaces we can test the overall storage
    functionality we're about to develop (which would be a part of map/reduce jobs
    later).

    As far as we learned from documentation/tutorials, to run hadoop we need to
    use it's own startup scripts, which looks like an overhead and a bit
    cumbersome to integrate in maven testing.

    Could you please advice?

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedNov 21, '07 at 3:22p
activeNov 22, '07 at 7:03p
posts3
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase