FAQ
Hi,

I've done a stat on the tests we have. Many of them are actually quite
simple: There are 226 test classes in HBase. 100 of them does not use a
cluster, and takes less than 2 seconds to execute. 24 do not use a cluster
but need between 2 and 40 seconds.

I tried to execute these with different maven settings.
fork: once or always, i.e. all the tests are executed in the same JVM or
there is a new JVM created for every test class
parallel; deactivated or activated, with 3 threads and a per class
parallelisation.

The 100 tests executes for the whole maven stuff, including the initial
dependency & task analysis
fork always: 1m50s
fork once: 30 seconds
parallel: 30s (=> no difference).

It works quite well, except with TestLruBlockCache that I had to remove.

I tried as well with the 24 'bigger' ones but no cluster, for the following
results:
fork always: 7m36s
parallel: 2m32s: 3 times faster.

However, I had to remove many of them, there are some conflicts that I have
not solved (nor tried to).

The fact that nearly 50% of the tests can be executed in 1 minutes is
interesting imho.

That makes me wonder if we should try to get a separation of the tests in 3
parts, small/medium/large as of
http://googletesting.blogspot.com/2009/07/software-testing-categorization.html

small would be:
- less than two seconds, can be executed in the same jvm
- all of them: less than 2 minute
- no cluster, nothing that can break the environment
- run by the developers as a part of the build

medium:
- less than 40 seconds
- all of them: would last less than 30 minutes, ideally 15 minutes
- run in an isolated jvm
- run explicitly by the developers before submitting a fix

large:
- run in an isolated jvm
- Should last less than 45minutes
- run on the central built or explicitly by the developers
- break the built when it fails.

Only one issue: it's not directly possible with maven, but there are known
workarounds :-)


What do you think?


Below the tests set for the 120 tests that seems to work altogether.


mvn test
-Dtest=org.apache.hadoop.hbase.avro.TestAvroUtil,org.apache.hadoop.hbase.client.TestAttributes,org.apache.hadoop.hbase.client.TestGet,org.apache.hadoop.hbase.client.TestOperation,org.apache.hadoop.hbase.client.TestResult,org.apache.hadoop.hbase.client.TestScan,org.apache.hadoop.hbase.coprocessor.TestRegionObserverStacking,org.apache.hadoop.hbase.filter.TestBitComparator,org.apache.hadoop.hbase.filter.TestColumnPaginationFilter,org.apache.hadoop.hbase.filter.TestDependentColumnFilter,org.apache.hadoop.hbase.filter.TestFilterList,org.apache.hadoop.hbase.filter.TestInclusiveStopFilter,org.apache.hadoop.hbase.filter.TestPageFilter,org.apache.hadoop.hbase.filter.TestParseFilter,org.apache.hadoop.hbase.filter.TestPrefixFilter,org.apache.hadoop.hbase.filter.TestRandomRowFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueExcludeFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueFilter,org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache,org.apache.hadoop.hbase.io.hfile.slab.TestSlab,org.apache.hadoop.hbase.io.hfile.TestBlockCacheColumnFamilySummary,org.apache.hadoop.hbase.io.hfile.TestCachedBlockQueue,org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer,org.apache.hadoop.hbase.io.hfile.TestHFileReaderV1,org.apache.hadoop.hbase.io.hfile.TestHFile,org.apache.hadoop.hbase.io.hfile.TestHFileWriterV2,org.apache.hadoop.hbase.io.hfile.TestReseekTo,org.apache.hadoop.hbase.io.hfile.TestSeekTo,org.apache.hadoop.hbase.io.TestHalfStoreFileReader,org.apache.hadoop.hbase.io.TestHbaseObjectWritable,org.apache.hadoop.hbase.io.TestHeapSize,org.apache.hadoop.hbase.io.TestImmutableBytesWritable,org.apache.hadoop.hbase.mapreduce.TestSimpleTotalOrderPartitioner,org.apache.hadoop.hbase.mapreduce.TestTableSplit,org.apache.hadoop.hbase.master.TestClockSkewDetection,org.apache.hadoop.hbase.master.TestDeadServer,org.apache.hadoop.hbase.master.TestDefaultLoadBalancer,org.apache.hadoop.hbase.master.TestMasterStatusServlet,org.apache.hadoop.hbase.metrics.TestMetricsMBeanBase,org.apache.hadoop.hbase.monitoring.TestMemoryBoundedLogMessageBuffer,org.apache.hadoop.hbase.monitoring.TestTaskMonitor,org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler,org.apache.hadoop.hbase.regionserver.TestCompactSelection,org.apache.hadoop.hbase.regionserver.TestExplicitColumnTracker,org.apache.hadoop.hbase.regionserver.TestHRegionInfo,org.apache.hadoop.hbase.regionserver.TestKeyValueHeap,org.apache.hadoop.hbase.regionserver.TestKeyValueScanFixture,org.apache.hadoop.hbase.regionserver.TestKeyValueSkipListSet,org.apache.hadoop.hbase.regionserver.TestMemStoreLAB,org.apache.hadoop.hbase.regionserver.TestMinVersions,org.apache.hadoop.hbase.regionserver.TestQueryMatcher,org.apache.hadoop.hbase.regionserver.TestRegionSplitPolicy,org.apache.hadoop.hbase.regionserver.TestResettingCounters,org.apache.hadoop.hbase.regionserver.TestRSStatusServlet,org.apache.hadoop.hbase.regionserver.TestScanDeleteTracker,org.apache.hadoop.hbase.regionserver.TestScanWildcardColumnTracker,org.apache.hadoop.hbase.regionserver.TestScanWithBloomError,org.apache.hadoop.hbase.regionserver.TestStoreScanner,org.apache.hadoop.hbase.regionserver.wal.TestHLogBench,org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods,org.apache.hadoop.hbase.regionserver.wal.TestWALActionsListener,org.apache.hadoop.hbase.rest.model.TestCellModel,org.apache.hadoop.hbase.rest.model.TestCellSetModel,org.apache.hadoop.hbase.rest.model.TestColumnSchemaModel,org.apache.hadoop.hbase.rest.model.TestRowModelo,rg.apache.hadoop.hbase.rest.model.TestScannerModel,org.apache.hadoop.hbase.rest.model.TestStorageClusterStatusModel,org.apache.hadoop.hbase.rest.model.TestStorageClusterVersionModel,org.apache.hadoop.hbase.rest.model.TestTableInfoModel,org.apache.hadoop.hbase.rest.model.TestTableListModel,org.apache.hadoop.hbase.rest.model.TestTableRegionModel,org.apache.hadoop.hbase.rest.model.TestTableSchemaModel,org.apache.hadoop.hbase.rest.model.TestVersionModel,org.apache.hadoop.hbase.security.TestUser,org.apache.hadoop.hbase.TestAcidGuarantees,org.apache.hadoop.hbase.TestCompare,org.apache.hadoop.hbase.TestHRegionLocation,org.apache.hadoop.hbase.TestHServerAddress,org.apache.hadoop.hbase.TestHServerInfo,org.apache.hadoop.hbase.TestKeyValue,org.apache.hadoop.hbase.TestSerialization,org.apache.hadoop.hbase.TestServerName,org.apache.hadoop.hbase.util.TestBase64,org.apache.hadoop.hbase.util.TestByteBloomFilter,org.apache.hadoop.hbase.util.TestBytes,org.apache.hadoop.hbase.util.TestCompressionTest,org.apache.hadoop.hbase.util.TestFSTableDescriptors,org.apache.hadoop.hbase.util.TestHBaseFsckComparator,org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge,org.apache.hadoop.hbase.util.TestKeying,org.apache.hadoop.hbase.util.TestPoolMap,org.apache.hadoop.hbase.util.TestRegionSplitCalculator,org.apache.hadoop.hbase.zookeeper.TestHQuorumPeer,org.apache.hadoop.hbase.zookeeper.TestZooKeeperMainServerArg,org.apache.hadoop.hbase.coprocessor.TestCoprocessorInterface,org.apache.hadoop.hbase.executor.TestExecutorService,org.apache.hadoop.hbase.filter.TestColumnPrefixFilter,org.apache.hadoop.hbase.filter.TestFilter,org.apache.hadoop.hbase.filter.TestMultipleColumnPrefixFilter,org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache,org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite,org.apache.hadoop.hbase.io.hfile.TestHFileBlockIndex,org.apache.hadoop.hbase.io.hfile.TestHFileBlock,org.apache.hadoop.hbase.io.hfile.TestHFilePerformance,org.apache.hadoop.hbase.io.hfile.TestHFileSeek,org.apache.hadoop.hbase.ipc.TestDelayedRpc,org.apache.hadoop.hbase.master.TestCatalogJanitor,org.apache.hadoop.hbase.regionserver.TestAtomicOperation,org.apache.hadoop.hbase.regionserver.TestMemStore,org.apache.hadoop.hbase.regionserver.TestReadWriteConsistencyControl,org.apache.hadoop.hbase.regionserver.TestSeekOptimizations,org.apache.hadoop.hbase.regionserver.TestSplitTransaction,org.apache.hadoop.hbase.regionserver.TestStore,org.apache.hadoop.hbase.replication.TestReplicationSource

Search Discussions

  • Todd Lipcon at Oct 20, 2011 at 8:49 pm
    +1 on separating into small/medium/large. One of the requirements of
    "small" should be that it can run parallel with other tests and in
    nofork mode.

    See SUREFIRE-329 for the necessary surefire patch. Until that's
    integrated upstream we'd have to have our own patched surefire :(

    -Todd
    On Thu, Oct 20, 2011 at 1:46 PM, N Keywal wrote:
    Hi,

    I've done a stat on the tests we have. Many of them are actually quite
    simple: There are 226 test classes in HBase.  100  of them does not use a
    cluster, and takes less than 2 seconds to execute. 24 do not use a cluster
    but need between 2 and 40 seconds.

    I tried to execute these with different maven settings.
    fork: once or always, i.e. all the tests are executed in the same JVM or
    there is a new JVM created for every test class
    parallel; deactivated or activated, with 3 threads and a per class
    parallelisation.

    The 100 tests executes for the whole maven stuff, including the initial
    dependency & task analysis
    fork always: 1m50s
    fork once: 30 seconds
    parallel: 30s (=> no difference).

    It works quite well, except with TestLruBlockCache that I had to remove.

    I tried as well with the 24 'bigger' ones but no cluster, for the following
    results:
    fork always: 7m36s
    parallel: 2m32s: 3 times faster.

    However, I had to remove many of them, there are some conflicts that I have
    not solved (nor tried to).

    The fact that nearly 50% of the tests can be executed in 1 minutes is
    interesting imho.

    That makes me wonder if we should try to get a separation of the tests in 3
    parts, small/medium/large as of
    http://googletesting.blogspot.com/2009/07/software-testing-categorization.html

    small would be:
    - less than two seconds, can be executed in the same jvm
    - all of them: less than 2 minute
    - no cluster, nothing that can break the environment
    - run by the developers as a part of the build

    medium:
    - less than 40 seconds
    - all of them: would last less than 30 minutes, ideally 15 minutes
    - run in an isolated jvm
    - run explicitly by the developers before submitting a fix

    large:
    - run in an isolated jvm
    - Should last less than 45minutes
    - run on the central built or explicitly by the developers
    - break the built when it fails.

    Only one issue: it's not directly possible with maven, but there are known
    workarounds :-)


    What do you think?


    Below the tests set for the 120 tests that seems to work altogether.


    mvn test
    -Dtest=org.apache.hadoop.hbase.avro.TestAvroUtil,org.apache.hadoop.hbase.client.TestAttributes,org.apache.hadoop.hbase.client.TestGet,org.apache.hadoop.hbase.client.TestOperation,org.apache.hadoop.hbase.client.TestResult,org.apache.hadoop.hbase.client.TestScan,org.apache.hadoop.hbase.coprocessor.TestRegionObserverStacking,org.apache.hadoop.hbase.filter.TestBitComparator,org.apache.hadoop.hbase.filter.TestColumnPaginationFilter,org.apache.hadoop.hbase.filter.TestDependentColumnFilter,org.apache.hadoop.hbase.filter.TestFilterList,org.apache.hadoop.hbase.filter.TestInclusiveStopFilter,org.apache.hadoop.hbase.filter.TestPageFilter,org.apache.hadoop.hbase.filter.TestParseFilter,org.apache.hadoop.hbase.filter.TestPrefixFilter,org.apache.hadoop.hbase.filter.TestRandomRowFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueExcludeFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueFilter,org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache,org.apache.hadoop.hbase.io.hfile.slab.TestSlab,org.apache.hadoop.hbase.io.hfile.TestBlockCacheColumnFamilySummary,org.apache.hadoop.hbase.io.hfile.TestCachedBlockQueue,org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer,org.apache.hadoop.hbase.io.hfile.TestHFileReaderV1,org.apache.hadoop.hbase.io.hfile.TestHFile,org.apache.hadoop.hbase.io.hfile.TestHFileWriterV2,org.apache.hadoop.hbase.io.hfile.TestReseekTo,org.apache.hadoop.hbase.io.hfile.TestSeekTo,org.apache.hadoop.hbase.io.TestHalfStoreFileReader,org.apache.hadoop.hbase.io.TestHbaseObjectWritable,org.apache.hadoop.hbase.io.TestHeapSize,org.apache.hadoop.hbase.io.TestImmutableBytesWritable,org.apache.hadoop.hbase.mapreduce.TestSimpleTotalOrderPartitioner,org.apache.hadoop.hbase.mapreduce.TestTableSplit,org.apache.hadoop.hbase.master.TestClockSkewDetection,org.apache.hadoop.hbase.master.TestDeadServer,org.apache.hadoop.hbase.master.TestDefaultLoadBalancer,org.apache.hadoop.hbase.master.TestMasterStatusServlet,org.apache.hadoop.hbase.metrics.TestMetricsMBeanBase,org.apache.hadoop.hbase.monitoring.TestMemoryBoundedLogMessageBuffer,org.apache.hadoop.hbase.monitoring.TestTaskMonitor,org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler,org.apache.hadoop.hbase.regionserver.TestCompactSelection,org.apache.hadoop.hbase.regionserver.TestExplicitColumnTracker,org.apache.hadoop.hbase.regionserver.TestHRegionInfo,org.apache.hadoop.hbase.regionserver.TestKeyValueHeap,org.apache.hadoop.hbase.regionserver.TestKeyValueScanFixture,org.apache.hadoop.hbase.regionserver.TestKeyValueSkipListSet,org.apache.hadoop.hbase.regionserver.TestMemStoreLAB,org.apache.hadoop.hbase.regionserver.TestMinVersions,org.apache.hadoop.hbase.regionserver.TestQueryMatcher,org.apache.hadoop.hbase.regionserver.TestRegionSplitPolicy,org.apache.hadoop.hbase.regionserver.TestResettingCounters,org.apache.hadoop.hbase.regionserver.TestRSStatusServlet,org.apache.hadoop.hbase.regionserver.TestScanDeleteTracker,org.apache.hadoop.hbase.regionserver.TestScanWildcardColumnTracker,org.apache.hadoop.hbase.regionserver.TestScanWithBloomError,org.apache.hadoop.hbase.regionserver.TestStoreScanner,org.apache.hadoop.hbase.regionserver.wal.TestHLogBench,org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods,org.apache.hadoop.hbase.regionserver.wal.TestWALActionsListener,org.apache.hadoop.hbase.rest.model.TestCellModel,org.apache.hadoop.hbase.rest.model.TestCellSetModel,org.apache.hadoop.hbase.rest.model.TestColumnSchemaModel,org.apache.hadoop.hbase.rest.model.TestRowModelo,rg.apache.hadoop.hbase.rest.model.TestScannerModel,org.apache.hadoop.hbase.rest.model.TestStorageClusterStatusModel,org.apache.hadoop.hbase.rest.model.TestStorageClusterVersionModel,org.apache.hadoop.hbase.rest.model.TestTableInfoModel,org.apache.hadoop.hbase.rest.model.TestTableListModel,org.apache.hadoop.hbase.rest.model.TestTableRegionModel,org.apache.hadoop.hbase.rest.model.TestTableSchemaModel,org.apache.hadoop.hbase.rest.model.TestVersionModel,org.apache.hadoop.hbase.security.TestUser,org.apache.hadoop.hbase.TestAcidGuarantees,org.apache.hadoop.hbase.TestCompare,org.apache.hadoop.hbase.TestHRegionLocation,org.apache.hadoop.hbase.TestHServerAddress,org.apache.hadoop.hbase.TestHServerInfo,org.apache.hadoop.hbase.TestKeyValue,org.apache.hadoop.hbase.TestSerialization,org.apache.hadoop.hbase.TestServerName,org.apache.hadoop.hbase.util.TestBase64,org.apache.hadoop.hbase.util.TestByteBloomFilter,org.apache.hadoop.hbase.util.TestBytes,org.apache.hadoop.hbase.util.TestCompressionTest,org.apache.hadoop.hbase.util.TestFSTableDescriptors,org.apache.hadoop.hbase.util.TestHBaseFsckComparator,org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge,org.apache.hadoop.hbase.util.TestKeying,org.apache.hadoop.hbase.util.TestPoolMap,org.apache.hadoop.hbase.util.TestRegionSplitCalculator,org.apache.hadoop.hbase.zookeeper.TestHQuorumPeer,org.apache.hadoop.hbase.zookeeper.TestZooKeeperMainServerArg,org.apache.hadoop.hbase.coprocessor.TestCoprocessorInterface,org.apache.hadoop.hbase.executor.TestExecutorService,org.apache.hadoop.hbase.filter.TestColumnPrefixFilter,org.apache.hadoop.hbase.filter.TestFilter,org.apache.hadoop.hbase.filter.TestMultipleColumnPrefixFilter,org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache,org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite,org.apache.hadoop.hbase.io.hfile.TestHFileBlockIndex,org.apache.hadoop.hbase.io.hfile.TestHFileBlock,org.apache.hadoop.hbase.io.hfile.TestHFilePerformance,org.apache.hadoop.hbase.io.hfile.TestHFileSeek,org.apache.hadoop.hbase.ipc.TestDelayedRpc,org.apache.hadoop.hbase.master.TestCatalogJanitor,org.apache.hadoop.hbase.regionserver.TestAtomicOperation,org.apache.hadoop.hbase.regionserver.TestMemStore,org.apache.hadoop.hbase.regionserver.TestReadWriteConsistencyControl,org.apache.hadoop.hbase.regionserver.TestSeekOptimizations,org.apache.hadoop.hbase.regionserver.TestSplitTransaction,org.apache.hadoop.hbase.regionserver.TestStore,org.apache.hadoop.hbase.replication.TestReplicationSource


    --
    Todd Lipcon
    Software Engineer, Cloudera
  • Jonathan Gray at Oct 20, 2011 at 9:34 pm
    +1 on small/medium/large.
    -----Original Message-----
    From: Todd Lipcon
    Sent: Thursday, October 20, 2011 1:49 PM
    To: dev@hbase.apache.org
    Subject: Re: [unit tests] small tests vs. others

    +1 on separating into small/medium/large. One of the requirements of
    "small" should be that it can run parallel with other tests and in nofork mode.

    See SUREFIRE-329 for the necessary surefire patch. Until that's integrated
    upstream we'd have to have our own patched surefire :(

    -Todd
    On Thu, Oct 20, 2011 at 1:46 PM, N Keywal wrote:
    Hi,

    I've done a stat on the tests we have. Many of them are actually quite
    simple: There are 226 test classes in HBase.  100  of them does not
    use a cluster, and takes less than 2 seconds to execute. 24 do not use
    a cluster but need between 2 and 40 seconds.

    I tried to execute these with different maven settings.
    fork: once or always, i.e. all the tests are executed in the same JVM
    or there is a new JVM created for every test class parallel;
    deactivated or activated, with 3 threads and a per class
    parallelisation.

    The 100 tests executes for the whole maven stuff, including the
    initial dependency & task analysis fork always: 1m50s fork once: 30
    seconds
    parallel: 30s (=> no difference).

    It works quite well, except with TestLruBlockCache that I had to remove.

    I tried as well with the 24 'bigger' ones but no cluster, for the
    following
    results:
    fork always: 7m36s
    parallel: 2m32s: 3 times faster.

    However, I had to remove many of them, there are some conflicts that I
    have not solved (nor tried to).

    The fact that nearly 50% of the tests can be executed in 1 minutes is
    interesting imho.

    That makes me wonder if we should try to get a separation of the tests
    in 3 parts, small/medium/large as of
    http://googletesting.blogspot.com/2009/07/software-testing-categorizat
    ion.html

    small would be:
    - less than two seconds, can be executed in the same jvm
    - all of them: less than 2 minute
    - no cluster, nothing that can break the environment
    - run by the developers as a part of the build

    medium:
    - less than 40 seconds
    - all of them: would last less than 30 minutes, ideally 15 minutes
    - run in an isolated jvm
    - run explicitly by the developers before submitting a fix

    large:
    - run in an isolated jvm
    - Should last less than 45minutes
    - run on the central built or explicitly by the developers
    - break the built when it fails.

    Only one issue: it's not directly possible with maven, but there are
    known workarounds :-)


    What do you think?


    Below the tests set for the 120 tests that seems to work altogether.


    mvn test
    -
    Dtest=org.apache.hadoop.hbase.avro.TestAvroUtil,org.apache.hadoop.hba
    se.client.TestAttributes,org.apache.hadoop.hbase.client.TestGet,org.ap
    ache.hadoop.hbase.client.TestOperation,org.apache.hadoop.hbase.client.
    TestResult,org.apache.hadoop.hbase.client.TestScan,org.apache.hadoop.h
    base.coprocessor.TestRegionObserverStacking,org.apache.hadoop.hbase.fi
    lter.TestBitComparator,org.apache.hadoop.hbase.filter.TestColumnPagina
    tionFilter,org.apache.hadoop.hbase.filter.TestDependentColumnFilter,or
    g.apache.hadoop.hbase.filter.TestFilterList,org.apache.hadoop.hbase.fi
    lter.TestInclusiveStopFilter,org.apache.hadoop.hbase.filter.TestPageFi
    lter,org.apache.hadoop.hbase.filter.TestParseFilter,org.apache.hadoop.
    hbase.filter.TestPrefixFilter,org.apache.hadoop.hbase.filter.TestRando
    mRowFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueExclude
    Filter,org.apache.hadoop.hbase.filter.TestSingleColumnValueFilter,org.
    apache.hadoop.hbase.io.hfile.slab.TestSlabCache,org.apache.hadoop.hbas
    e.io.hfile.slab.TestSlab,org.apache.hadoop.hbase.io.hfile.TestBlockCac
    heColumnFamilySummary,org.apache.hadoop.hbase.io.hfile.TestCachedBloc
    k
    Queue,org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer,org.apache
    .hadoop.hbase.io.hfile.TestHFileReaderV1,org.apache.hadoop.hbase.io.hf
    ile.TestHFile,org.apache.hadoop.hbase.io.hfile.TestHFileWriterV2,org.a
    pache.hadoop.hbase.io.hfile.TestReseekTo,org.apache.hadoop.hbase.io.hf
    ile.TestSeekTo,org.apache.hadoop.hbase.io.TestHalfStoreFileReader,org.
    apache.hadoop.hbase.io.TestHbaseObjectWritable,org.apache.hadoop.hbas
    e

    .io.TestHeapSize,org.apache.hadoop.hbase.io.TestImmutableBytesWritable

    ,org.apache.hadoop.hbase.mapreduce.TestSimpleTotalOrderPartitioner,org
    .apache.hadoop.hbase.mapreduce.TestTableSplit,org.apache.hadoop.hbase
    .
    master.TestClockSkewDetection,org.apache.hadoop.hbase.master.TestDea
    dS

    erver,org.apache.hadoop.hbase.master.TestDefaultLoadBalancer,org.apach

    e.hadoop.hbase.master.TestMasterStatusServlet,org.apache.hadoop.hbase.
    metrics.TestMetricsMBeanBase,org.apache.hadoop.hbase.monitoring.Test
    Me
    moryBoundedLogMessageBuffer,org.apache.hadoop.hbase.monitoring.Test
    Tas

    kMonitor,org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionH

    andler,org.apache.hadoop.hbase.regionserver.TestCompactSelection,org.a
    pache.hadoop.hbase.regionserver.TestExplicitColumnTracker,org.apache.h
    adoop.hbase.regionserver.TestHRegionInfo,org.apache.hadoop.hbase.regio
    nserver.TestKeyValueHeap,org.apache.hadoop.hbase.regionserver.TestKey
    V

    alueScanFixture,org.apache.hadoop.hbase.regionserver.TestKeyValueSkipL
    istSet,org.apache.hadoop.hbase.regionserver.TestMemStoreLAB,org.apach
    e

    .hadoop.hbase.regionserver.TestMinVersions,org.apache.hadoop.hbase.reg

    ionserver.TestQueryMatcher,org.apache.hadoop.hbase.regionserver.TestRe
    gionSplitPolicy,org.apache.hadoop.hbase.regionserver.TestResettingCoun
    ters,org.apache.hadoop.hbase.regionserver.TestRSStatusServlet,org.apac
    he.hadoop.hbase.regionserver.TestScanDeleteTracker,org.apache.hadoop.h

    base.regionserver.TestScanWildcardColumnTracker,org.apache.hadoop.hbas

    e.regionserver.TestScanWithBloomError,org.apache.hadoop.hbase.regionse

    rver.TestStoreScanner,org.apache.hadoop.hbase.regionserver.wal.TestHLo
    gBench,org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods,org.a
    p

    ache.hadoop.hbase.regionserver.wal.TestWALActionsListener,org.apache.h
    adoop.hbase.rest.model.TestCellModel,org.apache.hadoop.hbase.rest.mod
    e
    l.TestCellSetModel,org.apache.hadoop.hbase.rest.model.TestColumnSchem
    a
    Model,org.apache.hadoop.hbase.rest.model.TestRowModelo,rg.apache.ha
    doo
    p.hbase.rest.model.TestScannerModel,org.apache.hadoop.hbase.rest.mod
    el

    .TestStorageClusterStatusModel,org.apache.hadoop.hbase.rest.model.Test

    StorageClusterVersionModel,org.apache.hadoop.hbase.rest.model.TestTabl

    eInfoModel,org.apache.hadoop.hbase.rest.model.TestTableListModel,org.a
    pache.hadoop.hbase.rest.model.TestTableRegionModel,org.apache.hadoop
    .h
    base.rest.model.TestTableSchemaModel,org.apache.hadoop.hbase.rest.mo
    de
    l.TestVersionModel,org.apache.hadoop.hbase.security.TestUser,org.apach
    e.hadoop.hbase.TestAcidGuarantees,org.apache.hadoop.hbase.TestCompar
    e,
    org.apache.hadoop.hbase.TestHRegionLocation,org.apache.hadoop.hbase.T
    e

    stHServerAddress,org.apache.hadoop.hbase.TestHServerInfo,org.apache.ha
    doop.hbase.TestKeyValue,org.apache.hadoop.hbase.TestSerialization,org.
    apache.hadoop.hbase.TestServerName,org.apache.hadoop.hbase.util.TestB
    a
    se64,org.apache.hadoop.hbase.util.TestByteBloomFilter,org.apache.hadoo
    p.hbase.util.TestBytes,org.apache.hadoop.hbase.util.TestCompressionTes
    t,org.apache.hadoop.hbase.util.TestFSTableDescriptors,org.apache.hadoo
    p.hbase.util.TestHBaseFsckComparator,org.apache.hadoop.hbase.util.Test
    IncrementingEnvironmentEdge,org.apache.hadoop.hbase.util.TestKeying,or

    g.apache.hadoop.hbase.util.TestPoolMap,org.apache.hadoop.hbase.util.Te
    stRegionSplitCalculator,org.apache.hadoop.hbase.zookeeper.TestHQuorum
    P
    eer,org.apache.hadoop.hbase.zookeeper.TestZooKeeperMainServerArg,org
    .a

    pache.hadoop.hbase.coprocessor.TestCoprocessorInterface,org.apache.had
    oop.hbase.executor.TestExecutorService,org.apache.hadoop.hbase.filter.
    TestColumnPrefixFilter,org.apache.hadoop.hbase.filter.TestFilter,org.a
    pache.hadoop.hbase.filter.TestMultipleColumnPrefixFilter,org.apache.ha
    doop.hbase.io.hfile.slab.TestSingleSizeCache,org.apache.hadoop.hbase.i
    o.hfile.TestCacheOnWrite,org.apache.hadoop.hbase.io.hfile.TestHFileBlo
    ckIndex,org.apache.hadoop.hbase.io.hfile.TestHFileBlock,org.apache.had
    oop.hbase.io.hfile.TestHFilePerformance,org.apache.hadoop.hbase.io.hfi
    le.TestHFileSeek,org.apache.hadoop.hbase.ipc.TestDelayedRpc,org.apache

    .hadoop.hbase.master.TestCatalogJanitor,org.apache.hadoop.hbase.region
    server.TestAtomicOperation,org.apache.hadoop.hbase.regionserver.TestM
    e
    mStore,org.apache.hadoop.hbase.regionserver.TestReadWriteConsistencyC
    o
    ntrol,org.apache.hadoop.hbase.regionserver.TestSeekOptimizations,org.a
    pache.hadoop.hbase.regionserver.TestSplitTransaction,org.apache.hadoop
    .hbase.regionserver.TestStore,org.apache.hadoop.hbase.replication.Test
    ReplicationSource


    --
    Todd Lipcon
    Software Engineer, Cloudera
  • Gary Helmling at Oct 20, 2011 at 9:50 pm
    +1 on small/medium/large

    I have a test mvn repo up on people.apache.org that we can use for a
    patched surefire build, if that helps.

    On Thu, Oct 20, 2011 at 2:33 PM, Jonathan Gray wrote:
    +1 on small/medium/large.
    -----Original Message-----
    From: Todd Lipcon
    Sent: Thursday, October 20, 2011 1:49 PM
    To: dev@hbase.apache.org
    Subject: Re: [unit tests] small tests vs. others

    +1 on separating into small/medium/large. One of the requirements of
    "small" should be that it can run parallel with other tests and in nofork mode.

    See SUREFIRE-329 for the necessary surefire patch. Until that's integrated
    upstream we'd have to have our own patched surefire :(

    -Todd
    On Thu, Oct 20, 2011 at 1:46 PM, N Keywal wrote:
    Hi,

    I've done a stat on the tests we have. Many of them are actually quite
    simple: There are 226 test classes in HBase.  100  of them does not
    use a cluster, and takes less than 2 seconds to execute. 24 do not use
    a cluster but need between 2 and 40 seconds.

    I tried to execute these with different maven settings.
    fork: once or always, i.e. all the tests are executed in the same JVM
    or there is a new JVM created for every test class parallel;
    deactivated or activated, with 3 threads and a per class
    parallelisation.

    The 100 tests executes for the whole maven stuff, including the
    initial dependency & task analysis fork always: 1m50s fork once: 30
    seconds
    parallel: 30s (=> no difference).

    It works quite well, except with TestLruBlockCache that I had to remove.

    I tried as well with the 24 'bigger' ones but no cluster, for the
    following
    results:
    fork always: 7m36s
    parallel: 2m32s: 3 times faster.

    However, I had to remove many of them, there are some conflicts that I
    have not solved (nor tried to).

    The fact that nearly 50% of the tests can be executed in 1 minutes is
    interesting imho.

    That makes me wonder if we should try to get a separation of the tests
    in 3 parts, small/medium/large as of
    http://googletesting.blogspot.com/2009/07/software-testing-categorizat
    ion.html

    small would be:
    - less than two seconds, can be executed in the same jvm
    - all of them: less than 2 minute
    - no cluster, nothing that can break the environment
    - run by the developers as a part of the build

    medium:
    - less than 40 seconds
    - all of them: would last less than 30 minutes, ideally 15 minutes
    - run in an isolated jvm
    - run explicitly by the developers before submitting a fix

    large:
    - run in an isolated jvm
    - Should last less than 45minutes
    - run on the central built or explicitly by the developers
    - break the built when it fails.

    Only one issue: it's not directly possible with maven, but there are
    known workarounds :-)


    What do you think?


    Below the tests set for the 120 tests that seems to work altogether.


    mvn test
    -
    Dtest=org.apache.hadoop.hbase.avro.TestAvroUtil,org.apache.hadoop.hba
    se.client.TestAttributes,org.apache.hadoop.hbase.client.TestGet,org.ap
    ache.hadoop.hbase.client.TestOperation,org.apache.hadoop.hbase.client.
    TestResult,org.apache.hadoop.hbase.client.TestScan,org.apache.hadoop.h
    base.coprocessor.TestRegionObserverStacking,org.apache.hadoop.hbase.fi
    lter.TestBitComparator,org.apache.hadoop.hbase.filter.TestColumnPagina
    tionFilter,org.apache.hadoop.hbase.filter.TestDependentColumnFilter,or
    g.apache.hadoop.hbase.filter.TestFilterList,org.apache.hadoop.hbase.fi
    lter.TestInclusiveStopFilter,org.apache.hadoop.hbase.filter.TestPageFi
    lter,org.apache.hadoop.hbase.filter.TestParseFilter,org.apache.hadoop.
    hbase.filter.TestPrefixFilter,org.apache.hadoop.hbase.filter.TestRando
    mRowFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueExclude
    Filter,org.apache.hadoop.hbase.filter.TestSingleColumnValueFilter,org.
    apache.hadoop.hbase.io.hfile.slab.TestSlabCache,org.apache.hadoop.hbas
    e.io.hfile.slab.TestSlab,org.apache.hadoop.hbase.io.hfile.TestBlockCac
    heColumnFamilySummary,org.apache.hadoop.hbase.io.hfile.TestCachedBloc
    k
    Queue,org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer,org.apache
    .hadoop.hbase.io.hfile.TestHFileReaderV1,org.apache.hadoop.hbase.io.hf
    ile.TestHFile,org.apache.hadoop.hbase.io.hfile.TestHFileWriterV2,org.a
    pache.hadoop.hbase.io.hfile.TestReseekTo,org.apache.hadoop.hbase.io.hf
    ile.TestSeekTo,org.apache.hadoop.hbase.io.TestHalfStoreFileReader,org.
    apache.hadoop.hbase.io.TestHbaseObjectWritable,org.apache.hadoop.hbas
    e

    .io.TestHeapSize,org.apache.hadoop.hbase.io.TestImmutableBytesWritable

    ,org.apache.hadoop.hbase.mapreduce.TestSimpleTotalOrderPartitioner,org
    .apache.hadoop.hbase.mapreduce.TestTableSplit,org.apache.hadoop.hbase
    .
    master.TestClockSkewDetection,org.apache.hadoop.hbase.master.TestDea
    dS

    erver,org.apache.hadoop.hbase.master.TestDefaultLoadBalancer,org.apach

    e.hadoop.hbase.master.TestMasterStatusServlet,org.apache.hadoop.hbase.
    metrics.TestMetricsMBeanBase,org.apache.hadoop.hbase.monitoring.Test
    Me
    moryBoundedLogMessageBuffer,org.apache.hadoop.hbase.monitoring.Test
    Tas

    kMonitor,org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionH

    andler,org.apache.hadoop.hbase.regionserver.TestCompactSelection,org.a
    pache.hadoop.hbase.regionserver.TestExplicitColumnTracker,org.apache.h
    adoop.hbase.regionserver.TestHRegionInfo,org.apache.hadoop.hbase.regio
    nserver.TestKeyValueHeap,org.apache.hadoop.hbase.regionserver.TestKey
    V

    alueScanFixture,org.apache.hadoop.hbase.regionserver.TestKeyValueSkipL
    istSet,org.apache.hadoop.hbase.regionserver.TestMemStoreLAB,org.apach
    e

    .hadoop.hbase.regionserver.TestMinVersions,org.apache.hadoop.hbase.reg

    ionserver.TestQueryMatcher,org.apache.hadoop.hbase.regionserver.TestRe
    gionSplitPolicy,org.apache.hadoop.hbase.regionserver.TestResettingCoun
    ters,org.apache.hadoop.hbase.regionserver.TestRSStatusServlet,org.apac
    he.hadoop.hbase.regionserver.TestScanDeleteTracker,org.apache.hadoop.h

    base.regionserver.TestScanWildcardColumnTracker,org.apache.hadoop.hbas

    e.regionserver.TestScanWithBloomError,org.apache.hadoop.hbase.regionse

    rver.TestStoreScanner,org.apache.hadoop.hbase.regionserver.wal.TestHLo
    gBench,org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods,org.a
    p

    ache.hadoop.hbase.regionserver.wal.TestWALActionsListener,org.apache.h
    adoop.hbase.rest.model.TestCellModel,org.apache.hadoop.hbase.rest.mod
    e
    l.TestCellSetModel,org.apache.hadoop.hbase.rest.model.TestColumnSchem
    a
    Model,org.apache.hadoop.hbase.rest.model.TestRowModelo,rg.apache.ha
    doo
    p.hbase.rest.model.TestScannerModel,org.apache.hadoop.hbase.rest.mod
    el

    .TestStorageClusterStatusModel,org.apache.hadoop.hbase.rest.model.Test

    StorageClusterVersionModel,org.apache.hadoop.hbase.rest.model.TestTabl

    eInfoModel,org.apache.hadoop.hbase.rest.model.TestTableListModel,org.a
    pache.hadoop.hbase.rest.model.TestTableRegionModel,org.apache.hadoop
    .h
    base.rest.model.TestTableSchemaModel,org.apache.hadoop.hbase.rest.mo
    de
    l.TestVersionModel,org.apache.hadoop.hbase.security.TestUser,org.apach
    e.hadoop.hbase.TestAcidGuarantees,org.apache.hadoop.hbase.TestCompar
    e,
    org.apache.hadoop.hbase.TestHRegionLocation,org.apache.hadoop.hbase.T
    e

    stHServerAddress,org.apache.hadoop.hbase.TestHServerInfo,org.apache.ha
    doop.hbase.TestKeyValue,org.apache.hadoop.hbase.TestSerialization,org.
    apache.hadoop.hbase.TestServerName,org.apache.hadoop.hbase.util.TestB
    a
    se64,org.apache.hadoop.hbase.util.TestByteBloomFilter,org.apache.hadoo
    p.hbase.util.TestBytes,org.apache.hadoop.hbase.util.TestCompressionTes
    t,org.apache.hadoop.hbase.util.TestFSTableDescriptors,org.apache.hadoo
    p.hbase.util.TestHBaseFsckComparator,org.apache.hadoop.hbase.util.Test
    IncrementingEnvironmentEdge,org.apache.hadoop.hbase.util.TestKeying,or

    g.apache.hadoop.hbase.util.TestPoolMap,org.apache.hadoop.hbase.util.Te
    stRegionSplitCalculator,org.apache.hadoop.hbase.zookeeper.TestHQuorum
    P
    eer,org.apache.hadoop.hbase.zookeeper.TestZooKeeperMainServerArg,org
    .a

    pache.hadoop.hbase.coprocessor.TestCoprocessorInterface,org.apache.had
    oop.hbase.executor.TestExecutorService,org.apache.hadoop.hbase.filter.
    TestColumnPrefixFilter,org.apache.hadoop.hbase.filter.TestFilter,org.a
    pache.hadoop.hbase.filter.TestMultipleColumnPrefixFilter,org.apache.ha
    doop.hbase.io.hfile.slab.TestSingleSizeCache,org.apache.hadoop.hbase.i
    o.hfile.TestCacheOnWrite,org.apache.hadoop.hbase.io.hfile.TestHFileBlo
    ckIndex,org.apache.hadoop.hbase.io.hfile.TestHFileBlock,org.apache.had
    oop.hbase.io.hfile.TestHFilePerformance,org.apache.hadoop.hbase.io.hfi
    le.TestHFileSeek,org.apache.hadoop.hbase.ipc.TestDelayedRpc,org.apache

    .hadoop.hbase.master.TestCatalogJanitor,org.apache.hadoop.hbase.region
    server.TestAtomicOperation,org.apache.hadoop.hbase.regionserver.TestM
    e
    mStore,org.apache.hadoop.hbase.regionserver.TestReadWriteConsistencyC
    o
    ntrol,org.apache.hadoop.hbase.regionserver.TestSeekOptimizations,org.a
    pache.hadoop.hbase.regionserver.TestSplitTransaction,org.apache.hadoop
    .hbase.regionserver.TestStore,org.apache.hadoop.hbase.replication.Test
    ReplicationSource


    --
    Todd Lipcon
    Software Engineer, Cloudera
  • N Keywal at Nov 14, 2011 at 1:52 pm
    Hi Gary,

    The hbase specific versions for surefire & junit are ready (see HBASE-4763
    for all the details).
    Would it be possible to use your repository? What would you need from me?
    I've never deployed anything in a maven repo. Surefire works with maven, so
    it should be easy, JUnit does not, so there could be more work to do.

    I copy the dev list for information.

    Regards,

    N.


    ---------- Forwarded message ----------
    From: Gary Helmling <ghelmling@gmail.com>
    Date: Thu, Oct 20, 2011 at 11:50 PM
    Subject: Re: [unit tests] small tests vs. others
    To: dev@hbase.apache.org


    +1 on small/medium/large

    I have a test mvn repo up on people.apache.org that we can use for a
    patched surefire build, if that helps.

    On Thu, Oct 20, 2011 at 2:33 PM, Jonathan Gray wrote:
    +1 on small/medium/large.
    -----Original Message-----
    From: Todd Lipcon
    Sent: Thursday, October 20, 2011 1:49 PM
    To: dev@hbase.apache.org
    Subject: Re: [unit tests] small tests vs. others

    +1 on separating into small/medium/large. One of the requirements of
    "small" should be that it can run parallel with other tests and in
    nofork mode.
    See SUREFIRE-329 for the necessary surefire patch. Until that's
    integrated
    upstream we'd have to have our own patched surefire :(

    -Todd
    On Thu, Oct 20, 2011 at 1:46 PM, N Keywal wrote:
    Hi,

    I've done a stat on the tests we have. Many of them are actually quite
    simple: There are 226 test classes in HBase. 100 of them does not
    use a cluster, and takes less than 2 seconds to execute. 24 do not use
    a cluster but need between 2 and 40 seconds.

    I tried to execute these with different maven settings.
    fork: once or always, i.e. all the tests are executed in the same JVM
    or there is a new JVM created for every test class parallel;
    deactivated or activated, with 3 threads and a per class
    parallelisation.

    The 100 tests executes for the whole maven stuff, including the
    initial dependency & task analysis fork always: 1m50s fork once: 30
    seconds
    parallel: 30s (=> no difference).

    It works quite well, except with TestLruBlockCache that I had to
    remove.
    I tried as well with the 24 'bigger' ones but no cluster, for the
    following
    results:
    fork always: 7m36s
    parallel: 2m32s: 3 times faster.

    However, I had to remove many of them, there are some conflicts that I
    have not solved (nor tried to).

    The fact that nearly 50% of the tests can be executed in 1 minutes is
    interesting imho.

    That makes me wonder if we should try to get a separation of the tests
    in 3 parts, small/medium/large as of
    http://googletesting.blogspot.com/2009/07/software-testing-categorizat
    ion.html

    small would be:
    - less than two seconds, can be executed in the same jvm
    - all of them: less than 2 minute
    - no cluster, nothing that can break the environment
    - run by the developers as a part of the build

    medium:
    - less than 40 seconds
    - all of them: would last less than 30 minutes, ideally 15 minutes
    - run in an isolated jvm
    - run explicitly by the developers before submitting a fix

    large:
    - run in an isolated jvm
    - Should last less than 45minutes
    - run on the central built or explicitly by the developers
    - break the built when it fails.

    Only one issue: it's not directly possible with maven, but there are
    known workarounds :-)


    What do you think?


    Below the tests set for the 120 tests that seems to work altogether.


    mvn test
    -
    Dtest=org.apache.hadoop.hbase.avro.TestAvroUtil,org.apache.hadoop.hba
    se.client.TestAttributes,org.apache.hadoop.hbase.client.TestGet,org.ap
    ache.hadoop.hbase.client.TestOperation,org.apache.hadoop.hbase.client.
    TestResult,org.apache.hadoop.hbase.client.TestScan,org.apache.hadoop.h
    base.coprocessor.TestRegionObserverStacking,org.apache.hadoop.hbase.fi
    lter.TestBitComparator,org.apache.hadoop.hbase.filter.TestColumnPagina
    tionFilter,org.apache.hadoop.hbase.filter.TestDependentColumnFilter,or
    g.apache.hadoop.hbase.filter.TestFilterList,org.apache.hadoop.hbase.fi
    lter.TestInclusiveStopFilter,org.apache.hadoop.hbase.filter.TestPageFi
    lter,org.apache.hadoop.hbase.filter.TestParseFilter,org.apache.hadoop.
    hbase.filter.TestPrefixFilter,org.apache.hadoop.hbase.filter.TestRando
    mRowFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueExclude
    Filter,org.apache.hadoop.hbase.filter.TestSingleColumnValueFilter,org.
    apache.hadoop.hbase.io.hfile.slab.TestSlabCache,org.apache.hadoop.hbas
    e.io.hfile.slab.TestSlab,org.apache.hadoop.hbase.io.hfile.TestBlockCac
    heColumnFamilySummary,org.apache.hadoop.hbase.io.hfile.TestCachedBloc
    k
    Queue,org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer,org.apache
    .hadoop.hbase.io.hfile.TestHFileReaderV1,org.apache.hadoop.hbase.io.hf
    ile.TestHFile,org.apache.hadoop.hbase.io.hfile.TestHFileWriterV2,org.a
    pache.hadoop.hbase.io.hfile.TestReseekTo,org.apache.hadoop.hbase.io.hf
    ile.TestSeekTo,org.apache.hadoop.hbase.io.TestHalfStoreFileReader,org.
    apache.hadoop.hbase.io.TestHbaseObjectWritable,org.apache.hadoop.hbas
    e

    .io.TestHeapSize,org.apache.hadoop.hbase.io.TestImmutableBytesWritable

    ,org.apache.hadoop.hbase.mapreduce.TestSimpleTotalOrderPartitioner,org
    .apache.hadoop.hbase.mapreduce.TestTableSplit,org.apache.hadoop.hbase
    .
    master.TestClockSkewDetection,org.apache.hadoop.hbase.master.TestDea
    dS

    erver,org.apache.hadoop.hbase.master.TestDefaultLoadBalancer,org.apach

    e.hadoop.hbase.master.TestMasterStatusServlet,org.apache.hadoop.hbase.
    metrics.TestMetricsMBeanBase,org.apache.hadoop.hbase.monitoring.Test
    Me
    moryBoundedLogMessageBuffer,org.apache.hadoop.hbase.monitoring.Test
    Tas

    kMonitor,org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionH

    andler,org.apache.hadoop.hbase.regionserver.TestCompactSelection,org.a
    pache.hadoop.hbase.regionserver.TestExplicitColumnTracker,org.apache.h
    adoop.hbase.regionserver.TestHRegionInfo,org.apache.hadoop.hbase.regio
    nserver.TestKeyValueHeap,org.apache.hadoop.hbase.regionserver.TestKey
    V

    alueScanFixture,org.apache.hadoop.hbase.regionserver.TestKeyValueSkipL
    istSet,org.apache.hadoop.hbase.regionserver.TestMemStoreLAB,org.apach
    e

    .hadoop.hbase.regionserver.TestMinVersions,org.apache.hadoop.hbase.reg

    ionserver.TestQueryMatcher,org.apache.hadoop.hbase.regionserver.TestRe
    gionSplitPolicy,org.apache.hadoop.hbase.regionserver.TestResettingCoun
    ters,org.apache.hadoop.hbase.regionserver.TestRSStatusServlet,org.apac
    he.hadoop.hbase.regionserver.TestScanDeleteTracker,org.apache.hadoop.h

    base.regionserver.TestScanWildcardColumnTracker,org.apache.hadoop.hbas

    e.regionserver.TestScanWithBloomError,org.apache.hadoop.hbase.regionse

    rver.TestStoreScanner,org.apache.hadoop.hbase.regionserver.wal.TestHLo
    gBench,org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods,org.a
    p

    ache.hadoop.hbase.regionserver.wal.TestWALActionsListener,org.apache.h
    adoop.hbase.rest.model.TestCellModel,org.apache.hadoop.hbase.rest.mod
    e
    l.TestCellSetModel,org.apache.hadoop.hbase.rest.model.TestColumnSchem
    a
    Model,org.apache.hadoop.hbase.rest.model.TestRowModelo,rg.apache.ha
    doo
    p.hbase.rest.model.TestScannerModel,org.apache.hadoop.hbase.rest.mod
    el

    .TestStorageClusterStatusModel,org.apache.hadoop.hbase.rest.model.Test

    StorageClusterVersionModel,org.apache.hadoop.hbase.rest.model.TestTabl

    eInfoModel,org.apache.hadoop.hbase.rest.model.TestTableListModel,org.a
    pache.hadoop.hbase.rest.model.TestTableRegionModel,org.apache.hadoop
    .h
    base.rest.model.TestTableSchemaModel,org.apache.hadoop.hbase.rest.mo
    de
    l.TestVersionModel,org.apache.hadoop.hbase.security.TestUser,org.apach
    e.hadoop.hbase.TestAcidGuarantees,org.apache.hadoop.hbase.TestCompar
    e,
    org.apache.hadoop.hbase.TestHRegionLocation,org.apache.hadoop.hbase.T
    e

    stHServerAddress,org.apache.hadoop.hbase.TestHServerInfo,org.apache.ha
    doop.hbase.TestKeyValue,org.apache.hadoop.hbase.TestSerialization,org.
    apache.hadoop.hbase.TestServerName,org.apache.hadoop.hbase.util.TestB
    a
    se64,org.apache.hadoop.hbase.util.TestByteBloomFilter,org.apache.hadoo
    p.hbase.util.TestBytes,org.apache.hadoop.hbase.util.TestCompressionTes
    t,org.apache.hadoop.hbase.util.TestFSTableDescriptors,org.apache.hadoo
    p.hbase.util.TestHBaseFsckComparator,org.apache.hadoop.hbase.util.Test
    IncrementingEnvironmentEdge,org.apache.hadoop.hbase.util.TestKeying,or

    g.apache.hadoop.hbase.util.TestPoolMap,org.apache.hadoop.hbase.util.Te
    stRegionSplitCalculator,org.apache.hadoop.hbase.zookeeper.TestHQuorum
    P
    eer,org.apache.hadoop.hbase.zookeeper.TestZooKeeperMainServerArg,org
    .a

    pache.hadoop.hbase.coprocessor.TestCoprocessorInterface,org.apache.had
    oop.hbase.executor.TestExecutorService,org.apache.hadoop.hbase.filter.
    TestColumnPrefixFilter,org.apache.hadoop.hbase.filter.TestFilter,org.a
    pache.hadoop.hbase.filter.TestMultipleColumnPrefixFilter,org.apache.ha
    doop.hbase.io.hfile.slab.TestSingleSizeCache,org.apache.hadoop.hbase.i
    o.hfile.TestCacheOnWrite,org.apache.hadoop.hbase.io.hfile.TestHFileBlo
    ckIndex,org.apache.hadoop.hbase.io.hfile.TestHFileBlock,org.apache.had
    oop.hbase.io.hfile.TestHFilePerformance,org.apache.hadoop.hbase.io.hfi
    le.TestHFileSeek,org.apache.hadoop.hbase.ipc.TestDelayedRpc,org.apache

    .hadoop.hbase.master.TestCatalogJanitor,org.apache.hadoop.hbase.region
    server.TestAtomicOperation,org.apache.hadoop.hbase.regionserver.TestM
    e
    mStore,org.apache.hadoop.hbase.regionserver.TestReadWriteConsistencyC
    o
    ntrol,org.apache.hadoop.hbase.regionserver.TestSeekOptimizations,org.a
    pache.hadoop.hbase.regionserver.TestSplitTransaction,org.apache.hadoop
    .hbase.regionserver.TestStore,org.apache.hadoop.hbase.replication.Test
    ReplicationSource


    --
    Todd Lipcon
    Software Engineer, Cloudera
  • Gary Helmling at Nov 14, 2011 at 7:47 pm
    Hi N,

    I can push up the modified surefire and junit build with appropriate versions.

    Is there anything special to either build that would complicate
    building with the supplied patches? If not, then I can do the builds
    and push up the artifacts. Then we coordinate the testing. How does
    that sound?

    --gh

    On Mon, Nov 14, 2011 at 5:51 AM, N Keywal wrote:
    Hi Gary,

    The hbase specific versions for surefire & junit are ready (see HBASE-4763
    for all the details).
    Would it be possible to use your repository? What would you need from me?
    I've never deployed anything in a maven repo. Surefire works with maven, so
    it should be easy, JUnit does not, so there could be more work to do.

    I copy the dev list for information.

    Regards,

    N.


    ---------- Forwarded message ----------
    From: Gary Helmling <ghelmling@gmail.com>
    Date: Thu, Oct 20, 2011 at 11:50 PM
    Subject: Re: [unit tests] small tests vs. others
    To: dev@hbase.apache.org


    +1 on small/medium/large

    I have a test mvn repo up on people.apache.org that we can use for a
    patched surefire build, if that helps.

    On Thu, Oct 20, 2011 at 2:33 PM, Jonathan Gray wrote:
    +1 on small/medium/large.
    -----Original Message-----
    From: Todd Lipcon
    Sent: Thursday, October 20, 2011 1:49 PM
    To: dev@hbase.apache.org
    Subject: Re: [unit tests] small tests vs. others

    +1 on separating into small/medium/large. One of the requirements of
    "small" should be that it can run parallel with other tests and in nofork
    mode.

    See SUREFIRE-329 for the necessary surefire patch. Until that's
    integrated
    upstream we'd have to have our own patched surefire :(

    -Todd
    On Thu, Oct 20, 2011 at 1:46 PM, N Keywal wrote:
    Hi,

    I've done a stat on the tests we have. Many of them are actually quite
    simple: There are 226 test classes in HBase.  100  of them does not
    use a cluster, and takes less than 2 seconds to execute. 24 do not use
    a cluster but need between 2 and 40 seconds.

    I tried to execute these with different maven settings.
    fork: once or always, i.e. all the tests are executed in the same JVM
    or there is a new JVM created for every test class parallel;
    deactivated or activated, with 3 threads and a per class
    parallelisation.

    The 100 tests executes for the whole maven stuff, including the
    initial dependency & task analysis fork always: 1m50s fork once: 30
    seconds
    parallel: 30s (=> no difference).

    It works quite well, except with TestLruBlockCache that I had to
    remove.

    I tried as well with the 24 'bigger' ones but no cluster, for the
    following
    results:
    fork always: 7m36s
    parallel: 2m32s: 3 times faster.

    However, I had to remove many of them, there are some conflicts that I
    have not solved (nor tried to).

    The fact that nearly 50% of the tests can be executed in 1 minutes is
    interesting imho.

    That makes me wonder if we should try to get a separation of the tests
    in 3 parts, small/medium/large as of
    http://googletesting.blogspot.com/2009/07/software-testing-categorizat
    ion.html

    small would be:
    - less than two seconds, can be executed in the same jvm
    - all of them: less than 2 minute
    - no cluster, nothing that can break the environment
    - run by the developers as a part of the build

    medium:
    - less than 40 seconds
    - all of them: would last less than 30 minutes, ideally 15 minutes
    - run in an isolated jvm
    - run explicitly by the developers before submitting a fix

    large:
    - run in an isolated jvm
    - Should last less than 45minutes
    - run on the central built or explicitly by the developers
    - break the built when it fails.

    Only one issue: it's not directly possible with maven, but there are
    known workarounds :-)


    What do you think?


    Below the tests set for the 120 tests that seems to work altogether.


    mvn test
    -
    Dtest=org.apache.hadoop.hbase.avro.TestAvroUtil,org.apache.hadoop.hba
    se.client.TestAttributes,org.apache.hadoop.hbase.client.TestGet,org.ap
    ache.hadoop.hbase.client.TestOperation,org.apache.hadoop.hbase.client.
    TestResult,org.apache.hadoop.hbase.client.TestScan,org.apache.hadoop.h
    base.coprocessor.TestRegionObserverStacking,org.apache.hadoop.hbase.fi
    lter.TestBitComparator,org.apache.hadoop.hbase.filter.TestColumnPagina
    tionFilter,org.apache.hadoop.hbase.filter.TestDependentColumnFilter,or
    g.apache.hadoop.hbase.filter.TestFilterList,org.apache.hadoop.hbase.fi
    lter.TestInclusiveStopFilter,org.apache.hadoop.hbase.filter.TestPageFi
    lter,org.apache.hadoop.hbase.filter.TestParseFilter,org.apache.hadoop.
    hbase.filter.TestPrefixFilter,org.apache.hadoop.hbase.filter.TestRando
    mRowFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueExclude
    Filter,org.apache.hadoop.hbase.filter.TestSingleColumnValueFilter,org.
    apache.hadoop.hbase.io.hfile.slab.TestSlabCache,org.apache.hadoop.hbas
    e.io.hfile.slab.TestSlab,org.apache.hadoop.hbase.io.hfile.TestBlockCac
    heColumnFamilySummary,org.apache.hadoop.hbase.io.hfile.TestCachedBloc
    k
    Queue,org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer,org.apache
    .hadoop.hbase.io.hfile.TestHFileReaderV1,org.apache.hadoop.hbase.io.hf
    ile.TestHFile,org.apache.hadoop.hbase.io.hfile.TestHFileWriterV2,org.a
    pache.hadoop.hbase.io.hfile.TestReseekTo,org.apache.hadoop.hbase.io.hf
    ile.TestSeekTo,org.apache.hadoop.hbase.io.TestHalfStoreFileReader,org.
    apache.hadoop.hbase.io.TestHbaseObjectWritable,org.apache.hadoop.hbas
    e

    .io.TestHeapSize,org.apache.hadoop.hbase.io.TestImmutableBytesWritable

    ,org.apache.hadoop.hbase.mapreduce.TestSimpleTotalOrderPartitioner,org
    .apache.hadoop.hbase.mapreduce.TestTableSplit,org.apache.hadoop.hbase
    .
    master.TestClockSkewDetection,org.apache.hadoop.hbase.master.TestDea
    dS

    erver,org.apache.hadoop.hbase.master.TestDefaultLoadBalancer,org.apach

    e.hadoop.hbase.master.TestMasterStatusServlet,org.apache.hadoop.hbase.
    metrics.TestMetricsMBeanBase,org.apache.hadoop.hbase.monitoring.Test
    Me
    moryBoundedLogMessageBuffer,org.apache.hadoop.hbase.monitoring.Test
    Tas

    kMonitor,org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionH

    andler,org.apache.hadoop.hbase.regionserver.TestCompactSelection,org.a
    pache.hadoop.hbase.regionserver.TestExplicitColumnTracker,org.apache.h
    adoop.hbase.regionserver.TestHRegionInfo,org.apache.hadoop.hbase.regio
    nserver.TestKeyValueHeap,org.apache.hadoop.hbase.regionserver.TestKey
    V

    alueScanFixture,org.apache.hadoop.hbase.regionserver.TestKeyValueSkipL
    istSet,org.apache.hadoop.hbase.regionserver.TestMemStoreLAB,org.apach
    e

    .hadoop.hbase.regionserver.TestMinVersions,org.apache.hadoop.hbase.reg

    ionserver.TestQueryMatcher,org.apache.hadoop.hbase.regionserver.TestRe
    gionSplitPolicy,org.apache.hadoop.hbase.regionserver.TestResettingCoun
    ters,org.apache.hadoop.hbase.regionserver.TestRSStatusServlet,org.apac
    he.hadoop.hbase.regionserver.TestScanDeleteTracker,org.apache.hadoop.h

    base.regionserver.TestScanWildcardColumnTracker,org.apache.hadoop.hbas

    e.regionserver.TestScanWithBloomError,org.apache.hadoop.hbase.regionse

    rver.TestStoreScanner,org.apache.hadoop.hbase.regionserver.wal.TestHLo
    gBench,org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods,org.a
    p

    ache.hadoop.hbase.regionserver.wal.TestWALActionsListener,org.apache.h
    adoop.hbase.rest.model.TestCellModel,org.apache.hadoop.hbase.rest.mod
    e
    l.TestCellSetModel,org.apache.hadoop.hbase.rest.model.TestColumnSchem
    a
    Model,org.apache.hadoop.hbase.rest.model.TestRowModelo,rg.apache.ha
    doo
    p.hbase.rest.model.TestScannerModel,org.apache.hadoop.hbase.rest.mod
    el

    .TestStorageClusterStatusModel,org.apache.hadoop.hbase.rest.model.Test

    StorageClusterVersionModel,org.apache.hadoop.hbase.rest.model.TestTabl

    eInfoModel,org.apache.hadoop.hbase.rest.model.TestTableListModel,org.a
    pache.hadoop.hbase.rest.model.TestTableRegionModel,org.apache.hadoop
    .h
    base.rest.model.TestTableSchemaModel,org.apache.hadoop.hbase.rest.mo
    de
    l.TestVersionModel,org.apache.hadoop.hbase.security.TestUser,org.apach
    e.hadoop.hbase.TestAcidGuarantees,org.apache.hadoop.hbase.TestCompar
    e,
    org.apache.hadoop.hbase.TestHRegionLocation,org.apache.hadoop.hbase.T
    e

    stHServerAddress,org.apache.hadoop.hbase.TestHServerInfo,org.apache.ha
    doop.hbase.TestKeyValue,org.apache.hadoop.hbase.TestSerialization,org.
    apache.hadoop.hbase.TestServerName,org.apache.hadoop.hbase.util.TestB
    a
    se64,org.apache.hadoop.hbase.util.TestByteBloomFilter,org.apache.hadoo
    p.hbase.util.TestBytes,org.apache.hadoop.hbase.util.TestCompressionTes
    t,org.apache.hadoop.hbase.util.TestFSTableDescriptors,org.apache.hadoo
    p.hbase.util.TestHBaseFsckComparator,org.apache.hadoop.hbase.util.Test
    IncrementingEnvironmentEdge,org.apache.hadoop.hbase.util.TestKeying,or

    g.apache.hadoop.hbase.util.TestPoolMap,org.apache.hadoop.hbase.util.Te
    stRegionSplitCalculator,org.apache.hadoop.hbase.zookeeper.TestHQuorum
    P
    eer,org.apache.hadoop.hbase.zookeeper.TestZooKeeperMainServerArg,org
    .a

    pache.hadoop.hbase.coprocessor.TestCoprocessorInterface,org.apache.had
    oop.hbase.executor.TestExecutorService,org.apache.hadoop.hbase.filter.
    TestColumnPrefixFilter,org.apache.hadoop.hbase.filter.TestFilter,org.a
    pache.hadoop.hbase.filter.TestMultipleColumnPrefixFilter,org.apache.ha
    doop.hbase.io.hfile.slab.TestSingleSizeCache,org.apache.hadoop.hbase.i
    o.hfile.TestCacheOnWrite,org.apache.hadoop.hbase.io.hfile.TestHFileBlo
    ckIndex,org.apache.hadoop.hbase.io.hfile.TestHFileBlock,org.apache.had
    oop.hbase.io.hfile.TestHFilePerformance,org.apache.hadoop.hbase.io.hfi
    le.TestHFileSeek,org.apache.hadoop.hbase.ipc.TestDelayedRpc,org.apache

    .hadoop.hbase.master.TestCatalogJanitor,org.apache.hadoop.hbase.region
    server.TestAtomicOperation,org.apache.hadoop.hbase.regionserver.TestM
    e
    mStore,org.apache.hadoop.hbase.regionserver.TestReadWriteConsistencyC
    o
    ntrol,org.apache.hadoop.hbase.regionserver.TestSeekOptimizations,org.a
    pache.hadoop.hbase.regionserver.TestSplitTransaction,org.apache.hadoop
    .hbase.regionserver.TestStore,org.apache.hadoop.hbase.replication.Test
    ReplicationSource


    --
    Todd Lipcon
    Software Engineer, Cloudera
  • N Keywal at Nov 14, 2011 at 8:26 pm
    That sounds perfect :-).

    For JUnit, there is nothing specific I used the script provided with JUnit
    (build_tests.sh).
    For surefire, I had to use -DskipTests to get it installed, even before my
    changes. Except this (!) it's pure standard maven.

    Then there are two use cases:
    - short term, we just need to change the versions and the repository list
    in the .pom to see how it behaves.
    - medium term (3/4 days) we should publish the category split, I created
    4781 to put the .pom I am currently testing.

    Thanks for your help!

    On Mon, Nov 14, 2011 at 8:47 PM, Gary Helmling wrote:

    Hi N,

    I can push up the modified surefire and junit build with appropriate
    versions.

    Is there anything special to either build that would complicate
    building with the supplied patches? If not, then I can do the builds
    and push up the artifacts. Then we coordinate the testing. How does
    that sound?

    --gh

    On Mon, Nov 14, 2011 at 5:51 AM, N Keywal wrote:
    Hi Gary,

    The hbase specific versions for surefire & junit are ready (see
    HBASE-4763
    for all the details).
    Would it be possible to use your repository? What would you need from me?
    I've never deployed anything in a maven repo. Surefire works with maven, so
    it should be easy, JUnit does not, so there could be more work to do.

    I copy the dev list for information.

    Regards,

    N.


    ---------- Forwarded message ----------
    From: Gary Helmling <ghelmling@gmail.com>
    Date: Thu, Oct 20, 2011 at 11:50 PM
    Subject: Re: [unit tests] small tests vs. others
    To: dev@hbase.apache.org


    +1 on small/medium/large

    I have a test mvn repo up on people.apache.org that we can use for a
    patched surefire build, if that helps.

    On Thu, Oct 20, 2011 at 2:33 PM, Jonathan Gray wrote:
    +1 on small/medium/large.
    -----Original Message-----
    From: Todd Lipcon
    Sent: Thursday, October 20, 2011 1:49 PM
    To: dev@hbase.apache.org
    Subject: Re: [unit tests] small tests vs. others

    +1 on separating into small/medium/large. One of the requirements of
    "small" should be that it can run parallel with other tests and in
    nofork
    mode.

    See SUREFIRE-329 for the necessary surefire patch. Until that's
    integrated
    upstream we'd have to have our own patched surefire :(

    -Todd
    On Thu, Oct 20, 2011 at 1:46 PM, N Keywal wrote:
    Hi,

    I've done a stat on the tests we have. Many of them are actually
    quite
    simple: There are 226 test classes in HBase. 100 of them does not
    use a cluster, and takes less than 2 seconds to execute. 24 do not
    use
    a cluster but need between 2 and 40 seconds.

    I tried to execute these with different maven settings.
    fork: once or always, i.e. all the tests are executed in the same JVM
    or there is a new JVM created for every test class parallel;
    deactivated or activated, with 3 threads and a per class
    parallelisation.

    The 100 tests executes for the whole maven stuff, including the
    initial dependency & task analysis fork always: 1m50s fork once: 30
    seconds
    parallel: 30s (=> no difference).

    It works quite well, except with TestLruBlockCache that I had to
    remove.

    I tried as well with the 24 'bigger' ones but no cluster, for the
    following
    results:
    fork always: 7m36s
    parallel: 2m32s: 3 times faster.

    However, I had to remove many of them, there are some conflicts that
    I
    have not solved (nor tried to).

    The fact that nearly 50% of the tests can be executed in 1 minutes is
    interesting imho.

    That makes me wonder if we should try to get a separation of the
    tests
    in 3 parts, small/medium/large as of
    http://googletesting.blogspot.com/2009/07/software-testing-categorizat
    ion.html

    small would be:
    - less than two seconds, can be executed in the same jvm
    - all of them: less than 2 minute
    - no cluster, nothing that can break the environment
    - run by the developers as a part of the build

    medium:
    - less than 40 seconds
    - all of them: would last less than 30 minutes, ideally 15 minutes
    - run in an isolated jvm
    - run explicitly by the developers before submitting a fix

    large:
    - run in an isolated jvm
    - Should last less than 45minutes
    - run on the central built or explicitly by the developers
    - break the built when it fails.

    Only one issue: it's not directly possible with maven, but there are
    known workarounds :-)


    What do you think?


    Below the tests set for the 120 tests that seems to work altogether.


    mvn test
    -
    Dtest=org.apache.hadoop.hbase.avro.TestAvroUtil,org.apache.hadoop.hba
    se.client.TestAttributes,org.apache.hadoop.hbase.client.TestGet,org.ap
    ache.hadoop.hbase.client.TestOperation,org.apache.hadoop.hbase.client.
    TestResult,org.apache.hadoop.hbase.client.TestScan,org.apache.hadoop.h
    base.coprocessor.TestRegionObserverStacking,org.apache.hadoop.hbase.fi
    lter.TestBitComparator,org.apache.hadoop.hbase.filter.TestColumnPagina
    tionFilter,org.apache.hadoop.hbase.filter.TestDependentColumnFilter,or
    g.apache.hadoop.hbase.filter.TestFilterList,
    org.apache.hadoop.hbase.fi
    lter.TestInclusiveStopFilter,org.apache.hadoop.hbase.filter.TestPageFi
    lter,org.apache.hadoop.hbase.filter.TestParseFilter,org.apache.hadoop.
    hbase.filter.TestPrefixFilter,org.apache.hadoop.hbase.filter.TestRando
    mRowFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueExclude
    Filter,org.apache.hadoop.hbase.filter.TestSingleColumnValueFilter,org.
    apache.hadoop.hbase.io.hfile.slab.TestSlabCache,org.apache.hadoop.hbas
    e.io.hfile.slab.TestSlab,org.apache.hadoop.hbase.io.hfile.TestBlockCac
    heColumnFamilySummary,org.apache.hadoop.hbase.io.hfile.TestCachedBloc
    k
    Queue,org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer,org.apache
    .hadoop.hbase.io.hfile.TestHFileReaderV1,org.apache.hadoop.hbase.io.hf
    ile.TestHFile,org.apache.hadoop.hbase.io.hfile.TestHFileWriterV2,org.a
    pache.hadoop.hbase.io.hfile.TestReseekTo,org.apache.hadoop.hbase.io.hf
    ile.TestSeekTo,org.apache.hadoop.hbase.io.TestHalfStoreFileReader,org.
    apache.hadoop.hbase.io.TestHbaseObjectWritable,org.apache.hadoop.hbas
    e

    .io.TestHeapSize,org.apache.hadoop.hbase.io.TestImmutableBytesWritable

    ,org.apache.hadoop.hbase.mapreduce.TestSimpleTotalOrderPartitioner,org
    .apache.hadoop.hbase.mapreduce.TestTableSplit,org.apache.hadoop.hbase
    .
    master.TestClockSkewDetection,org.apache.hadoop.hbase.master.TestDea
    dS

    erver,org.apache.hadoop.hbase.master.TestDefaultLoadBalancer,org.apach

    e.hadoop.hbase.master.TestMasterStatusServlet,org.apache.hadoop.hbase.
    metrics.TestMetricsMBeanBase,org.apache.hadoop.hbase.monitoring.Test
    Me
    moryBoundedLogMessageBuffer,org.apache.hadoop.hbase.monitoring.Test
    Tas

    kMonitor,org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionH
    andler,org.apache.hadoop.hbase.regionserver.TestCompactSelection,org.a
    pache.hadoop.hbase.regionserver.TestExplicitColumnTracker,org.apache.h
    adoop.hbase.regionserver.TestHRegionInfo,org.apache.hadoop.hbase.regio
    nserver.TestKeyValueHeap,org.apache.hadoop.hbase.regionserver.TestKey
    V

    alueScanFixture,org.apache.hadoop.hbase.regionserver.TestKeyValueSkipL
    istSet,org.apache.hadoop.hbase.regionserver.TestMemStoreLAB,org.apach
    e

    .hadoop.hbase.regionserver.TestMinVersions,org.apache.hadoop.hbase.reg
    ionserver.TestQueryMatcher,org.apache.hadoop.hbase.regionserver.TestRe
    gionSplitPolicy,org.apache.hadoop.hbase.regionserver.TestResettingCoun
    ters,org.apache.hadoop.hbase.regionserver.TestRSStatusServlet,org.apac
    he.hadoop.hbase.regionserver.TestScanDeleteTracker,org.apache.hadoop.h

    base.regionserver.TestScanWildcardColumnTracker,org.apache.hadoop.hbas

    e.regionserver.TestScanWithBloomError,org.apache.hadoop.hbase.regionse

    rver.TestStoreScanner,org.apache.hadoop.hbase.regionserver.wal.TestHLo
    gBench,org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods,org.a
    p

    ache.hadoop.hbase.regionserver.wal.TestWALActionsListener,org.apache.h
    adoop.hbase.rest.model.TestCellModel,org.apache.hadoop.hbase.rest.mod
    e
    l.TestCellSetModel,org.apache.hadoop.hbase.rest.model.TestColumnSchem
    a
    Model,org.apache.hadoop.hbase.rest.model.TestRowModelo,rg.apache.ha
    doo
    p.hbase.rest.model.TestScannerModel,org.apache.hadoop.hbase.rest.mod
    el

    .TestStorageClusterStatusModel,org.apache.hadoop.hbase.rest.model.Test

    StorageClusterVersionModel,org.apache.hadoop.hbase.rest.model.TestTabl

    eInfoModel,org.apache.hadoop.hbase.rest.model.TestTableListModel,org.a
    pache.hadoop.hbase.rest.model.TestTableRegionModel,org.apache.hadoop
    .h
    base.rest.model.TestTableSchemaModel,org.apache.hadoop.hbase.rest.mo
    de
    l.TestVersionModel,org.apache.hadoop.hbase.security.TestUser,org.apach
    e.hadoop.hbase.TestAcidGuarantees,org.apache.hadoop.hbase.TestCompar
    e,
    org.apache.hadoop.hbase.TestHRegionLocation,org.apache.hadoop.hbase.T
    e
    stHServerAddress,org.apache.hadoop.hbase.TestHServerInfo,org.apache.ha
    doop.hbase.TestKeyValue,org.apache.hadoop.hbase.TestSerialization,org.
    apache.hadoop.hbase.TestServerName,org.apache.hadoop.hbase.util.TestB
    a
    se64,org.apache.hadoop.hbase.util.TestByteBloomFilter,org.apache.hadoo
    p.hbase.util.TestBytes,org.apache.hadoop.hbase.util.TestCompressionTes
    t,org.apache.hadoop.hbase.util.TestFSTableDescriptors,org.apache.hadoo
    p.hbase.util.TestHBaseFsckComparator,org.apache.hadoop.hbase.util.Test
    IncrementingEnvironmentEdge,org.apache.hadoop.hbase.util.TestKeying,or

    g.apache.hadoop.hbase.util.TestPoolMap,org.apache.hadoop.hbase.util.Te
    stRegionSplitCalculator,org.apache.hadoop.hbase.zookeeper.TestHQuorum
    P
    eer,org.apache.hadoop.hbase.zookeeper.TestZooKeeperMainServerArg,org
    .a
    pache.hadoop.hbase.coprocessor.TestCoprocessorInterface,org.apache.had
    oop.hbase.executor.TestExecutorService,org.apache.hadoop.hbase.filter.
    TestColumnPrefixFilter,org.apache.hadoop.hbase.filter.TestFilter,org.a
    pache.hadoop.hbase.filter.TestMultipleColumnPrefixFilter,org.apache.ha
    doop.hbase.io.hfile.slab.TestSingleSizeCache,org.apache.hadoop.hbase.i
    o.hfile.TestCacheOnWrite,org.apache.hadoop.hbase.io.hfile.TestHFileBlo
    ckIndex,org.apache.hadoop.hbase.io.hfile.TestHFileBlock,org.apache.had
    oop.hbase.io.hfile.TestHFilePerformance,org.apache.hadoop.hbase.io.hfi
    le.TestHFileSeek,org.apache.hadoop.hbase.ipc.TestDelayedRpc,org.apache

    .hadoop.hbase.master.TestCatalogJanitor,org.apache.hadoop.hbase.region
    server.TestAtomicOperation,org.apache.hadoop.hbase.regionserver.TestM
    e
    mStore,org.apache.hadoop.hbase.regionserver.TestReadWriteConsistencyC
    o
    ntrol,org.apache.hadoop.hbase.regionserver.TestSeekOptimizations,org.a
    pache.hadoop.hbase.regionserver.TestSplitTransaction,org.apache.hadoop
    .hbase.regionserver.TestStore,org.apache.hadoop.hbase.replication.Test
    ReplicationSource


    --
    Todd Lipcon
    Software Engineer, Cloudera
  • Stack at Oct 20, 2011 at 8:56 pm
    Sounds good Nicolas.
    St.Ack
    On Thu, Oct 20, 2011 at 1:46 PM, N Keywal wrote:
    Hi,

    I've done a stat on the tests we have. Many of them are actually quite
    simple: There are 226 test classes in HBase.  100  of them does not use a
    cluster, and takes less than 2 seconds to execute. 24 do not use a cluster
    but need between 2 and 40 seconds.

    I tried to execute these with different maven settings.
    fork: once or always, i.e. all the tests are executed in the same JVM or
    there is a new JVM created for every test class
    parallel; deactivated or activated, with 3 threads and a per class
    parallelisation.

    The 100 tests executes for the whole maven stuff, including the initial
    dependency & task analysis
    fork always: 1m50s
    fork once: 30 seconds
    parallel: 30s (=> no difference).

    It works quite well, except with TestLruBlockCache that I had to remove.

    I tried as well with the 24 'bigger' ones but no cluster, for the following
    results:
    fork always: 7m36s
    parallel: 2m32s: 3 times faster.

    However, I had to remove many of them, there are some conflicts that I have
    not solved (nor tried to).

    The fact that nearly 50% of the tests can be executed in 1 minutes is
    interesting imho.

    That makes me wonder if we should try to get a separation of the tests in 3
    parts, small/medium/large as of
    http://googletesting.blogspot.com/2009/07/software-testing-categorization.html

    small would be:
    - less than two seconds, can be executed in the same jvm
    - all of them: less than 2 minute
    - no cluster, nothing that can break the environment
    - run by the developers as a part of the build

    medium:
    - less than 40 seconds
    - all of them: would last less than 30 minutes, ideally 15 minutes
    - run in an isolated jvm
    - run explicitly by the developers before submitting a fix

    large:
    - run in an isolated jvm
    - Should last less than 45minutes
    - run on the central built or explicitly by the developers
    - break the built when it fails.

    Only one issue: it's not directly possible with maven, but there are known
    workarounds :-)


    What do you think?


    Below the tests set for the 120 tests that seems to work altogether.


    mvn test
    -Dtest=org.apache.hadoop.hbase.avro.TestAvroUtil,org.apache.hadoop.hbase.client.TestAttributes,org.apache.hadoop.hbase.client.TestGet,org.apache.hadoop.hbase.client.TestOperation,org.apache.hadoop.hbase.client.TestResult,org.apache.hadoop.hbase.client.TestScan,org.apache.hadoop.hbase.coprocessor.TestRegionObserverStacking,org.apache.hadoop.hbase.filter.TestBitComparator,org.apache.hadoop.hbase.filter.TestColumnPaginationFilter,org.apache.hadoop.hbase.filter.TestDependentColumnFilter,org.apache.hadoop.hbase.filter.TestFilterList,org.apache.hadoop.hbase.filter.TestInclusiveStopFilter,org.apache.hadoop.hbase.filter.TestPageFilter,org.apache.hadoop.hbase.filter.TestParseFilter,org.apache.hadoop.hbase.filter.TestPrefixFilter,org.apache.hadoop.hbase.filter.TestRandomRowFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueExcludeFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueFilter,org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache,org.apache.hadoop.hbase.io.hfile.slab.TestSlab,org.apache.hadoop.hbase.io.hfile.TestBlockCacheColumnFamilySummary,org.apache.hadoop.hbase.io.hfile.TestCachedBlockQueue,org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer,org.apache.hadoop.hbase.io.hfile.TestHFileReaderV1,org.apache.hadoop.hbase.io.hfile.TestHFile,org.apache.hadoop.hbase.io.hfile.TestHFileWriterV2,org.apache.hadoop.hbase.io.hfile.TestReseekTo,org.apache.hadoop.hbase.io.hfile.TestSeekTo,org.apache.hadoop.hbase.io.TestHalfStoreFileReader,org.apache.hadoop.hbase.io.TestHbaseObjectWritable,org.apache.hadoop.hbase.io.TestHeapSize,org.apache.hadoop.hbase.io.TestImmutableBytesWritable,org.apache.hadoop.hbase.mapreduce.TestSimpleTotalOrderPartitioner,org.apache.hadoop.hbase.mapreduce.TestTableSplit,org.apache.hadoop.hbase.master.TestClockSkewDetection,org.apache.hadoop.hbase.master.TestDeadServer,org.apache.hadoop.hbase.master.TestDefaultLoadBalancer,org.apache.hadoop.hbase.master.TestMasterStatusServlet,org.apache.hadoop.hbase.metrics.TestMetricsMBeanBase,org.apache.hadoop.hbase.monitoring.TestMemoryBoundedLogMessageBuffer,org.apache.hadoop.hbase.monitoring.TestTaskMonitor,org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler,org.apache.hadoop.hbase.regionserver.TestCompactSelection,org.apache.hadoop.hbase.regionserver.TestExplicitColumnTracker,org.apache.hadoop.hbase.regionserver.TestHRegionInfo,org.apache.hadoop.hbase.regionserver.TestKeyValueHeap,org.apache.hadoop.hbase.regionserver.TestKeyValueScanFixture,org.apache.hadoop.hbase.regionserver.TestKeyValueSkipListSet,org.apache.hadoop.hbase.regionserver.TestMemStoreLAB,org.apache.hadoop.hbase.regionserver.TestMinVersions,org.apache.hadoop.hbase.regionserver.TestQueryMatcher,org.apache.hadoop.hbase.regionserver.TestRegionSplitPolicy,org.apache.hadoop.hbase.regionserver.TestResettingCounters,org.apache.hadoop.hbase.regionserver.TestRSStatusServlet,org.apache.hadoop.hbase.regionserver.TestScanDeleteTracker,org.apache.hadoop.hbase.regionserver.TestScanWildcardColumnTracker,org.apache.hadoop.hbase.regionserver.TestScanWithBloomError,org.apache.hadoop.hbase.regionserver.TestStoreScanner,org.apache.hadoop.hbase.regionserver.wal.TestHLogBench,org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods,org.apache.hadoop.hbase.regionserver.wal.TestWALActionsListener,org.apache.hadoop.hbase.rest.model.TestCellModel,org.apache.hadoop.hbase.rest.model.TestCellSetModel,org.apache.hadoop.hbase.rest.model.TestColumnSchemaModel,org.apache.hadoop.hbase.rest.model.TestRowModelo,rg.apache.hadoop.hbase.rest.model.TestScannerModel,org.apache.hadoop.hbase.rest.model.TestStorageClusterStatusModel,org.apache.hadoop.hbase.rest.model.TestStorageClusterVersionModel,org.apache.hadoop.hbase.rest.model.TestTableInfoModel,org.apache.hadoop.hbase.rest.model.TestTableListModel,org.apache.hadoop.hbase.rest.model.TestTableRegionModel,org.apache.hadoop.hbase.rest.model.TestTableSchemaModel,org.apache.hadoop.hbase.rest.model.TestVersionModel,org.apache.hadoop.hbase.security.TestUser,org.apache.hadoop.hbase.TestAcidGuarantees,org.apache.hadoop.hbase.TestCompare,org.apache.hadoop.hbase.TestHRegionLocation,org.apache.hadoop.hbase.TestHServerAddress,org.apache.hadoop.hbase.TestHServerInfo,org.apache.hadoop.hbase.TestKeyValue,org.apache.hadoop.hbase.TestSerialization,org.apache.hadoop.hbase.TestServerName,org.apache.hadoop.hbase.util.TestBase64,org.apache.hadoop.hbase.util.TestByteBloomFilter,org.apache.hadoop.hbase.util.TestBytes,org.apache.hadoop.hbase.util.TestCompressionTest,org.apache.hadoop.hbase.util.TestFSTableDescriptors,org.apache.hadoop.hbase.util.TestHBaseFsckComparator,org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge,org.apache.hadoop.hbase.util.TestKeying,org.apache.hadoop.hbase.util.TestPoolMap,org.apache.hadoop.hbase.util.TestRegionSplitCalculator,org.apache.hadoop.hbase.zookeeper.TestHQuorumPeer,org.apache.hadoop.hbase.zookeeper.TestZooKeeperMainServerArg,org.apache.hadoop.hbase.coprocessor.TestCoprocessorInterface,org.apache.hadoop.hbase.executor.TestExecutorService,org.apache.hadoop.hbase.filter.TestColumnPrefixFilter,org.apache.hadoop.hbase.filter.TestFilter,org.apache.hadoop.hbase.filter.TestMultipleColumnPrefixFilter,org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache,org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite,org.apache.hadoop.hbase.io.hfile.TestHFileBlockIndex,org.apache.hadoop.hbase.io.hfile.TestHFileBlock,org.apache.hadoop.hbase.io.hfile.TestHFilePerformance,org.apache.hadoop.hbase.io.hfile.TestHFileSeek,org.apache.hadoop.hbase.ipc.TestDelayedRpc,org.apache.hadoop.hbase.master.TestCatalogJanitor,org.apache.hadoop.hbase.regionserver.TestAtomicOperation,org.apache.hadoop.hbase.regionserver.TestMemStore,org.apache.hadoop.hbase.regionserver.TestReadWriteConsistencyControl,org.apache.hadoop.hbase.regionserver.TestSeekOptimizations,org.apache.hadoop.hbase.regionserver.TestSplitTransaction,org.apache.hadoop.hbase.regionserver.TestStore,org.apache.hadoop.hbase.replication.TestReplicationSource
  • Andrew Purtell at Oct 20, 2011 at 10:05 pm
    +1, this sounds great.

    In 0.20 I had the REST (aka Stargate at that time) tests running all in single JVM but in the port to 0.90 had to be updated for the current surefire config. If there were a small/medium convention at that time the relative speed could have been preserved.

        - Andy




    ________________________________
    From: N Keywal <nkeywal@gmail.com>
    To: dev@hbase.apache.org
    Sent: Thursday, October 20, 2011 1:46 PM
    Subject: [unit tests] small tests vs. others

    Hi,

    I've done a stat on the tests we have. Many of them are actually quite
    simple: There are 226 test classes in HBase.  100  of them does not use a
    cluster, and takes less than 2 seconds to execute. 24 do not use a cluster
    but need between 2 and 40 seconds.

    I tried to execute these with different maven settings.
    fork: once or always, i.e. all the tests are executed in the same JVM or
    there is a new JVM created for every test class
    parallel; deactivated or activated, with 3 threads and a per class
    parallelisation.

    The 100 tests executes for the whole maven stuff, including the initial
    dependency & task analysis
    fork always: 1m50s
    fork once: 30 seconds
    parallel: 30s (=> no difference).

    It works quite well, except with TestLruBlockCache that I had to remove.

    I tried as well with the 24 'bigger' ones but no cluster, for the following
    results:
    fork always: 7m36s
    parallel: 2m32s: 3 times faster.

    However, I had to remove many of them, there are some conflicts that I have
    not solved (nor tried to).

    The fact that nearly 50% of the tests can be executed in 1 minutes is
    interesting imho.

    That makes me wonder if we should try to get a separation of the tests in 3
    parts, small/medium/large as of
    http://googletesting.blogspot.com/2009/07/software-testing-categorization.html

    small would be:
    - less than two seconds, can be executed in the same jvm
    - all of them: less than 2 minute
    - no cluster, nothing that can break the environment
    - run by the developers as a part of the build

    medium:
    - less than 40 seconds
    - all of them: would last less than 30 minutes, ideally 15 minutes
    - run in an isolated jvm
    - run explicitly by the developers before submitting a fix

    large:
    - run in an isolated jvm
    - Should last less than 45minutes
    - run on the central built or explicitly by the developers
    - break the built when it fails.

    Only one issue: it's not directly possible with maven, but there are known
    workarounds :-)


    What do you think?


    Below the tests set for the 120 tests that seems to work altogether.


    mvn test
    -Dtest=org.apache.hadoop.hbase.avro.TestAvroUtil,org.apache.hadoop.hbase.client.TestAttributes,org.apache.hadoop.hbase.client.TestGet,org.apache.hadoop.hbase.client.TestOperation,org.apache.hadoop.hbase.client.TestResult,org.apache.hadoop.hbase.client.TestScan,org.apache.hadoop.hbase.coprocessor.TestRegionObserverStacking,org.apache.hadoop.hbase.filter.TestBitComparator,org.apache.hadoop.hbase.filter.TestColumnPaginationFilter,org.apache.hadoop.hbase.filter.TestDependentColumnFilter,org.apache.hadoop.hbase.filter.TestFilterList,org.apache.hadoop.hbase.filter.TestInclusiveStopFilter,org.apache.hadoop.hbase.filter.TestPageFilter,org.apache.hadoop.hbase.filter.TestParseFilter,org.apache.hadoop.hbase.filter.TestPrefixFilter,org.apache.hadoop.hbase.filter.TestRandomRowFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueExcludeFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueFilter,org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache,org.apac
    he.hadoop.hbase.io.hfile.slab.TestSlab,org.apache.hadoop.hbase.io.hfile.TestBlockCacheColumnFamilySummary,org.apache.hadoop.hbase.io.hfile.TestCachedBlockQueue,org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer,org.apache.hadoop.hbase.io.hfile.TestHFileReaderV1,org.apache.hadoop.hbase.io.hfile.TestHFile,org.apache.hadoop.hbase.io.hfile.TestHFileWriterV2,org.apache.hadoop.hbase.io.hfile.TestReseekTo,org.apache.hadoop.hbase.io.hfile.TestSeekTo,org.apache.hadoop.hbase.io.TestHalfStoreFileReader,org.apache.hadoop.hbase.io.TestHbaseObjectWritable,org.apache.hadoop.hbase.io.TestHeapSize,org.apache.hadoop.hbase.io.TestImmutableBytesWritable,org.apache.hadoop.hbase.mapreduce.TestSimpleTotalOrderPartitioner,org.apache.hadoop.hbase.mapreduce.TestTableSplit,org.apache.hadoop.hbase.master.TestClockSkewDetection,org.apache.hadoop.hbase.master.TestDeadServer,org.apache.hadoop.hbase.master.TestDefaultLoadBalancer,org.apache.hadoop.hbase.master.TestMasterStatusServl
    et,org.apache.hadoop.hbase.metrics.TestMetricsMBeanBase,org.apache.hadoop.hbase.monitoring.TestMemoryBoundedLogMessageBuffer,org.apache.hadoop.hbase.monitoring.TestTaskMonitor,org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler,org.apache.hadoop.hbase.regionserver.TestCompactSelection,org.apache.hadoop.hbase.regionserver.TestExplicitColumnTracker,org.apache.hadoop.hbase.regionserver.TestHRegionInfo,org.apache.hadoop.hbase.regionserver.TestKeyValueHeap,org.apache.hadoop.hbase.regionserver.TestKeyValueScanFixture,org.apache.hadoop.hbase.regionserver.TestKeyValueSkipListSet,org.apache.hadoop.hbase.regionserver.TestMemStoreLAB,org.apache.hadoop.hbase.regionserver.TestMinVersions,org.apache.hadoop.hbase.regionserver.TestQueryMatcher,org.apache.hadoop.hbase.regionserver.TestRegionSplitPolicy,org.apache.hadoop.hbase.regionserver.TestResettingCounters,org.apache.hadoop.hbase.regionserver.TestRSStatusServlet,org.apache.hadoop.hbase.regionserver.T
    estScanDeleteTracker,org.apache.hadoop.hbase.regionserver.TestScanWildcardColumnTracker,org.apache.hadoop.hbase.regionserver.TestScanWithBloomError,org.apache.hadoop.hbase.regionserver.TestStoreScanner,org.apache.hadoop.hbase.regionserver.wal.TestHLogBench,org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods,org.apache.hadoop.hbase.regionserver.wal.TestWALActionsListener,org.apache.hadoop.hbase.rest.model.TestCellModel,org.apache.hadoop.hbase.rest.model.TestCellSetModel,org.apache.hadoop.hbase.rest.model.TestColumnSchemaModel,org.apache.hadoop.hbase.rest.model.TestRowModelo,rg.apache.hadoop.hbase.rest.model.TestScannerModel,org.apache.hadoop.hbase.rest.model.TestStorageClusterStatusModel,org.apache.hadoop.hbase.rest.model.TestStorageClusterVersionModel,org.apache.hadoop.hbase.rest.model.TestTableInfoModel,org.apache.hadoop.hbase.rest.model.TestTableListModel,org.apache.hadoop.hbase.rest.model.TestTableRegionModel,org.apache.hadoop.hbase.rest.model.T
    estTableSchemaModel,org.apache.hadoop.hbase.rest.model.TestVersionModel,org.apache.hadoop.hbase.security.TestUser,org.apache.hadoop.hbase.TestAcidGuarantees,org.apache.hadoop.hbase.TestCompare,org.apache.hadoop.hbase.TestHRegionLocation,org.apache.hadoop.hbase.TestHServerAddress,org.apache.hadoop.hbase.TestHServerInfo,org.apache.hadoop.hbase.TestKeyValue,org.apache.hadoop.hbase.TestSerialization,org.apache.hadoop.hbase.TestServerName,org.apache.hadoop.hbase.util.TestBase64,org.apache.hadoop.hbase.util.TestByteBloomFilter,org.apache.hadoop.hbase.util.TestBytes,org.apache.hadoop.hbase.util.TestCompressionTest,org.apache.hadoop.hbase.util.TestFSTableDescriptors,org.apache.hadoop.hbase.util.TestHBaseFsckComparator,org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge,org.apache.hadoop.hbase.util.TestKeying,org.apache.hadoop.hbase.util.TestPoolMap,org.apache.hadoop.hbase.util.TestRegionSplitCalculator,org.apache.hadoop.hbase.zookeeper.TestHQuorumPeer,
    org.apache.hadoop.hbase.zookeeper.TestZooKeeperMainServerArg,org.apache.hadoop.hbase.coprocessor.TestCoprocessorInterface,org.apache.hadoop.hbase.executor.TestExecutorService,org.apache.hadoop.hbase.filter.TestColumnPrefixFilter,org.apache.hadoop.hbase.filter.TestFilter,org.apache.hadoop.hbase.filter.TestMultipleColumnPrefixFilter,org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache,org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite,org.apache.hadoop.hbase.io.hfile.TestHFileBlockIndex,org.apache.hadoop.hbase.io.hfile.TestHFileBlock,org.apache.hadoop.hbase.io.hfile.TestHFilePerformance,org.apache.hadoop.hbase.io.hfile.TestHFileSeek,org.apache.hadoop.hbase.ipc.TestDelayedRpc,org.apache.hadoop.hbase.master.TestCatalogJanitor,org.apache.hadoop.hbase.regionserver.TestAtomicOperation,org.apache.hadoop.hbase.regionserver.TestMemStore,org.apache.hadoop.hbase.regionserver.TestReadWriteConsistencyControl,org.apache.hadoop.hbase.regionserver.TestSeekOptimizati
    ons,org.apache.hadoop.hbase.regionserver.TestSplitTransaction,org.apache.hadoop.hbase.regionserver.TestStore,org.apache.hadoop.hbase.replication.TestReplicationSource
  • Akash Ashok at Oct 21, 2011 at 1:17 am
    Ooo +1 on this. This sounds Just wonderful. :)

    Cheers,
    Akash A
    On Fri, Oct 21, 2011 at 3:34 AM, Andrew Purtell wrote:

    +1, this sounds great.

    In 0.20 I had the REST (aka Stargate at that time) tests running all in
    single JVM but in the port to 0.90 had to be updated for the current
    surefire config. If there were a small/medium convention at that time the
    relative speed could have been preserved.

    - Andy




    ________________________________
    From: N Keywal <nkeywal@gmail.com>
    To: dev@hbase.apache.org
    Sent: Thursday, October 20, 2011 1:46 PM
    Subject: [unit tests] small tests vs. others

    Hi,

    I've done a stat on the tests we have. Many of them are actually quite
    simple: There are 226 test classes in HBase. 100 of them does not use a
    cluster, and takes less than 2 seconds to execute. 24 do not use a cluster
    but need between 2 and 40 seconds.

    I tried to execute these with different maven settings.
    fork: once or always, i.e. all the tests are executed in the same JVM or
    there is a new JVM created for every test class
    parallel; deactivated or activated, with 3 threads and a per class
    parallelisation.

    The 100 tests executes for the whole maven stuff, including the initial
    dependency & task analysis
    fork always: 1m50s
    fork once: 30 seconds
    parallel: 30s (=> no difference).

    It works quite well, except with TestLruBlockCache that I had to remove.

    I tried as well with the 24 'bigger' ones but no cluster, for the following
    results:
    fork always: 7m36s
    parallel: 2m32s: 3 times faster.

    However, I had to remove many of them, there are some conflicts that I have
    not solved (nor tried to).

    The fact that nearly 50% of the tests can be executed in 1 minutes is
    interesting imho.

    That makes me wonder if we should try to get a separation of the tests in 3
    parts, small/medium/large as of
    http://googletesting.blogspot.com/2009/07/software-testing-categorization.html
    small would be:
    - less than two seconds, can be executed in the same jvm
    - all of them: less than 2 minute
    - no cluster, nothing that can break the environment
    - run by the developers as a part of the build

    medium:
    - less than 40 seconds
    - all of them: would last less than 30 minutes, ideally 15 minutes
    - run in an isolated jvm
    - run explicitly by the developers before submitting a fix

    large:
    - run in an isolated jvm
    - Should last less than 45minutes
    - run on the central built or explicitly by the developers
    - break the built when it fails.

    Only one issue: it's not directly possible with maven, but there are known
    workarounds :-)


    What do you think?


    Below the tests set for the 120 tests that seems to work altogether.


    mvn test
    -Dtest=org.apache.hadoop.hbase.avro.TestAvroUtil,org.apache.hadoop.hbase.client.TestAttributes,org.apache.hadoop.hbase.client.TestGet,org.apache.hadoop.hbase.client.TestOperation,org.apache.hadoop.hbase.client.TestResult,org.apache.hadoop.hbase.client.TestScan,org.apache.hadoop.hbase.coprocessor.TestRegionObserverStacking,org.apache.hadoop.hbase.filter.TestBitComparator,org.apache.hadoop.hbase.filter.TestColumnPaginationFilter,org.apache.hadoop.hbase.filter.TestDependentColumnFilter,org.apache.hadoop.hbase.filter.TestFilterList,org.apache.hadoop.hbase.filter.TestInclusiveStopFilter,org.apache.hadoop.hbase.filter.TestPageFilter,org.apache.hadoop.hbase.filter.TestParseFilter,org.apache.hadoop.hbase.filter.TestPrefixFilter,org.apache.hadoop.hbase.filter.TestRandomRowFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueExcludeFilter,org.apache.hadoop.hbase.filter.TestSingleColumnValueFilter,org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache,org.apac
    he.hadoop.hbase.io.hfile.slab.TestSlab,org.apache.hadoop.hbase.io.hfile.TestBlockCacheColumnFamilySummary,org.apache.hadoop.hbase.io.hfile.TestCachedBlockQueue,org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer,org.apache.hadoop.hbase.io.hfile.TestHFileReaderV1,org.apache.hadoop.hbase.io.hfile.TestHFile,org.apache.hadoop.hbase.io.hfile.TestHFileWriterV2,org.apache.hadoop.hbase.io.hfile.TestReseekTo,org.apache.hadoop.hbase.io.hfile.TestSeekTo,org.apache.hadoop.hbase.io.TestHalfStoreFileReader,org.apache.hadoop.hbase.io.TestHbaseObjectWritable,org.apache.hadoop.hbase.io.TestHeapSize,org.apache.hadoop.hbase.io.TestImmutableBytesWritable,org.apache.hadoop.hbase.mapreduce.TestSimpleTotalOrderPartitioner,org.apache.hadoop.hbase.mapreduce.TestTableSplit,org.apache.hadoop.hbase.master.TestClockSkewDetection,org.apache.hadoop.hbase.master.TestDeadServer,org.apache.hadoop.hbase.master.TestDefaultLoadBalancer,org.apache.hadoop.hbase.master.TestMasterStatusServl

    et,org.apache.hadoop.hbase.metrics.TestMetricsMBeanBase,org.apache.hadoop.hbase.monitoring.TestMemoryBoundedLogMessageBuffer,org.apache.hadoop.hbase.monitoring.TestTaskMonitor,org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler,org.apache.hadoop.hbase.regionserver.TestCompactSelection,org.apache.hadoop.hbase.regionserver.TestExplicitColumnTracker,org.apache.hadoop.hbase.regionserver.TestHRegionInfo,org.apache.hadoop.hbase.regionserver.TestKeyValueHeap,org.apache.hadoop.hbase.regionserver.TestKeyValueScanFixture,org.apache.hadoop.hbase.regionserver.TestKeyValueSkipListSet,org.apache.hadoop.hbase.regionserver.TestMemStoreLAB,org.apache.hadoop.hbase.regionserver.TestMinVersions,org.apache.hadoop.hbase.regionserver.TestQueryMatcher,org.apache.hadoop.hbase.regionserver.TestRegionSplitPolicy,org.apache.hadoop.hbase.regionserver.TestResettingCounters,org.apache.hadoop.hbase.regionserver.TestRSStatusServlet,org.apache.hadoop.hbase.regionserver.T

    estScanDeleteTracker,org.apache.hadoop.hbase.regionserver.TestScanWildcardColumnTracker,org.apache.hadoop.hbase.regionserver.TestScanWithBloomError,org.apache.hadoop.hbase.regionserver.TestStoreScanner,org.apache.hadoop.hbase.regionserver.wal.TestHLogBench,org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods,org.apache.hadoop.hbase.regionserver.wal.TestWALActionsListener,org.apache.hadoop.hbase.rest.model.TestCellModel,org.apache.hadoop.hbase.rest.model.TestCellSetModel,org.apache.hadoop.hbase.rest.model.TestColumnSchemaModel,org.apache.hadoop.hbase.rest.model.TestRowModelo,rg.apache.hadoop.hbase.rest.model.TestScannerModel,org.apache.hadoop.hbase.rest.model.TestStorageClusterStatusModel,org.apache.hadoop.hbase.rest.model.TestStorageClusterVersionModel,org.apache.hadoop.hbase.rest.model.TestTableInfoModel,org.apache.hadoop.hbase.rest.model.TestTableListModel,org.apache.hadoop.hbase.rest.model.TestTableRegionModel,org.apache.hadoop.hbase.rest.model.T

    estTableSchemaModel,org.apache.hadoop.hbase.rest.model.TestVersionModel,org.apache.hadoop.hbase.security.TestUser,org.apache.hadoop.hbase.TestAcidGuarantees,org.apache.hadoop.hbase.TestCompare,org.apache.hadoop.hbase.TestHRegionLocation,org.apache.hadoop.hbase.TestHServerAddress,org.apache.hadoop.hbase.TestHServerInfo,org.apache.hadoop.hbase.TestKeyValue,org.apache.hadoop.hbase.TestSerialization,org.apache.hadoop.hbase.TestServerName,org.apache.hadoop.hbase.util.TestBase64,org.apache.hadoop.hbase.util.TestByteBloomFilter,org.apache.hadoop.hbase.util.TestBytes,org.apache.hadoop.hbase.util.TestCompressionTest,org.apache.hadoop.hbase.util.TestFSTableDescriptors,org.apache.hadoop.hbase.util.TestHBaseFsckComparator,org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge,org.apache.hadoop.hbase.util.TestKeying,org.apache.hadoop.hbase.util.TestPoolMap,org.apache.hadoop.hbase.util.TestRegionSplitCalculator,org.apache.hadoop.hbase.zookeeper.TestHQuorumPeer,

    org.apache.hadoop.hbase.zookeeper.TestZooKeeperMainServerArg,org.apache.hadoop.hbase.coprocessor.TestCoprocessorInterface,org.apache.hadoop.hbase.executor.TestExecutorService,org.apache.hadoop.hbase.filter.TestColumnPrefixFilter,org.apache.hadoop.hbase.filter.TestFilter,org.apache.hadoop.hbase.filter.TestMultipleColumnPrefixFilter,org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache,org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite,org.apache.hadoop.hbase.io.hfile.TestHFileBlockIndex,org.apache.hadoop.hbase.io.hfile.TestHFileBlock,org.apache.hadoop.hbase.io.hfile.TestHFilePerformance,org.apache.hadoop.hbase.io.hfile.TestHFileSeek,org.apache.hadoop.hbase.ipc.TestDelayedRpc,org.apache.hadoop.hbase.master.TestCatalogJanitor,org.apache.hadoop.hbase.regionserver.TestAtomicOperation,org.apache.hadoop.hbase.regionserver.TestMemStore,org.apache.hadoop.hbase.regionserver.TestReadWriteConsistencyControl,org.apache.hadoop.hbase.regionserver.TestSeekOptimizati

    ons,org.apache.hadoop.hbase.regionserver.TestSplitTransaction,org.apache.hadoop.hbase.regionserver.TestStore,org.apache.hadoop.hbase.replication.TestReplicationSource

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categorieshbase, hadoop
postedOct 20, '11 at 8:46p
activeNov 14, '11 at 8:26p
posts10
users7
websitehbase.apache.org

People

Translate

site design / logo © 2022 Grokbase