FAQ
Some history:
The unit testing inside hive is good at doing what it does. Essentially it
runs a hive .q file and diffs the file against previous known runs.

This does some nice things for hive:

1) We are sure the query planner/parsers evaluate the query the same way.
2) We are sure the query returns the same physical results

The not so nice side:
1) Pretty intensive to run tests, creates tables before hand
2) Not really ideal for developing something not a part of hive, inhouse
UDF)
3) No way to integrate with standard unit tests (asserts etc)

Alternative solution:

hive_test
Unit test framework for hive and hive-service
https://github.com/edwardcapriolo/hive_test

Now that all the components of hive live in maven, we can bring the
dependencies of hive into other projects. We can extend the HadoopTestCase
into several other test cases. Currently we have abstract Test Cases for
embedded hive and a hive-service.

The end result is the user can now write end to end unit tests like this:


public class ServiceExampleTest extends HiveTestService {

public ServiceExampleTest() throws IOException {
super();
}

public void testExecute() throws Exception {
Path p = new Path(this.ROOT_DIR,"afile");
FSDataOutputStream o= this.getFileSystem().create(p);
BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(o));
bw.write("1\n");
bw.write("2\n");
bw.close();

client.execute("create table atest (num int)");
client.execute("load data local inpath '"+p.toString() +"' into table
atest" );
client.execute("select count(1) as cnt from atest");
String row = client.fetchOne();
assertEquals(row, "2");
client.execute("drop table atest");
}

}

You can also use the embedded capability to test workflows, no more
untestable integrations like this.

doIt.sh
in=$1
out=$2
`hadoop jar wordcount /$in /$out`
`hive -e "create table wordcount ..."
`hive -e "load data inpath $out into wordcount"`

Now these can be done in single java process!

Search Discussions

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshive, hadoop
postedSep 16, '11 at 4:06p
activeSep 16, '11 at 4:06p
posts1
users1
websitehive.apache.org

1 user in discussion

Edward Capriolo: 1 post

People

Translate

site design / logo © 2022 Grokbase