[Hadoop-common-dev] [VOTE] Release Hadoop 0.21.0 (candidate 0)

A problem was found with Job.getCounters() returning null (see
https://issues.apache.org/jira/browse/MAPREDUCE-1920). I plan to roll
a new release candidate when this is fixed.

Cheers,
Tom

Hi Tom,

I'm trying to build Hadoop 0.21.0 locally, but its failing because the
hadoop root dir is missing the build.xml in the tar.gz. Is there a new
build process ? I'm currently using the ant clean tar test-core
directives.

Regards
Steve Watt



From:
Tom White <tom@cloudera.com>
To:
common-dev <common-dev@hadoop.apache.org>
Date:
07/08/2010 07:16 AM
Subject:
Re: [VOTE] Release Hadoop 0.21.0 (candidate 0)



A problem was found with Job.getCounters() returning null (see
https://issues.apache.org/jira/browse/MAPREDUCE-1920). I plan to roll
a new release candidate when this is fixed.

Cheers,
Tom

Hi Tom,

Just want to let you know that back then when I tried to circumvent
this problem, I used job.submit() and retrieve the counter
asynchronously. I found out that the counter values always zero during
the execution (so the job is still running halfway). So, in this case,
the job is not retired yet, right? so, it might suggest another
problem?

Felix Halim

Steve, Yes, this is a consequence of the project split. It's still
possible to build Common, HDFS, MapReduce individually (from their
respective distributions), but I agree it would be nice to be able to
build from the combined distribution. When I create the next RC, I'll
see if I can provide instructions on how to do this.

Felix, I don't think this is a bug since counters are not stable until
the job is complete, so the values you see may be zero if you retrieve
them asynchronously.

Cheers,
Tom

PS. I'm travelling/on vacation for the next couple of weeks, so I
won't be working on the next release candidate until August. If anyone
wants to work on the (blocker) JIRAs I listed in my previous email in
this thread then be my guest!

Just to clarify.. these counters that are retrieved asynchronously
will be updated as soon as one mapper or reducer completes, right?
Or is it going to be delayed arbitrarily until the job is finished?

BTW, I have a warning when running a job:

10/07/20 17:44:19 WARN conf.Configuration: mapred.task.id is
deprecated. Instead, use mapreduce.task.attempt.id

What causes this warning? I don't use mapred.task.id anywhere in the
configurations...

Felix Halim

To be clear, the counters objects will not be updated, you have to
retrieve them again, and you can only be sure they are correct after
the job completes.

I downloaded the "combined" tarball of 0.21.0-rc0 and set it up as a
pseudo-distributed Hadoop cluster.

Everything seems to work; basic smoke tests pass. Did not run internal unit
tests. I tested Sqoop 1.0.0 against this release. All sqoop unit tests pass.
Sqoop can operate on the command-line as a "normal" Hadoop application
against 0.21.0 successfully as long as MAPREDUCE-1920 is applied (or the
appropriate configuration changes are made in mapred-site.xml).

The other problem I found is that build-utils.xml is missing which prevents
recompilation.

I am a (nonbinding) +1 to this release candidate along with modifications to
include MR-1920 and fix the missing build-utils.xml issue.

- Aaron

A number of issues have been identified with RC0. The following JIRAs
need fixing before the next release candidate can be built:

HDFS-1288 - start-all.sh / stop-all.sh does not seem to work with HDFS
MAPREDUCE-1920 - Job.getCounters() returns null when using a cluster
MAPREDUCE-1926 - MapReduce distribution is missing build-utils.xml

In addition, the following JIRAs need fixing before the release is finalized:

HDFS-1292 - Allow artifacts to be published to the staging Apache
Nexus Maven Repository
MAPREDUCE-1929 - Allow artifacts to be published to the staging Apache
Nexus Maven Repository

Tom

This is likely a result of how things are now being built post
project-split, but previously, for the hadoop-0.20.x releases there was a
top level build.xml file which would orchestrate building the sub-projects
which were split underneath the src directory, resulting in a final
hadoop-20.x-core.jar. I see now each sub-project (common, hdfs and mapred)
has its own subdirectory off of the root, as opposed to forking under the
src/ directory and in turn their own sub-project jar files and there is
now no aggregated hadoop-x-core.jar file.

I also saw Tom's JIRA with the 0.21 build scripts. My questions are:

1) Is this the way we're going to be building going forward for the
foreseeable future?
2) Could we consider still keeping a top level build file (i.e committing
a variant of tom's scripts) that orchestrates the building of all the
sub-projects?

Kind regards
Steve Watt