What is mrjob?
mrjob is a Python package that helps you write and run Hadoop Streaming

mrjob fully supports Amazon's Elastic MapReduce (EMR) service, which allows
you to buy time on a Hadoop cluster on an hourly basis. It also works with
your own Hadoop cluster.

Some important features:

* Run jobs on EMR, your own Hadoop cluster, or locally (for testing).
* Write multi-step jobs (one map-reduce step feeds into the next)
* Duplicate your production environment inside Hadoop
* Upload your source tree and put it in your job's $PYTHONPATH
* Run make and other setup scripts
* Set environment variables (e.g. $TZ)
* Easily install python packages from tarballs (EMR only)
* Setup handled transparently by mrjob.conf config file
* Automatically interpret error logs from EMR
* SSH tunnel to hadoop job tracker on EMR
* Minimal setup
* To run on your Hadoop cluster, install simplejson and make sure
$HADOOP_HOME is set.

More info:

* Install mrjob: python setup.py install
* Documentation: http://packages.python.org/mrjob/
* PyPI: http://pypi.python.org/pypi/mrjob
* Development is hosted at github: http://github.com/Yelp/mrjob

What's new?
Big thank you to Yelp intern Steve Johnson, who wrote the majority of the
code for this release. Wahbeh Qardaji, another Yelp intern, contributed as
well, and has been working hard on features for v0.3.0.

v0.2.7, 2011-07-12 -- Hooray for interns!
* All runner options can be set from the command line (Issue #121)
* Including for mrjob.tools.emr.create_job_flow (Issue #142)
* New EMR options:
* availability_zone (Issue #72)
* bootstrap_actions (Issue #69)
* enable_emr_debugging (Issue #133)
* Read counters from EMR log files (Issue #134)
* Clean old files out of S3 with mrjob.tools.emr.s3_tmpwatch (Issue #9)
* EMR parses and reports job failure due to steps timing out (Issue #15)
* EMR boostrap files are no longer made public on S3 (Issue #70)
* mrjob.tools.emr.terminate_idle_job_flows handles custom hadoop streaming
jars correctly (Issue #116)
* LocalMRJobRunner separates out counters by step (Issue #28)
* bootstrap_python_packages works regardless of tarball name (Issue #49)
* mrjob always creates temp buckets in the correct AWS region (Issue #64)
* Catch abuse of __main__ in jobs (Issue #78)
* Added mr_travelling_salesman example
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20110712/1efe01ee/attachment-0001.html>

Search Discussions

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
postedJul 13, '11 at 12:51a
activeJul 13, '11 at 12:51a

1 user in discussion

Jimmy Retzlaff: 1 post



site design / logo © 2022 Grokbase