Grokbase › Groups › Hadoop › common-user › July 2011
FAQ
I'm passing the whole ruby home to Hadoop, which contains thousands of small
files. The packaging process takes few minutes, any tips to speed this up?


-files ruby-1.9.2-p180
-D
mapred.child.env=PATH=ruby-1.9.2-p180/bin:'$PATH',GEM_HOME=ruby-1.9.2-p180,LD_LIBRARY_PATH=ruby-1.9.2-p180/lib,GEM_PATH=ruby-1.9.2-p180,RUBYLIB=ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1:ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/x86_64-linux:ruby-1.9.2-p180/lib/ruby/site_ruby:ruby-1.9.2-p180/lib/ruby/vendor_ruby/1.9.1:ruby-1.9.2-p180/lib/ruby/vendor_ruby/1.9.1/x86_64-linux:ruby-1.9.2-p180/lib/ruby/vendor_ruby:ruby-1.9.2-p180/lib/ruby/1.9.1:ruby-1.9.2-p180/lib/ruby/1.9.1/x86_64-linux
\

Search Discussions

  • Guang-Nan Cheng at Jul 9, 2011 at 1:57 pm ⇧
    Sorry, wrong question. I found it's not caused by packageJobJar. The
    slowness happens while putting those small files to HDFS.
    On Tue, Jul 5, 2011 at 2:18 PM, Guang-Nan Cheng wrote:

    I'm passing the whole ruby home to Hadoop, which contains thousands of
    small files. The packaging process takes few minutes, any tips to speed
    this up?


    -files ruby-1.9.2-p180
    -D
    mapred.child.env=PATH=ruby-1.9.2-p180/bin:'$PATH',GEM_HOME=ruby-1.9.2-p180,LD_LIBRARY_PATH=ruby-1.9.2-p180/lib,GEM_PATH=ruby-1.9.2-p180,RUBYLIB=ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1:ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/x86_64-linux:ruby-1.9.2-p180/lib/ruby/site_ruby:ruby-1.9.2-p180/lib/ruby/vendor_ruby/1.9.1:ruby-1.9.2-p180/lib/ruby/vendor_ruby/1.9.1/x86_64-linux:ruby-1.9.2-p180/lib/ruby/vendor_ruby:ruby-1.9.2-p180/lib/ruby/1.9.1:ruby-1.9.2-p180/lib/ruby/1.9.1/x86_64-linux
    \

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedJul 5, '11 at 6:20a
activeJul 9, '11 at 1:57p
posts2
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Guang-Nan Cheng: 2 posts

Content

People

Support

Translate

site design / logo © 2023 Grokbase