FAQ
Hi,

I am not sure if this question has been asked. Its more of a hadoop fs
question. I am trying to execute the following hadoop fs command :

hadoop fs -copyToLocal s3n://<Access Key>:<Secret Key>@<bucket
name>/file.txt /home/hadoop/workspace/file.txt

When I execute this command directly from the Terminal shell, it works
perfectly fine, however the above command from code doesn't execute. In fact,
it says :

Exception in thread "main" copyToLocal: null

Please note I am using Runtime.getRunTime().exec(cmdStr), where cmdStr is the
above hadoop command. Also, please note that hadoop fs -cp or hadoop fs -rmr
commands work fine with source and destination being both Amazon AWS locations.
In the above command (hadoop fs -copyToLocal) the destination is local location
to my machine(Ubuntu installed).

Your help would be greatly appreciated.

Thanks,

Neeral

Search Discussions

  • Mapred Learn at May 31, 2011 at 11:51 pm
    try using complete path for where you hadoop binary is present. For eg
    /usr/bin/hadoop instead of hadoop...


    On Tue, May 31, 2011 at 3:56 PM, neeral beladia wrote:

    Hi,

    I am not sure if this question has been asked. Its more of a hadoop fs
    question. I am trying to execute the following hadoop fs command :

    hadoop fs -copyToLocal s3n://<Access Key>:<Secret Key>@<bucket
    name>/file.txt /home/hadoop/workspace/file.txt

    When I execute this command directly from the Terminal shell, it works
    perfectly fine, however the above command from code doesn't execute. In
    fact,
    it says :

    Exception in thread "main" copyToLocal: null

    Please note I am using Runtime.getRunTime().exec(cmdStr), where cmdStr is
    the
    above hadoop command. Also, please note that hadoop fs -cp or hadoop fs
    -rmr
    commands work fine with source and destination being both Amazon AWS
    locations.
    In the above command (hadoop fs -copyToLocal) the destination is local
    location
    to my machine(Ubuntu installed).

    Your help would be greatly appreciated.

    Thanks,

    Neeral
  • Mapred Learn at Jun 1, 2011 at 12:01 am
    Oops.. reading again, the command is working.
    what is the exact string that you have in cmdStr ?



    On Tue, May 31, 2011 at 4:51 PM, Mapred Learn wrote:

    try using complete path for where you hadoop binary is present. For eg
    /usr/bin/hadoop instead of hadoop...


    On Tue, May 31, 2011 at 3:56 PM, neeral beladia wrote:

    Hi,

    I am not sure if this question has been asked. Its more of a hadoop fs
    question. I am trying to execute the following hadoop fs command :

    hadoop fs -copyToLocal s3n://<Access Key>:<Secret Key>@<bucket
    name>/file.txt /home/hadoop/workspace/file.txt

    When I execute this command directly from the Terminal shell, it works
    perfectly fine, however the above command from code doesn't execute. In
    fact,
    it says :

    Exception in thread "main" copyToLocal: null

    Please note I am using Runtime.getRunTime().exec(cmdStr), where cmdStr is
    the
    above hadoop command. Also, please note that hadoop fs -cp or hadoop fs
    -rmr
    commands work fine with source and destination being both Amazon AWS
    locations.
    In the above command (hadoop fs -copyToLocal) the destination is local
    location
    to my machine(Ubuntu installed).

    Your help would be greatly appreciated.

    Thanks,

    Neeral
  • Neeral beladia at Jun 1, 2011 at 5:00 am
    Hi,

    I have a file on amazon aws under :

    s3n://<Access Key>:<Secret Key>@<Bucket Name>/file.txt

    I want this file to be accessible by the slave nodes via Distributed Cache.

    I put the following after the job configuration statements in the Driver program
    :

    DistributedCache.addCacheFile(new Path("s3n://<Access Key>:<Secret Key>@<Bucket
    Name>/file.txt").toUri(), job.getConfiguration());

    Also in my setup method in the mapper class, I have the below statement :
    Path[] cacheFiles =
    DistributedCache.getLocalCacheFiles(context.getConfiguration());

    cacheFiles is gettng assigned null.

    Could you please let me know what I am doing wrong here ? The file does exist on
    S3.

    Thanks,

    Neeral
  • Neeral beladia at Jun 1, 2011 at 5:24 pm
    btw, just to let you know that I am running my job in a pseudo-distributed mode.

    Thanks,

    Neeral




    ________________________________
    From: neeral beladia <neeral_beladia@yahoo.com>
    To: common-user@hadoop.apache.org
    Sent: Tue, May 31, 2011 10:00:00 PM
    Subject: DistributedCache - getLocalCacheFiles method returns null

    Hi,

    I have a file on amazon aws under :

    s3n://<Access Key>:<Secret Key>@<Bucket Name>/file.txt

    I want this file to be accessible by the slave nodes via Distributed Cache.

    I put the following after the job configuration statements in the Driver program

    :

    DistributedCache.addCacheFile(new Path("s3n://<Access Key>:<Secret Key>@<Bucket
    Name>/file.txt").toUri(), job.getConfiguration());

    Also in my setup method in the mapper class, I have the below statement :
    Path[] cacheFiles =
    DistributedCache.getLocalCacheFiles(context.getConfiguration());

    cacheFiles is gettng assigned null.

    Could you please let me know what I am doing wrong here ? The file does exist on

    S3.

    Thanks,

    Neeral

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedMay 31, '11 at 10:56p
activeJun 1, '11 at 5:24p
posts5
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Neeral beladia: 3 posts Mapred Learn: 2 posts

People

Translate

site design / logo © 2022 Grokbase