FAQ
I have sort of the same problem I feel.
I've a large segment I cannot index because there's not enough room, nor
-copyToLocal, local being the server.
How can I instead -copyToLocal where local is my local terminal machine,
instead of the server?

On Tue, Jun 7, 2011 at 9:30 PM, Joe Greenawalt wrote:

Thanks, and I've seen this example. I think after I can connect, i'm ok,
but i'm not sure how to do it remotely. I'm writing a groovy script just
to
test connection and i'll paste it below so you can see what i'm trying to
do.

@GrabResolver(name='org.apache.mahout.hadoop', root='
http://mymavenrepo/nexus/content/repositories/thirdparty')
@Grab(group='org.apache.mahout.hadoop', module='hadoop-core',
version='0.20.203.0')
import org.apache.hadoop.hdfs.DFSClient
import org.apache.hadoop.hdfs.DistributedFileSystem
import org.apache.hadoop.conf.Configuration
org.apache.hadoop.hdfs.protocol.HdfsFileStatus
def DIR_HADOOP = "1.1.1.1";
def PORT_HADOOP = "9000";
def config = new Configuration()
config.set("fs.default.name", DIR_HADOOP + ":" + PORT_HADOOP) //got this
from from some site
def dfs = new DistributedFileSystem()
def dfsClient = dfs.getClient()
def fileInfo = dfsClient.getFileInfo("/DEV")
println fileInfo.isDir()

On Tue, Jun 7, 2011 at 2:11 PM, Habermaas, William <
William.Habermaas@fatwire.com> wrote:
Joe,

Take a look at http://wiki.apache.org/hadoop/HadoopDfsReadWriteExample

It should give you an idea of how to read and write HDFS files. This page
is somewhat old and the package names have changed a bit between versions
but I hope it will get you on the right track. If you don't want to write
code there are HDFS copy utilities that you can use instead from shell
scripts.

Bill

-----Original Message-----
From: Joe Greenawalt
Sent: Tuesday, June 07, 2011 1:38 PM
To: common-user@hadoop.apache.org
Subject: Re: remotely downloading file

Bill,
thanks for the reply, is there a resource that you have available that i
can
look at the correct way to connect remotely?
I seem to be seeing conflicting ways on doing that.

I'm looking at:
http://hadoop.apache.org/hdfs/docs/current/api/org/apache/hadoop/hdfs/DFSClient.html
http://hadoop.apache.org/hdfs/docs/current/api/org/apache/hadoop/hdfs/DistributedFileSystem.html
But the examples i'm seeing are using the Configuration but i don't see
that
being used in those classes.

Thanks again,

Joe

On Fri, Jun 3, 2011 at 5:05 PM, Habermaas, William <
William.Habermaas@fatwire.com> wrote:
You can access HDFS for reading and writing from other machines. The
API
works through the HDFS client which can be anywhere on the network and not
just on the namenode. You just have to have the Hadoop core jar with
your
application wherever it is going to run.

Bill

-----Original Message-----
From: Joe Greenawalt
Sent: Friday, June 03, 2011 4:55 PM
To: common-user@hadoop.apache.org
Subject: remotely downloading file

Hi,
We're interested in using HDFS to store several large file sets to be
available for download from our customers in the following paradigm:

Customer <- | APPSERVER-CLUSTER {app1,app2,app3} | <- | HDFS |

I had assumed that pulling the file from HDFS to the APPSERVER-CLUSTER
could
be done program-ably remotely after browsing the documentation. But after
reading the API, it seems that writing Java code to interface with HDFS
needs to happen locally? Is that correct?

If it is correct, what is the best/recommended way to
deliver downloadables to the APPSERVERS (and vice versa) which are
hosted
in
the same network but on different machines?

Thanks,
Joe


--
Regards,
K. Gabriele

--- unchanged since 20/9/10 ---
P.S. If the subject contains "[LON]" or the addressee acknowledges the
receipt within 48 hours then I don't resend the email.
subject(this) ∈ L(LON*) ∨ ∃x. (x ∈ MyInbox ∧ Acknowledges(x, this) ∧ time(x)
< Now + 48h) ⇒ ¬resend(I, this).

If an email is sent by a sender that is not a trusted contact or the email
does not contain a valid code then the email is not received. A valid code
starts with a hyphen and ends with "X".
∀x. x ∈ MyInbox ⇒ from(x) ∈ MySafeSenderList ∨ (∃y. y ∈ subject(x) ∧ y ∈
L(-[a-z]+[0-9]X)).

Search Discussions

  • Gabriele Kahlout at Jul 31, 2011 at 10:31 pm
    It looks like karmasphere can handle it:

    INFO
    com.karmasphere.studio.hadoop.netbeans.filesystem.CommonsVfsBrowserTransferHandler
    importData: Past filenode: FileNode for file:///Users/simpatico/Documents,
    state=Updated; type=folder
    INFO com.karmasphere.studio.hadoop.netbeans.filesystem.FileOperation$Type$2
    run: Moving Files: 1 to FileNode for file:///Users/simpatico/Documents,
    state=Updated; type=folder



    On Sun, Jul 31, 2011 at 8:41 PM, Gabriele Kahlout
    wrote:
    I have sort of the same problem I feel.
    I've a large segment I cannot index because there's not enough room, nor
    -copyToLocal, local being the server.
    How can I instead -copyToLocal where local is my local terminal machine,
    instead of the server?

    On Tue, Jun 7, 2011 at 9:30 PM, Joe Greenawalt wrote:

    Thanks, and I've seen this example. I think after I can connect, i'm ok,
    but i'm not sure how to do it remotely. I'm writing a groovy script just
    to
    test connection and i'll paste it below so you can see what i'm trying to
    do.

    @GrabResolver(name='org.apache.mahout.hadoop', root='
    http://mymavenrepo/nexus/content/repositories/thirdparty')
    @Grab(group='org.apache.mahout.hadoop', module='hadoop-core',
    version='0.20.203.0')
    import org.apache.hadoop.hdfs.DFSClient
    import org.apache.hadoop.hdfs.DistributedFileSystem
    import org.apache.hadoop.conf.Configuration
    org.apache.hadoop.hdfs.protocol.HdfsFileStatus
    def DIR_HADOOP = "1.1.1.1";
    def PORT_HADOOP = "9000";
    def config = new Configuration()
    config.set("fs.default.name", DIR_HADOOP + ":" + PORT_HADOOP) //got this
    from from some site
    def dfs = new DistributedFileSystem()
    def dfsClient = dfs.getClient()
    def fileInfo = dfsClient.getFileInfo("/DEV")
    println fileInfo.isDir()

    On Tue, Jun 7, 2011 at 2:11 PM, Habermaas, William <
    William.Habermaas@fatwire.com> wrote:
    Joe,

    Take a look at http://wiki.apache.org/hadoop/HadoopDfsReadWriteExample

    It should give you an idea of how to read and write HDFS files. This page
    is somewhat old and the package names have changed a bit between versions
    but I hope it will get you on the right track. If you don't want to write
    code there are HDFS copy utilities that you can use instead from shell
    scripts.

    Bill

    -----Original Message-----
    From: Joe Greenawalt
    Sent: Tuesday, June 07, 2011 1:38 PM
    To: common-user@hadoop.apache.org
    Subject: Re: remotely downloading file

    Bill,
    thanks for the reply, is there a resource that you have available that i
    can
    look at the correct way to connect remotely?
    I seem to be seeing conflicting ways on doing that.

    I'm looking at:
    http://hadoop.apache.org/hdfs/docs/current/api/org/apache/hadoop/hdfs/DFSClient.html
    http://hadoop.apache.org/hdfs/docs/current/api/org/apache/hadoop/hdfs/DistributedFileSystem.html
    But the examples i'm seeing are using the Configuration but i don't see
    that
    being used in those classes.

    Thanks again,

    Joe

    On Fri, Jun 3, 2011 at 5:05 PM, Habermaas, William <
    William.Habermaas@fatwire.com> wrote:
    You can access HDFS for reading and writing from other machines. The
    API
    works through the HDFS client which can be anywhere on the network and not
    just on the namenode. You just have to have the Hadoop core jar with
    your
    application wherever it is going to run.

    Bill

    -----Original Message-----
    From: Joe Greenawalt
    Sent: Friday, June 03, 2011 4:55 PM
    To: common-user@hadoop.apache.org
    Subject: remotely downloading file

    Hi,
    We're interested in using HDFS to store several large file sets to be
    available for download from our customers in the following paradigm:

    Customer <- | APPSERVER-CLUSTER {app1,app2,app3} | <- | HDFS |

    I had assumed that pulling the file from HDFS to the APPSERVER-CLUSTER
    could
    be done program-ably remotely after browsing the documentation. But after
    reading the API, it seems that writing Java code to interface with
    HDFS
    needs to happen locally? Is that correct?

    If it is correct, what is the best/recommended way to
    deliver downloadables to the APPSERVERS (and vice versa) which are
    hosted
    in
    the same network but on different machines?

    Thanks,
    Joe


    --
    Regards,
    K. Gabriele

    --- unchanged since 20/9/10 ---
    P.S. If the subject contains "[LON]" or the addressee acknowledges the
    receipt within 48 hours then I don't resend the email.
    subject(this) ∈ L(LON*) ∨ ∃x. (x ∈ MyInbox ∧ Acknowledges(x, this) ∧
    time(x) < Now + 48h) ⇒ ¬resend(I, this).

    If an email is sent by a sender that is not a trusted contact or the email
    does not contain a valid code then the email is not received. A valid code
    starts with a hyphen and ends with "X".
    ∀x. x ∈ MyInbox ⇒ from(x) ∈ MySafeSenderList ∨ (∃y. y ∈ subject(x) ∧ y ∈
    L(-[a-z]+[0-9]X)).

    --
    Regards,
    K. Gabriele

    --- unchanged since 20/9/10 ---
    P.S. If the subject contains "[LON]" or the addressee acknowledges the
    receipt within 48 hours then I don't resend the email.
    subject(this) ∈ L(LON*) ∨ ∃x. (x ∈ MyInbox ∧ Acknowledges(x, this) ∧ time(x)
    < Now + 48h) ⇒ ¬resend(I, this).

    If an email is sent by a sender that is not a trusted contact or the email
    does not contain a valid code then the email is not received. A valid code
    starts with a hyphen and ends with "X".
    ∀x. x ∈ MyInbox ⇒ from(x) ∈ MySafeSenderList ∨ (∃y. y ∈ subject(x) ∧ y ∈
    L(-[a-z]+[0-9]X)).

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedJul 31, '11 at 6:42p
activeJul 31, '11 at 10:31p
posts2
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Gabriele Kahlout: 2 posts

People

Translate

site design / logo © 2022 Grokbase