FAQ
It looks like karmasphere can handle it:

INFO
com.karmasphere.studio.hadoop.netbeans.filesystem.CommonsVfsBrowserTransferHandler
importData: Past filenode: FileNode for file:///Users/simpatico/Documents,
state=Updated; type=folder
INFO com.karmasphere.studio.hadoop.netbeans.filesystem.FileOperation$Type$2
run: Moving Files: 1 to FileNode for file:///Users/simpatico/Documents,
state=Updated; type=folder



On Sun, Jul 31, 2011 at 8:41 PM, Gabriele Kahlout
wrote:
I have sort of the same problem I feel.
I've a large segment I cannot index because there's not enough room, nor
-copyToLocal, local being the server.
How can I instead -copyToLocal where local is my local terminal machine,
instead of the server?

On Tue, Jun 7, 2011 at 9:30 PM, Joe Greenawalt wrote:

Thanks, and I've seen this example. I think after I can connect, i'm ok,
but i'm not sure how to do it remotely. I'm writing a groovy script just
to
test connection and i'll paste it below so you can see what i'm trying to
do.

@GrabResolver(name='org.apache.mahout.hadoop', root='
http://mymavenrepo/nexus/content/repositories/thirdparty')
@Grab(group='org.apache.mahout.hadoop', module='hadoop-core',
version='0.20.203.0')
import org.apache.hadoop.hdfs.DFSClient
import org.apache.hadoop.hdfs.DistributedFileSystem
import org.apache.hadoop.conf.Configuration
org.apache.hadoop.hdfs.protocol.HdfsFileStatus
def DIR_HADOOP = "1.1.1.1";
def PORT_HADOOP = "9000";
def config = new Configuration()
config.set("fs.default.name", DIR_HADOOP + ":" + PORT_HADOOP) //got this
from from some site
def dfs = new DistributedFileSystem()
def dfsClient = dfs.getClient()
def fileInfo = dfsClient.getFileInfo("/DEV")
println fileInfo.isDir()

On Tue, Jun 7, 2011 at 2:11 PM, Habermaas, William <
William.Habermaas@fatwire.com> wrote:
Joe,

Take a look at http://wiki.apache.org/hadoop/HadoopDfsReadWriteExample

It should give you an idea of how to read and write HDFS files. This page
is somewhat old and the package names have changed a bit between versions
but I hope it will get you on the right track. If you don't want to write
code there are HDFS copy utilities that you can use instead from shell
scripts.

Bill

-----Original Message-----
From: Joe Greenawalt
Sent: Tuesday, June 07, 2011 1:38 PM
To: common-user@hadoop.apache.org
Subject: Re: remotely downloading file

Bill,
thanks for the reply, is there a resource that you have available that i
can
look at the correct way to connect remotely?
I seem to be seeing conflicting ways on doing that.

I'm looking at:
http://hadoop.apache.org/hdfs/docs/current/api/org/apache/hadoop/hdfs/DFSClient.html
http://hadoop.apache.org/hdfs/docs/current/api/org/apache/hadoop/hdfs/DistributedFileSystem.html
But the examples i'm seeing are using the Configuration but i don't see
that
being used in those classes.

Thanks again,

Joe

On Fri, Jun 3, 2011 at 5:05 PM, Habermaas, William <
William.Habermaas@fatwire.com> wrote:
You can access HDFS for reading and writing from other machines. The
API
works through the HDFS client which can be anywhere on the network and not
just on the namenode. You just have to have the Hadoop core jar with
your
application wherever it is going to run.

Bill

-----Original Message-----
From: Joe Greenawalt
Sent: Friday, June 03, 2011 4:55 PM
To: common-user@hadoop.apache.org
Subject: remotely downloading file

Hi,
We're interested in using HDFS to store several large file sets to be
available for download from our customers in the following paradigm:

Customer <- | APPSERVER-CLUSTER {app1,app2,app3} | <- | HDFS |

I had assumed that pulling the file from HDFS to the APPSERVER-CLUSTER
could
be done program-ably remotely after browsing the documentation. But after
reading the API, it seems that writing Java code to interface with
HDFS
needs to happen locally? Is that correct?

If it is correct, what is the best/recommended way to
deliver downloadables to the APPSERVERS (and vice versa) which are
hosted
in
the same network but on different machines?

Thanks,
Joe


--
Regards,
K. Gabriele

--- unchanged since 20/9/10 ---
P.S. If the subject contains "[LON]" or the addressee acknowledges the
receipt within 48 hours then I don't resend the email.
subject(this) ∈ L(LON*) ∨ ∃x. (x ∈ MyInbox ∧ Acknowledges(x, this) ∧
time(x) < Now + 48h) ⇒ ¬resend(I, this).

If an email is sent by a sender that is not a trusted contact or the email
does not contain a valid code then the email is not received. A valid code
starts with a hyphen and ends with "X".
∀x. x ∈ MyInbox ⇒ from(x) ∈ MySafeSenderList ∨ (∃y. y ∈ subject(x) ∧ y ∈
L(-[a-z]+[0-9]X)).

--
Regards,
K. Gabriele

--- unchanged since 20/9/10 ---
P.S. If the subject contains "[LON]" or the addressee acknowledges the
receipt within 48 hours then I don't resend the email.
subject(this) ∈ L(LON*) ∨ ∃x. (x ∈ MyInbox ∧ Acknowledges(x, this) ∧ time(x)
< Now + 48h) ⇒ ¬resend(I, this).

If an email is sent by a sender that is not a trusted contact or the email
does not contain a valid code then the email is not received. A valid code
starts with a hyphen and ends with "X".
∀x. x ∈ MyInbox ⇒ from(x) ∈ MySafeSenderList ∨ (∃y. y ∈ subject(x) ∧ y ∈
L(-[a-z]+[0-9]X)).

Search Discussions

Discussion Posts

Previous

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 2 of 2 | next ›
Discussion Overview
groupcommon-user @
categorieshadoop
postedJul 31, '11 at 6:42p
activeJul 31, '11 at 10:31p
posts2
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Gabriele Kahlout: 2 posts

People

Translate

site design / logo © 2022 Grokbase