Search Discussions

93 discussions - 406 posts

  • We are looking at using HDFS as a long term storage solution. We want to use it to stored lots of files. The file could be big and small, they are images, videos etc... We only write the files once, ...
    Dongsheng WangDongsheng Wang
    Sep 5, 2007 at 8:03 pm
    Sep 7, 2007 at 6:46 pm
  • Hi all, I'm trying to get Hadoop 0.14.0 to talk to S3 but its not working out for me. My credentials are definitely right (checked multiple times and that's not the issue in any case) and the bucket ...
    Toby DiPasqualeToby DiPasquale
    Sep 5, 2007 at 7:20 pm
    Sep 23, 2007 at 9:57 pm
  • According the reference of HDFS(http://lucene.apache.org/hadoop/hdfs_design.html), a file in the HDFS will be split into one or more blocks and these blocks are stored in a set of Datanodes. I ...
    ChaoChun LiangChaoChun Liang
    Sep 5, 2007 at 8:58 am
    Sep 13, 2007 at 10:55 pm
  • running r578879. I shut everything down, wiped out all logs, then ran bin/start-all.sh when the jobtracker starts up i get 939,805 lines composed of some startup messages, and then on stack trace ...
    Kate rhodesKate rhodes
    Sep 26, 2007 at 2:21 pm
    Sep 26, 2007 at 10:26 pm
  • Consider two row based files. The first has fields: A B C the second has fields: B D E I want to join these files on the key B, to create records of the form: A B C D E So B can be thought of as a ...
    C GC G
    Sep 13, 2007 at 2:11 pm
    Sep 18, 2007 at 8:42 pm
  • Hi all-- I'm new to using Hadoop so I'm hoping to get a little guidance on what the best way to solve a particular class of problems would be. The general use case is this: from a very small set of ...
    Chris DyerChris Dyer
    Sep 27, 2007 at 7:23 pm
    Oct 3, 2007 at 2:37 am
  • Guys, I was wondering if people would be interested in an informal Hadoop Get Together. Could be just a simple format with people meeting someplace in the SF Bay Area to exchange ideas. Let me know ...
    Erich NachbarErich Nachbar
    Sep 24, 2007 at 9:58 pm
    Sep 26, 2007 at 12:27 am
  • What are reasonable hardware specifications for a Hadoop node? Can we document this somewhere (maybe in the wiki as HowToConfigureHardware?) Obviously this will be a moving target, but some guidance ...
    John HeidemannJohn Heidemann
    Sep 10, 2007 at 11:56 pm
    Sep 25, 2007 at 7:35 pm
  • The default is set to 60s. many of my dfs -put commands would seem to hang - and lowering the timeout (to 1s) seems to have made things a whole lot better. General curiosity - isn't 60s just huge for ...
    Joydeep Sen SarmaJoydeep Sen Sarma
    Sep 5, 2007 at 6:56 am
    Sep 13, 2007 at 9:51 pm
  • Hi All: In the context of using the aggregation classes, is there anyway to send output to multiple files? In my case, I am processing columnar records that are very wide. I have to do a variety of ...
    C GC G
    Sep 21, 2007 at 8:20 pm
    Sep 24, 2007 at 9:09 pm
  • Please excuse a possibly heretical question... My colleagues and I have been working with Hadoop lately, and I keep getting asked the same question: what is the performance impact of having the ...
    Steve SchlosserSteve Schlosser
    Sep 5, 2007 at 9:02 pm
    Sep 14, 2007 at 1:18 pm
  • Hi, I set up a 2-node Hadoop cluster, whose nodes are all in the same network and ran the 'grep' example. The map tasks were distributed among the two machines and ran without any problem. However, ...
    Ming YangMing Yang
    Sep 30, 2007 at 5:13 pm
    Oct 1, 2007 at 7:09 pm
  • Hi, Thanks for all the feedback regarding a Hadoop Get-Together! I create an Upcoming event for it: http://upcoming.yahoo.com/event/ 271501/?ps=6 I will bring tags & pens, but let me know if there is ...
    Erich NachbarErich Nachbar
    Sep 26, 2007 at 10:55 pm
    Oct 2, 2007 at 4:04 pm
  • Hey, I am a Pig newbie, I am trying to setup Pig over a solaris cluster with hadoop 13.1. I tried both ways of installing pig 1) I tried using the pig jar, I get the grunt prompt but with these ...
    Bhupesh bansalBhupesh bansal
    Sep 19, 2007 at 10:28 pm
    Sep 24, 2007 at 2:10 am
  • hey, has anyone leveraged the ability of datanodes to specify which datacenter and rack they live in? if so, any evidence of performance improvements? it seems that rack-awareness is only leveraged ...
    Jeff HammerbacherJeff Hammerbacher
    Sep 18, 2007 at 2:33 am
    Sep 18, 2007 at 7:13 pm
  • Greetings! We are happy to announce the release of Kosmos Filesystem (KFS) as an open source project. KFS was designed and implemented at Kosmix Corp. The initial release of KFS is version 0.1 ...
    Sriram RaoSriram Rao
    Sep 28, 2007 at 12:57 am
    Oct 1, 2007 at 11:27 pm
  • I've got a hadoop cluster running the example files just fine, but am having difficulty running a custom job. Specifically, the job starts, and then each task that is scheduled fails, with the ...
    Ross BoucherRoss Boucher
    Sep 19, 2007 at 9:31 pm
    Sep 21, 2007 at 6:01 pm
  • Hello all, I was trying to upgrade hadoop 0.13.1 to 0.14.1, but when I follow the instruction at http://wiki.apache.org/lucene-hadoop/Hadoop_0.14_Upgrade, running "./start-dfs.sh -upgrad", I found no ...
    Open StudyOpen Study
    Sep 12, 2007 at 6:29 pm
    Sep 12, 2007 at 8:00 pm
  • Hi, all -- I was trying to configure hadoop to work on two machines. The dfs seems to work fine. But when I tried the 'grep' example in 'hadoop-0.13.1-examples.jar', it always hang upon the finish of ...
    Xiaoguang QiXiaoguang Qi
    Sep 7, 2007 at 3:51 am
    Sep 12, 2007 at 1:40 am
  • Hey gang, We're getting ready to deploy our first cluster, and while deciding on the node layout, we ran into an interesting question. The cluster will be behind a firewall, and a few clients will be ...
    Stu HoodStu Hood
    Sep 11, 2007 at 9:41 pm
    Sep 11, 2007 at 10:36 pm
  • Hi, If the one of the hdfs's datanode fail or reboot and no one know that, could the hdfs auto restart the datanode daemons? Thank you Regards
    Sep 30, 2007 at 8:33 am
    Oct 6, 2007 at 3:16 am
  • Hi All, I am a complete newbie to Hadoop, not having tested or installed yet, but reading up for about a month now in spare time, and following the list. I think it's really exciting to provide this ...
    Jonathan HendlerJonathan Hendler
    Sep 21, 2007 at 3:25 am
    Sep 26, 2007 at 10:12 pm
  • Hi, I have a couple of problems that I think the development team could enhance. I'm currently running a job that takes a whole day to finish. 1) Adjusting input set dynamically At the start, I had ...
    Nathan WangNathan Wang
    Sep 25, 2007 at 5:33 pm
    Sep 26, 2007 at 7:17 am
  • Hi All: Two quick questions, thanks for any guidance... I'd like to run nodes with around 2T of local disk set up as JBOD. So I would have 4 separate file systems per machine, for example /hdfs_a, ...
    C GC G
    Sep 13, 2007 at 1:01 pm
    Sep 14, 2007 at 12:25 am
  • Curious to see if anyone has been considering using HDFS for a general storage platform. I have been playing around with using HDFS to store video type assets. wondering what type of mileage folks ...
    Lance BoomerangLance Boomerang
    Sep 6, 2007 at 9:43 pm
    Sep 6, 2007 at 10:46 pm
  • This is FYI. We at Yahoo! could successfully run hadoop (upto date trunk version) on a cluster of 2000 nodes. The programs we ran were RandomWriter and Sort. Sort performance was pretty good - we ...
    Devaraj DasDevaraj Das
    Sep 5, 2007 at 9:30 am
    Sep 6, 2007 at 12:54 pm
  • Dear developers: Hi,my name is floodhong,a student in china. Recently I want to make a cluster on hadoop, but now I have a question, How to let one data file only only be excuted by one task? which ...
    Sep 24, 2007 at 5:56 pm
    Sep 30, 2007 at 2:58 am
  • Dear all, Greetings and thank you very much in advance for your time. I have successfully installed hadoop and went over the source code very basically. I am interested in how does hadoop manage ...
    Khalil HonsaliKhalil Honsali
    Sep 26, 2007 at 8:53 pm
    Sep 26, 2007 at 10:26 pm
  • You are correct... in part. There are two problems that I discovered with this. 1) JobConf does not implement an interface it extends Configuration and the only thing Configuration implements is ...
    Kate rhodesKate rhodes
    Sep 22, 2007 at 9:59 pm
    Sep 24, 2007 at 9:11 pm
  • I've been running a program to count search terms in log files, which is basically a small modification of the wordcount program. This doesn't have a reduce phase, so the only tasks for the reduce ...
    Ross BoucherRoss Boucher
    Sep 21, 2007 at 7:02 pm
    Sep 23, 2007 at 1:18 am
  • Can anyone point me to some examples of unit tests for Mapper and Reducer classes? I'm finding plenty of tests for the infrastructure but no good examples of how to test that the mappers and reducers ...
    Kate rhodesKate rhodes
    Sep 20, 2007 at 9:54 pm
    Sep 21, 2007 at 4:31 pm
  • Hello everyone, I'm new to Hadoop and to this mailing list so: Hello. =) I'm experiencing a problem that I can't understand; I'm performing a wordcount task (from the examples in the source) on a ...
    Luca TelloliLuca Telloli
    Sep 14, 2007 at 2:17 pm
    Sep 17, 2007 at 6:21 pm
  • Hi, I've tried setting up hadoop on a single computer, and I'm experiencing a problem with the datanode. when i run the start-all.sh script it seems to run smoothly, including setting up the ...
    Sep 4, 2007 at 7:17 pm
    Sep 5, 2007 at 8:16 am
  • All: I am interested in hearing any success stories around deploying Hadoop in a commercial/non-academic environment. My interest is mostly around generating collateral for justifying our own ...
    C GC G
    Sep 4, 2007 at 2:00 pm
    Sep 5, 2007 at 6:26 am
  • Hi, Can I access the hdfs through Http? And load balanced? Thank you. Regards
    Sep 30, 2007 at 8:20 am
    Oct 8, 2007 at 2:54 am
  • Hi, Not sure whether this is the right place to ask, but I'll give it a go: Is Hadoop able to distribute tasks to individual cores in multicore nodes? I realise the framework is designed for running ...
    Ger-Jan te DorsthorstGer-Jan te Dorsthorst
    Sep 30, 2007 at 10:58 am
    Oct 1, 2007 at 6:18 pm
  • I saw a similar post (http://www.mail-archive.com/hadoop-user@lucene.apache.org/msg01112.html) but the answer was not very satisfactory. Image I used Hadoop as a fault-tolerance storage. I had 10 ...
    Nathan WangNathan Wang
    Sep 28, 2007 at 2:28 am
    Sep 28, 2007 at 4:23 pm
  • Hi All: I am trying to build an app using the data_join classes. When compiling via ant I am failing on an import: [javac] Compiling 8 source files to /home/cg/hadoop-0.14.1/build/proto [javac] ...
    C GC G
    Sep 27, 2007 at 4:12 am
    Sep 27, 2007 at 4:20 pm
  • when i use nutch's search, i'm getting error that java.io.IOException: No FileSystem for scheme: file org.apache.hadoop.fs.FileSystem.get(FileSystem.java:157) ...
    Sep 24, 2007 at 9:35 am
    Sep 27, 2007 at 9:46 am
  • Hi, I am having problem configuring my 2-node Hadoop cluster. The two machines, named A and B, are in the same network, having IP address of and, respectively. Machine A ...
    Ming YangMing Yang
    Sep 26, 2007 at 8:27 pm
    Sep 26, 2007 at 9:07 pm
  • https://issues.apache.org/jira/browse/HADOOP-1936 contrib tests -1. The patch failed contrib unit tests. The patch WAS unit tests. Actually it was one unit test, one new class, and one test for the ...
    Kate rhodesKate rhodes
    Sep 22, 2007 at 11:36 pm
    Sep 24, 2007 at 9:02 pm
  • i use this command "${HADOOP_HOME}/bin/hadoop jar ${NUTCHWAX_HOME}/nutchwax.jar all /tmp/inputs /tmp/outputs test" it show this error - ImportArcs segment: outputs/segments/25500920083644-test, src: ...
    Sep 20, 2007 at 2:38 am
    Sep 21, 2007 at 6:21 am
  • Hi All, Just popping my head up over the lurkers parapet for a second to let you know about some development work I've been doing regarding getting Hadoop to run in an OSGi environment. Sorry if this ...
    David SavageDavid Savage
    Sep 5, 2007 at 4:05 pm
    Sep 6, 2007 at 8:50 am
  • hi all. We have a small cluster set up now and have hit an interesting issue when trying to get people running jobs on it. My user kicked off the cluster. Other users with a copy of the same conf ...
    Jason gessnerJason gessner
    Sep 14, 2007 at 2:56 pm
    Oct 10, 2007 at 6:05 pm
  • Hi, Are there any examples using HDFS in java program. My requirement is simple: read and writer on the HDFS. Thank you. Regards HeQi
    Sep 24, 2007 at 4:32 am
    Sep 27, 2007 at 2:07 am
  • Has anyone been able to get Nutch 0.9 working with SOLR? Any help would be appreciated. ~~~~~~~~~~~~~~~~~~~~~ Daniel Clark, President DAC Systems, Inc. (703) 403-0340 ~~~~~~~~~~~~~~~~~~~~~
    Daniel ClarkDaniel Clark
    Sep 25, 2007 at 6:24 pm
    Sep 25, 2007 at 7:01 pm
  • I set up a hadoop task to run, and after 45 minutes it had completed all but one task. This one task had been killed and retried 3 times already, so I left it overnight, and 615 attempts later, it ...
    Ross BoucherRoss Boucher
    Sep 19, 2007 at 4:11 pm
    Sep 25, 2007 at 7:07 am
  • I just added 10 datanodes to a small cluster and turned up the replication on many of the files to balance the storage out a bit. I expected to see a uniform-ish distribution of blocks on the new ...
    Ted DunningTed Dunning
    Sep 20, 2007 at 2:46 am
    Sep 25, 2007 at 1:01 am
  • Hi All: Please indulge an embarrassing question for which I am sure there is a simple answer. Consider an aggregator which takes input like: A A 2 A B 5 A C 10 A A 4 A B 9 A D 5 and returns A A 6 A B ...
    C GC G
    Sep 19, 2007 at 10:00 pm
    Sep 20, 2007 at 4:48 am
  • Hi Everyone, I'm a newer to hadoop. I want to write a mapreduce program to implement the inverted index. My question is which input format should I use? It seems that the TextInputFormat's key is the ...
    贺皓\(He Hao\)贺皓\(He Hao\)
    Sep 19, 2007 at 1:58 pm
    Sep 20, 2007 at 1:55 am
Group Navigation
period‹ prev | Sep 2007 | next ›
Group Overview
groupcommon-user @

102 users for September 2007

Ted Dunning: 48 posts C G: 20 posts Kate rhodes: 17 posts Owen O'Malley: 15 posts Devaraj Das: 13 posts Joydeep Sen Sarma: 13 posts Arun C Murthy: 11 posts Doug Cutting: 11 posts Ross Boucher: 11 posts Toby DiPasquale: 11 posts Dhruba Borthakur: 10 posts Bhupesh bansal: 8 posts ChaoChun Liang: 8 posts Michael Bieniosek: 7 posts Stu Hood: 7 posts Tom White: 7 posts Torsten Curdt: 7 posts Dongsheng Wang: 6 posts Earney, Billy C.: 6 posts Enis Soztutar: 6 posts
show more