Search Discussions

136 discussions - 517 posts

  • Hi, there is a performance penalty in Windows (pardon the expression) if you put too many files in the same directory. The OS becomes very slow, stops seeing them, and lies about their status to my ...
    Mark KerznerMark Kerzner
    Jan 23, 2009 at 11:03 pm
    Jan 28, 2009 at 7:30 am
  • Hi, I'm looking for an advice. I need to process a directed graph encoded as a list of <from, to pairs. The goal is to compute a list of longest paths in the graph. There is no guarantee that the ...
    Andrzej BialeckiAndrzej Bialecki
    Jan 29, 2009 at 5:20 pm
    Feb 2, 2009 at 6:46 am
  • Hi, Is there a tool that one could run on a datanode to scrub all the blocks on that node? Sriram
    Sriram RaoSriram Rao
    Jan 28, 2009 at 10:11 pm
    Jan 29, 2009 at 2:21 pm
  • hi, i am new to hadoop. i am trying to set it up for the first time as a single node cluster. at present the snag is that i cannot seem to find the correct path for setting the JAVA_HOME variable. i ...
    Jan 30, 2009 at 9:49 pm
    Feb 2, 2009 at 5:36 pm
  • Hey all, I'm currently installing a new cluster, and noticed something a little confusing. My DFS is *completely* empty - 0 files in DFS. However, in the namenode web interface, the reported ...
    Bryan DuxburyBryan Duxbury
    Jan 29, 2009 at 11:23 pm
    Feb 1, 2009 at 8:09 am
  • I plan to use hadoop to do some log processing and I'm working on a method to load the files (probably nightly) into hdfs. My plan is to have a web server on each machine with logs that serves up the ...
    Derek YoungDerek Young
    Jan 21, 2009 at 9:33 pm
    Jan 23, 2009 at 8:35 pm
  • The problem we are having is that datanodes periodically stall for 10-15 minutes and drop off the active list and then come back. What is going on is that a long operation set is holding the lock on ...
    Jason VennerJason Venner
    Jan 9, 2009 at 12:05 am
    Jan 12, 2009 at 9:03 pm
  • Hello, I'm now using hadoop-0.18.0 and testing it on a cluster with 1 master and 4 slaves. In hadoop-site.xml the value of "mapred.map.tasks" is 10. Because the values "throughput" and "average IO ...
    Jan 6, 2009 at 3:05 pm
    May 22, 2011 at 5:10 pm
  • I'm running 0.19.0 on a 10 node cluster (8 core, 16GB RAM, 4x1.5TB). The current status of my FS is approximately 1 million files and directories, 950k blocks, and heap size of 7GB (16GB reserved). ...
    Sean KnappSean Knapp
    Jan 31, 2009 at 10:20 pm
    Feb 4, 2009 at 6:32 pm
  • How well does Hadoop handle multiple independent disks per node? I have a cluster with 4 identical disks per node. I plan to use one disk for OS and temporary storage, and dedicate the other three to ...
    David B. RitchDavid B. Ritch
    Jan 11, 2009 at 9:23 pm
    Jan 15, 2009 at 3:27 pm
  • I am experimenting with Hadoop backed by Amazon s3 filesystem as one of our backup storage solution. Just the hadoop and s3(block based since it overcomes the 5gb limit) so far seems to be fine. My ...
    Roopa SudheendraRoopa Sudheendra
    Jan 28, 2009 at 6:59 pm
    Jan 29, 2009 at 12:50 pm
  • Hi Apple provides opensource discovery service called Bonjour (zeroconf). Is it possible to integrate Zeroconf with Hadoop so that discovery of nodes become automatic ? Presently for setting up ...
    Nitesh bhatiaNitesh bhatia
    Jan 25, 2009 at 4:45 pm
    Jan 27, 2009 at 10:08 pm
  • Hi all, we've made our first steps in evaluating hadoop. The setup of 2 VMs as a hadoop grid was very easy and works fine. Now our operations team wonders why hadoop has to be able to connect to the ...
    Matthias SchererMatthias Scherer
    Jan 21, 2009 at 12:25 pm
    Jan 23, 2009 at 8:42 pm
  • Might there be a reason for why this seems to routinely happen to me when using Hadoop 0.19.0 on Amazon EC2? 09/01/23 11:45:52 INFO hdfs.DFSClient: Could not obtain block ...
    Zak, Richard [USA]Zak, Richard [USA]
    Jan 23, 2009 at 5:20 pm
    Jan 23, 2009 at 7:42 pm
  • I was able to decommission a datanode successfully without having to stop my cluster. But I noticed that after a node has been decommissioned, it shows up as a dead node in the web base interface to ...
    Bill AuBill Au
    Jan 27, 2009 at 10:08 pm
    Feb 4, 2009 at 3:46 pm
  • Hi, I am a new user and was setting up the HDFS on 3 nodes as of now. I could get them to run individual pseudo distributed setups but am unable to get the cluster going together. The site ...
    Amandeep KhuranaAmandeep Khurana
    Jan 30, 2009 at 10:50 pm
    Feb 2, 2009 at 4:04 am
  • Hi. Today I noticed when I ran a Solr Indexing job through our Hadoop cluster that the master MySQL database where screaming about "Too Many Connections". I wondered how that could happen so I logged ...
    Marcus HerouMarcus Herou
    Jan 25, 2009 at 4:43 pm
    Jan 27, 2009 at 9:21 pm
  • Hi I am trying to setup Hadoop 0.19 on OS X. Current Java Version is java version "1.6.0_07" Java(TM) SE Runtime Environment (build 1.6.0_07-b06-153) Java HotSpot(TM) 64-Bit Server VM (build ...
    Nitesh bhatiaNitesh bhatia
    Jan 24, 2009 at 10:36 pm
    Jan 26, 2009 at 8:10 pm
  • Hi, I have a task to process large quantities of files by converting them into other formats. Each file is processed as a whole and converted to a target format. Since there are 100's of GB of data I ...
    Darren GovoniDarren Govoni
    Jan 21, 2009 at 1:08 pm
    Jan 21, 2009 at 8:12 pm
  • Hi, I need to lookup a large number of key/value pairs in my map(). Is there any indexed hashtable available as a part of Hadoop I/O API? I find Hbase an overkill for my application; something on the ...
    Delip RaoDelip Rao
    Jan 15, 2009 at 2:47 am
    Jan 16, 2009 at 12:51 pm
  • hi, I have some questions about Map-Reduce that I'm not sure, hope that you guys can help me. - Does Map-Reduce support parallel writing/reading ? I think not because I don't find anything like that ...
    Jan 11, 2009 at 1:51 pm
    Jan 14, 2009 at 6:04 pm
  • Hi, Since I¹ve upgraded to 0.19.0, I¹ve been getting the following exceptions when restarting jobs, or even when a failed reducer is being restarted by the job tracker. It appears that stale file ...
    Stefan WillStefan Will
    Jan 23, 2009 at 7:24 pm
    Feb 17, 2009 at 11:53 pm
  • We have a small test cluster, a double master (NameNode+JobTracker) plus 2 slaves, running 0.18.1. We are seeing an intermittent problem where our application logs failures out of DFSClient, thus: ...
    Karl KleinpasteKarl Kleinpaste
    Jan 30, 2009 at 5:00 pm
    Feb 4, 2009 at 11:33 pm
  • Hello, I have a clarifying question about Hadoop streaming. I'm new to the list and didn't see anything posted that covers my questions - my apologies if I overlooked a relevant post. I have an input ...
    S DS D
    Jan 29, 2009 at 6:51 pm
    Feb 3, 2009 at 3:50 am
  • Hi, I wanted to ask, if HDFS is a good solution just as a distributed db (no running jobs, only get and put commands) A review says that "HDFS is not designed for low latency" and besides, it's ...
    Rasit OZDASRasit OZDAS
    Jan 27, 2009 at 9:28 pm
    Jan 29, 2009 at 9:03 am
  • Hi Hadoop Users, I am trying to build a storage system for the office of about 20-30 users which will store everything. (600mb) which are generated every hour. Is Hadoop suitable for this kind of ...
    Jan 29, 2009 at 6:03 am
    Jan 29, 2009 at 8:53 am
  • Any one knows Netbeans or Eclipse plugin for Hadoop Map -Reduce job. I want to make plugin for netbeans http://vinayakkatkar.wordpress.com -- Vinayak Katkar Sun Campus Ambassador Sun ...
    Vinayak katkarVinayak katkar
    Jan 25, 2009 at 3:57 pm
    Jan 28, 2009 at 2:34 pm
  • Why do we not use the Remaining % in place of use Used % when we are selecting datanode for new data and when running the balancer. form what I can tell we are using the use % used and we do not ...
    Billy PearsonBilly Pearson
    Jan 20, 2009 at 6:29 am
    Jan 26, 2009 at 9:19 pm
  • Hello all, I am trying to test hdfs_test.c provided with hadoop installation. libhdfs.so and hdfs_test are built fine after making a few changes in $(HADOOP_HOME)/src/c++/libhdfs/Makefile. But when I ...
    Arifa NisarArifa Nisar
    Jan 17, 2009 at 9:20 am
    Jan 24, 2009 at 10:46 am
  • Hello Hadoop Users, I was hoping someone would be able to answer a question about node decommissioning. I have a test Hadoop cluster set up which only consists of my computer and a master node. I am ...
    Hargraves, AlyssaHargraves, Alyssa
    Jan 22, 2009 at 12:35 am
    Jan 22, 2009 at 9:23 pm
  • Hello, Any tips would be greatly appreciated. Is there a way to set the order of the keys in reduce as shown below, no matter what order the collection in MAP occurs in. Thanks, Brian public void ...
    Brian MacKayBrian MacKay
    Jan 22, 2009 at 3:24 pm
    Jan 22, 2009 at 5:38 pm
  • Hello, The original map-reduce paper states: "After successful completion, the output of the map-reduce execution is available in the R output files (one per reduce task, with file names as specified by ...
    Jim TwenskyJim Twensky
    Jan 11, 2009 at 7:56 am
    Jan 15, 2009 at 12:34 am
  • Hi, I want to use an lzo file as input for a mapper. The record reader determines the codec using a CompressionCodecFactory, like this: (Hadoop version 0.19.0) compressionCodecs = new ...
    Gert PfeiferGert Pfeifer
    Jan 13, 2009 at 3:30 pm
    Jan 14, 2009 at 9:30 pm
  • Hello, I would just like to confirm, when does the Combiner run(since it might not be run at all,see below). I read somewhere that it is run, if there is at least one reduce (which in my case i can ...
    Saptarshi GuhaSaptarshi Guha
    Jan 2, 2009 at 5:57 pm
    Jan 8, 2009 at 6:33 am
  • Hello, When I check the job tracker web page, and look at the Map Input records read,the map input records goes up to say 1.4MN and then drops to 410K and then goes up again. The same happens with ...
    Saptarshi GuhaSaptarshi Guha
    Jan 5, 2009 at 4:35 pm
    Jan 6, 2009 at 12:40 am
  • Hello, Could someone point me toward some more documentation on how to write one's own partition class? I have having quite a bit of trouble getting mine to work. So far, it looks something like ...
    Jan 30, 2009 at 9:33 pm
    Feb 4, 2009 at 1:09 am
  • Is there any way to cancel a job after it has been submitted? Bill
    Bill AuBill Au
    Jan 30, 2009 at 10:41 pm
    Feb 2, 2009 at 9:56 pm
  • I have a MapReduce application in which I configure 16 reducers to run on 15 machines. My mappers output exactly 16 keys, IntWritable's from 0 to 15. However, only 12 out of the 15 machines are used ...
    Nathan MarzNathan Marz
    Jan 30, 2009 at 3:06 am
    Feb 2, 2009 at 1:55 am
  • We've been running 0.18.2 for over a month on an 8 node cluster. Last week we added 4 more nodes to the cluster and have experienced 2 failures to the tasktrackers since then. The namenodes are ...
    David J. O'DellDavid J. O'Dell
    Jan 28, 2009 at 4:26 pm
    Jan 29, 2009 at 1:55 am
  • Yuanyuan TianYuanyuan Tian
    Jan 27, 2009 at 10:09 pm
    Jan 27, 2009 at 10:09 pm
  • Is it possible to call a mapreduce job from inside another, if yes how? and is it possible to disable the reducer completely that is suspend the job immediately after call to map has been terminated. ...
    Aditya DesaiAditya Desai
    Jan 18, 2009 at 8:17 pm
    Jan 19, 2009 at 9:37 am
  • I'm trying to use use map reduce to merge two classes of files, each class using the same keys for grouping. An example: class 1 input file: id_1 A metadatum id_2 A metadatum id_1 A metadatum class 2 ...
    Meng MaoMeng Mao
    Jan 5, 2009 at 10:08 pm
    Jan 6, 2009 at 8:30 am
  • I am trying to add nodes to an existing working cluster. Do I need to bring the entire cluster down or just shutting down and restarting the namenode after adding the new machine list to the slaves ...
    Amandeep KhuranaAmandeep Khurana
    Jan 31, 2009 at 2:55 am
    Feb 2, 2009 at 1:42 am
  • I'm running Hadoop 0.19.0 on Solaris (SunOS 5.10 on x86) and many jobs are failing with this exception: Error initializing attempt_200901281655_0004_m_000025_0: java.io.IOException: Cannot run ...
    Andy LiuAndy Liu
    Jan 29, 2009 at 3:43 am
    Jan 30, 2009 at 7:13 pm
  • hi everyone, I got a question, maybe you can help me. - how can we get the meta data of a file on HDFS ? For example: If I have a file with e.g. 2 GB on HDFS, this file is split into many chunks and ...
    Jan 23, 2009 at 11:25 pm
    Jan 27, 2009 at 9:06 pm
  • Hi. I'm running streaming on relatively big (2Tb) dataset, which is being split by hadoop in 64mb pieces. One of the problems I have with that is my map tasks take very long time to initialize (they ...
    Dmitry PushkarevDmitry Pushkarev
    Jan 21, 2009 at 1:25 am
    Jan 21, 2009 at 8:01 am
  • Hello All, I'm currently trying to upgrade a hadoop 0.18.0 cluster to 0.19. The wrinkle is that I would like to include https://issues.apache.org/jira/browse/HADOOP-4906 into the build as well. Would ...
    Jan 16, 2009 at 7:47 pm
    Jan 19, 2009 at 6:56 pm
  • I recognize that Windows support is, um, limited :-) But, any ideas what exactly would need to be changed to support Windows (without cygwin) if someone such as myself were so motivated? The most ...
    Dan DiephouseDan Diephouse
    Jan 19, 2009 at 4:13 pm
    Jan 19, 2009 at 5:02 pm
  • Hi: I'm using Hadoop 0.17.1 and I'm encountering EOFException reading the FSEdits file. I don't have a clear understanding what is causing this and how to prevent this. Has anyone seen this and can ...
    Joe MontanezJoe Montanez
    Jan 13, 2009 at 5:54 pm
    Jan 19, 2009 at 8:49 am
  • Dear friends, I am new at Hadoop and at MapReduce techniques. I've developed my first map-reduce application using hadoop but I can't manage to make it work. I get the following error at the very ...
    Pedro VivancosPedro Vivancos
    Jan 16, 2009 at 6:42 pm
    Jan 19, 2009 at 8:01 am
Group Navigation
period‹ prev | Jan 2009 | next ›
Group Overview
groupcommon-user @

176 users for January 2009

Mark Kerzner: 19 posts Raghu Angadi: 17 posts Tienduc_dinh: 15 posts Aaron Kimball: 14 posts Saptarshi Guha: 14 posts Rasit OZDAS: 13 posts Steve Loughran: 13 posts Doug Cutting: 11 posts Owen O'Malley: 11 posts Sagar Naik: 11 posts Amareshwari Sriramadasu: 10 posts Bill Au: 10 posts Konstantin Shvachko: 10 posts Tom White: 10 posts Zak, Richard [USA]: 9 posts Brian Bockelman: 8 posts Craig Macdonald: 8 posts Jason hadoop: 8 posts Jason Venner: 8 posts Jim Twensky: 8 posts
show more