FAQ

Hadoop > mapreduce-user > December 2011

Search Discussions

77 discussions - 249 posts

  • Hi, I want to understand a basic concept in MR. If a mapper creates an instance of some class (using the 'new' operator), then the created class exists ONCE in the VM of this node. For each node. ...
    Eyal GolanEyal Golan
    Dec 30, 2011 at 11:13 am
    Jan 9, 2012 at 11:36 pm
  • Hi, Costin: I work on HBase. I went over http://static.springsource.org/spring-hadoop/docs/current/reference/hbase.htmlbut didn't have time to download the source code. Is there a typo: 'does more ...
    Ted YuTed Yu
    Dec 30, 2011 at 10:14 am
    Dec 30, 2011 at 11:14 am
  • Hi, The release notes for 0.22 ( http://hadoop.apache.org/common/releases.html#10+December%2C+2011%3A+release+0.22.0+available) it says By Security missing, what all features are missing? Does it ...
    Praveen SripatiPraveen Sripati
    Dec 29, 2011 at 2:41 pm
    Dec 29, 2011 at 3:40 pm
  • Hi, I use Yarn as resource management to deploy my run-time computing system. I follow http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/YARN.html yarn-nodemanager-**.log: ...
    Bing JiangBing Jiang
    Dec 29, 2011 at 8:28 am
    Jan 6, 2012 at 8:49 am
  • Hello, Another newbie question. Suppose I want to use an external library (jar) in the mapper / reducer classes. (commons-lang, google's guava, etc.) In our environment, I added the jars into a ...
    Eyal GolanEyal Golan
    Dec 28, 2011 at 12:10 am
    Dec 28, 2011 at 11:43 am
  • Hi, I have a 7-node setup (1 - Namenode/JobTracker, 6 - Datanodes/TaskTrackers) running Hadoop version 0.20.203. I performed the following test: Initially cluster is running smoothly. Just before ...
    Rajat GoelRajat Goel
    Dec 27, 2011 at 1:09 pm
    Dec 27, 2011 at 1:09 pm
  • One key point I wanted to mention for Hadoop developers (but then check out the announcement). I implemented a version of sysstat (iostat, vmstat, etc) in Peregrine and would be more than happy to ...
    Kevin BurtonKevin Burton
    Dec 27, 2011 at 6:31 am
    Dec 27, 2011 at 11:13 am
  • Hi, We just ran run large scale Apache Nutch jobs in our evaluation of 20.205.0 and they all failed. Some of these jobs ran concurrently with the fair scheduler enabled. These were simple jobs ...
    Markus JelsmaMarkus Jelsma
    Dec 27, 2011 at 12:48 am
    Jan 2, 2012 at 6:30 pm
  • Bccing common-user and ccing mapred-user. Please use the correct mailing lists for your questions. You can use -Dstream.map.output.field.separator= for specifying the seperator.  The link below ...
    Mahadev KonarMahadev Konar
    Dec 27, 2011 at 12:12 am
    Dec 28, 2011 at 4:59 pm
  • Hi, We sometimes see reducers fail just when all mappers are finishing. All mappers finish roughly at the same time. The reducers only dump the following exception: java.lang.Throwable: Child Error ...
    Markus JelsmaMarkus Jelsma
    Dec 26, 2011 at 7:45 pm
    Dec 27, 2011 at 11:14 am
  • Hi, We're sometimes seeing this exception if a map task already failed before due to, for example, an OOM error. Any ideas on how to address this issue? ...
    Markus JelsmaMarkus Jelsma
    Dec 26, 2011 at 5:39 pm
    Dec 27, 2011 at 11:17 am
  • Hi everyone, This is my first post in this list, as I am a newb with Hadoop. I am looking in the web for some documentation and example on how to use DI framework with Hadoop. Basically I want to ...
    Eyal GolanEyal Golan
    Dec 26, 2011 at 10:13 am
    Dec 30, 2011 at 5:29 pm
  • I know Hadoop Yarn can support MapReduce job well, but I have not found DAG model task. Can you give me some demonstration I missed out , and point out how to build my own programming models in the ...
    Bing JiangBing Jiang
    Dec 26, 2011 at 9:57 am
    Dec 27, 2011 at 8:07 am
  • Hi, In the past few weeks we evaluated and partially migrated from Hadoop 0.20.203.0 to 0.22.0. Most stuff works fine locally and simple jobs do well on the cluster. However, the most essential part ...
    Markus JelsmaMarkus Jelsma
    Dec 23, 2011 at 2:19 pm
    Dec 23, 2011 at 2:19 pm
  • Hi. I'm testing Apache Nutch on Hadoop 0.22.0 and migrated from 0.20.203. Many more tasks fail for unknown reasons (they timeout) while they didn't on the other cluster that was much less high-end. I ...
    Markus JelsmaMarkus Jelsma
    Dec 22, 2011 at 2:05 pm
    Dec 22, 2011 at 2:05 pm
  • I want to try hadoop 0.22. Its possible to downgrade it later without loosing HDFS content (namenode, datanodes)?
    Radim KolarRadim Kolar
    Dec 22, 2011 at 6:57 am
    Dec 22, 2011 at 6:57 am
  • Hi guys ! If we neglect the shuffle part, can reduce phase be CPU/IO bound ? Can anyone suggest some benchmark or example where we can see this ? Arun
    Arun kArun k
    Dec 22, 2011 at 6:16 am
    Dec 22, 2011 at 6:16 am
  • Hi, I want to know where has information regarding completed MapTask been stored? i.e. how reduce task know about completed map output data is available on which tasktracker? please let me know this. ...
    Hadoop anisHadoop anis
    Dec 22, 2011 at 6:14 am
    Jan 4, 2012 at 12:22 pm
  • Hi, I had the following questions related to Yarn: [1] How does the Application Master know where the data is, to give a list to Resource Manager? Is it talking to the Name Node? [2] How does ...
    Ann PalAnn Pal
    Dec 21, 2011 at 12:06 am
    Dec 21, 2011 at 12:48 am
  • The current hadoop implementation shuffles directly to disk and then those disk files are eventually requested by the target nodes which are responsible for doing the reduce() on the intermediate ...
    Kevin BurtonKevin Burton
    Dec 20, 2011 at 11:56 pm
    Dec 21, 2011 at 8:33 am
  • Hi, When resources are allocated in Map reduce Next gen, it can be based on cpu, memory, disk and network bandwidth. Is network bandwidth the bandwidth from server to the switch (TOR) it is connected ...
    Ann PalAnn Pal
    Dec 20, 2011 at 6:57 pm
    Dec 20, 2011 at 7:28 pm
  • Hi,all I am running hadoop 0.23 on 5 nodes. I could run any YARN application or Mapreduce Job on this cluster before. But, after I changed Resourcemanager Node from node4 to node5, when I run ...
    Jingui LeeJingui Lee
    Dec 20, 2011 at 1:15 pm
    Dec 21, 2011 at 1:38 pm
  • Hey, We use capacity scheduler and divide our map slots among queues. For a particular kind of job, we want to schedule at most one task per task tracker. How does one do this? We are using Hadoop ...
    Nitin KhandelwalNitin Khandelwal
    Dec 20, 2011 at 11:47 am
    Dec 21, 2011 at 5:03 am
  • Hi, On 0.22.0 we sometimes see a shuffle phase being stuck to a point where the framework does not kill it because of lack of progress. The reducer's tasktracker log keeps filling up with two ...
    Markus JelsmaMarkus Jelsma
    Dec 20, 2011 at 7:18 am
    Dec 20, 2011 at 7:18 am
  • Moving it to mapreduce-list. Sophie, This could just be a bug a 0.23. 0.23 does not have jobtrackers/tasktrackers. Could you see if you can recreate this? If yes, please do file a jira on this. ...
    Mahadev KonarMahadev Konar
    Dec 19, 2011 at 11:44 pm
    Dec 19, 2011 at 11:44 pm
  • Hi, We have many different jobs running on a 0.22.0 cluster, each with its own memory consumption. Some jobs can easily be run with a large amount of *.tasks per job and others require much more ...
    Markus JelsmaMarkus Jelsma
    Dec 19, 2011 at 11:04 pm
    Dec 20, 2011 at 2:19 pm
  • Hi, In the hadoop MapReduce, I've executed the webdatascan example, and the reduce output is in a SequeceFile. The result is shows here ( http://paste.lisp.org/display/126572). What's the trash ...
    Pedro CostaPedro Costa
    Dec 19, 2011 at 1:55 pm
    Dec 19, 2011 at 1:55 pm
  • Hi, Apologies for cross-posting. We're in the process of migrating data from an Apache Hadoop 0.20.203.0 cluster to a 0.22.0 cluster using distcp with a hftp source and hdfs dest as described in the ...
    Markus JelsmaMarkus Jelsma
    Dec 19, 2011 at 8:24 am
    Dec 19, 2011 at 12:30 pm
  • Hi, all I got the following exception when I submit a hadoop streaming job to my hadoop cluster. I wrote the mapper in perl langguage, and there is no reducer. the mapper script runs well on local ...
    Yu YangYu Yang
    Dec 19, 2011 at 3:45 am
    Dec 19, 2011 at 4:24 pm
  • Hi, I want to read a file that has 100MB of size and it is in the HDFS. How should I do it? Is it with IOUtils.readFully? Can anyone give me an example? -- Thanks, -- Thanks,
    Pedro CostaPedro Costa
    Dec 16, 2011 at 3:58 pm
    Dec 16, 2011 at 4:52 pm
  • Hi, I had some questions specifically on the Map-Reduce phase: [1] For the reduce phase, the TaskTrackers corresponding to the reduce node, poll the Job Tracker to know about maps that have completed ...
    Ann PalAnn Pal
    Dec 16, 2011 at 1:33 pm
    Dec 17, 2011 at 3:13 pm
  • Hi, I'm migrating Apache jobs to the new MapReduce API. I came across too many issues but there's one i can't seem to figure out: SequenceFile.Reader[] readers = ...
    Markus JelsmaMarkus Jelsma
    Dec 16, 2011 at 11:28 am
    Dec 16, 2011 at 11:28 am
  • Hai guys ! I have set up 5 node cluster with each of them in different racks. I have hadoop-0.20.2 set up on my Eclipse Helios. So, i ran Tracebuilder using Main Class: ...
    Arun kArun k
    Dec 16, 2011 at 6:52 am
    Dec 16, 2011 at 3:02 pm
  • (moving to mapreduce-user@, bcc'ing common-user@) Hi Joey - You'll want to change the value on all of your servers running tasktrackers and then restart each tasktracker to reread the configuration. ...
    James WarrenJames Warren
    Dec 15, 2011 at 11:38 pm
    Dec 29, 2011 at 4:33 pm
  • I am reporting on performance of a hadoop task on a cluster with about 50 nodes. I would like to be able to report performance on clusters of 5,10,20 nodes without changing int current cluster. Is ...
    Steve LewisSteve Lewis
    Dec 15, 2011 at 10:03 pm
    Dec 16, 2011 at 12:09 am
  • I've recently come across some interesting things happening within a 50-node cluster regarding the tasktrackers and task attempts. Essentially tasks are being created but they are sticking at 0.0% ...
    John MillerJohn Miller
    Dec 15, 2011 at 3:58 pm
    Dec 28, 2011 at 9:37 pm
  • Is there a way to pass a service to the output format? I have an object which I would like to initialize/configure outside and then pass in (since it must also be used elsewhere). So far I have been ...
    Adam PortleyAdam Portley
    Dec 15, 2011 at 12:45 am
    Dec 15, 2011 at 12:45 am
  • Hi, is it possible use side-effect file using streaming (python)? If it is, how can i do it? Thanks.
    Kadu canGica EduardoKadu canGica Eduardo
    Dec 14, 2011 at 5:24 pm
    Dec 14, 2011 at 5:24 pm
  • Hi Guys ! I want to analyse the completed Job counters like FILE/HDFS BYTES READ/WRITTEN along with other values like average map/reduce task run time. I see that Jobtracker GUI has this info but i ...
    Arun kArun k
    Dec 14, 2011 at 2:40 pm
    Dec 14, 2011 at 4:06 pm
  • Hi, there. I've run into an odd situation, and I'm wondering if there's a way around it; I'm trying to use Jackson for some JSON serialization in my program, and I wrote/unit-tested it to work with ...
    John ArmstrongJohn Armstrong
    Dec 14, 2011 at 1:21 pm
    Dec 14, 2011 at 5:36 pm
  • Hi, I am trying to run a shell command from within a mapper. The shell command is of the form: * hadoop jar somjarfile arg1 arg2 ...* Can i do this type of operation from within a mapper? Also, can i ...
    Souri dattaSouri datta
    Dec 14, 2011 at 11:24 am
    Dec 14, 2011 at 9:09 pm
  • Hi Friends, I want to know, where JobTracker stores Task's Information, i.e. which task is being executed on which tasktracker, and how JobTracker stores this information. If anyone know this please ...
    Hadoop anisHadoop anis
    Dec 14, 2011 at 9:15 am
    Dec 15, 2011 at 5:40 am
  • Hi, Can anyone give the procedure about how to run Distibuted shell example in hadoop yarn.So that i try to understand how applicatin master really works.
    Sri ramSri ram
    Dec 14, 2011 at 9:07 am
    Dec 16, 2011 at 2:23 am
  • Moving to mapreduce-user@, bcc common-user@. Please use project specific lists. Take a look at JobTracker.heartbeat - *Scheduler.assignTasks. After the scheduler 'assigns' tasks, the JT sends the ...
    Arun C MurthyArun C Murthy
    Dec 13, 2011 at 7:12 pm
    Dec 13, 2011 at 7:12 pm
  • HI guys ! I have a single node set up as per http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/ 1 I have put some sysout statements in Jobtracker and wordcount ...
    Arun kArun k
    Dec 13, 2011 at 2:37 pm
    Dec 13, 2011 at 2:37 pm
  • Hi, I receive the following error while starting datanode in secure mode of hadoop 0.23 2011-12-14 14:35:48,468 INFO http.HttpServer (HttpServer.java:addGlobalFilter(476)) - Added global filter ...
    Sri ramSri ram
    Dec 13, 2011 at 9:27 am
    Dec 13, 2011 at 9:27 am
  • Hi guys ! I have set up a single node cluster as per below link http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/#run-the-mapreduce-job I have tried to run ...
    Arun kArun k
    Dec 13, 2011 at 6:53 am
    Dec 13, 2011 at 6:53 am
  • Hi guys ! I have set up cluster according to http://ankitasblogger.blogspot.com/2011/01/hadoop-cluster-setup.html I have set topology.script.file.name in core-site.xml I tried this simple test ...
    Arun kArun k
    Dec 13, 2011 at 6:40 am
    Dec 13, 2011 at 6:40 am
  • Hi Hadoop users, In my company we have been using Hadoop for 2 years and we have need to pause and resume map reduce jobs. I was searching on Hadoop JIRA and there are couple of tickets which are not ...
    Dino KečoDino Kečo
    Dec 13, 2011 at 12:44 am
    Dec 13, 2011 at 1:40 am
  • Hi, I am trying to form a hadoop cluster of 0.23 version in secure mode. While starting nodemanager i get the following error 2011-12-12 15:37:26,874 INFO ipc.HadoopYarnRPC ...
    Sri ramSri ram
    Dec 12, 2011 at 10:15 am
    Dec 12, 2011 at 5:22 pm
Dec 2011
12
Group Navigation
period‹ prev | Dec 2011 | next ›
Group Overview
groupmapreduce-user @
categorieshadoop
discussions77
posts249
users66
websitehadoop.apache.org...
irc#hadoop

66 users for December 2011

Harsh J: 23 posts Markus Jelsma: 21 posts Arun C Murthy: 20 posts Arun k: 19 posts Praveen Sripati: 10 posts Robert Evans: 9 posts Eyal Golan: 8 posts Bejoy Ks: 7 posts Kevin Burton: 6 posts Mahadev Konar: 6 posts Raghavendhra rahul: 6 posts Avery Ching: 5 posts John Miller: 5 posts Nitin Khandelwal: 5 posts Ann Pal: 4 posts Costin Leau: 4 posts Hadoop anis: 4 posts Keren Ouaknine: 4 posts Sri ram: 4 posts Bing Jiang: 3 posts
show more
Archives