FAQ

Search Discussions

50 discussions - 184 posts

  • I loaded data into HDFS last week, and this morning I was greeted with this on the web interface: "WARNING : There are about 32 missing blocks. Please check the log or run fsck." I ran fsck and see ...
    Time LessTime Less
    May 18, 2011 at 12:13 am
    May 30, 2011 at 5:45 am
  • Hi, We've performed tests for ext3 and xfs filesystems using different settings. The results might be useful for anyone else. The datanode cluster consists of 15 slave nodes, each equipped with 1Gbit ...
    Ferdy GalemaFerdy Galema
    May 5, 2011 at 8:46 pm
    May 10, 2011 at 9:25 pm
  • Say I add a datanode to a pseudo cluster and I want to change the replication factor to 2. I see that I can either run hadoop fs -setrep or change the hdfs-site.xml value for dfs.replication. But do ...
    Steve CohenSteve Cohen
    May 18, 2011 at 10:33 pm
    May 19, 2011 at 5:55 pm
  • Hi Folks, We have asked this question in common-users hadoop mail list, but not resolved for a week. We try to get hbase and hadoop running on clusters, take 2 Solaris servers(also tried 1 linux, 1 ...
    Xu, RichardXu, Richard
    May 31, 2011 at 2:15 pm
    May 31, 2011 at 5:31 pm
  • Hi,all I set dfs.name.dir to a comma-delimited list of directories, dir1 is in /dev/sdb1 dir2 is in /dev/sdb2 and dir3 is nfs derectory. What happens if /dev/sdb1 disk error, so dir1 cannot be read ...
    CcxixiccCcxixicc
    May 25, 2011 at 6:39 am
    May 26, 2011 at 4:42 am
  • Is there a way to send a request to the name node to replicate block(s) to a specific DataNode? If not, what would be a way to do this? -Thanks
    Jason RutherglenJason Rutherglen
    May 26, 2011 at 6:37 pm
    May 27, 2011 at 3:49 pm
  • Did you check the requirements for that release? I don´t know if this version require at least a mayor version to 1.6.20. Did you test with the 1.6.24? I think that can be a bug. Take a time to ...
    Marcos OrtizMarcos Ortiz
    May 20, 2011 at 9:01 pm
    May 22, 2011 at 5:57 am
  • We have an 11 node Hadoop cluster running 20.2 that has been in production for 15 months now. The system is used to process log files that are ingested daily, and the oldest files in the HDFS are ...
    Kester, ScottKester, Scott
    May 13, 2011 at 5:41 pm
    May 16, 2011 at 3:51 pm
  • Hi all, is the read operation of 1 file stored in hdfs done in parallel? I mean let's say that I have 1 file split in 2 blocks (hdfs block) and each block is stored in 1 rack. When reading this file, ...
    Hassen RiahiHassen Riahi
    May 7, 2011 at 3:51 pm
    May 12, 2011 at 11:09 am
  • hi all, perhaps this is a dummy question but can anyone tell me that when the namenode saves a fsimage, are the Inodes saved in an alphabetical order? thanks Thanh
    Thanh DoThanh Do
    May 2, 2011 at 8:20 pm
    May 3, 2011 at 6:45 am
  • Hi All, I am just reading the book *Hadoop: The Definitive Guide *by Tom White, there is an example for the *FileSystem* class in chapter 3, example3-2: ...
    Yang XiaoliangYang Xiaoliang
    May 22, 2011 at 2:24 pm
    May 22, 2011 at 5:36 pm
  • hi all, I am using hdfs 0.21.0, and want to run Backup Node in the same host as the NameNode for experiment purpose. Can anyone shed some light on how to do that? Many thanks, Thanh
    Thanh DoThanh Do
    May 10, 2011 at 1:07 am
    May 12, 2011 at 2:18 am
  • Hi All, I need some help for my project. I plan to develop a Java program to manipulate remote HDFS. The code cannot pass the compilation with throwing some errs like "cannot find symbol FileSystem". ...
    Bo LiBo Li
    May 6, 2011 at 7:44 pm
    May 9, 2011 at 6:29 pm
  • 8 DataNodes (16-core CPU && 32G memory && 1000M NET CARD) 1 NameNode (16-core CPU && 32G memory && 1000M NET CARD) I really want to know how to make full use of the cluster . Some advice ? thank you. ...
    Fei PanFei Pan
    May 7, 2011 at 5:32 pm
    May 7, 2011 at 6:39 pm
  • Hello all, I was running HDFS in standalone mode where the machine got accidently turned off. On resuming it back to normal, I got this exception: =================================================== ...
    Himanshu VashishthaHimanshu Vashishtha
    May 4, 2011 at 6:34 pm
    May 4, 2011 at 8:53 pm
  • Hi guys, I asked this question earlier but did not get any response. So, posting again. Hope somebody can point to the right description: When you do hadoop fs -copyFromLocal or use API to call ...
    Mapred LearnMapred Learn
    May 31, 2011 at 11:57 pm
    Jun 3, 2011 at 10:32 am
  • Hi all I'm doing a test and need create lots of files ( 100 million ) in HDFS, I use a shell script to do this , it's very very slow, how to create a lot files in HDFS quickly? Thanks
    CcxixiccCcxixicc
    May 30, 2011 at 2:45 am
    May 30, 2011 at 3:51 pm
  • Resending ====
    Mapred LearnMapred Learn
    May 25, 2011 at 2:09 pm
    May 25, 2011 at 3:00 pm
  • I will preface this with a couple statements: a) it's almost 6am, and I've been up all night b) I'm drugged up from an allergic reaction, so I may not be firing on all 64 bits. Do I correctly ...
    Jonathan DisherJonathan Disher
    May 10, 2011 at 12:50 pm
    May 10, 2011 at 9:22 pm
  • using java new File("/tmp/common") but /tmp/common is a HDFS file how to implement this feature? thanks
    LtomunoLtomuno
    May 9, 2011 at 3:45 am
    May 9, 2011 at 5:49 am
  • Hi,all I found NameNode often lost heartbeat from DataNodes: org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.heartbeatCheck: lost heartbeat from 192.168.1.101:50010 ...
    CcxixiccCcxixicc
    May 31, 2011 at 3:17 am
    May 31, 2011 at 7:44 pm
  • Directory Modified Time is earlier than the files it contains for some of my folders on HDFS. Is this usual? I would like to be able to hadoop fs -rmr /path/to/some/dir if the dirs modified time is ...
    Tom HallTom Hall
    May 24, 2011 at 3:28 pm
    May 24, 2011 at 11:41 pm
  • hi hdfs users, Is anybody aware of a system that is similar to HDFS, in the sense that it has single master architecture, and the master also keeps an operation log. Thanks, Thanh
    Thanh DoThanh Do
    May 19, 2011 at 2:03 am
    May 19, 2011 at 2:29 am
  • Hello, We are running an hdfs cluster and we decided we wanted to add a new datanode. Since we are using a virtual machine, we just cloned an existing datanode. We added it to the slaves list and ...
    Steve CohenSteve Cohen
    May 11, 2011 at 9:00 pm
    May 11, 2011 at 10:33 pm
  • Is it possible to enforce a replication of 2 for a single node, so that replicas are spread out over disks? Currently with more replicas than nodes this results in "under-replicated" blocks. I ...
    Ferdy GalemaFerdy Galema
    May 9, 2011 at 11:24 am
    May 10, 2011 at 5:00 pm
  • hi all, any body deploy the Backup Node in your system. I am curious about the impact of the Backup Node to the NameNode throughput. To my understanding, NameNode streams edits log operation to the ...
    Thanh DoThanh Do
    May 5, 2011 at 4:28 pm
    May 6, 2011 at 12:56 pm
  • Hi, Can anyone point me to a doc describing how to port/use another clustered FS? Thanks. Anh-
    Anh NguyenAnh Nguyen
    May 4, 2011 at 5:01 pm
    May 6, 2011 at 7:44 am
  • I am writing multiple files using multiple FSOutputStreams through different threads in HDFS. All the files are getting written properly and I see that namenode and datanode logs have no error. The ...
    Sudhanshu aroraSudhanshu arora
    May 28, 2011 at 2:57 am
    May 30, 2011 at 5:58 am
  • Hi,all I'm testing hadoop with two secondary namenode ( I'm using hadoop-0.20.2) , and get some errors. So I did a few search. But first I got confused about SNN: From ...
    CcxixiccCcxixicc
    May 26, 2011 at 2:54 am
    May 26, 2011 at 3:42 am
  • Hi all I Googled 'namenode high available', and found the AvatarNode, but I cannot found the setup and configuration to get start with AvatarNode. Can anybody help? And are there any other approaches ...
    XixiXixi
    May 23, 2011 at 1:31 am
    May 23, 2011 at 3:44 am
  • hey all, I want to run a experimental cluster but my machines have limited disk space capacity. I want each node in my cluster to have around 50,000 thousand blocks. I don't want to have smaller the ...
    Thanh DoThanh Do
    May 16, 2011 at 2:42 pm
    May 16, 2011 at 4:24 pm
  • Hi list, I notice that whenever our Hadoop installation is put under a heavy load we lose one or two (on a total of five) datanodes. This results in IOExceptions, and affects the overall performance ...
    Evert LammertsEvert Lammerts
    May 11, 2011 at 9:26 am
    May 11, 2011 at 6:51 pm
  • Hi Ayon, Bobby, Thanks for the response. You mentioned that we read only one block at a time and keep it in client’s memory, but I have a few queries: · What if I am reading an image, the entire ...
    Shreya ChakravartyShreya Chakravarty
    May 4, 2011 at 5:39 am
    May 4, 2011 at 4:24 pm
  • Hello, I run hadoop jar hadoop-0.20.2-test.jar TestDFSIO -write -nrFiles 1000 -fileSize 128, because wanted to know what is the througput of my cluster (how many data per sec its able to write). Here ...
    PobPob
    May 3, 2011 at 2:39 pm
    May 3, 2011 at 3:42 pm
  • Hi, I have extracted the hadoop-0.20.2, hadoop-0.20.203.0 and hadoop-0.21.0 files. In the hadoop-0.21.0 folder the hadoop-hdfs-0.21.0.jar, hadoop-mapred-0.21.0.jar and the hadoop-common-0.21.0.jar ...
    Praveen SripatiPraveen Sripati
    May 30, 2011 at 1:46 pm
    May 30, 2011 at 1:46 pm
  • Hi All, I am running a process to extract feature vectors from images and write as SequenceFiles <text, DenseVector on HDFS. My dataset of images is very large (~half a million images). The writing ...
    Lokendra SinghLokendra Singh
    May 27, 2011 at 5:19 am
    May 27, 2011 at 5:19 am
  • I agree. Specially people like Harsh who are always there to answer everyone's queries !
    Mapred LearnMapred Learn
    May 26, 2011 at 4:11 pm
    May 26, 2011 at 4:11 pm
  • Hi, I have a question about practical limit on number of files per hdfs directory. (what's the hard limit btw?) What is a practical limit on a # of files in a hadoop directory so that glob selection ...
    Dmitriy LyubimovDmitriy Lyubimov
    May 19, 2011 at 9:51 pm
    May 19, 2011 at 9:51 pm
  • System: HDFS dirs across cluster as dataone, datatwo, datathree I recently had an issue where I lost a slave that resulted in a large amount of under replicated blocks. The replication was quite slow ...
    HighpointeHighpointe
    May 19, 2011 at 1:38 am
    May 19, 2011 at 1:38 am
  • Hello, We are running nutch in a hdfs distributed filesystem. on occasion, we get errors like: 2011-05-18 02:05:42,132 WARN hdfs.DFSClient - NotReplicatedYetException sleeping ...
    Steve CohenSteve Cohen
    May 18, 2011 at 6:57 pm
    May 18, 2011 at 6:57 pm
  • Hi. We are using a cluster of 2 computers (1 namenode and 2 secondarynodes) to store a large number of text files in the HDFS. The process had been running for atleast a couple of weeks when suddenly ...
    Vishaal JatavVishaal Jatav
    May 18, 2011 at 11:17 am
    May 18, 2011 at 11:17 am
  • Hi there, Apologies if this comes through twice but i sent the mail a few hours ago and haven't seen it on the mailing list. I'm experiencing some unusual behaviour on our 0.20.2 hadoop cluster. ...
    Sidney SimmonsSidney Simmons
    May 12, 2011 at 9:56 pm
    May 12, 2011 at 9:56 pm
  • Hi there, I'm experiencing some unusual behaviour on our 0.20.2 hadoop cluster. Randomly (periodically), we're getting "Call to namenode" failures on tasktrackers causing tasks to fail: 2011-05-12 ...
    Sidney SimmonsSidney Simmons
    May 12, 2011 at 7:12 pm
    May 12, 2011 at 7:12 pm
  • Hi, I get error like: java.lang.NullPointerException at org.apache.hadoop.io<http://org.apache.hadoop.io.serializer.serializationfactory.ge/ ...
    Mapred LearnMapred Learn
    May 11, 2011 at 5:01 am
    May 11, 2011 at 5:01 am
  • Hi all, Cheers to Hadoop, I was trying to understand Behavioral targeting for large data, though it is well explained in this document<http://www.cc.gatech.edu/~zha/CSE8801/ad/p209-chen.pdf , but it ...
    Sagar KohliSagar Kohli
    May 9, 2011 at 10:51 am
    May 9, 2011 at 10:51 am
  • Does anybody know what might be the root cause of this exception: 11/05/08 17:07:28 WARN hdfs.DFSClient: Problem renewing lease for DFSClient_1994238103 for a period of 0 seconds. Shutting down HDFS ...
    Viliam HolubViliam Holub
    May 9, 2011 at 2:03 am
    May 9, 2011 at 2:03 am
  • hey folks, BerlinBuzzwords 2011 is close only 33 days left until the big Search, Store and Scale opensource crowd is gathering in Berlin on June 6th/7th. The conference again focuses on the topics ...
    Simon WillnauerSimon Willnauer
    May 4, 2011 at 7:39 am
    May 4, 2011 at 7:39 am
  • Hi, I started off with a cluster of 8 nodes and did some benchmarks. Now I wanted to see how my application scales and decided to decommission 6= nodes an run the same benchmarks. That went quite ...
    Lai WillLai Will
    May 3, 2011 at 8:11 pm
    May 3, 2011 at 8:11 pm
  • Hi, I want to know I/O Performance of my Hadoop Cluster. Because of that I ran test.jar, hier is my Results; ----- TestDFSIO ----- : write Date & time: Mon May 02 14:38:29 CEST 2011 Number of files: ...
    Baran cakiciBaran cakici
    May 3, 2011 at 10:26 am
    May 3, 2011 at 10:26 am
  • any comments??? 2011/4/28 baran cakici <barancakici@gmail.com
    Baran cakiciBaran cakici
    May 2, 2011 at 11:38 am
    May 2, 2011 at 11:38 am
Group Navigation
period‹ prev | May 2011 | next ›
Group Overview
grouphdfs-user @
categorieshadoop
discussions50
posts184
users66
websitehadoop.apache.org...
irc#hadoop

66 users for May 2011

Thanh Do: 15 posts Harsh J: 12 posts Todd Lipcon: 10 posts Steve Cohen: 9 posts Jonathan Disher: 8 posts Joey Echeverria: 6 posts Marcos Ortiz Valmaseda: 6 posts Ccxixicc: 5 posts Mapred Learn: 5 posts Matthew Foley: 5 posts Time Less: 5 posts Allen Wittenauer: 4 posts Fei Pan: 4 posts Will Maier: 4 posts Xu, Richard: 4 posts Bharath Mundlapudi: 3 posts Bo Li: 3 posts Ferdy Galema: 3 posts Hassen Riahi: 3 posts Jason Rutherglen: 3 posts
show more