FAQ

Hadoop > hdfs-user > March 2011

Search Discussions

32 discussions - 110 posts

  • Let's say an HDFS client starts writing a file A (which is 10 blocks long) and 5 blocks have been writen to datanodes. At this time, if the HDFS client crashes (apparently without a close op), will ...
    Sean BigdatafunSean Bigdatafun
    Mar 13, 2011 at 4:52 pm
    Apr 25, 2011 at 2:12 am
  • I have a 2 rack cluster. All of my files have a replication factor of 2. How does hdfs determine what node to use when serving the data? Does it always use the first rack? or is there an algorithm ...
    RitaRita
    Mar 10, 2011 at 3:08 am
    Mar 14, 2011 at 4:29 pm
  • We have a bad disk on one of our datanode machines, and while we have dfs.datanode.failed.volumes.tolerated set to 2 and didn't see any problem while the DataNode process was running we are seeing a ...
    Adam PhelpsAdam Phelps
    Mar 24, 2011 at 5:31 pm
    Mar 25, 2011 at 4:49 pm
  • Parts of this may end up on the hbase list, but I thought I'd start here. My basic problem is: My cluster is getting full enough that having one data node go down does put a bit of pressure on the ...
    Stuart SmithStuart Smith
    Mar 17, 2011 at 7:14 pm
    Mar 24, 2011 at 9:01 pm
  • It's unsynchronized however it's creating a new BlockReader on each call, that seems like a problem?
    Jason RutherglenJason Rutherglen
    Mar 2, 2011 at 12:26 am
    Mar 2, 2011 at 11:48 pm
  • Hi, If I have is big gzip files ( block size) does the M/R will split a single file to multiple blocks and send them to different mappers ? The behavior I currently see is that a map is still open ...
    Lior SchachterLior Schachter
    Mar 17, 2011 at 1:10 pm
    Mar 17, 2011 at 4:15 pm
  • Using 0.21 When I have a filesystem (XFS) with 1TB it detects the datanode detects it immediately. When I create 3 identical file systems all 3TB are visible immediately. If I create a 6TB filesystem ...
    RitaRita
    Mar 26, 2011 at 4:07 am
    Mar 29, 2011 at 12:31 pm
  • Hi Everyone, How can I do a md5sum/sha1sum directly against files on HDFS ? A pretty common thing I do when archiving files is make an md5sum list eg) md5sum /archive/path/* md5sum-list.txt Then ...
    Scott GolbyScott Golby
    Mar 2, 2011 at 5:04 pm
    Mar 8, 2011 at 2:50 pm
  • If I'm using a replication factor of 3 and I write a block of data will my write operation block until the data is present on 3 nodes? or will it block until the data is present on 1 node and ...
    Andrew RothsteinAndrew Rothstein
    Mar 17, 2011 at 4:30 pm
    Mar 22, 2011 at 10:10 pm
  • Hi Everyone, I make a Project with Hadoop-MapRedeuce for my master-Thesis. I have a strange problem on my System. First of all, I use Hadoop-0.20.2 on Windows XP Pro with Eclipse Plug-In. When I ...
    Baran cakiciBaran cakici
    Mar 16, 2011 at 4:53 pm
    Mar 16, 2011 at 9:26 pm
  • All of my data nodes have 16GB of memory. Would increasing my Java virtual memory help in performance? -- --- Get your facts first, then you can distort them as you please.--
    RitaRita
    Mar 9, 2011 at 12:24 pm
    Mar 10, 2011 at 2:19 am
  • I would like to build a fast dataquery system. Basically I have several terabytes of time data I would like to analyze and I was wondering if hbase is the right tool? Currently, I have a hdfs cluster ...
    RitaRita
    Mar 8, 2011 at 1:16 pm
    Mar 9, 2011 at 3:10 am
  • Hi all, How "atomic" is the copyFromLocal call? i.e. on process is in the midst of uploading a file to HDFS is it possible for another process to start reading it before the upload is complete? I am ...
    Ishaaq ChandyIshaaq Chandy
    Mar 1, 2011 at 4:23 am
    Mar 1, 2011 at 9:30 pm
  • I'm interested in hearing how you get data into and out of HDFS. Are you using tools like Flume? Are you using fuse_dfs? Are you putting files on HDFS with "hadoop dfs -put ..."? And how does your ...
    EricEric
    Mar 29, 2011 at 1:21 pm
    Mar 29, 2011 at 2:22 pm
  • Hello all .. I write some programs to hadoop here .. and i had a problem with writeUTF .. It put a null in every line begin . See in attachement .. Whats happen? How to remove this null ? thanks
    Alessandro BinharaAlessandro Binhara
    Mar 25, 2011 at 1:37 pm
    Mar 25, 2011 at 5:08 pm
  • Hi all, size of a newly created file is reported to be zero even though I've written some data and hsync-ed them. Is that correct and expected effect? hadoop fs -cat will retrieve the data correctly. ...
    Viliam HolubViliam Holub
    Mar 18, 2011 at 4:30 pm
    Mar 19, 2011 at 2:15 am
  • Hi, We plan a 100T cluster with M/R jobs running on 15G gzip files. Should we configure HDFS block to be 128M or 256M. Thanks, Lior
    Lior SchachterLior Schachter
    Mar 17, 2011 at 8:58 am
    Mar 17, 2011 at 9:22 am
  • Hello, As far as I understand, since "hadoop fs -du" command uses Linux' "du" internally this mean that the number of replicas (at the moment of command run) affect the result. Is that correct? I ...
    Alex BaranauAlex Baranau
    Mar 14, 2011 at 2:35 pm
    Mar 15, 2011 at 7:51 pm
  • We see a lot of IOExceptions coming from HDFS during a job that does nothing but untar 100 files (1 per Mapper, sizes vary between 5GB and 80GB) that are in HDFS, to HDFS. DataNodes are also showing ...
    Evert LammertsEvert Lammerts
    Mar 9, 2011 at 11:30 am
    Mar 10, 2011 at 8:06 am
  • A better choice would likely be hdfs-user@, since this is really about watching files in HDFS. Two ways off the top of my head: 1) Read/watch the edits stream 2) Read/watch the HDFS audit log Given ...
    Allen WittenauerAllen Wittenauer
    Mar 25, 2011 at 4:47 pm
    Mar 29, 2011 at 9:00 am
  • Is there a way to determine how many complete replicas there are of a file? That is, can I find out that the async replication has completed? And how many replicas are complete? I'd like to do this ...
    Martin SerranoMartin Serrano
    Mar 28, 2011 at 8:18 pm
    Mar 28, 2011 at 8:29 pm
  • I download cloudera CDH3 beta:hadoop-0.20.2+228,and modified three files:hdfs.xml,core-site.xml and hadoop-env.sh.and I do have set JAVA_HOME in file:hadoop-env.sh,and then try to ...
    Alex LuyaAlex Luya
    Mar 16, 2011 at 3:20 pm
    Mar 16, 2011 at 3:44 pm
  • Hi, We have been using hadoop/HDSF for some while and we want to upgrade the namenode server to a new (and stronger) machine. Can you please advise on the correct way to do this while minimizing ...
    Lior SchachterLior Schachter
    Mar 16, 2011 at 2:18 pm
    Mar 16, 2011 at 2:30 pm
  • Hello ... I have a servlet on tomcat.. and it open a hdfs and write simple file with a content of post information. Well , in first test we had a 14.000 request per second. My servet start many trads ...
    Alessandro BinharaAlessandro Binhara
    Mar 14, 2011 at 5:49 pm
    Mar 14, 2011 at 6:03 pm
  • Hello, I am getting the below error while lunching the NameNode 2011-03-12 21:39:00,831 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemStatusMBean 2011-03-12 ...
    Clement JebakumarClement Jebakumar
    Mar 13, 2011 at 2:42 am
    Mar 13, 2011 at 3:04 am
  • Hi there, We've just set up a Namenode that backups its metadata (as recommended) on an NFS store. Since this NFS-Mount is not a netapp appliance and can go down anytime, I just checked what happens ...
    Tobias SchlottkeTobias Schlottke
    Mar 9, 2011 at 3:08 pm
    Mar 9, 2011 at 5:16 pm
  • I want to parse XML file in hadoop. I have my own mapper class called "*MyXMLMapper*"... $ ./bin/hadoop jar hadoop-streaming.jar -inputreader "StreamXmlRecordReader,begin='<Page',end='</Page '" -file ...
    Clement JebakumarClement Jebakumar
    Mar 6, 2011 at 6:52 pm
    Mar 6, 2011 at 7:32 pm
  • Hi all, A newbie question reg backupnode . I just started the namenode and backupnode and in backupnode web UI it shows "Upgrade for version -24 has been completed. Upgrade is not finalized". I did ...
    Gokulakannan MGokulakannan M
    Mar 22, 2011 at 1:32 pm
    Mar 22, 2011 at 1:32 pm
  • Got this error,configuration is : I have checked out source from:https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-append/ this is my build script: ...
    Alex LuyaAlex Luya
    Mar 19, 2011 at 2:59 pm
    Mar 19, 2011 at 2:59 pm
  • Alex LuyaAlex Luya
    Mar 16, 2011 at 3:08 pm
    Mar 16, 2011 at 3:08 pm
  • Hi, I am trying to write data to Seq File from hdfs client on a box outside of HDFS. When I create a Sequence.Write output object in a java program running on this box, can I call output.append(key, ...
    Mapred LearnMapred Learn
    Mar 11, 2011 at 6:21 pm
    Mar 11, 2011 at 6:21 pm
  • Hello, I am facing problems in running fuse-dfs over hdfs. I came across this thread while searching for my problem: http://www.mail-archive.com/hdfs-user@hadoop.apache.org/msg00341.html OR ...
    Aastha MehtaAastha Mehta
    Mar 1, 2011 at 5:51 pm
    Mar 1, 2011 at 5:51 pm
Mar 2011
Group Navigation
period‹ prev | Mar 2011 | next ›
Group Overview
grouphdfs-user @
categorieshadoop
discussions32
posts110
users40
websitehadoop.apache.org...
irc#hadoop

40 users for March 2011

Stuart Smith: 12 posts Rita: 9 posts Harsh J: 8 posts Allen Wittenauer: 6 posts Lior Schachter: 6 posts Todd Lipcon: 5 posts Eric: 4 posts Adam Phelps: 3 posts Alessandro Binhara: 3 posts Alex Luya: 3 posts Ayon Sinha: 3 posts Jason Rutherglen: 3 posts Ted Dunning: 3 posts Alex Baranau: 2 posts Andrew Rothstein: 2 posts Baran cakici: 2 posts Bharath Mundlapudi: 2 posts Clement Jebakumar: 2 posts Elton sky: 2 posts Evert Lammerts: 2 posts
show more
Archives