FAQ
I want to write to a common hdfs file from within my map method. Given that each task runs in a separate jvm (on separate machines) making a method syncronized will not work I assume. Are there any file locking or other methods to guarantee mutual exclusion on hdfs?

(I want to append to this file and I have the append option turned on.) Thanks.

Search Discussions

  • Bill Habermaas at May 7, 2010 at 2:48 pm
    I had a similar requirement. Hdfs has no locking that I am aware of, at
    least I have never run across it in reading the source. My solution was to
    build a distributed locking mechanism using ZooKeeper. You might want to
    visit http://hadoop.apache.org/zookeeper/docs/current/recipes.html
    For some ideas. The code you find there is a start but buggy.

    Bill

    -----Original Message-----
    From: Raymond Jennings III
    Sent: Friday, May 07, 2010 10:32 AM
    To: common-user@hadoop.apache.org
    Subject: How can I syncronize writing to an hdfs file

    I want to write to a common hdfs file from within my map method. Given that
    each task runs in a separate jvm (on separate machines) making a method
    syncronized will not work I assume. Are there any file locking or other
    methods to guarantee mutual exclusion on hdfs?

    (I want to append to this file and I have the append option turned on.)
    Thanks.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedMay 7, '10 at 2:32p
activeMay 7, '10 at 2:48p
posts2
users2
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase