I had a similar requirement. Hdfs has no locking that I am aware of, at
least I have never run across it in reading the source. My solution was to
build a distributed locking mechanism using ZooKeeper. You might want to
For some ideas. The code you find there is a start but buggy.
From: Raymond Jennings III
Sent: Friday, May 07, 2010 10:32 AM
Subject: How can I syncronize writing to an hdfs file
I want to write to a common hdfs file from within my map method. Given that
each task runs in a separate jvm (on separate machines) making a method
syncronized will not work I assume. Are there any file locking or other
methods to guarantee mutual exclusion on hdfs?
(I want to append to this file and I have the append option turned on.)