|| at Oct 14, 2010 at 12:45 am
Oh, now i see the problem.
The implication here is that some blocks might not be
scanned for every long time, because the scanner
may not finish scan all the blocks during 3 weeks,
then after that, it start over again, ...
Interesting, thanks for prompt reply, Brian.
On Wed, Oct 13, 2010 at 7:37 PM, Brian Bockelman wrote:
On Oct 13, 2010, at 7:29 PM, Thanh Do wrote:
If this is the case, then is there any chance that,
some how the DataBlockScanner cannot finishes
the verification for all the block in three weeks
(e.g, a node has a very large number of blocks)?
Yes. At some point, I'd really like to figure out what percentage of our
blocks actually get scanned at our site, I suspect some go very long without
On Wed, Oct 13, 2010 at 7:18 PM, Brian Bockelman <email@example.com
That is correct. Last time I read the code, Hadoop scheduled the block
verifications randomly throughout the period in order to avoid periodic
effects (i.e., high load every N minutes).
On Oct 13, 2010, at 7:14 PM, Thanh Do wrote:
When you say *attempt* to complete and *entire* node scan,
you mean for example, if a node has 100 block files, it will
try to verify all 100 block every 3 weeks?
That is in average, a block is scanned every (3 weeks / 100 time
On Wed, Oct 13, 2010 at 7:07 PM, Brian Bockelman <firstname.lastname@example.org
The scan period is the period that hadoop *attempts* to complete an
node scan. That is, if it's set to 3 weeks, HDFS will try to scan
block once every 3 weeks.
Obviously, depending on the bandwidth you have made available to the
scanning thread, you can specify impossibly small periods.
On Oct 13, 2010, at 7:01 PM, Thanh Do wrote:
Could any body explain to me about the scanning period
policy of DataBlockScanner? That is who often it wake up
and scan a block file.
When looking at the code, I found
static final long DEFAULT_SCAN_PERIOD_HOURS = 21*24L; // three weeks
but definitely it does not wake up and pick a random block
to verify every three weeks, right?
Thanks a lot,