FAQ
Hi,
I do not clearly understand how the dfs compare blocks.

The blockId is randomly generated but the compare method looks like:

if (getBlockId() < b.getBlockId()) {
return -1;
} else if (getBlockId() == b.getBlockId()) {
return 0;
} else {
return 1;
}

That wouldn't be a big problem if this would be used in the
FSNamesystem#processBlock
if (cmp == 0) {
// Do nothing, blocks are the same
...
} else if (cmp < 0) {
// The old report has a block the new one does not
removeStoredBlock(oldReport[oldPos], node);
...
} else {
// The new report has a block the old one does not
addStoredBlock(newReport[newPos], node);

I currently just miss some glue how result of a compare of two random
numbers can be used to decide if a block can be removed or not.

Any hints what I'm missing?

Thanks.
Stefan



---------------------------------------------
blog: http://www.find23.org
company: http://www.media-style.com

Search Discussions

  • Andrzej Bialecki at Mar 27, 2006 at 10:25 am

    Stefan Groschupf wrote:
    Hi,
    I do not clearly understand how the dfs compare blocks.
    I currently just miss some glue how result of a compare of two random
    numbers can be used to decide if a block can be removed or not.

    Any hints what I'm missing?
    Block reports are sorted according to the same compateTo (in
    TreeSet.toArray()). It's true that IDs are random, but if both oldReport
    and newReport lists are sorted, like this:

    OLD NEW
    1 2
    2 3
    3 4
    5 5

    Then the sequence of compares and pointer advances looks like this

    old[oldPos=0].compareTo(new[newPos=0]) == -1; remove(old[0]); oldPos++;
    old[oldPos=1].compareTo(new[newPos=0]) == 0; /* nothing */; oldPos++;
    newPos++;
    old[oldPos=2].compareTo(new[newPos=1]) == 0; /* nothing */; oldPos++;
    newPos++;
    old[oldPos=3].compareTo(new[newPos=2]) == 1; add(new[2]); newPos++;
    old[oldPos=3].compareTo(new[newPos=3]) == 0; /* nothing */; oldPos++;
    newPos++;

    --
    Best regards,
    Andrzej Bialecki <><
    ___. ___ ___ ___ _ _ __________________________________
    [__ || __|__/|__||\/| Information Retrieval, Semantic Web
    ___|||__|| \| || | Embedded Unix, System Integration
    http://www.sigram.com Contact: info at sigram dot com
  • Stefan Groschupf at Mar 27, 2006 at 12:20 pm
    Hi Andrzej,
    I see!
    Thanks for the hint.
    Stefan

    Am 27.03.2006 um 12:24 schrieb Andrzej Bialecki:
    Stefan Groschupf wrote:
    Hi,
    I do not clearly understand how the dfs compare blocks.
    I currently just miss some glue how result of a compare of two
    random numbers can be used to decide if a block can be removed or
    not.

    Any hints what I'm missing?
    Block reports are sorted according to the same compateTo (in
    TreeSet.toArray()). It's true that IDs are random, but if both
    oldReport and newReport lists are sorted, like this:

    OLD NEW
    1 2
    2 3
    3 4
    5 5

    Then the sequence of compares and pointer advances looks like this

    old[oldPos=0].compareTo(new[newPos=0]) == -1; remove(old[0]); oldPos
    ++;
    old[oldPos=1].compareTo(new[newPos=0]) == 0; /* nothing */; oldPos+
    +; newPos++;
    old[oldPos=2].compareTo(new[newPos=1]) == 0; /* nothing */; oldPos+
    +; newPos++;
    old[oldPos=3].compareTo(new[newPos=2]) == 1; add(new[2]); newPos++;
    old[oldPos=3].compareTo(new[newPos=3]) == 0; /* nothing */; oldPos+
    +; newPos++;

    --
    Best regards,
    Andrzej Bialecki <><
    ___. ___ ___ ___ _ _ __________________________________
    [__ || __|__/|__||\/| Information Retrieval, Semantic Web
    ___|||__|| \| || | Embedded Unix, System Integration
    http://www.sigram.com Contact: info at sigram dot com

    ---------------------------------------------------------------
    company: http://www.media-style.com
    forum: http://www.text-mining.org
    blog: http://www.find23.net

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedMar 26, '06 at 11:49p
activeMar 27, '06 at 12:20p
posts3
users2
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase