FAQ
I was just wondering the other day:

What if the the values for a key that get passed into the reducer do
not fit into memory?
After all a reducer should get all values per key from the whole job.
Is the iterator disk backed?

cheers
--
Torsten

Search Discussions

  • Owen O'Malley at Jun 22, 2010 at 5:48 am

    On Jun 21, 2010, at 5:14 PM, Torsten Curdt wrote:

    I was just wondering the other day:

    What if the the values for a key that get passed into the reducer do
    not fit into memory?
    After all a reducer should get all values per key from the whole job.
    Is the iterator disk backed?
    There is no assumption that all of the values fit into memory. The
    iterator is really the
    result of a merge sort from disk and/or memory.

    -- Owen=
  • Torsten Curdt at Jun 22, 2010 at 7:05 am
    Cool. Great :)
    On Tue, Jun 22, 2010 at 07:47, Owen O'Malley wrote:
    On Jun 21, 2010, at 5:14 PM, Torsten Curdt wrote:

    I was just wondering the other day:

    What if the the values for a key that get passed into the reducer do
    not fit into memory?
    After all a reducer should get all values per key from the whole job.
    Is the iterator disk backed?
    There is no assumption that all of the values fit into memory. The iterator
    is really the
    result of a merge sort from disk and/or memory.

    -- Owen=

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-user @
categorieshadoop
postedJun 22, '10 at 12:15a
activeJun 22, '10 at 7:05a
posts3
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Torsten Curdt: 2 posts Owen O'Malley: 1 post

People

Translate

site design / logo © 2022 Grokbase