Grokbase Groups Hive user April 2011
FAQ
When the following query was run with mapred.job.reuse.jvm.num.tasks=20, some of the map tasks failed with "Error: Java heap space", causing the job to fail. After changing to mapred.job.reuse.jvm.num.tasks=1, the job succeeded.

FROM (
FROM intable1
SELECT acct_id, esn) b
JOIN (
FROM intable2
SELECT acct_id, xid, devtype_id, esn, other_properties['cdndldist'] AS cdndldist
WHERE (dateint>=20110201 AND dateint<=20110228)
AND (other_properties['cdndldist'] is not null)
AND (client_msg_type='endplay')
AND (devtype_id=272 OR devtype_id=129 OR devtype_id=12 OR devtype_id=13 OR devtype_id=14)) c
ON (b.acct_id=c.acct_id) AND (b.esn=c.esn)
INSERT OVERWRITE TABLE outtable PARTITION (dateint='20110201-20110228')
SELECT b.acct_id, c.xid, c.devtype_id, c.esn, c.cdndldist;

I'm on a pre-release version of Hive 0.7, with Hadoop 0.20. Does anyone know about any Hive/Hadoop issue that may be related to this?

Thanks.
Steven

Search Discussions

  • Igor Tatarinov at Apr 8, 2011 at 4:50 pm
    I had a similar problem until I set this parameter to 1 (although 3 seems to
    work fine too).

    There is an explanation somewhere on the web. Basically, if you run 20 tasks
    and the garbage collector cannot catch up with accumulated garbage, the java
    process grows too big so when it finally decides to fork a new process, the
    forked process and the original one take up too much memory. Or something
    like that.

    I imagine the same kind thing would still happen even when setting reuse to
    1 if a task happens to consume a lot of memory and CPU cycles, so something
    else must be happening here. Perhaps, there is a memory 'leak' and some
    'garbage' never gets collected between task executions.

    On Thu, Apr 7, 2011 at 6:53 PM, Steven Wong wrote:

    When the following query was run with mapred.job.reuse.jvm.num.tasks=20,
    some of the map tasks failed with “Error: Java heap space”, causing the job
    to fail. After changing to mapred.job.reuse.jvm.num.tasks=1, the job
    succeeded.



    FROM (

    FROM intable1

    SELECT acct_id, esn) b

    JOIN (

    FROM intable2

    SELECT acct_id, xid, devtype_id, esn, other_properties['cdndldist'] AS
    cdndldist

    WHERE (dateint>=20110201 AND dateint<=20110228)

    AND (other_properties['cdndldist'] is not null)

    AND (client_msg_type='endplay')

    AND (devtype_id=272 OR devtype_id=129 OR devtype_id=12 OR devtype_id=13
    OR devtype_id=14)) c

    ON (b.acct_id=c.acct_id) AND (b.esn=c.esn)

    INSERT OVERWRITE TABLE outtable PARTITION (dateint='20110201-20110228')

    SELECT b.acct_id, c.xid, c.devtype_id, c.esn, c.cdndldist;



    I’m on a pre-release version of Hive 0.7, with Hadoop 0.20. Does anyone
    know about any Hive/Hadoop issue that may be related to this?



    Thanks.

    Steven
  • Steven Wong at Apr 8, 2011 at 11:39 pm
    A likely or possible cause is memory leaks. Another possibility may be heap fragmentation. The OOM happened when the mapper was trying to allocate the buffer whose size is controlled by io.sort.mb, and my setup has io.sort.mb=500 and -Xmx1000m.


    From: Igor Tatarinov
    Sent: Friday, April 08, 2011 9:50 AM
    To: user@hive.apache.org
    Cc: Steven Wong
    Subject: Re: Mapper OOMs disappear after disabling JVM reuse

    I had a similar problem until I set this parameter to 1 (although 3 seems to work fine too).

    There is an explanation somewhere on the web. Basically, if you run 20 tasks and the garbage collector cannot catch up with accumulated garbage, the java process grows too big so when it finally decides to fork a new process, the forked process and the original one take up too much memory. Or something like that.

    I imagine the same kind thing would still happen even when setting reuse to 1 if a task happens to consume a lot of memory and CPU cycles, so something else must be happening here. Perhaps, there is a memory 'leak' and some 'garbage' never gets collected between task executions.


    On Thu, Apr 7, 2011 at 6:53 PM, Steven Wong wrote:
    When the following query was run with mapred.job.reuse.jvm.num.tasks=20, some of the map tasks failed with "Error: Java heap space", causing the job to fail. After changing to mapred.job.reuse.jvm.num.tasks=1, the job succeeded.

    FROM (
    FROM intable1
    SELECT acct_id, esn) b
    JOIN (
    FROM intable2
    SELECT acct_id, xid, devtype_id, esn, other_properties['cdndldist'] AS cdndldist
    WHERE (dateint>=20110201 AND dateint<=20110228)
    AND (other_properties['cdndldist'] is not null)
    AND (client_msg_type='endplay')
    AND (devtype_id=272 OR devtype_id=129 OR devtype_id=12 OR devtype_id=13 OR devtype_id=14)) c
    ON (b.acct_id=c.acct_id) AND (b.esn=c.esn)
    INSERT OVERWRITE TABLE outtable PARTITION (dateint='20110201-20110228')
    SELECT b.acct_id, c.xid, c.devtype_id, c.esn, c.cdndldist;

    I'm on a pre-release version of Hive 0.7, with Hadoop 0.20. Does anyone know about any Hive/Hadoop issue that may be related to this?

    Thanks.
    Steven

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshive, hadoop
postedApr 8, '11 at 1:53a
activeApr 8, '11 at 11:39p
posts3
users2
websitehive.apache.org

2 users in discussion

Steven Wong: 2 posts Igor Tatarinov: 1 post

People

Translate

site design / logo © 2022 Grokbase