Grokbase › Groups › Hadoop › common-dev › March 2009
FAQ
hadoop command uses large JVM heap size
---------------------------------------

Key: HADOOP-5564
URL: https://issues.apache.org/jira/browse/HADOOP-5564
Project: Hadoop Core
Issue Type: Bug
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas
Fix For: 0.20.0


Command used to determine JAVA_PLATFORM in bin/hadoop command does not set the heap size. The command uses default 1GB heap size. The tasks invoking hadoop command end up using large heap size in streaming jobs. If the maximum memory that can be used by a task is restricted, this could result in map/reduce job failures.


--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Tsz Wo (Nicholas), SZE (JIRA) at Mar 24, 2009 at 6:37 pm ⇧
    [ https://issues.apache.org/jira/browse/HADOOP-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Tsz Wo (Nicholas), SZE updated HADOOP-5564:
    -------------------------------------------

    Component/s: build
    Priority: Blocker (was: Major)

    As Koji suggested, we probably should add -Xmx to limit the memory usage.
    hadoop command uses large JVM heap size
    ---------------------------------------

    Key: HADOOP-5564
    URL: https://issues.apache.org/jira/browse/HADOOP-5564
    Project: Hadoop Core
    Issue Type: Bug
    Components: build
    Reporter: Suresh Srinivas
    Assignee: Suresh Srinivas
    Priority: Blocker
    Fix For: 0.20.0


    Command used to determine JAVA_PLATFORM in bin/hadoop command does not set the heap size. The command uses default 1GB heap size. The tasks invoking hadoop command end up using large heap size in streaming jobs. If the maximum memory that can be used by a task is restricted, this could result in map/reduce job failures.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Suresh Srinivas (JIRA) at Mar 24, 2009 at 6:47 pm ⇧
    [ https://issues.apache.org/jira/browse/HADOOP-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Suresh Srinivas updated HADOOP-5564:
    ------------------------------------

    Attachment: 5564.patch

    Setting the heap size to 32MB
    hadoop command uses large JVM heap size
    ---------------------------------------

    Key: HADOOP-5564
    URL: https://issues.apache.org/jira/browse/HADOOP-5564
    Project: Hadoop Core
    Issue Type: Bug
    Components: build
    Reporter: Suresh Srinivas
    Assignee: Suresh Srinivas
    Priority: Blocker
    Fix For: 0.20.0

    Attachments: 5564.patch


    Command used to determine JAVA_PLATFORM in bin/hadoop command does not set the heap size. The command uses default 1GB heap size. The tasks invoking hadoop command end up using large heap size in streaming jobs. If the maximum memory that can be used by a task is restricted, this could result in map/reduce job failures.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Tsz Wo (Nicholas), SZE (JIRA) at Mar 24, 2009 at 6:51 pm ⇧
    [ https://issues.apache.org/jira/browse/HADOOP-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Tsz Wo (Nicholas), SZE updated HADOOP-5564:
    -------------------------------------------

    Hadoop Flags: [Reviewed]

    +1 patch looks good

    Talked to Koji and Suresh offline. We have considered using -client. However, the [tool doc|http://java.sun.com/javase/6/docs/technotes/tools/solaris/java.html] says that the -client option may be ignored by the jdk.
    hadoop command uses large JVM heap size
    ---------------------------------------

    Key: HADOOP-5564
    URL: https://issues.apache.org/jira/browse/HADOOP-5564
    Project: Hadoop Core
    Issue Type: Bug
    Components: build
    Reporter: Suresh Srinivas
    Assignee: Suresh Srinivas
    Priority: Blocker
    Fix For: 0.20.0

    Attachments: 5564.patch


    Command used to determine JAVA_PLATFORM in bin/hadoop command does not set the heap size. The command uses default 1GB heap size. The tasks invoking hadoop command end up using large heap size in streaming jobs. If the maximum memory that can be used by a task is restricted, this could result in map/reduce job failures.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Suresh Srinivas (JIRA) at Mar 24, 2009 at 6:53 pm ⇧
    [ https://issues.apache.org/jira/browse/HADOOP-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688792#action_12688792 ]

    Suresh Srinivas commented on HADOOP-5564:
    -----------------------------------------

    I have tested this manually by making sure {{hadoop}} command takes the patch of changed code and it prints the {{JAVA_PLATFORM}} successfully.
    hadoop command uses large JVM heap size
    ---------------------------------------

    Key: HADOOP-5564
    URL: https://issues.apache.org/jira/browse/HADOOP-5564
    Project: Hadoop Core
    Issue Type: Bug
    Components: build
    Reporter: Suresh Srinivas
    Assignee: Suresh Srinivas
    Priority: Blocker
    Fix For: 0.20.0

    Attachments: 5564.patch


    Command used to determine JAVA_PLATFORM in bin/hadoop command does not set the heap size. The command uses default 1GB heap size. The tasks invoking hadoop command end up using large heap size in streaming jobs. If the maximum memory that can be used by a task is restricted, this could result in map/reduce job failures.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Suresh Srinivas (JIRA) at Mar 24, 2009 at 7:01 pm ⇧
    [ https://issues.apache.org/jira/browse/HADOOP-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Suresh Srinivas updated HADOOP-5564:
    ------------------------------------

    Attachment: 5564.trunk.patch

    Attaching patch for the trunk. Tested the patch manually on trunk.
    hadoop command uses large JVM heap size
    ---------------------------------------

    Key: HADOOP-5564
    URL: https://issues.apache.org/jira/browse/HADOOP-5564
    Project: Hadoop Core
    Issue Type: Bug
    Components: build
    Reporter: Suresh Srinivas
    Assignee: Suresh Srinivas
    Priority: Blocker
    Fix For: 0.20.0

    Attachments: 5564.patch, 5564.trunk.patch


    Command used to determine JAVA_PLATFORM in bin/hadoop command does not set the heap size. The command uses default 1GB heap size. The tasks invoking hadoop command end up using large heap size in streaming jobs. If the maximum memory that can be used by a task is restricted, this could result in map/reduce job failures.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Suresh Srinivas (JIRA) at Mar 24, 2009 at 7:05 pm ⇧
    [ https://issues.apache.org/jira/browse/HADOOP-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Suresh Srinivas updated HADOOP-5564:
    ------------------------------------

    Attachment: 5564.trunk.patch

    Updated trunk patch
    hadoop command uses large JVM heap size
    ---------------------------------------

    Key: HADOOP-5564
    URL: https://issues.apache.org/jira/browse/HADOOP-5564
    Project: Hadoop Core
    Issue Type: Bug
    Components: build
    Reporter: Suresh Srinivas
    Assignee: Suresh Srinivas
    Priority: Blocker
    Fix For: 0.20.0

    Attachments: 5564.patch, 5564.trunk.patch, 5564.trunk.patch


    Command used to determine JAVA_PLATFORM in bin/hadoop command does not set the heap size. The command uses default 1GB heap size. The tasks invoking hadoop command end up using large heap size in streaming jobs. If the maximum memory that can be used by a task is restricted, this could result in map/reduce job failures.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Tsz Wo (Nicholas), SZE (JIRA) at Mar 24, 2009 at 7:09 pm ⇧
    [ https://issues.apache.org/jira/browse/HADOOP-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Tsz Wo (Nicholas), SZE resolved HADOOP-5564.
    --------------------------------------------

    Resolution: Fixed
    Fix Version/s: 0.21.0

    I have committed this to 0.20 and above. Thanks, Suresh!
    hadoop command uses large JVM heap size
    ---------------------------------------

    Key: HADOOP-5564
    URL: https://issues.apache.org/jira/browse/HADOOP-5564
    Project: Hadoop Core
    Issue Type: Bug
    Components: build
    Reporter: Suresh Srinivas
    Assignee: Suresh Srinivas
    Priority: Blocker
    Fix For: 0.20.0, 0.21.0

    Attachments: 5564.patch, 5564.trunk.patch, 5564.trunk.patch


    Command used to determine JAVA_PLATFORM in bin/hadoop command does not set the heap size. The command uses default 1GB heap size. The tasks invoking hadoop command end up using large heap size in streaming jobs. If the maximum memory that can be used by a task is restricted, this could result in map/reduce job failures.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Steve Loughran (JIRA) at Mar 24, 2009 at 7:34 pm ⇧
    [ https://issues.apache.org/jira/browse/HADOOP-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688815#action_12688815 ]

    Steve Loughran commented on HADOOP-5564:
    ----------------------------------------

    has anyone tested that everyting works ok on a 64 bit JVM? one with pointers twice the size?
    hadoop command uses large JVM heap size
    ---------------------------------------

    Key: HADOOP-5564
    URL: https://issues.apache.org/jira/browse/HADOOP-5564
    Project: Hadoop Core
    Issue Type: Bug
    Components: build
    Reporter: Suresh Srinivas
    Assignee: Suresh Srinivas
    Priority: Blocker
    Fix For: 0.20.0, 0.21.0

    Attachments: 5564.patch, 5564.trunk.patch, 5564.trunk.patch


    Command used to determine JAVA_PLATFORM in bin/hadoop command does not set the heap size. The command uses default 1GB heap size. The tasks invoking hadoop command end up using large heap size in streaming jobs. If the maximum memory that can be used by a task is restricted, this could result in map/reduce job failures.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hudson (JIRA) at Mar 26, 2009 at 1:17 am ⇧
    [ https://issues.apache.org/jira/browse/HADOOP-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689227#action_12689227 ]

    Hudson commented on HADOOP-5564:
    --------------------------------

    Integrated in Hadoop-trunk #790 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/790/])
    . Limit the JVM heap size in the java command for initializing JAVA_PLATFORM. (Suresh Srinivas via szetszwo)

    hadoop command uses large JVM heap size
    ---------------------------------------

    Key: HADOOP-5564
    URL: https://issues.apache.org/jira/browse/HADOOP-5564
    Project: Hadoop Core
    Issue Type: Bug
    Components: build
    Reporter: Suresh Srinivas
    Assignee: Suresh Srinivas
    Priority: Blocker
    Fix For: 0.20.0, 0.21.0

    Attachments: 5564.patch, 5564.trunk.patch, 5564.trunk.patch


    Command used to determine JAVA_PLATFORM in bin/hadoop command does not set the heap size. The command uses default 1GB heap size. The tasks invoking hadoop command end up using large heap size in streaming jobs. If the maximum memory that can be used by a task is restricted, this could result in map/reduce job failures.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedMar 24, '09 at 6:35p
activeMar 26, '09 at 1:17a
posts10
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Hudson (JIRA): 10 posts

Content

People

Support

Translate

site design / logo © 2023 Grokbase