FAQ
Pig leaves HOD processes behind if Ctrl-C is used before HOD connection is fully established
--------------------------------------------------------------------------------------------

Key: PIG-411
URL: https://issues.apache.org/jira/browse/PIG-411
Project: Pig
Issue Type: Bug
Reporter: Olga Natkovich


The code in doHod in backend/hadoop/executionengine/HExecutionEngine.java waits for hod connection to be fully established before setting a flag that indicates need to discunnect. The flag should be set as soon as HOD process is created so that we destroy it.

This change will not guarantee that process is not left behind but would make it much less likely

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Olga Natkovich (JIRA) at Sep 4, 2008 at 5:38 pm
    [ https://issues.apache.org/jira/browse/PIG-411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Olga Natkovich updated PIG-411:
    -------------------------------

    Fix Version/s: types_branch
    Affects Version/s: types_branch
    Pig leaves HOD processes behind if Ctrl-C is used before HOD connection is fully established
    --------------------------------------------------------------------------------------------

    Key: PIG-411
    URL: https://issues.apache.org/jira/browse/PIG-411
    Project: Pig
    Issue Type: Bug
    Affects Versions: types_branch
    Reporter: Olga Natkovich
    Fix For: types_branch


    The code in doHod in backend/hadoop/executionengine/HExecutionEngine.java waits for hod connection to be fully established before setting a flag that indicates need to discunnect. The flag should be set as soon as HOD process is created so that we destroy it.
    This change will not guarantee that process is not left behind but would make it much less likely
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Olga Natkovich (JIRA) at Sep 4, 2008 at 11:54 pm
    [ https://issues.apache.org/jira/browse/PIG-411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Olga Natkovich updated PIG-411:
    -------------------------------

    Assignee: Olga Natkovich
    Priority: Critical (was: Major)
    Pig leaves HOD processes behind if Ctrl-C is used before HOD connection is fully established
    --------------------------------------------------------------------------------------------

    Key: PIG-411
    URL: https://issues.apache.org/jira/browse/PIG-411
    Project: Pig
    Issue Type: Bug
    Affects Versions: types_branch
    Reporter: Olga Natkovich
    Assignee: Olga Natkovich
    Priority: Critical
    Fix For: types_branch


    The code in doHod in backend/hadoop/executionengine/HExecutionEngine.java waits for hod connection to be fully established before setting a flag that indicates need to discunnect. The flag should be set as soon as HOD process is created so that we destroy it.
    This change will not guarantee that process is not left behind but would make it much less likely
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Olga Natkovich (JIRA) at Sep 19, 2008 at 11:19 pm
    [ https://issues.apache.org/jira/browse/PIG-411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Olga Natkovich updated PIG-411:
    -------------------------------

    Status: Patch Available (was: Open)
    Pig leaves HOD processes behind if Ctrl-C is used before HOD connection is fully established
    --------------------------------------------------------------------------------------------

    Key: PIG-411
    URL: https://issues.apache.org/jira/browse/PIG-411
    Project: Pig
    Issue Type: Bug
    Affects Versions: types_branch
    Reporter: Olga Natkovich
    Assignee: Olga Natkovich
    Priority: Critical
    Fix For: types_branch

    Attachments: PIG-411.patch


    The code in doHod in backend/hadoop/executionengine/HExecutionEngine.java waits for hod connection to be fully established before setting a flag that indicates need to discunnect. The flag should be set as soon as HOD process is created so that we destroy it.
    This change will not guarantee that process is not left behind but would make it much less likely
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Olga Natkovich (JIRA) at Sep 19, 2008 at 11:19 pm
    [ https://issues.apache.org/jira/browse/PIG-411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Olga Natkovich updated PIG-411:
    -------------------------------

    Attachment: PIG-411.patch
    Pig leaves HOD processes behind if Ctrl-C is used before HOD connection is fully established
    --------------------------------------------------------------------------------------------

    Key: PIG-411
    URL: https://issues.apache.org/jira/browse/PIG-411
    Project: Pig
    Issue Type: Bug
    Affects Versions: types_branch
    Reporter: Olga Natkovich
    Assignee: Olga Natkovich
    Priority: Critical
    Fix For: types_branch

    Attachments: PIG-411.patch


    The code in doHod in backend/hadoop/executionengine/HExecutionEngine.java waits for hod connection to be fully established before setting a flag that indicates need to discunnect. The flag should be set as soon as HOD process is created so that we destroy it.
    This change will not guarantee that process is not left behind but would make it much less likely
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Pradeep Kamath (JIRA) at Sep 22, 2008 at 6:21 pm
    [ https://issues.apache.org/jira/browse/PIG-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12633415#action_12633415 ]

    Pradeep Kamath commented on PIG-411:
    ------------------------------------

    Changes look good.

    Couple of comments:
    1)
    {code}
    private synchronized void closeHod(String server){
    if (hodProcess == null){
    // just cleanup the dir if it exists and return
    if (hodConfDir != null)
    deleteDir(server, hodConfDir);
    return;
    }

    // hod deallocate format: hod deallocate -d <conf dir>
    String[] cmdarray = new String[4];
    cmdarray[0] = "hod";
    cmdarray[1] = "deallocate";
    cmdarray[2] = "-d";
    if (remoteHodConfDir != null)
    cmdarray[3] = remoteHodConfDir;
    else
    cmdarray[3] = hodConfDir;

    log.info("Disconnecting from HOD...");
    log.debug("Disconnect command: " + cmdToString(cmdarray));

    try {
    runCommand(server, cmdarray, false);
    } catch (Exception e) {
    log.warn("Failed to disconnect from HOD; error: " + e.getMessage());
    } finally {
    if (remoteHodConfDir != null)
    deleteDir(server, remoteHodConfDir);
    deleteDir(LOCAL, hodConfDir);
    }

    hodProcess = null;
    }

    {code}

    Should the code in
    {code}
    if (hodProcess == null){
    ..
    }
    {code}

    be same as the code which deletes and cleans up in
    {code}
    } finally {
    if (remoteHodConfDir != null)
    deleteDir(server, remoteHodConfDir);
    deleteDir(LOCAL, hodConfDir);
    }
    {code}
    to be consistent in the way cleanup is done?

    2)
    If hodProcess != null, and if we failed to deallocate hod cluster in the catch{}, should we try to destroy hodProcess -(hodProcess.destroy()) so that we don't leave the hodProcess as a zombie?
    Pig leaves HOD processes behind if Ctrl-C is used before HOD connection is fully established
    --------------------------------------------------------------------------------------------

    Key: PIG-411
    URL: https://issues.apache.org/jira/browse/PIG-411
    Project: Pig
    Issue Type: Bug
    Affects Versions: types_branch
    Reporter: Olga Natkovich
    Assignee: Olga Natkovich
    Priority: Critical
    Fix For: types_branch

    Attachments: PIG-411.patch


    The code in doHod in backend/hadoop/executionengine/HExecutionEngine.java waits for hod connection to be fully established before setting a flag that indicates need to discunnect. The flag should be set as soon as HOD process is created so that we destroy it.
    This change will not guarantee that process is not left behind but would make it much less likely
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Olga Natkovich (JIRA) at Sep 22, 2008 at 7:49 pm
    [ https://issues.apache.org/jira/browse/PIG-411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Olga Natkovich updated PIG-411:
    -------------------------------

    Attachment: PIG-411_2.patch

    Thanks pradeep for the feedback. New patch attached.
    Pig leaves HOD processes behind if Ctrl-C is used before HOD connection is fully established
    --------------------------------------------------------------------------------------------

    Key: PIG-411
    URL: https://issues.apache.org/jira/browse/PIG-411
    Project: Pig
    Issue Type: Bug
    Affects Versions: types_branch
    Reporter: Olga Natkovich
    Assignee: Olga Natkovich
    Priority: Critical
    Fix For: types_branch

    Attachments: PIG-411.patch, PIG-411_2.patch


    The code in doHod in backend/hadoop/executionengine/HExecutionEngine.java waits for hod connection to be fully established before setting a flag that indicates need to discunnect. The flag should be set as soon as HOD process is created so that we destroy it.
    This change will not guarantee that process is not left behind but would make it much less likely
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Pradeep Kamath (JIRA) at Sep 22, 2008 at 8:10 pm
    [ https://issues.apache.org/jira/browse/PIG-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12633466#action_12633466 ]

    Pradeep Kamath commented on PIG-411:
    ------------------------------------

    Reviewed the changes - look good.
    Pig leaves HOD processes behind if Ctrl-C is used before HOD connection is fully established
    --------------------------------------------------------------------------------------------

    Key: PIG-411
    URL: https://issues.apache.org/jira/browse/PIG-411
    Project: Pig
    Issue Type: Bug
    Affects Versions: types_branch
    Reporter: Olga Natkovich
    Assignee: Olga Natkovich
    Priority: Critical
    Fix For: types_branch

    Attachments: PIG-411.patch, PIG-411_2.patch


    The code in doHod in backend/hadoop/executionengine/HExecutionEngine.java waits for hod connection to be fully established before setting a flag that indicates need to discunnect. The flag should be set as soon as HOD process is created so that we destroy it.
    This change will not guarantee that process is not left behind but would make it much less likely
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Olga Natkovich (JIRA) at Sep 22, 2008 at 8:22 pm
    [ https://issues.apache.org/jira/browse/PIG-411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Olga Natkovich updated PIG-411:
    -------------------------------

    Resolution: Fixed
    Status: Resolved (was: Patch Available)

    patch committed. thanks pradeep for reviews
    Pig leaves HOD processes behind if Ctrl-C is used before HOD connection is fully established
    --------------------------------------------------------------------------------------------

    Key: PIG-411
    URL: https://issues.apache.org/jira/browse/PIG-411
    Project: Pig
    Issue Type: Bug
    Affects Versions: types_branch
    Reporter: Olga Natkovich
    Assignee: Olga Natkovich
    Priority: Critical
    Fix For: types_branch

    Attachments: PIG-411.patch, PIG-411_2.patch


    The code in doHod in backend/hadoop/executionengine/HExecutionEngine.java waits for hod connection to be fully established before setting a flag that indicates need to discunnect. The flag should be set as soon as HOD process is created so that we destroy it.
    This change will not guarantee that process is not left behind but would make it much less likely
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categoriespig, hadoop
postedSep 4, '08 at 5:36p
activeSep 22, '08 at 8:22p
posts9
users1
websitepig.apache.org

1 user in discussion

Olga Natkovich (JIRA): 9 posts

People

Translate

site design / logo © 2022 Grokbase