FAQ
I have a running Hadoop/HBase cluster.
When I want to change hadoop parameters without stopping the cluster, can I
use org.apache.hadoop.conf.Configuration API?

I wrote following java source, but it didn't do anything.
-------------------------
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
public class ChangeConf
{
public static void main(String[] args) throws Exception
{
Configuration conf = new Configuration();
conf.addResource(new
Path("/hadoop-0.18.1/conf/hadoop-default.xml"));
conf.addResource(new Path("/hadoop-0.18.1/conf/hadoop-site.xml"));
conf.setStrings("dfs.data.dir",
"/hadoop-data/data.dir","/data4/hadoop-data");
return;
}
}
----------------------------

Actually, I know I can use "decommission" to add more directory without
stopping whole cluster.
But I'm very confused "org.apache.hadoop.conf.Configuration" API.

Could anyone please let me know clearly?

Search Discussions

  • Alex Loddengaard at Oct 22, 2008 at 6:19 pm
    Just to be clear, you want to persist a configuration change to your entire
    cluster without bringing it down, and you're hoping to use the Configuration
    API to do so. Did I get your question correct?

    I don't know of a way to do this without restarting the cluster, because I'm
    pretty sure Configuration changes will only affect the current job. Does
    anyone else have a suggestion?

    Alex
    On Wed, Oct 22, 2008 at 12:44 AM, Jinyeon Lee wrote:

    I have a running Hadoop/HBase cluster.
    When I want to change hadoop parameters without stopping the cluster, can I
    use org.apache.hadoop.conf.Configuration API?

    I wrote following java source, but it didn't do anything.
    -------------------------
    import org.apache.hadoop.conf.Configuration;
    import org.apache.hadoop.fs.Path;
    public class ChangeConf
    {
    public static void main(String[] args) throws Exception
    {
    Configuration conf = new Configuration();
    conf.addResource(new
    Path("/hadoop-0.18.1/conf/hadoop-default.xml"));
    conf.addResource(new Path("/hadoop-0.18.1/conf/hadoop-site.xml"));
    conf.setStrings("dfs.data.dir",
    "/hadoop-data/data.dir","/data4/hadoop-data");
    return;
    }
    }
    ----------------------------

    Actually, I know I can use "decommission" to add more directory without
    stopping whole cluster.
    But I'm very confused "org.apache.hadoop.conf.Configuration" API.

    Could anyone please let me know clearly?
  • Steve Loughran at Oct 23, 2008 at 12:29 pm

    Alex Loddengaard wrote:
    Just to be clear, you want to persist a configuration change to your entire
    cluster without bringing it down, and you're hoping to use the Configuration
    API to do so. Did I get your question correct?

    I don't know of a way to do this without restarting the cluster, because I'm
    pretty sure Configuration changes will only affect the current job. Does
    anyone else have a suggestion?

    Alex
    Its not doable with the current architecture
    -confs never get reread
    -they aren't persisted

    If you did persist them, it would get into a mess if every node in the
    cluster was reading the same shared file from an NFS mount; the last
    node to start up and set a value would set it for everything else.
    Better to have a persistent central configuration management ("CM")
    infrastructure and drive hadoop that way.

    Even with CM tooling, the idea of persisting all changes back from the
    apps is generally considered a bad thing. What is preferred is to have a
    gui for managing system state that you can view, rollback, compare
    configuration changes too, as having the app do it means you don't know
    what's going on -and your app can leave your system in a mess, which is
    exactly what CM tools try to prevent.

    -steve
  • Jinyeon Lee at Oct 23, 2008 at 10:36 pm
    Thank you!!!

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedOct 22, '08 at 7:44a
activeOct 23, '08 at 10:36p
posts4
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase