FAQ
Hello,
   I'm attempting to get started provisioning and configuring clusters using
the Java client for the CM API.

   The following code stub successfully creates a "cluster", adds services
and hosts to the cluster, assigns roles to hosts, and attempts to set
configuration to the role instance. The last part (config) is not
sticking, but the other items all show up in the CM Web UI.

   In the code below, the host already exists from a prior execution. The
role gets succesfully created but dfs replication does not get set to 2
(role.setConfig isn't working). If I go to the CM Web UI afterward and set
it to 2, it does stick.

Anyone have ideas on what's not working?

-Joe

ServicesResourceV4 svcResource =
clustersResource.getServicesResource(cluster.getName());
RolesResourceV4 hdfsRolesResource = svcResource.getRolesResource("HDFS");
RoleConfigGroupsResource rg = svcResource.getRoleConfigGroupsResource("HDFS");


ApiRoleList roleList = new ApiRoleList();
ApiRole role = new ApiRole();
role.setHostRef(new ApiHostRef("test0"));
role.setName("hdfs-nn1");
role.setType("NAMENODE");

ApiConfigList configList = new ApiConfigList();
ApiConfig config = new ApiConfig("dfs_replication", "2");
configList.add(config);

role.setConfig(configList);
roleList.add(role);

hdfsRolesResource.createRoles(roleList);

Search Discussions

  • Joe Travaglini at Jul 25, 2013 at 4:16 pm
    I had the theory that because dfs_replication is a DataNode config and not
    a HDFS service wide config, that this was not taking.

    So instead, I set a NameNode specific config before creating the role:

    ApiConfig config = new ApiConfig("dfs_name_dir_list", "/foo");
    configList.add(config);

    I confirmed that dfs_name_dir_list is a valid config key by navigating
    to /api/v4/clusters/mycluster/services/HDFS/roles/hdfs-nn1/config?view=full.

    However, this value is empty when I create the role via the Java code.

    Any ideas on why this isn't working? Is it expected behavior that
    ApiConfig provided to Role.setConfig() prior to RolesResource.createRole()
    will not be respected?



    On Thu, Jul 25, 2013 at 10:50 AM, Joe Travaglini
    wrote:
    Hello,
    I'm attempting to get started provisioning and configuring clusters
    using the Java client for the CM API.

    The following code stub successfully creates a "cluster", adds services
    and hosts to the cluster, assigns roles to hosts, and attempts to set
    configuration to the role instance. The last part (config) is not
    sticking, but the other items all show up in the CM Web UI.

    In the code below, the host already exists from a prior execution. The
    role gets succesfully created but dfs replication does not get set to 2
    (role.setConfig isn't working). If I go to the CM Web UI afterward and set
    it to 2, it does stick.

    Anyone have ideas on what's not working?

    -Joe

    ServicesResourceV4 svcResource = clustersResource.getServicesResource(cluster.getName());
    RolesResourceV4 hdfsRolesResource = svcResource.getRolesResource("HDFS");
    RoleConfigGroupsResource rg = svcResource.getRoleConfigGroupsResource("HDFS");


    ApiRoleList roleList = new ApiRoleList();
    ApiRole role = new ApiRole();
    role.setHostRef(new ApiHostRef("test0"));
    role.setName("hdfs-nn1");
    role.setType("NAMENODE");

    ApiConfigList configList = new ApiConfigList();
    ApiConfig config = new ApiConfig("dfs_replication", "2");
    configList.add(config);

    role.setConfig(configList);
    roleList.add(role);

    hdfsRolesResource.createRoles(roleList);
  • Philip Zeyliger at Jul 25, 2013 at 5:20 pm
    Hi Joe,

    dfs_replication is a service-level config, so it should be added to the
    ApiService, not a specific ApiRole. In your original snippet, you were
    adding the ApiConfig to the role.

    Configs passed to createRole() should be honored--if you've got a snippet
    of that failing, I'd be happy to check it out for you.

    -- Philip

    On Thu, Jul 25, 2013 at 9:16 AM, Joe Travaglini wrote:

    I had the theory that because dfs_replication is a DataNode config and not
    a HDFS service wide config, that this was not taking.

    So instead, I set a NameNode specific config before creating the role:

    ApiConfig config = new ApiConfig("dfs_name_dir_list", "/foo");
    configList.add(config);

    I confirmed that dfs_name_dir_list is a valid config key by navigating
    to /api/v4/clusters/mycluster/services/HDFS/roles/hdfs-nn1/config?view=full.

    However, this value is empty when I create the role via the Java code.

    Any ideas on why this isn't working? Is it expected behavior that
    ApiConfig provided to Role.setConfig() prior to RolesResource.createRole()
    will not be respected?



    On Thu, Jul 25, 2013 at 10:50 AM, Joe Travaglini <joe.travaglini@gmail.com
    wrote:
    Hello,
    I'm attempting to get started provisioning and configuring clusters
    using the Java client for the CM API.

    The following code stub successfully creates a "cluster", adds services
    and hosts to the cluster, assigns roles to hosts, and attempts to set
    configuration to the role instance. The last part (config) is not
    sticking, but the other items all show up in the CM Web UI.

    In the code below, the host already exists from a prior execution. The
    role gets succesfully created but dfs replication does not get set to 2
    (role.setConfig isn't working). If I go to the CM Web UI afterward and set
    it to 2, it does stick.

    Anyone have ideas on what's not working?

    -Joe

    ServicesResourceV4 svcResource = clustersResource.getServicesResource(cluster.getName());
    RolesResourceV4 hdfsRolesResource = svcResource.getRolesResource("HDFS");
    RoleConfigGroupsResource rg = svcResource.getRoleConfigGroupsResource("HDFS");


    ApiRoleList roleList = new ApiRoleList();
    ApiRole role = new ApiRole();
    role.setHostRef(new ApiHostRef("test0"));
    role.setName("hdfs-nn1");
    role.setType("NAMENODE");

    ApiConfigList configList = new ApiConfigList();
    ApiConfig config = new ApiConfig("dfs_replication", "2");
    configList.add(config);

    role.setConfig(configList);
    roleList.add(role);

    hdfsRolesResource.createRoles(roleList);
  • Joe Travaglini at Jul 25, 2013 at 5:23 pm
    Hey Philip,
    Yes, the same snippet doesn't appear to be working: - I had already created
    the cluster/services/hosts in some code that executes beforehand. The
    following creates the role, but does not set dfs.namedir:

    ServicesResourceV4 svcResource =
    clustersResource.getServicesResource(cluster.getName());
    RolesResourceV4 hdfsRolesResource = svcResource.getRolesResource("HDFS");
    RoleConfigGroupsResource rg = svcResource.getRoleConfigGroupsResource("HDFS");


    ApiRoleList roleList = new ApiRoleList();
    ApiRole role = new ApiRole();
    role.setHostRef(new ApiHostRef("test0"));
    role.setName("hdfs-nn1");
    role.setType("NAMENODE");

    ApiConfigList configList = new ApiConfigList();
    ApiConfig config = new ApiConfig("dfs_name_dir_list", "/foo");
    configList.add(config);

    role.setConfig(configList);
    roleList.add(role);

    hdfsRolesResource.createRoles(roleList);


    On Thu, Jul 25, 2013 at 1:20 PM, Philip Zeyliger wrote:

    Hi Joe,

    dfs_replication is a service-level config, so it should be added to the
    ApiService, not a specific ApiRole. In your original snippet, you were
    adding the ApiConfig to the role.

    Configs passed to createRole() should be honored--if you've got a snippet
    of that failing, I'd be happy to check it out for you.

    -- Philip

    On Thu, Jul 25, 2013 at 9:16 AM, Joe Travaglini wrote:

    I had the theory that because dfs_replication is a DataNode config and
    not a HDFS service wide config, that this was not taking.

    So instead, I set a NameNode specific config before creating the role:

    ApiConfig config = new ApiConfig("dfs_name_dir_list", "/foo");
    configList.add(config);

    I confirmed that dfs_name_dir_list is a valid config key by navigating
    to /api/v4/clusters/mycluster/services/HDFS/roles/hdfs-nn1/config?view=full.

    However, this value is empty when I create the role via the Java code.

    Any ideas on why this isn't working? Is it expected behavior that
    ApiConfig provided to Role.setConfig() prior to RolesResource.createRole()
    will not be respected?



    On Thu, Jul 25, 2013 at 10:50 AM, Joe Travaglini <
    joe.travaglini@gmail.com> wrote:
    Hello,
    I'm attempting to get started provisioning and configuring clusters
    using the Java client for the CM API.

    The following code stub successfully creates a "cluster", adds
    services and hosts to the cluster, assigns roles to hosts, and attempts to
    set configuration to the role instance. The last part (config) is not
    sticking, but the other items all show up in the CM Web UI.

    In the code below, the host already exists from a prior execution.
    The role gets succesfully created but dfs replication does not get set to
    2 (role.setConfig isn't working). If I go to the CM Web UI afterward and
    set it to 2, it does stick.

    Anyone have ideas on what's not working?

    -Joe

    ServicesResourceV4 svcResource = clustersResource.getServicesResource(cluster.getName());
    RolesResourceV4 hdfsRolesResource = svcResource.getRolesResource("HDFS");
    RoleConfigGroupsResource rg = svcResource.getRoleConfigGroupsResource("HDFS");


    ApiRoleList roleList = new ApiRoleList();
    ApiRole role = new ApiRole();
    role.setHostRef(new ApiHostRef("test0"));
    role.setName("hdfs-nn1");
    role.setType("NAMENODE");

    ApiConfigList configList = new ApiConfigList();
    ApiConfig config = new ApiConfig("dfs_replication", "2");
    configList.add(config);

    role.setConfig(configList);
    roleList.add(role);

    hdfsRolesResource.createRoles(roleList);
  • Philip Zeyliger at Jul 25, 2013 at 10:01 pm
    Hi Joe,

    You've uncovered what I think is a bug in this API. I'll file it and work
    to get it fixed.

    That said, you should actually be setting the configuration on the "Role
    Config Group" so that it applies to all namenodes, not just this one.
      That's this guy:
    http://cloudera.github.io/cm_api/apidocs/v4/el_ns0_roleConfigGroup.html

    Thanks!

    -- Philip


    On Thu, Jul 25, 2013 at 10:23 AM, Joe Travaglini
    wrote:
    Hey Philip,
    Yes, the same snippet doesn't appear to be working: - I had already
    created the cluster/services/hosts in some code that executes beforehand.
    The following creates the role, but does not set dfs.namedir:

    ServicesResourceV4 svcResource = clustersResource.getServicesResource(cluster.getName());
    RolesResourceV4 hdfsRolesResource = svcResource.getRolesResource("HDFS");
    RoleConfigGroupsResource rg = svcResource.getRoleConfigGroupsResource("HDFS");


    ApiRoleList roleList = new ApiRoleList();
    ApiRole role = new ApiRole();
    role.setHostRef(new ApiHostRef("test0"));
    role.setName("hdfs-nn1");
    role.setType("NAMENODE");

    ApiConfigList configList = new ApiConfigList();

    ApiConfig config = new ApiConfig("dfs_name_dir_list", "/foo");
    configList.add(config);


    role.setConfig(configList);
    roleList.add(role);

    hdfsRolesResource.createRoles(roleList);


    On Thu, Jul 25, 2013 at 1:20 PM, Philip Zeyliger wrote:

    Hi Joe,

    dfs_replication is a service-level config, so it should be added to the
    ApiService, not a specific ApiRole. In your original snippet, you were
    adding the ApiConfig to the role.

    Configs passed to createRole() should be honored--if you've got a snippet
    of that failing, I'd be happy to check it out for you.

    -- Philip


    On Thu, Jul 25, 2013 at 9:16 AM, Joe Travaglini <joe.travaglini@gmail.com
    wrote:
    I had the theory that because dfs_replication is a DataNode config and
    not a HDFS service wide config, that this was not taking.

    So instead, I set a NameNode specific config before creating the role:

    ApiConfig config = new ApiConfig("dfs_name_dir_list", "/foo");
    configList.add(config);

    I confirmed that dfs_name_dir_list is a valid config key by navigating
    to /api/v4/clusters/mycluster/services/HDFS/roles/hdfs-nn1/config?view=full.

    However, this value is empty when I create the role via the Java code.

    Any ideas on why this isn't working? Is it expected behavior that
    ApiConfig provided to Role.setConfig() prior to RolesResource.createRole()
    will not be respected?



    On Thu, Jul 25, 2013 at 10:50 AM, Joe Travaglini <
    joe.travaglini@gmail.com> wrote:
    Hello,
    I'm attempting to get started provisioning and configuring clusters
    using the Java client for the CM API.

    The following code stub successfully creates a "cluster", adds
    services and hosts to the cluster, assigns roles to hosts, and attempts to
    set configuration to the role instance. The last part (config) is not
    sticking, but the other items all show up in the CM Web UI.

    In the code below, the host already exists from a prior execution.
    The role gets succesfully created but dfs replication does not get set to
    2 (role.setConfig isn't working). If I go to the CM Web UI afterward and
    set it to 2, it does stick.

    Anyone have ideas on what's not working?

    -Joe

    ServicesResourceV4 svcResource = clustersResource.getServicesResource(cluster.getName());
    RolesResourceV4 hdfsRolesResource = svcResource.getRolesResource("HDFS");
    RoleConfigGroupsResource rg = svcResource.getRoleConfigGroupsResource("HDFS");


    ApiRoleList roleList = new ApiRoleList();
    ApiRole role = new ApiRole();
    role.setHostRef(new ApiHostRef("test0"));
    role.setName("hdfs-nn1");
    role.setType("NAMENODE");

    ApiConfigList configList = new ApiConfigList();
    ApiConfig config = new ApiConfig("dfs_replication", "2");
    configList.add(config);

    role.setConfig(configList);
    roleList.add(role);

    hdfsRolesResource.createRoles(roleList);

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedJul 25, '13 at 2:50p
activeJul 25, '13 at 10:01p
posts5
users2
websitecloudera.com
irc#hadoop

2 users in discussion

Joe Travaglini: 3 posts Philip Zeyliger: 2 posts

People

Translate

site design / logo © 2022 Grokbase