FAQ
Hi,

I'm trying to turn off replication on a 4 node cluster. In cloudera
manager, I went to Services > HDFS > Configuration > View & Edit > Service
Wide (on the righthand side) > Replication

and changed Replication Factor to 1. I then restarted the cluster.

When I go to the namenode web UI and look at a file I have put in HDFS
since this change, I still see replication at "3" for this file.

Is this a webui issue or am I setting the replication factor incorreectly?

Thanks

Search Discussions

  • Joe Travaglini at Mar 23, 2013 at 12:19 am
    John,
    The config change only applies to files that will be written after the
    configuration is set. It doesn't retroactively change files already
    written.

    If you wish to do this, you can do so by issuing a dfs setrep of 1
    recursively at the root. As the hdfs user:

    hadoop dfs -setrep -R -w 1 /

    On Fri, Mar 22, 2013 at 8:01 PM, John wrote:

    Hi,

    I'm trying to turn off replication on a 4 node cluster. In cloudera
    manager, I went to Services > HDFS > Configuration > View & Edit > Service
    Wide (on the righthand side) > Replication

    and changed Replication Factor to 1. I then restarted the cluster.

    When I go to the namenode web UI and look at a file I have put in HDFS
    since this change, I still see replication at "3" for this file.

    Is this a webui issue or am I setting the replication factor incorreectly?

    Thanks
  • John at Mar 25, 2013 at 4:34 pm
    Thanks, that is useful, but files I upload now still are uploading with
    replication 3. Is there a place I can look directly at the config files on
    the cluster to make sure that replication is 1 in those? The Cloudera
    Manager UI is still showing one and when I "Download Client Configuration"
    I see replication set to 1 in the hdfs-site.xml.
    On Friday, March 22, 2013 5:18:57 PM UTC-7, Joe Travaglini wrote:

    John,
    The config change only applies to files that will be written after the
    configuration is set. It doesn't retroactively change files already
    written.

    If you wish to do this, you can do so by issuing a dfs setrep of 1
    recursively at the root. As the hdfs user:

    hadoop dfs -setrep -R -w 1 /


    On Fri, Mar 22, 2013 at 8:01 PM, John <adl.su...@gmail.com <javascript:>>wrote:
    Hi,

    I'm trying to turn off replication on a 4 node cluster. In cloudera
    manager, I went to Services > HDFS > Configuration > View & Edit > Service
    Wide (on the righthand side) > Replication

    and changed Replication Factor to 1. I then restarted the cluster.

    When I go to the namenode web UI and look at a file I have put in HDFS
    since this change, I still see replication at "3" for this file.

    Is this a webui issue or am I setting the replication factor incorreectly?

    Thanks
  • Darren Lo at Mar 25, 2013 at 5:24 pm
    Make sure you deploy your client configuration after making any changes,
    otherwise it won't show up in /etc/hadoop/conf. You can do this from the
    cluster-wide actions menu on the main page.

    On Mon, Mar 25, 2013 at 9:34 AM, John wrote:

    Thanks, that is useful, but files I upload now still are uploading with
    replication 3. Is there a place I can look directly at the config files on
    the cluster to make sure that replication is 1 in those? The Cloudera
    Manager UI is still showing one and when I "Download Client Configuration"
    I see replication set to 1 in the hdfs-site.xml.

    On Friday, March 22, 2013 5:18:57 PM UTC-7, Joe Travaglini wrote:

    John,
    The config change only applies to files that will be written after the
    configuration is set. It doesn't retroactively change files already
    written.

    If you wish to do this, you can do so by issuing a dfs setrep of 1
    recursively at the root. As the hdfs user:

    hadoop dfs -setrep -R -w 1 /

    On Fri, Mar 22, 2013 at 8:01 PM, John wrote:

    Hi,

    I'm trying to turn off replication on a 4 node cluster. In cloudera
    manager, I went to Services > HDFS > Configuration > View & Edit > Service
    Wide (on the righthand side) > Replication

    and changed Replication Factor to 1. I then restarted the cluster.

    When I go to the namenode web UI and look at a file I have put in HDFS
    since this change, I still see replication at "3" for this file.

    Is this a webui issue or am I setting the replication factor
    incorreectly?

    Thanks

    --
    Thanks,
    Darren
  • John at Mar 25, 2013 at 6:34 pm
    Thanks, that did it. I thought I already did this, but I think I deployed
    from the HDFS service actions menu. Are the client configs that deploys
    different?

    Thanks again,
    John
    On Monday, March 25, 2013 10:24:28 AM UTC-7, Darren Lo wrote:

    Make sure you deploy your client configuration after making any changes,
    otherwise it won't show up in /etc/hadoop/conf. You can do this from the
    cluster-wide actions menu on the main page.


    On Mon, Mar 25, 2013 at 9:34 AM, John <adl.su...@gmail.com <javascript:>>wrote:
    Thanks, that is useful, but files I upload now still are uploading with
    replication 3. Is there a place I can look directly at the config files on
    the cluster to make sure that replication is 1 in those? The Cloudera
    Manager UI is still showing one and when I "Download Client Configuration"
    I see replication set to 1 in the hdfs-site.xml.

    On Friday, March 22, 2013 5:18:57 PM UTC-7, Joe Travaglini wrote:

    John,
    The config change only applies to files that will be written after the
    configuration is set. It doesn't retroactively change files already
    written.

    If you wish to do this, you can do so by issuing a dfs setrep of 1
    recursively at the root. As the hdfs user:

    hadoop dfs -setrep -R -w 1 /

    On Fri, Mar 22, 2013 at 8:01 PM, John wrote:

    Hi,

    I'm trying to turn off replication on a 4 node cluster. In cloudera
    manager, I went to Services > HDFS > Configuration > View & Edit > Service
    Wide (on the righthand side) > Replication

    and changed Replication Factor to 1. I then restarted the cluster.

    When I go to the namenode web UI and look at a file I have put in HDFS
    since this change, I still see replication at "3" for this file.

    Is this a webui issue or am I setting the replication factor
    incorreectly?

    Thanks

    --
    Thanks,
    Darren
  • Darren Lo at Mar 25, 2013 at 6:42 pm
    Hi John,

    MapReduce has a higher alternatives priority for client configs, so they'll
    take precedence over your HDFS ones if you have both HDFS and MR on the
    same host (which is usually the case). That's probably why when you
    deployed HDFS configs you didn't see it change, but deploying the
    cluster-wide ones worked.

    Thanks,
    Darren

    On Mon, Mar 25, 2013 at 11:33 AM, John wrote:

    Thanks, that did it. I thought I already did this, but I think I deployed
    from the HDFS service actions menu. Are the client configs that deploys
    different?

    Thanks again,
    John

    On Monday, March 25, 2013 10:24:28 AM UTC-7, Darren Lo wrote:

    Make sure you deploy your client configuration after making any changes,
    otherwise it won't show up in /etc/hadoop/conf. You can do this from the
    cluster-wide actions menu on the main page.

    On Mon, Mar 25, 2013 at 9:34 AM, John wrote:

    Thanks, that is useful, but files I upload now still are uploading with
    replication 3. Is there a place I can look directly at the config files on
    the cluster to make sure that replication is 1 in those? The Cloudera
    Manager UI is still showing one and when I "Download Client Configuration"
    I see replication set to 1 in the hdfs-site.xml.

    On Friday, March 22, 2013 5:18:57 PM UTC-7, Joe Travaglini wrote:

    John,
    The config change only applies to files that will be written after the
    configuration is set. It doesn't retroactively change files already
    written.

    If you wish to do this, you can do so by issuing a dfs setrep of 1
    recursively at the root. As the hdfs user:

    hadoop dfs -setrep -R -w 1 /

    On Fri, Mar 22, 2013 at 8:01 PM, John wrote:

    Hi,

    I'm trying to turn off replication on a 4 node cluster. In cloudera
    manager, I went to Services > HDFS > Configuration > View & Edit > Service
    Wide (on the righthand side) > Replication

    and changed Replication Factor to 1. I then restarted the cluster.

    When I go to the namenode web UI and look at a file I have put in HDFS
    since this change, I still see replication at "3" for this file.

    Is this a webui issue or am I setting the replication factor
    incorreectly?

    Thanks

    --
    Thanks,
    Darren

    --
    Thanks,
    Darren

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedMar 23, '13 at 12:01a
activeMar 25, '13 at 6:42p
posts6
users3
websitecloudera.com
irc#hadoop

3 users in discussion

John: 3 posts Darren Lo: 2 posts Joe Travaglini: 1 post

People

Translate

site design / logo © 2022 Grokbase