FAQ
Hello,

Is there any File systems size standrd in CDH4. like /users, /tmp,
/home, /opt...etc. like to know the standard on all Datanodes and
Namenoses?

thanks in advance.

Regards,
Chittu

Search Discussions

  • Harsh J at Oct 17, 2012 at 10:21 pm
    Hi,

    Are you asking for HDFS directory quota standard sizes or something else?

    Installation-wise, RPMs make use of /usr and /var directories, plus
    any directories you have configured for metadata and data persistence.
    The /var is used for logging, so you will need quite some amount of
    space there depending on how many days worth of logs you'll be
    retaining.
    On Thu, Oct 18, 2012 at 1:09 AM, wrote:
    Hello,

    Is there any File systems size standrd in CDH4. like /users, /tmp,
    /home, /opt...etc. like to know the standard on all Datanodes and
    Namenoses?

    thanks in advance.

    Regards,
    Chittu


    --
    Harsh J
  • Chittu546 at Oct 17, 2012 at 8:14 pm
    Thank you Harsh for the reply..

    I like to know 2 things.

    1) OS Disk File Systems Size like /tmp, /usr, /home, /var..etc
    2) and we have 12*2TB data disks, so how many File systems will be
    appropriate for data on datanodes.

    Also, we are implementing LVM's on top JBOD for datadisks. is there
    any concern on I/O performance?

    Thanks in advance.

    Regards,
    Chittu
    On Wed, Oct 17, 2012 at 12:48 PM, Harsh J wrote:
    Hi,

    Are you asking for HDFS directory quota standard sizes or something else?

    Installation-wise, RPMs make use of /usr and /var directories, plus
    any directories you have configured for metadata and data persistence.
    The /var is used for logging, so you will need quite some amount of
    space there depending on how many days worth of logs you'll be
    retaining.
    On Thu, Oct 18, 2012 at 1:09 AM, wrote:
    Hello,

    Is there any File systems size standrd in CDH4. like /users, /tmp,
    /home, /opt...etc. like to know the standard on all Datanodes and
    Namenoses?

    thanks in advance.

    Regards,
    Chittu


    --
    Harsh J
  • Joey Echeverria at Oct 17, 2012 at 10:16 pm
    I would skip LVM for the data disks and just mount each disk as a
    standalone filesystem. Usually I mount those to /data/1,/data/2,etc.

    As for OS partitions, you want about 100 GB for /var and about 30 GB
    on /usr to fit all of the package. Usually I just do one 100 GB
    partition for / and one 100 GB partition for /var and then use the
    rest of the OS disk for /data/1.

    -Joey
    On Wed, Oct 17, 2012 at 4:14 PM, wrote:
    Thank you Harsh for the reply..

    I like to know 2 things.

    1) OS Disk File Systems Size like /tmp, /usr, /home, /var..etc
    2) and we have 12*2TB data disks, so how many File systems will be
    appropriate for data on datanodes.

    Also, we are implementing LVM's on top JBOD for datadisks. is there
    any concern on I/O performance?

    Thanks in advance.

    Regards,
    Chittu
    On Wed, Oct 17, 2012 at 12:48 PM, Harsh J wrote:
    Hi,

    Are you asking for HDFS directory quota standard sizes or something else?

    Installation-wise, RPMs make use of /usr and /var directories, plus
    any directories you have configured for metadata and data persistence.
    The /var is used for logging, so you will need quite some amount of
    space there depending on how many days worth of logs you'll be
    retaining.
    On Thu, Oct 18, 2012 at 1:09 AM, wrote:
    Hello,

    Is there any File systems size standrd in CDH4. like /users, /tmp,
    /home, /opt...etc. like to know the standard on all Datanodes and
    Namenoses?

    thanks in advance.

    Regards,
    Chittu


    --
    Harsh J


    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.
  • Chittu546 at Oct 17, 2012 at 10:10 pm
    Thank you very much Joey

    Regards,
    Chittu
    On Wed, Oct 17, 2012 at 2:14 PM, Joey Echeverria wrote:
    I would skip LVM for the data disks and just mount each disk as a
    standalone filesystem. Usually I mount those to /data/1,/data/2,etc.

    As for OS partitions, you want about 100 GB for /var and about 30 GB
    on /usr to fit all of the package. Usually I just do one 100 GB
    partition for / and one 100 GB partition for /var and then use the
    rest of the OS disk for /data/1.

    -Joey
    On Wed, Oct 17, 2012 at 4:14 PM, wrote:
    Thank you Harsh for the reply..

    I like to know 2 things.

    1) OS Disk File Systems Size like /tmp, /usr, /home, /var..etc
    2) and we have 12*2TB data disks, so how many File systems will be
    appropriate for data on datanodes.

    Also, we are implementing LVM's on top JBOD for datadisks. is there
    any concern on I/O performance?

    Thanks in advance.

    Regards,
    Chittu
    On Wed, Oct 17, 2012 at 12:48 PM, Harsh J wrote:
    Hi,

    Are you asking for HDFS directory quota standard sizes or something else?

    Installation-wise, RPMs make use of /usr and /var directories, plus
    any directories you have configured for metadata and data persistence.
    The /var is used for logging, so you will need quite some amount of
    space there depending on how many days worth of logs you'll be
    retaining.
    On Thu, Oct 18, 2012 at 1:09 AM, wrote:
    Hello,

    Is there any File systems size standrd in CDH4. like /users, /tmp,
    /home, /opt...etc. like to know the standard on all Datanodes and
    Namenoses?

    thanks in advance.

    Regards,
    Chittu


    --
    Harsh J


    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.
  • Chittu546 at Oct 24, 2012 at 4:47 pm
    Hi Joey,

    we can implement this on all cluster nodes right, including data, name
    and edge nodes?

    Thanks,
    Chittu
    On Wed, Oct 17, 2012 at 2:14 PM, Joey Echeverria wrote:
    I would skip LVM for the data disks and just mount each disk as a
    standalone filesystem. Usually I mount those to /data/1,/data/2,etc.

    As for OS partitions, you want about 100 GB for /var and about 30 GB
    on /usr to fit all of the package. Usually I just do one 100 GB
    partition for / and one 100 GB partition for /var and then use the
    rest of the OS disk for /data/1.

    -Joey
    On Wed, Oct 17, 2012 at 4:14 PM, wrote:
    Thank you Harsh for the reply..

    I like to know 2 things.

    1) OS Disk File Systems Size like /tmp, /usr, /home, /var..etc
    2) and we have 12*2TB data disks, so how many File systems will be
    appropriate for data on datanodes.

    Also, we are implementing LVM's on top JBOD for datadisks. is there
    any concern on I/O performance?

    Thanks in advance.

    Regards,
    Chittu
    On Wed, Oct 17, 2012 at 12:48 PM, Harsh J wrote:
    Hi,

    Are you asking for HDFS directory quota standard sizes or something else?

    Installation-wise, RPMs make use of /usr and /var directories, plus
    any directories you have configured for metadata and data persistence.
    The /var is used for logging, so you will need quite some amount of
    space there depending on how many days worth of logs you'll be
    retaining.
    On Thu, Oct 18, 2012 at 1:09 AM, wrote:
    Hello,

    Is there any File systems size standrd in CDH4. like /users, /tmp,
    /home, /opt...etc. like to know the standard on all Datanodes and
    Namenoses?

    thanks in advance.

    Regards,
    Chittu


    --
    Harsh J


    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.
  • Joey Echeverria at Oct 25, 2012 at 10:56 am
    There can be some value to using RAID on the name nodes and edge nodes, but if you want to keep things simple, it shouldn't hurt to use the same FS layout for all nodes.

    -Joey

    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.
    On Oct 24, 2012, at 12:47, chittu546@gmail.com wrote:

    Hi Joey,

    we can implement this on all cluster nodes right, including data, name
    and edge nodes?

    Thanks,
    Chittu
    On Wed, Oct 17, 2012 at 2:14 PM, Joey Echeverria wrote:
    I would skip LVM for the data disks and just mount each disk as a
    standalone filesystem. Usually I mount those to /data/1,/data/2,etc.

    As for OS partitions, you want about 100 GB for /var and about 30 GB
    on /usr to fit all of the package. Usually I just do one 100 GB
    partition for / and one 100 GB partition for /var and then use the
    rest of the OS disk for /data/1.

    -Joey
    On Wed, Oct 17, 2012 at 4:14 PM, wrote:
    Thank you Harsh for the reply..

    I like to know 2 things.

    1) OS Disk File Systems Size like /tmp, /usr, /home, /var..etc
    2) and we have 12*2TB data disks, so how many File systems will be
    appropriate for data on datanodes.

    Also, we are implementing LVM's on top JBOD for datadisks. is there
    any concern on I/O performance?

    Thanks in advance.

    Regards,
    Chittu
    On Wed, Oct 17, 2012 at 12:48 PM, Harsh J wrote:
    Hi,

    Are you asking for HDFS directory quota standard sizes or something else?

    Installation-wise, RPMs make use of /usr and /var directories, plus
    any directories you have configured for metadata and data persistence.
    The /var is used for logging, so you will need quite some amount of
    space there depending on how many days worth of logs you'll be
    retaining.
    On Thu, Oct 18, 2012 at 1:09 AM, wrote:
    Hello,

    Is there any File systems size standrd in CDH4. like /users, /tmp,
    /home, /opt...etc. like to know the standard on all Datanodes and
    Namenoses?

    thanks in advance.

    Regards,
    Chittu


    --
    Harsh J


    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.
  • Chittu546 at Oct 31, 2012 at 6:32 pm
    Hi Joey,

    we are using LVM setup for our Data disks. and we like to create ext4
    filesystem and want to give mount point /data01. and then we will
    deploy CDH4 on top of it. will this way work or just we can leave data
    disks with partitioning and Cloudera manager will create file systems
    and mount points and create HDFS? please let me know.

    Thanks
    Chittu
    On Thu, Oct 25, 2012 at 3:54 AM, Joey Echeverria wrote:
    There can be some value to using RAID on the name nodes and edge nodes, but if you want to keep things simple, it shouldn't hurt to use the same FS layout for all nodes.

    -Joey

    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.
    On Oct 24, 2012, at 12:47, chittu546@gmail.com wrote:

    Hi Joey,

    we can implement this on all cluster nodes right, including data, name
    and edge nodes?

    Thanks,
    Chittu
    On Wed, Oct 17, 2012 at 2:14 PM, Joey Echeverria wrote:
    I would skip LVM for the data disks and just mount each disk as a
    standalone filesystem. Usually I mount those to /data/1,/data/2,etc.

    As for OS partitions, you want about 100 GB for /var and about 30 GB
    on /usr to fit all of the package. Usually I just do one 100 GB
    partition for / and one 100 GB partition for /var and then use the
    rest of the OS disk for /data/1.

    -Joey
    On Wed, Oct 17, 2012 at 4:14 PM, wrote:
    Thank you Harsh for the reply..

    I like to know 2 things.

    1) OS Disk File Systems Size like /tmp, /usr, /home, /var..etc
    2) and we have 12*2TB data disks, so how many File systems will be
    appropriate for data on datanodes.

    Also, we are implementing LVM's on top JBOD for datadisks. is there
    any concern on I/O performance?

    Thanks in advance.

    Regards,
    Chittu
    On Wed, Oct 17, 2012 at 12:48 PM, Harsh J wrote:
    Hi,

    Are you asking for HDFS directory quota standard sizes or something else?

    Installation-wise, RPMs make use of /usr and /var directories, plus
    any directories you have configured for metadata and data persistence.
    The /var is used for logging, so you will need quite some amount of
    space there depending on how many days worth of logs you'll be
    retaining.
    On Thu, Oct 18, 2012 at 1:09 AM, wrote:
    Hello,

    Is there any File systems size standrd in CDH4. like /users, /tmp,
    /home, /opt...etc. like to know the standard on all Datanodes and
    Namenoses?

    thanks in advance.

    Regards,
    Chittu


    --
    Harsh J


    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.
  • Joey Echeverria at Oct 31, 2012 at 9:30 pm
    You need to do the partitioning and formatting, but CM will take over
    from there. You don't need to use LVM as you want a single partition
    per data disk, but it won't cause too much overhead.

    -Joey
    On Wed, Oct 31, 2012 at 2:26 PM, wrote:
    Hi Joey,

    we are using LVM setup for our Data disks. and we like to create ext4
    filesystem and want to give mount point /data01. and then we will
    deploy CDH4 on top of it. will this way work or just we can leave data
    disks with partitioning and Cloudera manager will create file systems
    and mount points and create HDFS? please let me know.

    Thanks
    Chittu
    On Thu, Oct 25, 2012 at 3:54 AM, Joey Echeverria wrote:
    There can be some value to using RAID on the name nodes and edge nodes, but if you want to keep things simple, it shouldn't hurt to use the same FS layout for all nodes.

    -Joey

    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.
    On Oct 24, 2012, at 12:47, chittu546@gmail.com wrote:

    Hi Joey,

    we can implement this on all cluster nodes right, including data, name
    and edge nodes?

    Thanks,
    Chittu
    On Wed, Oct 17, 2012 at 2:14 PM, Joey Echeverria wrote:
    I would skip LVM for the data disks and just mount each disk as a
    standalone filesystem. Usually I mount those to /data/1,/data/2,etc.

    As for OS partitions, you want about 100 GB for /var and about 30 GB
    on /usr to fit all of the package. Usually I just do one 100 GB
    partition for / and one 100 GB partition for /var and then use the
    rest of the OS disk for /data/1.

    -Joey
    On Wed, Oct 17, 2012 at 4:14 PM, wrote:
    Thank you Harsh for the reply..

    I like to know 2 things.

    1) OS Disk File Systems Size like /tmp, /usr, /home, /var..etc
    2) and we have 12*2TB data disks, so how many File systems will be
    appropriate for data on datanodes.

    Also, we are implementing LVM's on top JBOD for datadisks. is there
    any concern on I/O performance?

    Thanks in advance.

    Regards,
    Chittu
    On Wed, Oct 17, 2012 at 12:48 PM, Harsh J wrote:
    Hi,

    Are you asking for HDFS directory quota standard sizes or something else?

    Installation-wise, RPMs make use of /usr and /var directories, plus
    any directories you have configured for metadata and data persistence.
    The /var is used for logging, so you will need quite some amount of
    space there depending on how many days worth of logs you'll be
    retaining.
    On Thu, Oct 18, 2012 at 1:09 AM, wrote:
    Hello,

    Is there any File systems size standrd in CDH4. like /users, /tmp,
    /home, /opt...etc. like to know the standard on all Datanodes and
    Namenoses?

    thanks in advance.

    Regards,
    Chittu


    --
    Harsh J


    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.


    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.
  • Chittu546 at Oct 31, 2012 at 8:04 pm
    Please also let me know which Filesystem is recommendable ext3/ext4 as
    we are using CDH4.

    much appreciate your help on this.

    Thanks
    Chittu
    On Wed, Oct 31, 2012 at 12:46 PM, Joey Echeverria wrote:
    You need to do the partitioning and formatting, but CM will take over
    from there. You don't need to use LVM as you want a single partition
    per data disk, but it won't cause too much overhead.

    -Joey
    On Wed, Oct 31, 2012 at 2:26 PM, wrote:
    Hi Joey,

    we are using LVM setup for our Data disks. and we like to create ext4
    filesystem and want to give mount point /data01. and then we will
    deploy CDH4 on top of it. will this way work or just we can leave data
    disks with partitioning and Cloudera manager will create file systems
    and mount points and create HDFS? please let me know.

    Thanks
    Chittu
    On Thu, Oct 25, 2012 at 3:54 AM, Joey Echeverria wrote:
    There can be some value to using RAID on the name nodes and edge nodes, but if you want to keep things simple, it shouldn't hurt to use the same FS layout for all nodes.

    -Joey

    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.
    On Oct 24, 2012, at 12:47, chittu546@gmail.com wrote:

    Hi Joey,

    we can implement this on all cluster nodes right, including data, name
    and edge nodes?

    Thanks,
    Chittu
    On Wed, Oct 17, 2012 at 2:14 PM, Joey Echeverria wrote:
    I would skip LVM for the data disks and just mount each disk as a
    standalone filesystem. Usually I mount those to /data/1,/data/2,etc.

    As for OS partitions, you want about 100 GB for /var and about 30 GB
    on /usr to fit all of the package. Usually I just do one 100 GB
    partition for / and one 100 GB partition for /var and then use the
    rest of the OS disk for /data/1.

    -Joey
    On Wed, Oct 17, 2012 at 4:14 PM, wrote:
    Thank you Harsh for the reply..

    I like to know 2 things.

    1) OS Disk File Systems Size like /tmp, /usr, /home, /var..etc
    2) and we have 12*2TB data disks, so how many File systems will be
    appropriate for data on datanodes.

    Also, we are implementing LVM's on top JBOD for datadisks. is there
    any concern on I/O performance?

    Thanks in advance.

    Regards,
    Chittu
    On Wed, Oct 17, 2012 at 12:48 PM, Harsh J wrote:
    Hi,

    Are you asking for HDFS directory quota standard sizes or something else?

    Installation-wise, RPMs make use of /usr and /var directories, plus
    any directories you have configured for metadata and data persistence.
    The /var is used for logging, so you will need quite some amount of
    space there depending on how many days worth of logs you'll be
    retaining.
    On Thu, Oct 18, 2012 at 1:09 AM, wrote:
    Hello,

    Is there any File systems size standrd in CDH4. like /users, /tmp,
    /home, /opt...etc. like to know the standard on all Datanodes and
    Namenoses?

    thanks in advance.

    Regards,
    Chittu


    --
    Harsh J


    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.


    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.
  • Joey Echeverria at Oct 31, 2012 at 8:34 pm
    ext4
    On Wed, Oct 31, 2012 at 4:04 PM, wrote:
    Please also let me know which Filesystem is recommendable ext3/ext4 as
    we are using CDH4.

    much appreciate your help on this.

    Thanks
    Chittu
    On Wed, Oct 31, 2012 at 12:46 PM, Joey Echeverria wrote:
    You need to do the partitioning and formatting, but CM will take over
    from there. You don't need to use LVM as you want a single partition
    per data disk, but it won't cause too much overhead.

    -Joey
    On Wed, Oct 31, 2012 at 2:26 PM, wrote:
    Hi Joey,

    we are using LVM setup for our Data disks. and we like to create ext4
    filesystem and want to give mount point /data01. and then we will
    deploy CDH4 on top of it. will this way work or just we can leave data
    disks with partitioning and Cloudera manager will create file systems
    and mount points and create HDFS? please let me know.

    Thanks
    Chittu
    On Thu, Oct 25, 2012 at 3:54 AM, Joey Echeverria wrote:
    There can be some value to using RAID on the name nodes and edge nodes, but if you want to keep things simple, it shouldn't hurt to use the same FS layout for all nodes.

    -Joey

    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.
    On Oct 24, 2012, at 12:47, chittu546@gmail.com wrote:

    Hi Joey,

    we can implement this on all cluster nodes right, including data, name
    and edge nodes?

    Thanks,
    Chittu
    On Wed, Oct 17, 2012 at 2:14 PM, Joey Echeverria wrote:
    I would skip LVM for the data disks and just mount each disk as a
    standalone filesystem. Usually I mount those to /data/1,/data/2,etc.

    As for OS partitions, you want about 100 GB for /var and about 30 GB
    on /usr to fit all of the package. Usually I just do one 100 GB
    partition for / and one 100 GB partition for /var and then use the
    rest of the OS disk for /data/1.

    -Joey
    On Wed, Oct 17, 2012 at 4:14 PM, wrote:
    Thank you Harsh for the reply..

    I like to know 2 things.

    1) OS Disk File Systems Size like /tmp, /usr, /home, /var..etc
    2) and we have 12*2TB data disks, so how many File systems will be
    appropriate for data on datanodes.

    Also, we are implementing LVM's on top JBOD for datadisks. is there
    any concern on I/O performance?

    Thanks in advance.

    Regards,
    Chittu
    On Wed, Oct 17, 2012 at 12:48 PM, Harsh J wrote:
    Hi,

    Are you asking for HDFS directory quota standard sizes or something else?

    Installation-wise, RPMs make use of /usr and /var directories, plus
    any directories you have configured for metadata and data persistence.
    The /var is used for logging, so you will need quite some amount of
    space there depending on how many days worth of logs you'll be
    retaining.
    On Thu, Oct 18, 2012 at 1:09 AM, wrote:
    Hello,

    Is there any File systems size standrd in CDH4. like /users, /tmp,
    /home, /opt...etc. like to know the standard on all Datanodes and
    Namenoses?

    thanks in advance.

    Regards,
    Chittu


    --
    Harsh J


    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.


    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.


    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.
  • Chris at Nov 1, 2012 at 9:47 pm
    Just to chime in here for other ideas--

    We use the following, we have datanodes with 4 drives each, and each bay
    has a 2TB SATA drive:

    *Disk 1 - /dev/sda*
    - 100GB / filesystem - ext4
    - 50GB /var filesystem for logging, ext4 - this way if logs go crazy and
    fill the filesystem up, it won't affect the actual operating system FS
    - Rest of disk for /data/disk0

    *Disk 2 - /dev/sdb*
    Entire disk, ext4 (no LVM!) - /data/disk1

    *Disk 3 - /dev/sdc*
    Entire disk, ext4 (no LVM!) - /data/disk2

    *Disk 4 - /dev/sdd*
    Entire disk, ext4 (no LVM!) - /data/disk3


    For Disk 1, we have aprox 1.5TB of space allocated to the /data/disk0
    filesystem. We do not add this into the drives to use for Hadoop/Cloudera
    - *we keep /data/disk0 around for emergencies*. Then, if there is an
    emergency and disk space is critically needed, we just add /data/disk0 into
    the lineup via Cloudera Manager, restart the datanode and extra space is
    immediately available.

    We keep this as an emergency drive on purpose, so that the OS can operate
    on a disk all of it's own and without hadoop data rolling into it which
    could cause some OS/disk contention issues under high loads.

    Disks 2, 3, and 4 are the main 2TB drives and can run without causing
    contention to the OS.

    If a disk goes out, we don't usually put in a new drive and newfs it, we
    simply pull the whole machine and reload it from scratch since we are a RF
    of 3. This also helps to balance the data out on a node basis so that when
    it comes online, all the incoming data gets round robin'd to the newly
    loaded machine.

    Kind of a long statement, but perhaps it might help someone decide things
    in the future - lastly, I don't think XFS would be a bad idea to use
    either, but my FS of choice for this type of installation is by far ext4 on
    a drive with no LVM.


    Thanks,

    -Chris


    On Wed, Oct 31, 2012 at 1:34 PM, Joey Echeverria wrote:

    ext4
    On Wed, Oct 31, 2012 at 4:04 PM, wrote:
    Please also let me know which Filesystem is recommendable ext3/ext4 as
    we are using CDH4.

    much appreciate your help on this.

    Thanks
    Chittu
    On Wed, Oct 31, 2012 at 12:46 PM, Joey Echeverria wrote:
    You need to do the partitioning and formatting, but CM will take over
    from there. You don't need to use LVM as you want a single partition
    per data disk, but it won't cause too much overhead.

    -Joey
    On Wed, Oct 31, 2012 at 2:26 PM, wrote:
    Hi Joey,

    we are using LVM setup for our Data disks. and we like to create ext4
    filesystem and want to give mount point /data01. and then we will
    deploy CDH4 on top of it. will this way work or just we can leave data
    disks with partitioning and Cloudera manager will create file systems
    and mount points and create HDFS? please let me know.

    Thanks
    Chittu
    On Thu, Oct 25, 2012 at 3:54 AM, Joey Echeverria wrote:
    There can be some value to using RAID on the name nodes and edge
    nodes, but if you want to keep things simple, it shouldn't hurt to use the
    same FS layout for all nodes.
    -Joey

    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.
    On Oct 24, 2012, at 12:47, chittu546@gmail.com wrote:

    Hi Joey,

    we can implement this on all cluster nodes right, including data,
    name
    and edge nodes?

    Thanks,
    Chittu
    On Wed, Oct 17, 2012 at 2:14 PM, Joey Echeverria wrote:
    I would skip LVM for the data disks and just mount each disk as a
    standalone filesystem. Usually I mount those to /data/1,/data/2,etc.

    As for OS partitions, you want about 100 GB for /var and about 30 GB
    on /usr to fit all of the package. Usually I just do one 100 GB
    partition for / and one 100 GB partition for /var and then use the
    rest of the OS disk for /data/1.

    -Joey
    On Wed, Oct 17, 2012 at 4:14 PM, wrote:
    Thank you Harsh for the reply..

    I like to know 2 things.

    1) OS Disk File Systems Size like /tmp, /usr, /home, /var..etc
    2) and we have 12*2TB data disks, so how many File systems will be
    appropriate for data on datanodes.

    Also, we are implementing LVM's on top JBOD for datadisks. is there
    any concern on I/O performance?

    Thanks in advance.

    Regards,
    Chittu
    On Wed, Oct 17, 2012 at 12:48 PM, Harsh J wrote:
    Hi,

    Are you asking for HDFS directory quota standard sizes or
    something else?
    Installation-wise, RPMs make use of /usr and /var directories,
    plus
    any directories you have configured for metadata and data
    persistence.
    The /var is used for logging, so you will need quite some amount
    of
    space there depending on how many days worth of logs you'll be
    retaining.
    On Thu, Oct 18, 2012 at 1:09 AM, wrote:
    Hello,

    Is there any File systems size standrd in CDH4. like /users,
    /tmp,
    /home, /opt...etc. like to know the standard on all Datanodes and
    Namenoses?

    thanks in advance.

    Regards,
    Chittu


    --
    Harsh J


    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.


    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.


    --
    Joey Echeverria
    Principal Solutions Architect
    Cloudera, Inc.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedOct 17, '12 at 8:14p
activeNov 1, '12 at 9:47p
posts12
users4
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase