Anaconda doesn't support raid10

So after troubleshooting this for about a week, I was finally able to
create a raid 10 device by installing the system, copying the md modules
onto a floppy, and loading the raid10 module during the install.

Now the problem is that I can't get it to show up in anaconda.  It 
detects the other arrays (raid0 and raid1) fine, but the raid10 array
won't show up.  Looking through the logs (Alt-F3), I see the following 
warning:

WARNING: raid level RAID10 not supported, skipping md10.

I'm starting to hate the installer more and more.  Why won't it let me 
install on this device, even though it's working perfectly from the
shell?  Why am I the only one having this problem?  Is nobody out there 
using md based raid10?

Russ

Most people install the OS on a 2 disk raid1, then create a separate
raid10 for data storage.

Anaconda was never designed to create RAID5/RAID10 during install.

-Ross

______________________________________________________________________
This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.

Whether or not it was designed to create a Raid5/raid10, it allows the
creating of raid5 and raid6 during install.  It doesn't, however, allow 
the use of raid10 even if it's created in the shell outside of anaconda
(or if you have an old installation on a raid10).

I've just installed the system as follows

Raid1 for /boot with 2 spares (200mb)
raid0 for swap  (1GB)
raid6 for / (10GB)

after installing, I was able to create a raid10 device and successfully
mount and automount by using /etc/fstab

Now to test what happens when a drive fails.  I pulled out the first 
drive - Box refuses to boot.  Going into rescue mode, I was able to 
mount /boot, was not able to mount the swap drive (as to be expected, as
it's a raid0), was also not able to mount the / for some reason, which
is a little surprising.

I was able to mount the raid10 parition just fine.

Maybe I messed up somewhere along the line.  I'll try again, but it's 
disheartening to see that a raid6 array would die after one drive
failure, even if it was somehow my fault.

Also assuming that the raid5 array could be recovered, what would I do
with the swap partition?  Would I just recreate it from the space in the 
leftover drives and would that be all that I need to boot?

Russ

Russ,

Nothing here to help you (again - :) just looking down the road a
little. If you do get this thing working the way you want, will you be
able to trust it to stay that way?

Well, it's been my experience, that in linux, unlike windows, it might
take a while to get things the way you want, but once you do, you can
pretty much trust it to stay that way.

So yea, this is what I'm looking to do here.  I want to set up a system, 
that will live after 1 (or possibly 2) drive failures.  I want to know 
what I need to do ahead of time, so that I can be confident in my set
up, and know what to do in case disaster strikes.

Russ

Ok, my bad raid5/6 can be created during install even if OS
can't boot from it.

I guess raid10 is the red headed stepchild of anaconda...

I suggest this:

/dev/md0 raid1, 128MB partition, all 4 drives, for /boot
/dev/md1 raid1, rest of drive space, first 2 drives, for lvm
/dev/md2 raid1, rest of drive space, second 2 drives, for lvm

lvm volgroup CentOS, comprised of /dev/md1 and /dev/md2
logical vol1, root, interleave 2, mount /, 16GB
logical vol2, swap, interleave 2, swapfs, 4GB

This will provide the same performance and fail-over as a raid10.

If you remove the first disk and boot make sure BIOS is set to boot
off of disk2!

-Ross

______________________________________________________________________
This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.

I don't seem to be able to control the interleave through anaconda.  Is 
this something that can be done post install?

Also I'm not very comfortable using LVM yet.  Just getting used to md. 

Russ

My guess is that software Raid10 is still in the eats its children
stage. Most of the time I found it reliable, but I remember the kernel
developers having lots of race condition problems at times. In a
limited memory install region that is anaconda.. these might creep up
more often to be only supported by hand.

Well it's not even supported by hand.  Anaconda flat out refuses to let 
you use it.  Even when I create the device by hand (be it by loading the 
raid10 module manually and doing mdadm, or by making 2 raid1's and
putting a raid0 on top of them), anaconda just flat out refuses to see
the final device.

Russ

If you have the hardware, or the money, you can make a system pretty durable.
But you get to a point that the gains aren't worth the cost. You can get a
system to 3 "9's" fairly easy, but the cost to get to 4 "9's" is much more. If
you want something better than 4 "9's", you will have to look at clustering,
because a single reboot in a month can shoot down your numbers.

If you want total reliability, you will need hot spares and a raid method that
builds quickly, and you will need regular backups.

I'm not looking for total reliability.  I am building a low budget 
file/backup server.  I would like it to be fairly reliable with good 
performance.  Basically if 1 drive fails, I would like to still be up 
and running, even if it requires slight reconfigurations (ie recreating
the swap partition).

If 2 drives fail, I would like to still be able to be up and running
assuming I wasn't unlucky enough to have 2 drives fail in the same
mirror set.

If 3 drives fail, I'm pretty much SOL.

The most important thing is that I can easily survive a single disk
failure.

Russ

I had 2 seperate 2 drive failures on 2 brand new HP servers. I burned the
servers in over 2 weeks running various drive exercising duties like creating
and deleting files and such, and they waited to fail within an hour of each
other. Not even enough time for the hot spare to rebuild. Then I have lost 2
more drives in single event failures. HP was great with sending new drives,
usually the next day, and stated that if one more drive failed from the
original set they would replace the whole lot. But that doesn't get your data
back, or keep the server going. We even ordered an extra drive per server, and
I stuck them in the hardware closet( after some burn-in time) just to be safe.

That also relates to my adaptec raid controller nightmare, but have since been
sleeping easy since I moved to 3ware's.

I don't think anaconda will support it or raid 6 for the foreseeable future.
You have to throw up an upstream complaint, and I doubt you would see anything
till Fedora Core 8 or so. RedHat's stance seems to be that if you need that
kind of reliability, buy hardware raid that does raid 10

I like to keep things simple-minded and not fight with anadconda.
During the install, put /boot, swap, and / on your first 2 drives as
RAID1.  After that works the way you want, build whatever layout you 
want with the rest of your space and either move your /home contents and
mount point over or mount it somewhere else.  A nice feature of this 
approach is that you can upgrade to pretty much any other version/distro
     by building a new set of system disks and swapping them in, keeping 
your data intact.  I also like to use disks in swappable carriers and to 
keep a spare chassis around.  That way you can use it for testing things 
and developing your next version but if your production motherboard
fails you can just move the drives to it and keep going.

> If 2 drives fail, I would like to still be able to be up and running
> assuming I wasn't unlucky enough to have 2 drives fail in the same
> mirror set.
> If 3 drives fail, I'm pretty much SOL.
> The most important thing is that I can easily survive a single disk
> failure.

If you can deal with the space constraints of partitions that match
single disk sizes by mounting them in appropriate places it's hard to
beat RAID1.  If everything fries except one drive you can still recover 
the data that was on it - plus it gives you natural boundaries for
backups which you shouldn't ignore just because you have raid.

Anaconda does now support raid6.  The only reason I didn't go with it is 
that I heard that performance would be very bad on it.

Russ

I have 4 500GB drives.  Seems kind of a waste to put just /boot swap and 
/ on the first 2 drives.
>> If 2 drives fail, I would like to still be able to be up and running
>> assuming I wasn't unlucky enough to have 2 drives fail in the same
>> mirror set.
>> If 3 drives fail, I'm pretty much SOL.
>> The most important thing is that I can easily survive a single disk
>> failure.
>
> If you can deal with the space constraints of partitions that match
> single disk sizes by mounting them in appropriate places it's hard to
> beat RAID1. If everything fries except one drive you can still
> recover the data that was on it - plus it gives you natural boundaries
> for backups which you shouldn't ignore just because you have raid.
>
Unfortunately this is my backup server, and also file server.  While I 
may move the file server part out to another box in the future, for now
it's going to be serving two roles.  I would like to be able to depend 
on it.

In the future I might set up a backup of this server to be on Amazon's
S3.  Is there a linux program that interfaces with it? 

Russ

I typically use 36 Gig scsi's for the system.  You can use that or even 
less for the first 3 partitions where you install, then add a 4th
partition on the same pair of drives.

>> If you can deal with the space constraints of partitions that match
>> single disk sizes by mounting them in appropriate places it's hard to
>> beat RAID1. If everything fries except one drive you can still
>> recover the data that was on it - plus it gives you natural boundaries
>> for backups which you shouldn't ignore just because you have raid.
>>
> Unfortunately this is my backup server, and also file server. While I
> may move the file server part out to another box in the future, for now
> it's going to be serving two roles. I would like to be able to depend
> on it.

You are living very dangereously there.  RAID can protect you from one 
of the more likely failures, but nowhere near all of them - and some
will kill all the data in the box in one step.

> In the future I might set up a backup of this server to be on Amazon's
> S3. Is there a linux program that interfaces with it?
> Russ

I'd toss two of the drives in some desktop linux box and run backuppc on
it - and get an external drive to periodically make an offsite copy.  If 
your data compresses well you could use drives about half the size for
backuppc.

If this is a file server too, may I suggest that you keep system stuff
separate from your data like Les says he does? Did you have to get real
old stuff for your add-on 'raid' controller? A newer and less quirky
si3124 based controller cannot be much more expensive than what you have
given that you have a four port card.

Please put boot and /tmp (maybe 512MB/1GB each) on its own mirrored
partition, then make two nice big mirrors from the rest and use lvm to
stripe them. That way, you don't have to make a lot of partitions to
balance usage to get a four big partitions to get your 'raid10' array.
anaconda supports this kind of configuration easily and you can also
create the volumes for swap, /, whatever_you_fancy_partition, maybe
/opt, /usr/local, /home with anaconda.

I'm not sure what you mean by "get real old stuff for your controller". The controller is brand new, although the pc is a few years old. The controller is si3114 based. What's so quirky about it vs si3124?

Here's the way I plan to set things up. Please let me know if this is worse then what you suggest.

4 partitions per drive
1st partition - 200 mb
2nd partition - 250 mb
3rd partition - 5gb
4th partition - 745gb

Md0 raid 1 with 2 spares - 1st partition of all drives - /boot
Md1 raid0 - 2nd partition of all drives - swap
Md2 raid6 - 3rd partition of all drives - /

After install create
Md10 raid10 - 4th partition of all drives - /data

What advantages, if any, would lvm have over this set up?   


Russ
Sent wirelessly via BlackBerry from T-Mobile.

I would not make swap a RAID 0. Ever. It is fairly rare on systems to
actually use swap, so they don't need to be *that* fast. And if you lose a
drive, the system might chug along nicely until you try to access swap, in
which case I imagine the machine would crap all over itself. I would make
those partitions RAID 1, or perhaps a pair of RAID 1 md's with equal
priority.

Matt