FAQ
I have an 8-core SuperMicro Xeon server with CentOS 6.3. The OS is
installed on a 120 GB SSD connected by SATA, the machine also contains
an Areca SAS controller with 24 drives connected. The motherboard is a
SuperMicro X9DA7.


When I installed the OS, I used the default options, which creates an
LVM volume group to contain / and /home, and keeps /boot and /boot/efi
outside the volume group.


The machine is a couple of months old, and has been stable. While
installing some new hardware, I decided to also clean up the cabling in
the box, since it was a bit messy. In doing this, I probably moved the
boot SSD disk to another port on the motherboard (it has a bunch, 2 SATA
6GBps and 6 SATA 3GBps).


When I booted the box after this, I got a kernel panic, the typical
"Can't find root device".


I read some docs, and first tried to boot from a rescue disc and
reinstal GRUB, but that didn't change anything. Further Googling got me
the rdshell kernel parameter, and that dropped me to a shell when it
failed to find the root device.


Reading https://fedoraproject.org/wiki/How_to_debug_Dracut_problems , I
did the following:


# lvm vgscan
# lvm vgchange -ay


And then


# ln -s /dev/mapper/<volumegroup>-<root_volume> /dev/root
# exit


After this, the box boots up normally, and everything works as it
should. However, when I reboot, it again fails to find the root device.


So, after all this, my question is, how do I make Dracut (I'm assuming)
understand that this LVM volume is my root device and pick it up
automatically?


And, is there a way to avoid this problem in the future, if I move
drives around? Surely it can't be normal for this to happen just because
I connect a drive to another port?


--
Joakim Ziegler - Supervisor de postproducci?n - Terminal
joakim at terminalmx.com - 044 55 2971 8514 - 5264 0864

Search Discussions

  • Barry Brimer at Mar 23, 2013 at 3:33 am

    When I booted the box after this, I got a kernel panic, the typical
    "Can't find root device".
    Reading https://fedoraproject.org/wiki/How_to_debug_Dracut_problems , I
    did the following:

    # lvm vgscan
    # lvm vgchange -ay

    And then

    # ln -s /dev/mapper/<volumegroup>-<root_volume> /dev/root
    # exit

    After this, the box boots up normally, and everything works as it
    should. However, when I reboot, it again fails to find the root device.

    So, after all this, my question is, how do I make Dracut (I'm assuming)
    understand that this LVM volume is my root device and pick it up
    automatically?

    What does your kernel line in grub look like?


    Barry
  • Joakim Ziegler at Mar 24, 2013 at 12:55 am
    Grub (in the menu) has the following commands:


    root (hd0,1)


    kernel /vmlinuz-2.6.32-279.el6.x86_64 ro
    root=/dev/mapper/vg_resolve02-lv_root rd_NO_LUKS LANG=en.US.UTF-8
    rd_NO_MD crashkernel8M rd_LVM_LV=vg_resolve02/lv_root
    SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb
    quiet pcie_aspm=off


    When I successfully booted manually, I removed "rhgb quiet" and added
    "rdshell" to that line.


    To the best of my memory, that line is stock, I don't recall ever
    changing it permanently.


    The names of the volume group and logical volume in that line correspond
    to my actual root device.


    --
    Joakim Ziegler - Supervisor de postproducci?n - Terminal
    joakim at terminalmx.com - 044 55 2971 8514 - 5264 0864

    On 22/03/13 21:33, Barry Brimer wrote:
    When I booted the box after this, I got a kernel panic, the typical
    "Can't find root device".
    Reading https://fedoraproject.org/wiki/How_to_debug_Dracut_problems , I
    did the following:

    # lvm vgscan
    # lvm vgchange -ay

    And then

    # ln -s /dev/mapper/<volumegroup>-<root_volume> /dev/root
    # exit

    After this, the box boots up normally, and everything works as it
    should. However, when I reboot, it again fails to find the root device.

    So, after all this, my question is, how do I make Dracut (I'm assuming)
    understand that this LVM volume is my root device and pick it up
    automatically?
    What does your kernel line in grub look like?

    Barry
    _______________________________________________
    CentOS mailing list
    CentOS at centos.org
    http://lists.centos.org/mailman/listinfo/centos
  • Joakim Ziegler at Mar 24, 2013 at 1:24 am
    Oh, and the exact Dracut error I get is:


    dracut Warning: No root device "block:/dev/mapper/vg_resolve02-lv_root"
    found




    dracut Warning: LVM vg_resolve02/lv_root not found




    But then:


    # lvm vgscan




    Found volume group "vg_resolve02" using metadata type lvm2


    # lvm vgchange -ay


    1 logical volume(s) in volume group "vg_resolve02" now active




    # ls /dev/mapper


    control vg_resolve02-lv_root


    # ln -s /dev/mapper/vg_resolve02-lv_root /dev/root


    ln: creating symbolic link "/dev/root": File exists


    # ls -l /dev/root


    /dev/root -> dm-0


    # rm /dev/root
    # ln -s /dev/mapper/vg_resolve02-lv_root /dev/root
    # exit


    And everything boots normally.


    Apologies if there are minor mistakes or omissions in this text. Since I
    can't copy/paste, I've transcribed it, excluding some parts, like the
    permissions of the symlink in the ls output. I have, however, double
    checked the important parts, like the names of devices and files.




    --
    Joakim Ziegler - Supervisor de postproducci?n - Terminal
    joakim at terminalmx.com - 044 55 2971 8514 - 5264 0864

    On 23/03/13 18:55, Joakim Ziegler wrote:
    Grub (in the menu) has the following commands:

    root (hd0,1)

    kernel /vmlinuz-2.6.32-279.el6.x86_64 ro
    root=/dev/mapper/vg_resolve02-lv_root rd_NO_LUKS LANG=en.US.UTF-8
    rd_NO_MD crashkernel8M rd_LVM_LV=vg_resolve02/lv_root
    SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb
    quiet pcie_aspm=off

    When I successfully booted manually, I removed "rhgb quiet" and added
    "rdshell" to that line.

    To the best of my memory, that line is stock, I don't recall ever
    changing it permanently.

    The names of the volume group and logical volume in that line correspond
    to my actual root device.
  • Gordon Messmer at Mar 25, 2013 at 5:29 am

    On 03/22/2013 08:27 PM, Joakim Ziegler wrote:
    So, after all this, my question is, how do I make Dracut (I'm assuming)
    understand that this LVM volume is my root device and pick it up
    automatically?

    I've looked through Dracut trying to spot circumstances that might cause
    the problem that you've described, but came up with nothing. udev
    should be scanning block devices as they become available, and setting
    up any logical volumes on all of the available block devices.


    It may be useful to capture some information in the debugging shell,
    before running vgscan.


    As suggested in the fedora debugging document, capture the output of the
    following commands to get a better idea of what the kernel knows about
    block devices before you manually start the volumes, and maybe that'll
    lead us to some conclusion about why the devices aren't found.


    lvm pvdisplay
    lvm vgdisplay
    lvm lvdisplay
    blkid
    dmesg
  • Joakim Ziegler at Mar 25, 2013 at 5:38 am
    Thank you, I will do this tomorrow. It'll take me a little time, since I
    need to transcribe everything manually, but I'll get it done. It's just
    a very weird problem all in all.


    --
    Joakim Ziegler - Supervisor de postproducci?n - Terminal
    joakim at terminalmx.com - 044 55 2971 8514 - 5264 0864

    On 24/03/13 23:29, Gordon Messmer wrote:
    On 03/22/2013 08:27 PM, Joakim Ziegler wrote:
    So, after all this, my question is, how do I make Dracut (I'm assuming)
    understand that this LVM volume is my root device and pick it up
    automatically?
    I've looked through Dracut trying to spot circumstances that might cause
    the problem that you've described, but came up with nothing. udev
    should be scanning block devices as they become available, and setting
    up any logical volumes on all of the available block devices.

    It may be useful to capture some information in the debugging shell,
    before running vgscan.

    As suggested in the fedora debugging document, capture the output of the
    following commands to get a better idea of what the kernel knows about
    block devices before you manually start the volumes, and maybe that'll
    lead us to some conclusion about why the devices aren't found.

    lvm pvdisplay
    lvm vgdisplay
    lvm lvdisplay
    blkid
    dmesg

    _______________________________________________
    CentOS mailing list
    CentOS at centos.org
    http://lists.centos.org/mailman/listinfo/centos
  • Gordon Messmer at Mar 25, 2013 at 5:41 am

    On 03/24/2013 10:38 PM, Joakim Ziegler wrote:
    Thank you, I will do this tomorrow. It'll take me a little time, since I
    need to transcribe everything manually, but I'll get it done. It's just
    a very weird problem all in all.

    You should be able to pipe the output into a file and copy it to the
    actual root filesystem after vgchange & mount. Unless there's
    absolutely no writable space in the rescue shell?
  • Joakim Ziegler at Mar 25, 2013 at 5:43 am
    I haven't actually tried writing anywhere in the rescue shell before
    vgchange and mount. I'll give it a try, that would simplify things.


    --
    Joakim Ziegler - Supervisor de postproducci?n - Terminal
    joakim at terminalmx.com - 044 55 2971 8514 - 5264 0864

    On 24/03/13 23:41, Gordon Messmer wrote:
    On 03/24/2013 10:38 PM, Joakim Ziegler wrote:
    Thank you, I will do this tomorrow. It'll take me a little time, since I
    need to transcribe everything manually, but I'll get it done. It's just
    a very weird problem all in all.
    You should be able to pipe the output into a file and copy it to the
    actual root filesystem after vgchange & mount. Unless there's
    absolutely no writable space in the rescue shell?
    _______________________________________________
    CentOS mailing list
    CentOS at centos.org
    http://lists.centos.org/mailman/listinfo/centos
  • Gordon Messmer at Mar 25, 2013 at 6:21 am

    On 03/24/2013 10:43 PM, Joakim Ziegler wrote:
    I haven't actually tried writing anywhere in the rescue shell before
    vgchange and mount. I'll give it a try, that would simplify things.

    If nothing else, you probably can fit most or all of that in the shell's
    environment. In case it's ever useful:


    # debug=$(lvm pvdisplay; lvm vgdisplay; lvm lvdisplay; blkid; dmesg)
    # vgchange -a y
    # mount ...
    # echo "$debug" > /mnt/sysroot/root/debug
  • Joakim Ziegler at Mar 26, 2013 at 1:26 am
    Hi, Gordon, this was indeed a very good idea. I'm attaching that file
    here, as it's a bit large. Thanks in advance for help and ideas.


    --
    Joakim Ziegler - Supervisor de postproducci?n - Terminal
    joakim at terminalmx.com - 044 55 2971 8514 - 5264 0864

    On 25/03/13 0:21, Gordon Messmer wrote:
    On 03/24/2013 10:43 PM, Joakim Ziegler wrote:
    I haven't actually tried writing anywhere in the rescue shell before
    vgchange and mount. I'll give it a try, that would simplify things.
    If nothing else, you probably can fit most or all of that in the shell's
    environment. In case it's ever useful:

    # debug=$(lvm pvdisplay; lvm vgdisplay; lvm lvdisplay; blkid; dmesg)
    # vgchange -a y
    # mount ...
    # echo "$debug" > /mnt/sysroot/root/debug

    _______________________________________________
    CentOS mailing list
    CentOS at centos.org
    http://lists.centos.org/mailman/listinfo/centos
  • Joakim Ziegler at Mar 26, 2013 at 1:35 am

    On 25/03/13 19:26, Joakim Ziegler wrote:
    Hi, Gordon, this was indeed a very good idea. I'm attaching that file
    here, as it's a bit large. Thanks in advance for help and ideas.

    Hm, it seems the list strips attachments, and just pasting it makes the
    mail too big to go through, so, pastebin to the rescue:


    http://pastebin.com/x4wqzHyc


    That's the output of, like you suggested:


    lvm pvdisplay; lvm vgdisplay; lvm lvdisplay; blkid; dmesg


    In that order.


    --
    Joakim Ziegler - Supervisor de postproducci?n - Terminal
    joakim at terminalmx.com - 044 55 2971 8514 - 5264 0864
  • Gordon Messmer at Mar 26, 2013 at 5:26 am

    On 03/25/2013 06:35 PM, Joakim Ziegler wrote:
    That's the output of, like you suggested:

    And you ran that before you ran "vgchange -a y"? That doesn't make any
    sense. The commands show the volume group active. I can't see any
    reason why the system wouldn't boot.


    I hate for you to keep rebooting your server, but do the device nodes
    look correct in both /dev/mapper and /dev/vg_resolve02 at that point?
  • Joakim Ziegler at Mar 26, 2013 at 5:35 am
    Yes, I ran that immediately after getting dropped to the shell. I can
    take a look at the device nodes tomorrow, but if I remember correctly,
    /dev/mapper contained only the file "control" before running vgchange
    -ay, that is, there was no "vg_resolve02-lv_root" device there. That
    device only shows up after I run vgchange -ay.


    I did not check whether /dev/vg_resolve02 exists, I can do that tomorrow.


    --
    Joakim Ziegler - Supervisor de postproducci?n - Terminal
    joakim at terminalmx.com - 044 55 2971 8514 - 5264 0864

    On 25/03/13 23:26, Gordon Messmer wrote:
    On 03/25/2013 06:35 PM, Joakim Ziegler wrote:
    That's the output of, like you suggested:
    And you ran that before you ran "vgchange -a y"? That doesn't make any
    sense. The commands show the volume group active. I can't see any
    reason why the system wouldn't boot.

    I hate for you to keep rebooting your server, but do the device nodes
    look correct in both /dev/mapper and /dev/vg_resolve02 at that point?

    _______________________________________________
    CentOS mailing list
    CentOS at centos.org
    http://lists.centos.org/mailman/listinfo/centos
  • Patrick Flaherty at Mar 26, 2013 at 2:08 pm

    On Tue, Mar 26, 2013 at 1:35 AM, Joakim Ziegler wrote:
    Yes, I ran that immediately after getting dropped to the shell. I can
    take a look at the device nodes tomorrow, but if I remember correctly,
    /dev/mapper contained only the file "control" before running vgchange
    -ay, that is, there was no "vg_resolve02-lv_root" device there. That
    device only shows up after I run vgchange -ay.

    I did not check whether /dev/vg_resolve02 exists, I can do that tomorrow.

    --
    Joakim Ziegler - Supervisor de postproducci?n - Terminal
    joakim at terminalmx.com - 044 55 2971 8514 - 5264 0864
    On 25/03/13 23:26, Gordon Messmer wrote:
    On 03/25/2013 06:35 PM, Joakim Ziegler wrote:
    That's the output of, like you suggested:
    And you ran that before you ran "vgchange -a y"? That doesn't make any
    sense. The commands show the volume group active. I can't see any
    reason why the system wouldn't boot.

    I hate for you to keep rebooting your server, but do the device nodes
    look correct in both /dev/mapper and /dev/vg_resolve02 at that point?



    Apologies if someone mentioned this already ( don't have the whole
    thread in my mailbox), but whenever I've had to re-name a root lvm
    volume, I also had to recreate initrd. I haven't done it on 6.X, but I
    assume it applies to initramfs as well. The notes in my corp wiki link
    back to this redhat bugzilla post,
    https://bugzilla.redhat.com/show_bug.cgi?id#0190 try that maybe?


    Patrick
  • Joakim Ziegler at Mar 26, 2013 at 8:52 pm
    I haven't actually renamed the root LVM volume, it's had the same name
    since install. I just moved some drives around on the SATA ports. Is it
    still worth recreating initrd?


    --
    Joakim Ziegler - Supervisor de postproducci?n - Terminal
    joakim at terminalmx.com - 044 55 2971 8514 - 5264 0864

    On 26/03/13 8:08, Patrick Flaherty wrote:
    On Tue, Mar 26, 2013 at 1:35 AM, Joakim Ziegler wrote:
    Yes, I ran that immediately after getting dropped to the shell. I can
    take a look at the device nodes tomorrow, but if I remember correctly,
    /dev/mapper contained only the file "control" before running vgchange
    -ay, that is, there was no "vg_resolve02-lv_root" device there. That
    device only shows up after I run vgchange -ay.

    I did not check whether /dev/vg_resolve02 exists, I can do that tomorrow.

    --
    Joakim Ziegler - Supervisor de postproducci?n - Terminal
    joakim at terminalmx.com - 044 55 2971 8514 - 5264 0864
    On 25/03/13 23:26, Gordon Messmer wrote:
    On 03/25/2013 06:35 PM, Joakim Ziegler wrote:
    That's the output of, like you suggested:
    And you ran that before you ran "vgchange -a y"? That doesn't make any
    sense. The commands show the volume group active. I can't see any
    reason why the system wouldn't boot.

    I hate for you to keep rebooting your server, but do the device nodes
    look correct in both /dev/mapper and /dev/vg_resolve02 at that point?

    Apologies if someone mentioned this already ( don't have the whole
    thread in my mailbox), but whenever I've had to re-name a root lvm
    volume, I also had to recreate initrd. I haven't done it on 6.X, but I
    assume it applies to initramfs as well. The notes in my corp wiki link
    back to this redhat bugzilla post,
    https://bugzilla.redhat.com/show_bug.cgi?id#0190 try that maybe?

    Patrick
    _______________________________________________
    CentOS mailing list
    CentOS at centos.org
    http://lists.centos.org/mailman/listinfo/centos
  • Gordon Messmer at Mar 27, 2013 at 12:47 am

    On 03/26/2013 01:52 PM, Joakim Ziegler wrote:
    I haven't actually renamed the root LVM volume, it's had the same name
    since install. I just moved some drives around on the SATA ports. Is it
    still worth recreating initrd?

    I wouldn't expect it to make a difference, but it probably wouldn't hurt
    anything. Copy or rename your existing initrd to a path in /boot, so
    that you can revert if anything goes wrong. After that, create a new
    one. If that fixes the problem, I'd be curious to know why. We can
    compare the content of the two if that changes anything, and I'll learn
    something. As far as I know, the path to the devices isn't included in
    the initrd.


    # mkinitrd /boot/initramfs-$(uname -r).img $(uname -r)
  • Joakim Ziegler at Mar 27, 2013 at 12:55 am
    Thanks, will try.


    --
    Joakim Ziegler - Supervisor de postproducci?n - Terminal
    joakim at terminalmx.com - 044 55 2971 8514 - 5264 0864

    On 26/03/13 18:47, Gordon Messmer wrote:
    On 03/26/2013 01:52 PM, Joakim Ziegler wrote:
    I haven't actually renamed the root LVM volume, it's had the same name
    since install. I just moved some drives around on the SATA ports. Is it
    still worth recreating initrd?
    I wouldn't expect it to make a difference, but it probably wouldn't hurt
    anything. Copy or rename your existing initrd to a path in /boot, so
    that you can revert if anything goes wrong. After that, create a new
    one. If that fixes the problem, I'd be curious to know why. We can
    compare the content of the two if that changes anything, and I'll learn
    something. As far as I know, the path to the devices isn't included in
    the initrd.

    # mkinitrd /boot/initramfs-$(uname -r).img $(uname -r)

    _______________________________________________
    CentOS mailing list
    CentOS at centos.org
    http://lists.centos.org/mailman/listinfo/centos
  • Joakim Ziegler at Mar 29, 2013 at 8:23 am
    Recreating initrd made no difference.


    Immediately after getting dropped to rdshell, I looked around in /dev,
    which brought me a few surprises...


    /dev/mapper contains only "control", that is, "vg_resolve02-lv_root" is
    missing.


    /dev/root is a symlink to /dev/dm-0


    Which is a bit surprising, since, when I do lvm vgscan and lvm vgchange
    -ya, /dev/mapper/vg_resolve02-lv_root appears, but I just now noticed
    that it's a symlink to /dev/dm-0, so, in effect, when I symlink
    /dev/root to /dev/vg_resolve02-lv_root, I'm just creating the same
    symlink that was already there, with one more level of redirection.


    That means /dev/root already is correct, so the only thing I'm actually
    changing to make the system boot is to scan for volume groups and
    activate them.


    The big question then becomes: Why do I have to do this manually? How do
    I make Dracut (I assume this is Dracut's job) make this automatically?


    --
    Joakim Ziegler - Supervisor de postproducci?n - Terminal
    joakim at terminalmx.com - 044 55 2971 8514 - 5264 0864

    On 26/03/13 18:55, Joakim Ziegler wrote:
    Thanks, will try.
  • Gordon Messmer at Mar 29, 2013 at 4:38 pm

    On 03/29/2013 01:23 AM, Joakim Ziegler wrote:
    Immediately after getting dropped to rdshell, I looked around in /dev,
    which brought me a few surprises...

    /dev/mapper contains only "control", that is, "vg_resolve02-lv_root" is
    missing.

    Did you get to look at or for /dev/vg_resolve02 as well?

    /dev/root is a symlink to /dev/dm-0

    Does /dev/dm-0 exist?


    Does the system boot if you just "exit" from the rdshell? What about if
    you "vgchange -a y" without changing the symlink?

    That means /dev/root already is correct, so the only thing I'm actually
    changing to make the system boot is to scan for volume groups and
    activate them.

    The big question then becomes: Why do I have to do this manually? How do
    I make Dracut (I assume this is Dracut's job) make this automatically?

    udev should be doing this. And... I was just looking at this again,
    because the last time I came up with nothing useful. Look at
    /usr/share/dracut/modules.d/90lvm/64-lvm.rules. If I'm reading this
    correctly, udev will look for dm-0 in /sys and will not run lvm_scan if
    it's found. I wonder if it's possible that the /sys nodes are getting
    set up, but device-mapper isn't setting up the nodes in /dev?


    I'm really at a loss... it seems like a much simpler explanation is
    simply that the devices take so long to detect that init gives up. When
    you run vgchange, they've had the time they need. That idea is
    inconsistent with the fact that your dmesg output shows what I assume is
    the correct devices and partition tables.


    You could try adding "rdinitdebug rdudevdebug" to your kernel command
    line, but you're going to see a LOT of output, and it's only really
    going to be meaningful if you've read the /init script that Dracut
    creates, and understand more or less what it's doing, particularly in
    the "main_loop" section.
  • Joakim Ziegler at Mar 30, 2013 at 1:18 pm

    On 29/03/13 10:38, Gordon Messmer wrote:
    On 03/29/2013 01:23 AM, Joakim Ziegler wrote:
    Immediately after getting dropped to rdshell, I looked around in /dev,
    which brought me a few surprises...
    /dev/mapper contains only "control", that is, "vg_resolve02-lv_root" is
    missing.
    Did you get to look at or for /dev/vg_resolve02 as well?
    /dev/root is a symlink to /dev/dm-0
    Does /dev/dm-0 exist?
    Does the system boot if you just "exit" from the rdshell? What about if
    you "vgchange -a y" without changing the symlink?

    I checked this a bit more thoroughly. The status is as follows:


    When I boot up and get dropped to rdshell, neither /dev/root nor
    /dev/vg_resolve02, nor /dev/dm-0 exist. Just exiting at this point drops
    me back into rdshell. Waiting a few minutes makes no difference.


    Doing lvm vgscan finds the volume group, but creates no device nodes.
    Just exiting at this point drops me back into rdshell as well.


    When I do lvm vgchange -ay, /dev/dm-0 is created, /dev/root is created
    as a symlink to it, as well as /dev/vg_resolve02/ with lv_root inside
    it, and /dev/mapper/vg_resolve02-lv_root. I don't need to change the
    symlink or do anything else, if I exit after doing lvm vgchange -ay,
    everything is ok.



    That means /dev/root already is correct, so the only thing I'm actually
    changing to make the system boot is to scan for volume groups and
    activate them.

    The big question then becomes: Why do I have to do this manually? How do
    I make Dracut (I assume this is Dracut's job) make this automatically?
    udev should be doing this. And... I was just looking at this again,
    because the last time I came up with nothing useful. Look at
    /usr/share/dracut/modules.d/90lvm/64-lvm.rules. If I'm reading this
    correctly, udev will look for dm-0 in /sys and will not run lvm_scan if
    it's found. I wonder if it's possible that the /sys nodes are getting
    set up, but device-mapper isn't setting up the nodes in /dev?

    It turns out I was wrong about dm-0 already existing, it's created on
    vgchange -ay. I'm looking at the file you mention, but I'm afraid I
    don't know LVM well enough to make that much sense of it. From what I
    can tell, it calls lvm_scan for each device, and there's an lvm_scan.sh
    in there that looks like it should be doing lvchange -ay, but if dm-0
    doesn't already exist, I don't think this will do anything, am I wrong?



    I'm really at a loss... it seems like a much simpler explanation is
    simply that the devices take so long to detect that init gives up. When
    you run vgchange, they've had the time they need. That idea is
    inconsistent with the fact that your dmesg output shows what I assume is
    the correct devices and partition tables.
    You could try adding "rdinitdebug rdudevdebug" to your kernel command
    line, but you're going to see a LOT of output, and it's only really
    going to be meaningful if you've read the /init script that Dracut
    creates, and understand more or less what it's doing, particularly in
    the "main_loop" section.

    I can try this, but it might be a bit beyond my area of expertise, I'm
    afraid.


    If I were to just try a brute force approach, what RPM packages should I
    reinstall/update to get all this stuff reinstalled as it was the first
    time I installed the system?


    --
    Joakim Ziegler - Supervisor de postproducci?n - Terminal
    joakim at terminalmx.com - 044 55 2971 8514 - 5264 0864
  • Joakim Ziegler at Apr 1, 2013 at 10:53 pm

    On 30/03/13 7:18, Joakim Ziegler wrote:
    On 29/03/13 10:38, Gordon Messmer wrote:
    On 03/29/2013 01:23 AM, Joakim Ziegler wrote:
    Immediately after getting dropped to rdshell, I looked around in /dev,
    which brought me a few surprises...
    /dev/mapper contains only "control", that is, "vg_resolve02-lv_root" is
    missing.
    Did you get to look at or for /dev/vg_resolve02 as well?
    /dev/root is a symlink to /dev/dm-0
    Does /dev/dm-0 exist?
    Does the system boot if you just "exit" from the rdshell? What about if
    you "vgchange -a y" without changing the symlink?
    I checked this a bit more thoroughly. The status is as follows:
    When I boot up and get dropped to rdshell, neither /dev/root nor
    /dev/vg_resolve02, nor /dev/dm-0 exist. Just exiting at this point drops
    me back into rdshell. Waiting a few minutes makes no difference.
    Doing lvm vgscan finds the volume group, but creates no device nodes.
    Just exiting at this point drops me back into rdshell as well.
    When I do lvm vgchange -ay, /dev/dm-0 is created, /dev/root is created
    as a symlink to it, as well as /dev/vg_resolve02/ with lv_root inside
    it, and /dev/mapper/vg_resolve02-lv_root. I don't need to change the
    symlink or do anything else, if I exit after doing lvm vgchange -ay,
    everything is ok.


    That means /dev/root already is correct, so the only thing I'm actually
    changing to make the system boot is to scan for volume groups and
    activate them.
    The big question then becomes: Why do I have to do this manually? How do
    I make Dracut (I assume this is Dracut's job) make this automatically?
    udev should be doing this. And... I was just looking at this again,
    because the last time I came up with nothing useful. Look at
    /usr/share/dracut/modules.d/90lvm/64-lvm.rules. If I'm reading this
    correctly, udev will look for dm-0 in /sys and will not run lvm_scan if
    it's found. I wonder if it's possible that the /sys nodes are getting
    set up, but device-mapper isn't setting up the nodes in /dev?
    It turns out I was wrong about dm-0 already existing, it's created on
    vgchange -ay. I'm looking at the file you mention, but I'm afraid I
    don't know LVM well enough to make that much sense of it. From what I
    can tell, it calls lvm_scan for each device, and there's an lvm_scan.sh
    in there that looks like it should be doing lvchange -ay, but if dm-0
    doesn't already exist, I don't think this will do anything, am I wrong?


    I'm really at a loss... it seems like a much simpler explanation is
    simply that the devices take so long to detect that init gives up. When
    you run vgchange, they've had the time they need. That idea is
    inconsistent with the fact that your dmesg output shows what I assume is
    the correct devices and partition tables.
    You could try adding "rdinitdebug rdudevdebug" to your kernel command
    line, but you're going to see a LOT of output, and it's only really
    going to be meaningful if you've read the /init script that Dracut
    creates, and understand more or less what it's doing, particularly in
    the "main_loop" section.
    I can try this, but it might be a bit beyond my area of expertise, I'm
    afraid.
    If I were to just try a brute force approach, what RPM packages should I
    reinstall/update to get all this stuff reinstalled as it was the first
    time I installed the system?

    Just bumping this up, any ideas about this? It's a little annoying not
    having this box boot by itself...


    --
    Joakim Ziegler - Supervisor de postproducci?n - Terminal
    joakim at terminalmx.com - 044 55 2971 8514 - 5264 0864
  • Joakim Ziegler at Apr 22, 2013 at 5:11 pm

    On 01/04/13 16:53, Joakim Ziegler wrote:
    On 30/03/13 7:18, Joakim Ziegler wrote:
    On 29/03/13 10:38, Gordon Messmer wrote:
    On 03/29/2013 01:23 AM, Joakim Ziegler wrote:
    Immediately after getting dropped to rdshell, I looked around in /dev,
    which brought me a few surprises...
    /dev/mapper contains only "control", that is, "vg_resolve02-lv_root" is
    missing.
    Did you get to look at or for /dev/vg_resolve02 as well?
    /dev/root is a symlink to /dev/dm-0
    Does /dev/dm-0 exist?
    Does the system boot if you just "exit" from the rdshell? What about if
    you "vgchange -a y" without changing the symlink?
    I checked this a bit more thoroughly. The status is as follows:
    When I boot up and get dropped to rdshell, neither /dev/root nor
    /dev/vg_resolve02, nor /dev/dm-0 exist. Just exiting at this point drops
    me back into rdshell. Waiting a few minutes makes no difference.
    Doing lvm vgscan finds the volume group, but creates no device nodes.
    Just exiting at this point drops me back into rdshell as well.
    When I do lvm vgchange -ay, /dev/dm-0 is created, /dev/root is created
    as a symlink to it, as well as /dev/vg_resolve02/ with lv_root inside
    it, and /dev/mapper/vg_resolve02-lv_root. I don't need to change the
    symlink or do anything else, if I exit after doing lvm vgchange -ay,
    everything is ok.
    That means /dev/root already is correct, so the only thing I'm actually
    changing to make the system boot is to scan for volume groups and
    activate them.
    The big question then becomes: Why do I have to do this manually? How do
    I make Dracut (I assume this is Dracut's job) make this automatically?
    udev should be doing this. And... I was just looking at this again,
    because the last time I came up with nothing useful. Look at
    /usr/share/dracut/modules.d/90lvm/64-lvm.rules. If I'm reading this
    correctly, udev will look for dm-0 in /sys and will not run lvm_scan if
    it's found. I wonder if it's possible that the /sys nodes are getting
    set up, but device-mapper isn't setting up the nodes in /dev?
    It turns out I was wrong about dm-0 already existing, it's created on
    vgchange -ay. I'm looking at the file you mention, but I'm afraid I
    don't know LVM well enough to make that much sense of it. From what I
    can tell, it calls lvm_scan for each device, and there's an lvm_scan.sh
    in there that looks like it should be doing lvchange -ay, but if dm-0
    doesn't already exist, I don't think this will do anything, am I wrong?
    I'm really at a loss... it seems like a much simpler explanation is
    simply that the devices take so long to detect that init gives up. When
    you run vgchange, they've had the time they need. That idea is
    inconsistent with the fact that your dmesg output shows what I assume is
    the correct devices and partition tables.
    You could try adding "rdinitdebug rdudevdebug" to your kernel command
    line, but you're going to see a LOT of output, and it's only really
    going to be meaningful if you've read the /init script that Dracut
    creates, and understand more or less what it's doing, particularly in
    the "main_loop" section.
    I can try this, but it might be a bit beyond my area of expertise, I'm
    afraid.
    If I were to just try a brute force approach, what RPM packages should I
    reinstall/update to get all this stuff reinstalled as it was the first
    time I installed the system?
    Just bumping this up, any ideas about this? It's a little annoying not
    having this box boot by itself...

    And bumping this again... Any ideas? Anyone?


    --
    Joakim Ziegler - Supervisor de postproducci?n - Terminal
    joakim at terminalmx.com - 044 55 2971 8514 - 5264 0864

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcentos @
categoriescentos
postedMar 23, '13 at 3:27a
activeApr 22, '13 at 5:11p
posts22
users4
websitecentos.org
irc#centos

People

Translate

site design / logo © 2021 Grokbase