FAQ
At the beginning of January I encountered a problem where
several vm guests on a single host somehow managed to see
the the virtual disks assigned to other guests on the same
hosts. I was unable to resolve this situation and
shutdown the affected guests after creating new guest
instances and moving the services and data off the
corrupted guests.

I have since removed these guests via virt-manager but all
attempts to remove from the host the logical volumes
associated with the former VirtIO disks fail. The volumes
are considered open by lvremove and nothing I have tried
can get them to close for removal. The --force option has
no effect on this situation.

# /sbin/lvremove -f /dev/vg_vhost01/lv_vm_base
Can't remove open logical volume "lv_vm_base"

# dmsetup info -c vg_vhost01-lv_vm_base
Name Maj Min Stat Open Targ Event UUID
vg_vhost01-lv_vm_base 253 5 L--w 2 1 0
LVM-gXMt00E1RDjpSX3INLZ35Prtg66aX36BeAOlKIkmfSNQRNol3Hni920R4YVaZr52

# dmsetup remove vg_vhost01-lv_vm_base
device-mapper: remove ioctl failed: Device or resource busy
Command failed


There are several bugs filed on similar issues and udev is
sometimes identified as the culprit. If I kill the udev
daemon with T=`pidof -x udevd`; kill $T and rerun the
lvremove -f command then I see this change in behaviour:

# /sbin/lvremove -f /dev/vg_vhost01/lv_vm_base
Found duplicate PV djM23m6YebBQ2xgPh9ORMtdX2iOu9xBQ:
using /dev/mapper/vg_vhost01-lv_vm_pas.harte--lyne.cap2
not
/dev/mapper/vg_vhost01-lv_vm_pgsql--dbms.harte--lyne.ca_00p2
Found duplicate PV djM23m6YebBQ2xgPh9ORMtdX2iOu9xBQ:
using /dev/mapper/vg_vhost01-lv_vm_basep2 not
/dev/mapper/vg_vhost01-lv_vm_pas.harte--lyne.cap2
Can't remove open logical volume "lv_vm_base"

I need to get this system stable and return the lost disk
space to the storage pool. Does anyone have any
suggestions as to how to proceed?

If I cannot solve this using the available system commands
then prudence dictates that I have to re-install the
server OS and rebuild all of the vm guests. As these
guests have been laboriously transferred from other hosts
during the past month this is a task I would rather not
have to do.

Any help is gratefully accepted.

--
*** E-Mail is NOT a SECURE channel ***
James B. Byrne mailto:ByrneJB at Harte-Lyne.ca
Harte & Lyne Limited http://www.harte-lyne.ca
9 Brockley Drive vox: +1 905 561 1241
Hamilton, Ontario fax: +1 905 561 0757
Canada L8E 3C3

Search Discussions

  • Phil Schaffner at Jan 28, 2012 at 12:19 pm
    Pardon the top-post and cross-post, but this apparently should have gone
    to centos-virt at centos.org

    Phil

    chris procter wrote on 01/27/2012 03:29 PM:
    Hi,

    Its ages since I came across this problem so my memory is a little hazey but something is obviously holding on to the lv so you'll need to figure out whats holding it and kill that. The open value returned by "dmsetup info" is how many things have the device file open), it looks to be 2 at the moment and you cant "dmsetup remove" untill its zero (lvm might be one of them I cant remember)


    A few things worth trying are


    1) make sure its not mounted anywhere!
    2) if multipathd is running try stopping that.

    3) either lsof or fuser on the device file may be able to tell you which process has it open
    4) something vm related might not have let go properly, are there any deamons/processes etc still running?

    5) reboot, the sledgehammer aproach to killing off processes!!

    6) you could try hitting it with dmsetup again, you need to suspend the device first using "dmsetup suspend" which *may* persuade the holding process to let go, if it does reduce the open count you'll need to "dmsetup resume" and then suspend again untill open reaches zero when "dmsetup remove" should work. I'd try and avoid this option if you can, you're messing beneath the lvm layer and it may not like that, should be ok but...


    if none of that is possible/works you could try asking on the lvm-linux list.


    Reinstalling really shouldn't be necesary.


    chris




    ----- Original Message -----
    From: James B. Byrne<byrnejb at harte-lyne.ca>
    To: centos-virt at centos.org; centos at centos.org
    Cc:
    Sent: Friday, 27 January 2012, 17:50
    Subject: [CentOS] Cannot remove lvs associated with deleted vm guests

    At the beginning of January I encountered a problem where
    several vm guests on a single host somehow managed to see
    the the virtual disks assigned to other guests on the same
    hosts. I was unable to resolve this situation and
    shutdown the affected guests after creating new guest
    instances and moving the services and data off the
    corrupted guests.

    I have since removed these guests via virt-manager but all
    attempts to remove from the host the logical volumes
    associated with the former VirtIO disks fail. The volumes
    are considered open by lvremove and nothing I have tried
    can get them to close for removal. The --force option has
    no effect on this situation.

    # /sbin/lvremove -f /dev/vg_vhost01/lv_vm_base
    Can't remove open logical volume "lv_vm_base"

    # dmsetup info -c vg_vhost01-lv_vm_base
    Name Maj Min Stat Open Targ Event UUID
    vg_vhost01-lv_vm_base 253 5 L--w 2 1 0
    LVM-gXMt00E1RDjpSX3INLZ35Prtg66aX36BeAOlKIkmfSNQRNol3Hni920R4YVaZr52

    # dmsetup remove vg_vhost01-lv_vm_base
    device-mapper: remove ioctl failed: Device or resource busy
    Command failed


    There are several bugs filed on similar issues and udev is
    sometimes identified as the culprit. If I kill the udev
    daemon with T=`pidof -x udevd`; kill $T and rerun the
    lvremove -f command then I see this change in behaviour:

    # /sbin/lvremove -f /dev/vg_vhost01/lv_vm_base
    Found duplicate PV djM23m6YebBQ2xgPh9ORMtdX2iOu9xBQ:
    using /dev/mapper/vg_vhost01-lv_vm_pas.harte--lyne.cap2
    not
    /dev/mapper/vg_vhost01-lv_vm_pgsql--dbms.harte--lyne.ca_00p2
    Found duplicate PV djM23m6YebBQ2xgPh9ORMtdX2iOu9xBQ:
    using /dev/mapper/vg_vhost01-lv_vm_basep2 not
    /dev/mapper/vg_vhost01-lv_vm_pas.harte--lyne.cap2
    Can't remove open logical volume "lv_vm_base"

    I need to get this system stable and return the lost disk
    space to the storage pool. Does anyone have any
    suggestions as to how to proceed?

    If I cannot solve this using the available system commands
    then prudence dictates that I have to re-install the
    server OS and rebuild all of the vm guests. As these
    guests have been laboriously transferred from other hosts
    during the past month this is a task I would rather not
    have to do.

    Any help is gratefully accepted.

    --
    *** E-Mail is NOT a SECURE channel ***
    James B. Byrne mailto:ByrneJB at Harte-Lyne.ca
    Harte& Lyne Limited http://www.harte-lyne.ca
    9 Brockley Drive vox: +1 905 561 1241
    Hamilton, Ontario fax: +1 905 561 0757
    Canada L8E 3C3

    _______________________________________________
    CentOS mailing list
    CentOS at centos.org
    http://lists.centos.org/mailman/listinfo/centos
    _______________________________________________
    CentOS mailing list
    CentOS at centos.org
    http://lists.centos.org/mailman/listinfo/centos
  • James B. Byrne at Jan 30, 2012 at 10:10 am

    On Sat, January 28, 2012 12:19, Phil Schaffner wrote:
    Pardon the top-post and cross-post, but this apparently
    should have gone
    to centos-virt at centos.org

    Phil

    chris procter wrote on 01/27/2012 03:29 PM:
    Hi,

    Its ages since I came across this problem so my memory
    is a little hazey but something is obviously holding on
    to the lv so you'll need to figure out whats holding it
    and kill that. The open value returned by "dmsetup info"
    is how many things have the device file open), it looks
    to be 2 at the moment and you cant "dmsetup remove"
    untill its zero (lvm might be one of them I cant
    remember)


    A few things worth trying are


    1) make sure its not mounted anywhere!
    [root at vhost01 ~]# umount /dev/vg_vhost01/lv_vm_base
    umount: /dev/vg_vhost01/lv_vm_base: not mounted
    2) if multipathd is running try stopping that.
    [root at vhost01 ~]# ps -ef | grep multipathd
    root 465 307 0 09:40 pts/0 00:00:00 grep
    multipathd
    [root at vhost01 ~]# service multipathd status
    multipathd: unrecognized service
    [root at vhost01 ~]# find / -name multipathd
    [root at vhost01 ~]#
    3) either lsof or fuser on the device file may be able
    [root at vhost01 ~]# fuser /dev/vg_vhost01/lv_vm_base
    [root at vhost01 ~]# lsof /dev/vg_vhost01/lv_vm_base
    [root at vhost01 ~]#
    4) something vm related might not have let go properly,
    are there any deamons/processes etc still running?
    There are many vm qeum processes running, non contain any
    reference to lv_vm_base

    [root at vhost01 ~]# ps -ef | grep vm_base
    root 549 307 0 09:44 pts/0 00:00:00 grep vm_base
    [root at vhost01 ~]#
    5) reboot, the sledgehammer aproach to killing off
    processes!!

    6) you could try hitting it with dmsetup again, you need
    to suspend the device first using "dmsetup suspend"
    which *may* persuade the holding process to let go, if
    it does reduce the open count you'll need to "dmsetup
    resume" and then suspend again untill open reaches zero
    when "dmsetup remove" should work. I'd try and avoid
    this option if you can, you're messing beneath the lvm
    layer and it may not like that, should be ok but...
    [root at vhost01 ~]# dmsetup info -c vg_vhost01-lv_vm_base
    Name Maj Min Stat Open Targ Event UUID
    vg_vhost01-lv_vm_base 253 5 L--w 2 1 0
    LVM-gXMt00E1RDjpSX3INLZ35Prtg66aX36BeAOlKIkmfSNQRNol3Hni920R4YVaZr52
    [root at vhost01 ~]# dmsetup suspend vg_vhost01-lv_vm_base
    [root at vhost01 ~]# dmsetup info -c vg_vhost01-lv_vm_base
    Name Maj Min Stat Open Targ Event UUID
    vg_vhost01-lv_vm_base 253 5 L-sw 2 1 0
    LVM-gXMt00E1RDjpSX3INLZ35Prtg66aX36BeAOlKIkmfSNQRNol3Hni920R4YVaZr52
    if none of that is possible/works you could try asking
    on the lvm-linux list.
    Am so doing.

    Reinstalling really shouldn't be necesary.
    Yes. However. . .


    --
    *** E-Mail is NOT a SECURE channel ***
    James B. Byrne mailto:ByrneJB at Harte-Lyne.ca
    Harte & Lyne Limited http://www.harte-lyne.ca
    9 Brockley Drive vox: +1 905 561 1241
    Hamilton, Ontario fax: +1 905 561 0757
    Canada L8E 3C3
  • James B. Byrne at Jan 30, 2012 at 12:45 pm
    This problem was finally resolved with the generous help
    of Bryn M. Reeves on the inux-lvm at redhat.com list.

    The difficulty was that the utility kpartx had the lvs
    opened via their mappings. Further, the mappings were
    created with the -pp option of kpartx and that option had
    to be provided to the delete action in order for that to
    work:

    # kpartx -d -pp /path/to/logical/volume/name

    Once the mappings were removed then the lvremove also
    worked and the volumes were removed.

    A small point, failure to provide the -pp option to the
    kpartx -d action does not raise an error. Nor does it
    remove the mapping however.


    --
    *** E-Mail is NOT a SECURE channel ***
    James B. Byrne mailto:ByrneJB at Harte-Lyne.ca
    Harte & Lyne Limited http://www.harte-lyne.ca
    9 Brockley Drive vox: +1 905 561 1241
    Hamilton, Ontario fax: +1 905 561 0757
    Canada L8E 3C3

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcentos-virt @
categoriescentos
postedJan 27, '12 at 12:50p
activeJan 30, '12 at 12:45p
posts4
users2
websitecentos.org
irc#centos

2 users in discussion

James B. Byrne: 3 posts Phil Schaffner: 1 post

People

Translate

site design / logo © 2022 Grokbase