FAQ

[CentOS-virt] Setting up a pci passthrough device

James B. Byrne
Feb 3, 2012 at 10:34 am
I have been investigating pci pass-through for virtualized
guests and the documentation I have found seems to me to
lack a certain consistency in its example. This may be
due to my not understanding what it is trying to inform
me.

What I wish to do is to configure a pci multi-port serial
i/o card for use by a single virtual host.

I start by running lspci -v on the host to identify the
serial card:

03:00.0 Serial controller: Oxford Semiconductor Ltd
OX16PCI954 (Quad 16950 UART) function 0 (Uart) (prog-if 06
[16950])
Subsystem: Oxford Semiconductor Ltd Device 0000
Flags: medium devsel, IRQ 17
I/O ports at d040 [size2]
Memory at d0702000 (32-bit, non-prefetchable)
[size=4K]
I/O ports at d020 [size2]
Memory at d0701000 (32-bit, non-prefetchable)
[size=4K]
Capabilities: [40] Power Management version 2
Kernel driver in use: serial

I then check for possible multiple IRQ assignment:

lspci -v | grep ' IRQ 17'
Flags: medium devsel, IRQ 17

I next use lspci -n to identify the vendor codes:

lspci -n | grep '00:03.0'
00:03.0 0780: 8086:2e24 (rev 03)

So this is an Intel chipset and the device id is 2e24.
Now this is the point in the example where the
documentation and I part company. In the examples I have
found, although the pci device ids listed from virsh
nodedev-list are uniformly of the form pci_0000_00_03_0
those used in the examples then switch and use the form
pci_8086_3a6c for the subsequent steps.

This pattern appears to be the prefix pci followed by the
manufacturer's code followed by the device id. There is no
other mapping to the pci device ids previously reported by
virsh nodedev-list and lspci in the examples that I can
discern.

However, if I attempt to use the manufacturer and device
ids in the next step of the example, substituting those
used in the example with those reported on my own system,
then I get a device not found reported:

virsh nodedev-dumpxml pci_8086_2e24
error: Could not find matching device 'pci_8086_2e24'
error: Node device not found

If instead I use the pci device ids exactly as reported by
virsh nodedev-list then I get what I expect:

virsh nodedev-dumpxml pci_0000_00_03_0
<device>
<name>pci_0000_00_03_0</name>
<parent>computer</parent>
<capability type='pci'>
<domain>0</domain>
<bus>0</bus>
<slot>3</slot>
<function>0</function>
<product id='0x2e24'>4 Series Chipset HECI
Controller</product>
<vendor id='0x8086'>Intel Corporation</vendor>
<capability type='virt_functions'>
</capability>
</capability>
</device>

My question is: Why does the documentation change the form
of the pci identifiers used in the second half of the
example from those reported previously in the same
example? Is this change significant? What does it mean?
Am I missing something important here?


--
*** E-Mail is NOT a SECURE channel ***
James B. Byrne mailto:ByrneJB at Harte-Lyne.ca
Harte & Lyne Limited http://www.harte-lyne.ca
9 Brockley Drive vox: +1 905 561 1241
Hamilton, Ontario fax: +1 905 561 0757
Canada L8E 3C3
reply

Search Discussions

6 responses

  • James B. Byrne at Feb 3, 2012 at 11:14 am

    On Fri, February 3, 2012 10:34, James B. Byrne wrote:
    I have been investigating pci pass-through for virtualized
    guests and the documentation I have found seems to me to
    lack a certain consistency in its example. This may be
    due to my not understanding what it is trying to inform
    me.
    Following along in the example and working on the basis
    that what is reported by virsh nodedev-dumpxml
    pci_0000_00_03_0 is what I should use I get to the point
    where I try the readline command, and discover there is no
    such link.

    Proceeding past this point I detach the pci device from
    the host. I then edit the guest config file as given in
    the example only to discover that the changes are not
    saved. it reported that the configuration was edted by
    opening the configuration in virtsh edit a second time
    shows that the added <hostdev></hostdev> section was not
    saved. Perhaps this is because the example provides no
    context as to where it is nested within the xml file and
    my placing it directly within the <domain> </domain> tags
    is invalid.

    Further, the requirement to set setsebool -P
    virt_manage_sysfs 1 cannot be met since an SELinux boolean
    of that name does not exist (apparently it was renamed to
    virt_use_nfs).

    The reference I am using is found at:

    http://www.linuxtopia.org/online_books/rhel6/rhel_6_virtualization/rhel_6_virtualization_chap-Virtualization-PCI_passthrough.html

    However, I do not think that I can recommend it based on
    my experience.



    --
    *** E-Mail is NOT a SECURE channel ***
    James B. Byrne mailto:ByrneJB at Harte-Lyne.ca
    Harte & Lyne Limited http://www.harte-lyne.ca
    9 Brockley Drive vox: +1 905 561 1241
    Hamilton, Ontario fax: +1 905 561 0757
    Canada L8E 3C3
  • James B. Byrne at Feb 3, 2012 at 11:32 am
    Evidently I should be using

    http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Host_Configuration_and_Guest_Installation_Guide/chap-Virtualization_Host_Configuration_and_Guest_Installation_Guide-PCI_Assignment.html

    Which I had looked for but google apparently does not
    report. I had to search the RedHat web site using their
    search interface to locate it. The other site purports to
    be a rhel6 essentials book.

    Nonetheless, while the inconsistencies of the previous
    documents are resolved in this new reference the example
    edit of the virtual guest configuration still fails to
    provide a context for the insertion:

    # virsh edit guest1-rhel6-64
    <hostdev mode='subsystem' type='pci' managed='yes'>
    <source>
    <address domain='0x000' bus='0x03' slot='0x00'
    function='0x0'/>
    </source>
    </hostdev>


    Where does this go inside the rest of the guest
    configuration?

    <domain type='kvm'>
    <name>inet08.harte-lyne.ca</name>
    <uuid>6409d721-cfcf-2169-f65e-8f583b685f58</uuid>
    <description>Inet08 [216.185.71.28] virtual hosts:
    none</description>
    <memory>4194304</memory>
    <currentMemory>4194304</currentMemory>
    <vcpu>1</vcpu>
    <os>
    <type arch='x86_64' machine='rhel6.2.0'>hvm</type>
    <boot dev='hd'/>
    </os>
    <features>
    <acpi/>
    <apic/>
    <pae/>
    </features>
    <clock offset='utc'/>
    <on_poweroff>destroy</on_poweroff>
    <on_reboot>restart</on_reboot>
    <on_crash>restart</on_crash>
    <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='block' device='disk'>
    <driver name='qemu' type='raw' cache='none'
    io='native'/>
    <source
    dev='/dev/vg_vhost01/lv_vm_inet08.harte-lyne.ca_00'/>
    <target dev='vda' bus='virtio'/>
    <address type='pci' domain='0x0000' bus='0x00'
    slot='0x05' function='0x0'/>
    </disk>
    <disk type='block' device='cdrom'>
    <driver name='qemu' type='raw'/>
    <target dev='hdc' bus='ide'/>
    <readonly/>
    <address type='drive' controller='0' bus='1' unit='0'/>
    </disk>
    <controller type='ide' index='0'>
    <address type='pci' domain='0x0000' bus='0x00'
    slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
    <mac address='52:54:00:bf:e9:ac'/>
    <source bridge='br0'/>
    <model type='virtio'/>
    <address type='pci' domain='0x0000' bus='0x00'
    slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
    <target port='0'/>
    </serial>
    <serial type='dev'>
    <source path='/dev/ttyS0'/>
    <target port='1'/>
    </serial>
    <console type='pty'>
    <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'/>
    <sound model='ich6'>
    <address type='pci' domain='0x0000' bus='0x00'
    slot='0x04' function='0x0'/>
    </sound>
    <video>
    <model type='cirrus' vram='9216' heads='1'/>
    <address type='pci' domain='0x0000' bus='0x00'
    slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
    <address type='pci' domain='0x0000' bus='0x00'
    slot='0x06' function='0x0'/>
    </memballoon>
    </devices>
    </domain>

    I have tried placing the <hostdev> tags after the
    <devices> tag but the changes simply disappear.

    --
    *** E-Mail is NOT a SECURE channel ***
    James B. Byrne mailto:ByrneJB at Harte-Lyne.ca
    Harte & Lyne Limited http://www.harte-lyne.ca
    9 Brockley Drive vox: +1 905 561 1241
    Hamilton, Ontario fax: +1 905 561 0757
    Canada L8E 3C3
  • Nenad Opsenica at Feb 4, 2012 at 10:39 am

    On 02/03/2012 05:32 PM, James B. Byrne wrote:
    Where does this go inside the rest of the guest
    configuration?
    virt-manager GUI places PCI device pass-through inside <devices> ...
    </devices>:

    <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='cdrom'>
    <driver name='qemu' type='raw' io='threads'/>
    <target dev='hdc' bus='ide'/>
    <readonly/>
    <address type='drive' controller='0' bus='1' unit='0'/>
    </disk>
    <disk type='block' device='disk'>
    <driver name='qemu' type='raw' cache='none'/>
    <source dev='/dev/storage/win7'/>
    <target dev='hda' bus='ide'/>
    <address type='drive' controller='0' bus='0' unit='0'/>
    </disk>
    <controller type='ide' index='0'>
    <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='network'>
    <mac address='52:54:fe:ab:28:c0'/>
    <source network='default'/>
    <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
    <target port='0'/>
    </serial>
    <console type='pty'>
    <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1'/>
    <sound model='ich6'>
    <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
    <model type='vga' vram='9216' heads='1'/>
    <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
    <source>
    <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </source>
    <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
    <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
    </devices>
  • James B. Byrne at Feb 6, 2012 at 10:48 am

    On Sat, February 4, 2012 10:39, Nenad Opsenica wrote:
    On 02/03/2012 05:32 PM, James B. Byrne wrote:
    Where does this go inside the rest of the guest
    configuration?
    virt-manager GUI places PCI device pass-through inside
    <devices> ...
    http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Host_Configuration_and_Guest_Installation_Guide/chap-Virtualization_Host_Configuration_and_Guest_Installation_Guide-PCI_Assignment.html

    I have followed the instructions in the RedHat reference
    above to the best of my ability to understand them. I add
    the pci multi-port serial io card through virt-manager and
    it was indeed entered into the virtual machine's
    configuration file inside the <devices> tag:

    <hostdev mode='subsystem' type='pci' managed='yes'>
    <source>
    <address domain='0x0000' bus='0x00' slot='0x03'
    function='0x0'/>
    </source>
    <address type='pci' domain='0x0000' bus='0x00'
    slot='0x07' function='0x0'/>
    </hostdev>


    However, when I try and start the virtual machine I get
    this error:

    Error starting domain: internal error Unable to reset PCI
    device 0000:00:03.0: no FLR, PM reset or bus reset
    available

    Traceback (most recent call last):
    File "/usr/share/virt-manager/virtManager/asyncjob.py",
    line 44, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
    File "/usr/share/virt-manager/virtManager/asyncjob.py",
    line 65, in tmpcb
    callback(*args, **kwargs)
    File "/usr/share/virt-manager/virtManager/domain.py",
    line 1050, in startup
    self._backend.create()
    File "/usr/lib64/python2.6/site-packages/libvirt.py",
    line 511, in create
    if ret == -1: raise libvirtError ('virDomainCreate()
    failed', dom=self)
    libvirtError: internal error Unable to reset PCI device
    0000:00:03.0: no FLR, PM reset or bus reset available



    The steps I followed were:

    1. Check VT-D extensions available and enabled in BIOS - yes

    2. Restart virtual host - yes

    3. Identify device - yes
    <address domain='0x000' bus='0x00' slot='0x03'
    function='0x0'/>

    4. Add device to virtual machine configuration - yes
    . . .
    <hostdev mode='subsystem' type='pci' managed='yes'>
    <source>
    <address domain='0x0000' bus='0x00' slot='0x03'
    function='0x0'/>
    </source>
    <address type='pci' domain='0x0000' bus='0x00'
    slot='0x07' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
    <address type='pci' domain='0x0000' bus='0x00'
    slot='0x06' function='0x0'/>
    </memballoon>
    </devices>
    </domain>

    5. Enable SELinux boolean - yes
    getsebool virt_use_sysfs
    virt_use_sysfs --> on

    6. Start virtual machine - fails

    Am I making any obvious errors? Has anyone here
    configured and managed to get a multi-port serial card
    working with a virtual guest?

    --
    *** E-Mail is NOT a SECURE channel ***
    James B. Byrne mailto:ByrneJB at Harte-Lyne.ca
    Harte & Lyne Limited http://www.harte-lyne.ca
    9 Brockley Drive vox: +1 905 561 1241
    Hamilton, Ontario fax: +1 905 561 0757
    Canada L8E 3C3
  • Ken Bass at Feb 6, 2012 at 6:05 pm
    Take a look at http://wiki.xensource.com/xenwiki/VTdHowTo

    Two things in particular about PCI passthrough:

    - Only devices with FLR capabilities are supported.
    - Some motherboards are buggy. They advertised that they support Vt-d
    but do not correctly handle it (those with a broken ACPI DMAR table)

    I think lspci -vv will tell you if the device supports FLR. It will show
    'FLReset+' I believe.
  • James B. Byrne at Feb 8, 2012 at 9:17 am

    On Mon, February 6, 2012 18:05, Ken Bass wrote:
    Take a look at http://wiki.xensource.com/xenwiki/VTdHowTo

    Two things in particular about PCI passthrough:

    - Only devices with FLR capabilities are supported.
    - Some motherboards are buggy. They advertised that they
    support Vt-d
    but do not correctly handle it (those with a broken ACPI
    DMAR table)

    I think lspci -vv will tell you if the device supports
    FLR. It will show
    'FLReset+' I believe.
    03:00.0 Serial controller: Oxford Semiconductor Ltd
    OX16PCI954 (Quad 16950 UART) function 0 (Uart) (prog-if 06
    [16950])
    Subsystem: Oxford Semiconductor Ltd Device 0000
    Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV-
    VGASnoop- ParErr- Stepping- SERR- FastB2B-
    DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B+ ParErr-
    DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR-
    <PERR- INTx-
    Interrupt: pin A routed to IRQ 17
    Region 0: I/O ports at d040 [size2]
    Region 1: Memory at d0702000 (32-bit,
    non-prefetchable) [size=4K]
    Region 2: I/O ports at d020 [size2]
    Region 3: Memory at d0701000 (32-bit,
    non-prefetchable) [size=4K]
    Capabilities: [40] Power Management version 2
    Flags: PMEClk- DSI- D1- D2+ AuxCurrent=0mA
    PME(D0+,D1-,D2+,D3hot+,D3cold-)
    Status: D0 NoSoftRst- PME-Enable- DSel=0
    DScale=0 PME-
    Kernel driver in use: serial


    No FLR string is present. So pci pass through is a dead
    end I take it?


    I increased the number of uarts available to the host
    system at boot with the 8250.nr_uarts option. This
    gives me the following:

    # setserial -g /dev/ttyS*
    /dev/ttyS0, UART: 16550A, Port: 0x03f8, IRQ: 4
    /dev/ttyS1, UART: unknown, Port: 0x02f8, IRQ: 3
    /dev/ttyS2, UART: unknown, Port: 0x03e8, IRQ: 4
    /dev/ttyS3, UART: unknown, Port: 0x02e8, IRQ: 3
    /dev/ttyS4, UART: 16950/954, Port: 0xd040, IRQ: 17
    /dev/ttyS5, UART: 16950/954, Port: 0xd048, IRQ: 17
    /dev/ttyS6, UART: 16950/954, Port: 0xd050, IRQ: 17
    /dev/ttyS7, UART: 16950/954, Port: 0xd058, IRQ: 17
    /dev/ttyS8, UART: unknown, Port: 0x0000, IRQ: 0
    /dev/ttyS9, UART: unknown, Port: 0x0000, IRQ: 0

    With this change I now can add one serial port
    (/dev/ttyS4) to a virtual guest using virt-manager and
    have the guest start, but no more than one. Any more that
    one and the guest fails to run with the same irq conflict
    error as before. I still have not tried to see if the
    serial port actually works in this case, just that the
    system starts.

    I ran across this thread relating to serial devices in
    qemu from some time ago:

    http://www.mail-archive.com/qemu-devel at nongnu.org/msg27354.html

    Which seems to me to imply that it is not possible for a
    qemu guest to have more than 2 serial ports, one of which
    I gather has to be the console.

    However, this statement attracted my attention:
    This is wrong. Two devices should never be manipulating
    the same qemu_irq object. If you want multiple devices
    connected to the same IRQ then you need an explicit
    multiplexer. e.g. arm_timer.c:sp804_set_irq.
    And in a later message in the same thread:
    Two devices have the same s->irq. Give each on its own
    qemu_irq, and feed it into a multiplexer that ORs them
    together and sends the result to the interrupt
    controller's qemu_irq:

    Is there a way to set irqs in quem to map to specific
    ports on a pci card as this seems to imply? How is it
    done?




    --
    *** E-Mail is NOT a SECURE channel ***
    James B. Byrne mailto:ByrneJB at Harte-Lyne.ca
    Harte & Lyne Limited http://www.harte-lyne.ca
    9 Brockley Drive vox: +1 905 561 1241
    Hamilton, Ontario fax: +1 905 561 0757
    Canada L8E 3C3

Related Discussions

Discussion Navigation
viewthread | post