1 .. include:: <isonum.txt>
7 :Copyright: |copy| 2016, NVIDIA CORPORATION. All rights reserved.
8 :Author: Neo Jia <cjia@nvidia.com>
9 :Author: Kirti Wankhede <kwankhede@nvidia.com>
11 This program is free software; you can redistribute it and/or modify
12 it under the terms of the GNU General Public License version 2 as
13 published by the Free Software Foundation.
16 Virtual Function I/O (VFIO) Mediated devices[1]
17 ===============================================
19 The number of use cases for virtualizing DMA devices that do not have built-in
20 SR_IOV capability is increasing. Previously, to virtualize such devices,
21 developers had to create their own management interfaces and APIs, and then
22 integrate them with user space software. To simplify integration with user space
23 software, we have identified common requirements and a unified management
24 interface for such devices.
26 The VFIO driver framework provides unified APIs for direct device access. It is
27 an IOMMU/device-agnostic framework for exposing direct device access to user
28 space in a secure, IOMMU-protected environment. This framework is used for
29 multiple devices, such as GPUs, network adapters, and compute accelerators. With
30 direct device access, virtual machines or user space applications have direct
31 access to the physical device. This framework is reused for mediated devices.
33 The mediated core driver provides a common interface for mediated device
34 management that can be used by drivers of different devices. This module
35 provides a generic interface to perform these operations:
37 * Create and destroy a mediated device
38 * Add a mediated device to and remove it from a mediated bus driver
39 * Add a mediated device to and remove it from an IOMMU group
41 The mediated core driver also provides an interface to register a bus driver.
42 For example, the mediated VFIO mdev driver is designed for mediated devices and
43 supports VFIO APIs. The mediated bus driver adds a mediated device to and
44 removes it from a VFIO group.
46 The following high-level block diagram shows the main components and interfaces
47 in the VFIO mediated driver framework. The diagram shows NVIDIA, Intel, and IBM
48 devices as examples, as these devices are the first devices to use this module::
52 | +-----------+ | mdev_register_driver() +--------------+
53 | | | +<------------------------+ |
55 | | bus | +------------------------>+ vfio_mdev.ko |<-> VFIO user
56 | | driver | | probe()/remove() | | APIs
57 | | | | +--------------+
63 | +-----------+ | mdev_register_device() +--------------+
64 | | | +<------------------------+ |
65 | | | | | nvidia.ko |<-> physical
66 | | | +------------------------>+ | device
67 | | | | callbacks +--------------+
69 | | device | | mdev_register_device() +--------------+
70 | | interface | |<------------------------+ |
71 | | | | | i915.ko |<-> physical
72 | | | +------------------------>+ | device
73 | | | | callbacks +--------------+
75 | | | | mdev_register_device() +--------------+
76 | | | +<------------------------+ |
77 | | | | | ccw_device.ko|<-> physical
78 | | | +------------------------>+ | device
79 | | | | callbacks +--------------+
84 Registration Interfaces
85 =======================
87 The mediated core driver provides the following types of registration
90 * Registration interface for a mediated bus driver
91 * Physical device driver interface
93 Registration Interface for a Mediated Bus Driver
94 ------------------------------------------------
96 The registration interface for a mediated bus driver provides the following
97 structure to represent a mediated device's driver::
100 * struct mdev_driver [2] - Mediated device's driver
101 * @probe: called when new device created
102 * @remove: called when device removed
103 * @driver: device driver structure
106 int (*probe) (struct mdev_device *dev);
107 void (*remove) (struct mdev_device *dev);
108 struct device_driver driver;
111 A mediated bus driver for mdev should use this structure in the function calls
112 to register and unregister itself with the core driver:
116 extern int mdev_register_driver(struct mdev_driver *drv);
120 extern void mdev_unregister_driver(struct mdev_driver *drv);
122 The mediated bus driver is responsible for adding mediated devices to the VFIO
123 group when devices are bound to the driver and removing mediated devices from
124 the VFIO when devices are unbound from the driver.
127 Physical Device Driver Interface
128 --------------------------------
130 The physical device driver interface provides the mdev_parent_ops[3] structure
131 to define the APIs to manage work in the mediated core driver that is related
132 to the physical device.
134 The structures in the mdev_parent_ops structure are as follows:
136 * dev_attr_groups: attributes of the parent device
137 * mdev_attr_groups: attributes of the mediated device
138 * supported_config: attributes to define supported configurations
140 The functions in the mdev_parent_ops structure are as follows:
142 * create: allocate basic resources in a driver for a mediated device
143 * remove: free resources in a driver when a mediated device is destroyed
145 (Note that mdev-core provides no implicit serialization of create/remove
146 callbacks per mdev parent device, per mdev type, or any other categorization.
147 Vendor drivers are expected to be fully asynchronous in this respect or
148 provide their own internal resource protection.)
150 The callbacks in the mdev_parent_ops structure are as follows:
152 * open: open callback of mediated device
153 * close: close callback of mediated device
154 * ioctl: ioctl callback of mediated device
155 * read : read emulation callback
156 * write: write emulation callback
157 * mmap: mmap emulation callback
159 A driver should use the mdev_parent_ops structure in the function call to
160 register itself with the mdev core driver::
162 extern int mdev_register_device(struct device *dev,
163 const struct mdev_parent_ops *ops);
165 However, the mdev_parent_ops structure is not required in the function call
166 that a driver should use to unregister itself with the mdev core driver::
168 extern void mdev_unregister_device(struct device *dev);
171 Mediated Device Management Interface Through sysfs
172 ==================================================
174 The management interface through sysfs enables user space software, such as
175 libvirt, to query and configure mediated devices in a hardware-agnostic fashion.
176 This management interface provides flexibility to the underlying physical
177 device's driver to support features such as:
179 * Mediated device hot plug
180 * Multiple mediated devices in a single virtual machine
181 * Multiple mediated devices from different physical devices
183 Links in the mdev_bus Class Directory
184 -------------------------------------
185 The /sys/class/mdev_bus/ directory contains links to devices that are registered
186 with the mdev core driver.
188 Directories and files under the sysfs for Each Physical Device
189 --------------------------------------------------------------
193 |- [parent physical device]
194 |--- Vendor-specific-attributes [optional]
195 |--- [mdev_supported_types]
199 | | |--- available_instances
206 | | |--- available_instances
213 | |--- available_instances
218 * [mdev_supported_types]
220 The list of currently supported mediated device types and their details.
222 [<type-id>], device_api, and available_instances are mandatory attributes
223 that should be provided by vendor driver.
227 The [<type-id>] name is created by adding the device driver string as a prefix
228 to the string provided by the vendor driver. This format of this name is as
231 sprintf(buf, "%s-%s", dev_driver_string(parent->dev), group->name);
233 (or using mdev_parent_dev(mdev) to arrive at the parent device outside
234 of the core mdev code)
238 This attribute should show which device API is being created, for example,
239 "vfio-pci" for a PCI device.
241 * available_instances
243 This attribute should show the number of devices of type <type-id> that can be
248 This directory contains links to the devices of type <type-id> that have been
253 This attribute should show human readable name. This is optional attribute.
257 This attribute should show brief features/description of the type. This is
260 Directories and Files Under the sysfs for Each mdev Device
261 ----------------------------------------------------------
265 |- [parent phy device]
268 |--- mdev_type {link to its type}
269 |--- vendor-specific-attributes [optional]
271 * remove (write only)
273 Writing '1' to the 'remove' file destroys the mdev device. The vendor driver can
274 fail the remove() callback if that device is active and the vendor driver
275 doesn't support hot unplug.
279 # echo 1 > /sys/bus/mdev/devices/$mdev_UUID/remove
281 Mediated device Hot plug
282 ------------------------
284 Mediated devices can be created and assigned at runtime. The procedure to hot
285 plug a mediated device is the same as the procedure to hot plug a PCI device.
287 Translation APIs for Mediated Devices
288 =====================================
290 The following APIs are provided for translating user pfn to host pfn in a VFIO
293 extern int vfio_pin_pages(struct device *dev, unsigned long *user_pfn,
294 int npage, int prot, unsigned long *phys_pfn);
296 extern int vfio_unpin_pages(struct device *dev, unsigned long *user_pfn,
299 These functions call back into the back-end IOMMU module by using the pin_pages
300 and unpin_pages callbacks of the struct vfio_iommu_driver_ops[4]. Currently
301 these callbacks are supported in the TYPE1 IOMMU module. To enable them for
302 other IOMMU backend modules, such as PPC64 sPAPR module, they need to provide
303 these two callback functions.
305 Using the Sample Code
306 =====================
308 mtty.c in samples/vfio-mdev/ directory is a sample driver program to
309 demonstrate how to use the mediated device framework.
311 The sample driver creates an mdev device that simulates a serial port over a PCI
314 1. Build and load the mtty.ko module.
316 This step creates a dummy device, /sys/devices/virtual/mtty/mtty/
318 Files in this device directory in sysfs are similar to the following::
320 # tree /sys/devices/virtual/mtty/mtty/
321 /sys/devices/virtual/mtty/mtty/
322 |-- mdev_supported_types
324 | | |-- available_instances
330 | |-- available_instances
336 | `-- sample_mtty_dev
338 | |-- autosuspend_delay_ms
340 | |-- runtime_active_time
342 | `-- runtime_suspended_time
343 |-- subsystem -> ../../../../class/mtty
346 2. Create a mediated device by using the dummy device that you created in the
349 # echo "83b8f4f2-509f-382f-3c1e-e6bfe0fa1001" > \
350 /sys/devices/virtual/mtty/mtty/mdev_supported_types/mtty-2/create
352 3. Add parameters to qemu-kvm::
355 sysfsdev=/sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001
359 In the Linux guest VM, with no hardware on the host, the device appears
362 # lspci -s 00:05.0 -xxvv
363 00:05.0 Serial controller: Device 4348:3253 (rev 10) (prog-if 02 [16550])
364 Subsystem: Device 4348:3253
366 Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
367 Stepping- SERR- FastB2B- DisINTx-
368 Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
369 <TAbort- <MAbort- >SERR- <PERR- INTx-
370 Interrupt: pin A routed to IRQ 10
371 Region 0: I/O ports at c150 [size=8]
372 Region 1: I/O ports at c158 [size=8]
373 Kernel driver in use: serial
374 00: 48 43 53 32 01 00 00 02 10 02 00 07 00 00 00 00
375 10: 51 c1 00 00 59 c1 00 00 00 00 00 00 00 00 00 00
376 20: 00 00 00 00 00 00 00 00 00 00 00 00 48 43 53 32
377 30: 00 00 00 00 00 00 00 00 00 00 00 00 0a 01 00 00
379 In the Linux guest VM, dmesg output for the device is as follows:
381 serial 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10
382 0000:00:05.0: ttyS1 at I/O 0xc150 (irq = 10) is a 16550A
383 0000:00:05.0: ttyS2 at I/O 0xc158 (irq = 10) is a 16550A
386 5. In the Linux guest VM, check the serial ports::
388 # setserial -g /dev/ttyS*
389 /dev/ttyS0, UART: 16550A, Port: 0x03f8, IRQ: 4
390 /dev/ttyS1, UART: 16550A, Port: 0xc150, IRQ: 10
391 /dev/ttyS2, UART: 16550A, Port: 0xc158, IRQ: 10
393 6. Using minicom or any terminal emulation program, open port /dev/ttyS1 or
394 /dev/ttyS2 with hardware flow control disabled.
396 7. Type data on the minicom terminal or send data to the terminal emulation
397 program and read the data.
399 Data is loop backed from hosts mtty driver.
401 8. Destroy the mediated device that you created::
403 # echo 1 > /sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001/remove
408 1. See Documentation/driver-api/vfio.rst for more information on VFIO.
409 2. struct mdev_driver in include/linux/mdev.h
410 3. struct mdev_parent_ops in include/linux/mdev.h
411 4. struct vfio_iommu_driver_ops in include/linux/vfio.h