VFIO device Migration
Migration of virtual machine involves saving the state for each device that the guest is running on source host and restoring this saved state on the destination host. This document details how saving and restoring of VFIO devices is done in QEMU.
Migration of VFIO devices consists of two phases: the optional pre-copy phase, and the stop-and-copy phase. The pre-copy phase is iterative and allows to accommodate VFIO devices that have a large amount of data that needs to be transferred. The iterative pre-copy phase of migration allows for the guest to continue whilst the VFIO device state is transferred to the destination, this helps to reduce the total downtime of the VM. VFIO devices can choose to skip the pre-copy phase of migration by returning pending_bytes as zero during the pre-copy phase.
A detailed description of the UAPI for VFIO device migration can be found in
the comment for the
vfio_device_migration_info structure in the header
VFIO implements the device hooks for the iterative approach as follows:
save_setupfunction that sets up the migration region and sets _SAVING flag in the VFIO device state.
load_setupfunction that sets up the migration region on the destination and sets _RESUMING flag in the VFIO device state.
save_live_pendingfunction that reads pending_bytes from the vendor driver, which indicates the amount of data that the vendor driver has yet to save for the VFIO device.
save_live_iteratefunction that reads the VFIO device’s data from the vendor driver through the migration region during iterative phase.
save_statefunction to save the device config space if it is present.
save_live_complete_precopyfunction that resets _RUNNING flag from the VFIO device state and iteratively copies the remaining data for the VFIO device until the vendor driver indicates that no data remains (pending bytes is zero).
load_statefunction that loads the config section and the data sections that are generated by the save functions above
cleanupfunctions for both save and load that perform any migration related cleanup, including unmapping the migration region
The VFIO migration code uses a VM state change handler to change the VFIO device state when the VM state changes from running to not-running, and vice versa.
Similarly, a migration state change handler is used to trigger a transition of the VFIO device state when certain changes of the migration state occur. For example, the VFIO device state is transitioned back to _RUNNING in case a migration failed or was canceled.
System memory dirty pages tracking
log_global_stop memory listener callback informs
the VFIO IOMMU module to start and stop dirty page tracking. A
memory listener callback marks those system memory pages as dirty which are
used for DMA by the VFIO device. The dirty pages bitmap is queried per
container. All pages pinned by the vendor driver through external APIs have to
be marked as dirty during migration. When there are CPU writes, CPU dirty page
tracking can identify dirtied pages, but any page pinned by the vendor driver
can also be written by the device. There is currently no device or IOMMU
support for dirty page tracking in hardware.
By default, dirty pages are tracked when the device is in pre-copy as well as stop-and-copy phase. So, a page pinned by the vendor driver will be copied to the destination in both phases. Copying dirty pages in pre-copy phase helps QEMU to predict if it can achieve its downtime tolerances. If QEMU during pre-copy phase keeps finding dirty pages continuously, then it understands that even in stop-and-copy phase, it is likely to find dirty pages and can predict the downtime accordingly.
QEMU also provides a per device opt-out option
which disables querying the dirty bitmap during pre-copy phase. If it is set to
off, all dirty pages will be copied to the destination in stop-and-copy phase
System memory dirty pages tracking when vIOMMU is enabled
With vIOMMU, an IO virtual address range can get unmapped while in pre-copy phase of migration. In that case, the unmap ioctl returns any dirty pages in that range and QEMU reports corresponding guest physical pages dirty. During stop-and-copy phase, an IOMMU notifier is used to get a callback for mapped pages and then dirty pages bitmap is fetched from VFIO IOMMU modules for those mapped ranges.
Flow of state changes during Live migration
Below is the flow of state change during live migration. The values in the brackets represent the VM state, the migration state, and the VFIO device state, respectively.
Live migration save path
QEMU normal running state (RUNNING, _NONE, _RUNNING) | migrate_init spawns migration_thread Migration thread then calls each device's .save_setup() (RUNNING, _SETUP, _RUNNING|_SAVING) | (RUNNING, _ACTIVE, _RUNNING|_SAVING) If device is active, get pending_bytes by .save_live_pending() If total pending_bytes >= threshold_size, call .save_live_iterate() Data of VFIO device for pre-copy phase is copied Iterate till total pending bytes converge and are less than threshold | On migration completion, vCPU stops and calls .save_live_complete_precopy for each active device. The VFIO device is then transitioned into _SAVING state (FINISH_MIGRATE, _DEVICE, _SAVING) | For the VFIO device, iterate in .save_live_complete_precopy until pending data is 0 (FINISH_MIGRATE, _DEVICE, _STOPPED) | (FINISH_MIGRATE, _COMPLETED, _STOPPED) Migraton thread schedules cleanup bottom half and exits
Live migration resume path
Incoming migration calls .load_setup for each device (RESTORE_VM, _ACTIVE, _STOPPED) | For each device, .load_state is called for that device section data (RESTORE_VM, _ACTIVE, _RESUMING) | At the end, .load_cleanup is called for each device and vCPUs are started (RUNNING, _NONE, _RUNNING)
Postcopy migration is currently not supported for VFIO devices.