Warm Migrations

Warm migrations in mtv-api-tests are the powered-on migration path. The suite starts from a running source VM, lets MTV perform one or more precopy rounds, and then completes the migration at a scheduled cutover time. If you are setting up or tuning warm coverage, the main knobs are warm_migration, snapshots_interval, and mins_before_cutover.

What You Need To Set

A warm scenario in this repository is driven by a small set of plan fields:

  • warm_migration: True tells the suite to create a warm MTV plan.
  • source_vm_power: "on" makes the source VM run before migration starts.
  • guest_agent: True enables guest-agent validation after cutover.
  • target_power_state lets you enforce the expected destination VM power state.

Example from tests/tests_config/config.py:

"test_sanity_warm_mtv_migration": {
    "virtual_machines": [
        {
            "name": "mtv-tests-rhel8",
            "source_vm_power": "on",
            "guest_agent": True,
        },
    ],
    "warm_migration": True,
    "preserve_static_ips": True,
},

In practice, warm tests in this repository also clone the source VM before migration. That makes repeated test runs safer and gives the suite a place to apply warm-specific preparation.

The Five-Step Warm Pattern

Warm classes use the same five test methods as cold classes, with one behavioral difference in test_migrate_vms: it passes a computed cut_over timestamp.

  1. test_create_storagemap
  2. test_create_networkmap
  3. test_create_plan
  4. test_migrate_vms
  5. test_check_vms

The ClusterRole warm classes are the exception to the five-step shape. TestClusterroleWarmMtvMigration (tests/warm/test_mtv_warm_clusterrole_migration.py) stops after test_migrate_vms and has no test_check_vms, because its subject is the ClusterRole binding rather than post-migration VM validation. TestClusterroleWarmWithSccMigration keeps the full five steps and does run check_vms().

Warm classes that exercise the precopy interval — the standard warm scenarios — also activate the session-scoped precopy_interval_forkliftcontroller fixture through @pytest.mark.usefixtures, which patches the live ForkliftController before any migration starts. The two ClusterRole warm classes (TestClusterroleWarmMtvMigration and TestClusterroleWarmWithSccMigration) request only cleanup_migrated_vms, so they run without that patch.

Two warm variants extend the five-step pattern:

  • 6-step XFS pattern in tests/warm/test_warm_migration_xfs.py (TestWarmMigrationXfs): storagemap -> networkmap -> plan -> migrate -> test_verify_xfs_version -> check_vms

  • 6-step copy-offload pattern in tests/copyoffload/test_copyoffload_migration.py (TestCopyoffloadWarmMigration): storagemap -> networkmap -> plan -> migrate -> test_check_xcopy_used -> check_vms

How Warm Source Preparation Works

For vSphere sources, the suite explicitly enables Change Block Tracking (CBT) on the cloned VM before warm migration starts. prepared_plan passes enable_ctk=warm_migration in the clone options, and the vSphere provider applies it during clone:

# Enable Change Block Tracking (CBT) only for warm migrations
enable_ctk = kwargs.get("enable_ctk", False)
if enable_ctk:
    LOGGER.info("Enabling Change Block Tracking (CBT) for warm migration")
    cbt_option = vim.option.OptionValue()
    cbt_option.key = "ctkEnabled"
    cbt_option.value = "true"

CBT is enabled at the VM level and then per-disk on each virtual disk's SCSI controller.

Tip: The VM name suffix generated by get_vm_suffix() embeds the migration type, so warm clones end in -warm and cold clones in -cold. That suffix is part of the VM name and is truncated to fit the 63-character DNS limit.

Precopy Interval Tuning

Warm timing knobs live in tests/tests_config/config.py, which is the default test configuration file loaded by pytest.ini:

snapshots_interval: int = 2
mins_before_cutover: int = 5
plan_wait_timeout: int = 3600

The snapshots_interval value is applied through the precopy_interval_forkliftcontroller fixture, which patches the live ForkliftController custom resource in the MTV namespace:

current_interval = getattr(forklift_controller.instance.spec, "controller_precopy_interval", None)

if str(current_interval) == str(snapshots_interval):
    LOGGER.info(
        f"ForkliftController controller_precopy_interval already set to {snapshots_interval}, skipping update"
    )
    yield
    return

LOGGER.info(
    f"Setting ForkliftController controller_precopy_interval from {current_interval!r} to {snapshots_interval}"
)
ResourceEditor(patches={forklift_controller: {"spec": {"controller_precopy_interval": snapshots_interval}}}).update(
    backup_resources=False
)

The fixture intentionally does not restore the previous interval. Its docstring (conftest.py:781) records why: with ResourceEditor restoring at teardown, one parallel worker ending its run would clobber the interval other workers are still migrating with. The patched value therefore stays on the controller for the rest of the session.

What these values do:

  • snapshots_interval controls how often Forklift schedules warm precopy snapshots.
  • mins_before_cutover controls how far in the future cutover is scheduled.
  • plan_wait_timeout controls how long the suite waits for the migration to finish.

The defaults in this repository are a fast 2 second precopy interval and a 5 minute cutover delay.

Warning: snapshots_interval is not treated as a per-plan setting in the test suite. Warm tests patch the live forklift-controller ForkliftController resource while the fixture is active.

Cutover Timing

Warm tests do not cut over immediately by default. They compute the cutover timestamp in UTC from mins_before_cutover:

def get_cutover_value(current_cutover: bool = False) -> datetime:
    datetime_utc = datetime.now(pytz.utc)
    if current_cutover:
        return datetime_utc

    return datetime_utc + timedelta(minutes=int(py_config["mins_before_cutover"]))

That helper is passed into the migration execution step:

di_callback = (
    create_di_capture_callback(plan=self.plan_resource, fixture_store=fixture_store)
    if source_provider.type == Provider.ProviderType.VSPHERE
    else None
)
execute_migration(
    ocp_admin_client=ocp_admin_client,
    fixture_store=fixture_store,
    plan=self.plan_resource,
    target_namespace=target_namespace,
    cut_over=get_cutover_value(),
    on_status_poll=di_callback,
)

This gives the warm migration time to accumulate precopy work before the final switchover. On vSphere the status-poll callback also captures Deep Inspection output during execution, and check_vms() verifies the captured DI results afterwards.

Use these tuning rules:

  • Increase mins_before_cutover if you want a longer warm phase before downtime.
  • Decrease mins_before_cutover if you want the test to reach final cutover sooner.
  • Increase plan_wait_timeout if your environment is slow and the plan needs more time to finish.

Tip: The helper also supports immediate cutover with current_cutover=True, although the warm test classes in this repository use the delayed default.

Supported Providers

Warm support is not expressed per test class. It is enforced at collection time by a pytest_collection_modifyitems hook in the root conftest.py, based on the source provider type in .providers.json:

# Skip warm migration tests for providers that do not support warm migration.
warm_unsupported = (
    Provider.ProviderType.OPENSTACK,
    Provider.ProviderType.OPENSHIFT,
    Provider.ProviderType.OVA,
    Provider.ProviderType.HYPERV,
)
if source_provider_type in warm_unsupported:
    warm_skip = pytest.mark.skip(reason=f"{source_provider_type} warm migration is not supported.")
    for item in items:
        if "warm" in item.keywords:
            item.add_marker(warm_skip)
Source provider Warm status in this repo Notes
vSphere Supported Main warm path. Also used for warm copy-offload and warm XFS coverage.
RHV Supported Warm classes carry @pytest.mark.rhv alongside @pytest.mark.vsphere.
OpenStack Not supported Skipped by the collection hook.
OpenShift Not supported as a warm source Skipped by the collection hook.
OVA Not supported Skipped by the collection hook.
Hyper-V Not supported Skipped by the collection hook.

Note that the class-level @pytest.mark.vsphere and @pytest.mark.rhv markers on warm classes are provider-selection markers for -m filtering. They do not by themselves skip a class; the collection hook is what actually gates execution.

The repository also includes remote-destination warm scenarios:

  • TestWarmRemoteOcp runs a warm migration to a remote OpenShift provider.
  • TestClusterroleWarmMtvMigration and TestClusterroleWarmWithSccMigration test the ClusterRole-based destination provider with and without an SCC binding.
  • All three require remote_ocp_cluster to be configured and carry both @pytest.mark.warm and @pytest.mark.remote.

Warm copy-offload coverage is vSphere-only, enforced separately by the copyoffload_config fixture in tests/copyoffload/conftest.py, which raises if the source provider is not vSphere.

Marker definitions come from pytest.ini:

markers =
    tier0: Core functionality tests (smoke tests)
    tier1: Extended functionality tests
    remote: Remote cluster migration tests
    warm: Warm migration tests
    copyoffload: Copy-offload (XCOPY) tests

Note: Most warm suites are selected with the warm marker. The remote OpenShift warm scenarios live in tests/warm/test_mtv_warm_migration.py and tests/warm/test_mtv_warm_clusterrole_migration.py, and they are selected through remote in addition to warm.

Warm Scenarios Included In The Repository

These are the main warm configurations currently defined in tests/tests_config/config.py:

  • test_sanity_warm_mtv_migration: basic warm migration of a powered-on RHEL VM, with static IP preservation enabled
  • test_mtv_migration_warm_2disks2nics: warm migration of a multi-disk, multi-NIC VM
  • test_warm_remote_ocp: warm migration to a remote OpenShift destination
  • test_mtv_clusterrole_warm_migration: warm migration using the ClusterRole-based destination provider
  • test_mtv_clusterrole_warm_migration_with_scc: same, plus an SCC RoleBinding for the default ServiceAccount
  • test_warm_migration_comprehensive: warm migration with static IP, PVC naming, target labels, affinity, and custom VM namespace coverage
  • test_warm_migration_xfs: warm migration that verifies XFS v4 filesystem compatibility after cutover
  • test_copyoffload_warm_migration: warm migration through the vSphere copy-offload path

What Warm Tests Validate

Warm tests use the standard post-migration validation path in check_vms(). That means a warm run is not only checking whether MTV finishes, but also whether the migrated VM looks correct afterward.

The standard validation path covers:

  • VM power state, including target_power_state when it is set
  • CPU and memory
  • network mapping
  • storage mapping
  • provider SSL configuration for VMware, RHV, and OpenStack providers
  • guest agent availability when the plan expects it
  • SSH connectivity when the destination VM is powered on
  • static IP preservation and, for vSphere, NIC name preservation
  • nested-virtualization CPU features when enable_nested_virtualization is False
  • plan-driven Deep Inspection results for warm vSphere migrations

For vSphere sources the suite additionally checks snapshot tracking, serial preservation, and boot configuration.

This is especially useful for warm migrations because the destination VM is usually powered on after cutover, which allows the suite to verify guest-agent and SSH behavior immediately.

Note: SSH-based checks run only when the destination VM is powered on. If you set a powered-off target state, those checks are skipped.

The Comprehensive Warm Validation Path

The most feature-rich warm scenario in the repository is test_warm_migration_comprehensive:

"test_warm_migration_comprehensive": {
    "virtual_machines": [
        {
            "name": "mtv-win2022-ip-3disks",
            "source_vm_power": "on",
            "guest_agent": True,
        },
    ],
    "warm_migration": True,
    "target_power_state": "on",
    "preserve_static_ips": True,
    "enable_nested_virtualization": False,
    "vm_target_namespace": f"mtv-vms-warm-comprehensive-{uuid.uuid4().hex[:4]}",
    "multus_namespace": "default",  # Cross-namespace NAD access
    # Keys must be Provider.ProviderType string values (e.g. "vsphere") or "default".
    # vsphere uses .FileName; default truncates VmName to stay within 63 chars.
    "pvc_name_template": {
        "vsphere": '{{ .FileName | trimSuffix ".vmdk" | replace "_" "-" }}-{{.DiskIndex}}',
        "default": '{{ .VmName | trunc 32 | trimSuffix "-" }}-{{ .VmName | trunc -4 }}-disk-{{.DiskIndex}}',
    },
    "pvc_name_template_use_generate_name": True,
    "target_labels": {
        "mtv-comprehensive-test": None,  # None = auto-generate with session_uuid
        "static-label": "static-value",
    },
    "target_affinity": {
        "podAffinity": {
            "preferredDuringSchedulingIgnoredDuringExecution": [
                {
                    "podAffinityTerm": {
                        "labelSelector": {"matchLabels": {"app": "comprehensive-test"}},
                        "topologyKey": "kubernetes.io/hostname",
                    },
                    "weight": 75,
                }
            ]
        }
    },
    "guest_agent_timeout": 600,
},

This configuration adds several warm-specific checks on top of the baseline check_vms() flow:

  • preserve_static_ips triggers static IP validation after cutover
  • enable_nested_virtualization: False asserts that nested-virtualization CPU features are disabled
  • vm_target_namespace moves the migrated VM into a custom namespace
  • pvc_name_template enables PVC name verification against the rendered Forklift template
  • pvc_name_template_use_generate_name switches PVC validation to prefix matching, because Kubernetes adds a random suffix
  • target_labels verifies labels on the migrated VM
  • target_affinity verifies the affinity block on the destination VM

A few details are easy to miss:

  • In target_labels, a value of None means the suite replaces that value with the session UUID at runtime.
  • When vm_target_namespace is set, the fixture creates that namespace if needed before migration.
  • pvc_name_template may be a provider-keyed mapping. prepared_plan resolves it with resolve_pvc_name_template(), which falls back to the "default" key.
  • PVC template rendering supports Go template syntax and Sprig functions in the validation path.

Note: The {{.FileName}} template token is resolved from vSphere inventory disk paths only. For a non-vSphere source provider, check_pvc_names() logs a warning and skips PVC name verification entirely.

Warm XFS: The Extra Verification Step

TestWarmMigrationXfs in tests/warm/test_warm_migration_xfs.py is a warm vSphere class that inserts test_verify_xfs_version between migration and validation. It runs a configured command on the migrated VM and asserts on the output:

"test_warm_migration_xfs": {
    "virtual_machines": [
        {
            "name": "mtv-feature-rhel7-xfs",
            "source_vm_power": "on",
            "guest_agent": True,
        },
    ],
    "warm_migration": True,
    "target_power_state": "on",
    "xfs_compatibility": True,
    "xfs_check": {
        "command": "/usr/sbin/xfs_info",
        "mount_point": "/",
        "expected_output": "crc=0",
    },
},

xfs_compatibility: True makes the Plan request the XFS-compatible virt-v2v image. The xfs_check dict is required for these tests: command, mount_point, and expected_output are all read directly by test_verify_xfs_version.

Warm Copy-Offload: The Extra Validation Path

The warm copy-offload scenario adds one more validation step after the migration. In tests/copyoffload/test_copyoffload_migration.py, test_check_xcopy_used runs first to prove the accelerated transfer path was used, and test_check_vms then runs the standard verification followed by an explicit disk-count check:

def test_check_xcopy_used(
    self, ocp_admin_client: DynamicClient, target_namespace: str, fixture_store: dict[str, Any]
) -> None:
    """Verify XCOPY acceleration was used for all disks.

    Args:
        ocp_admin_client (DynamicClient): OpenShift admin client.
        target_namespace (str): Namespace where populate pods exist.
        fixture_store (dict[str, Any]): Fixture store containing cached populate pod logs.
    """
    verify_xcopy_used(
        ocp_admin_client=ocp_admin_client,
        plan=self.plan_resource,
        target_namespace=target_namespace,
        expected_xcopy_used=True,
        fixture_store=fixture_store,
    )

That extra assertion matters for the copy-offload path because it confirms that the migrated VM still has the expected disk inventory after the accelerated transfer flow.

The corresponding warm test plan looks like this:

"test_copyoffload_warm_migration": {
    "virtual_machines": [
        {
            "name": "xcopy-template-test",
            "source_vm_power": "on",
            "guest_agent": True,
            "clone": True,
            "disk_type": "thin",
        },
    ],
    "warm_migration": True,
    "copyoffload": True,
},

This path is stricter about provider requirements than the standard warm path:

  • the source provider must be vSphere
  • the provider configuration must include a copyoffload section
  • the storage credentials and required copy-offload fields must be present

Warning: Warm copy-offload is not a generic warm migration path. The fixture fails early unless the source provider is vSphere and the required copyoffload settings are configured.

Provider Configuration For Full Warm Validation

If you want the full warm validation path, especially guest-agent and SSH-based checks, your provider entry needs guest VM credentials. The vSphere example in .providers.json.example includes them:

"vsphere": {
    "type": "vsphere",
    "version": "<SERVER VERSION>",
    "fqdn": "SERVER FQDN/IP",
    "api_url": "<SERVER FQDN/IP>/sdk",
    "username": "USERNAME",
    "password": "PASSWORD",  # pragma: allowlist secret
    "guest_vm_linux_user": "LINUX VMS USERNAME",
    "guest_vm_linux_password": "LINUX VMS PASSWORD",  # pragma: allowlist secret
    "guest_vm_win_user": "WINDOWS VMS USERNAME",
    "guest_vm_win_password": "WINDOWS VMS PASSWORD",  # pragma: allowlist secret
    "vddk_init_image": "<PATH TO VDDK INIT IMAGE>",
    "endpoint_type": "vcenter"
}

The warm copy-offload path expects a copyoffload subsection under the vSphere provider entry. The example file includes fields such as storage_vendor_product, datastore_id, storage_hostname, storage_username, and storage_password.

Note: The example provider file uses comments for secret-scanning exceptions. Those comments are fine in the example file, but they are not valid in strict JSON.

Practical Guidance

  • Start with test_sanity_warm_mtv_migration when you want to prove your environment can complete a basic warm cycle.
  • Move to test_mtv_migration_warm_2disks2nics when you want more disk and network coverage without turning on every advanced feature.
  • Use test_warm_migration_comprehensive when you specifically need to validate static IP preservation, PVC naming, target labels, affinity, and custom VM namespace placement.
  • Use test_copyoffload_warm_migration only when your environment is already prepared for vSphere copy-offload.
  • Increase mins_before_cutover if your goal is to observe more precopy work before final downtime.
  • Keep target_power_state: "on" when you want SSH-based post-migration validation to run.