# mtv-api-tests > Pytest-based integration suite for validating Migration Toolkit for Virtualization migrations into OpenShift Virtualization. --- Source: introduction.md # Introduction `mtv-api-tests` is an end-to-end validation suite for Migration Toolkit for Virtualization (MTV). It is built on `pytest`, but it is not a typical unit-test project: it connects to real source providers, creates real MTV custom resources on OpenShift, runs actual migrations, and then checks the migrated virtual machines on the destination cluster. That makes it useful when you need to answer practical questions such as: - Can MTV migrate this VM from my provider into OpenShift Virtualization? - Do warm migrations still behave correctly after an MTV or cluster upgrade? - Did advanced plan settings such as hooks, copy-offload, PVC naming, labels, affinity, or target placement actually take effect? ## What mtv-api-tests is for This project is aimed at people who need confidence in real migration behavior, not just API-level validation: - QE and release-validation teams qualifying MTV across supported migration paths - Platform engineers testing migrations in their own OpenShift environments - Storage, provider, and partner teams validating feature-specific scenarios such as copy-offload - Operators who need proof that a migrated VM still behaves the way they expect after the move > **Warning:** `mtv-api-tests` is not a mock-based local test harness. It expects a live OpenShift environment with MTV installed, real source-provider credentials, and real VMs or templates to migrate. ## What it validates The repository covers the full MTV workflow, not just a single API call or resource: | Area | What the suite covers | | --- | --- | | Source providers | vSphere, RHV/oVirt, OpenStack, OVA, and OpenShift source-provider flows | | Destination | OpenShift Virtualization, including remote-cluster-style scenarios | | Migration types | Cold migration, warm migration, copy-offload, hook-based flows, and comprehensive feature combinations | | MTV resources | `Provider`, `StorageMap`, `NetworkMap`, `Plan`, `Hook`, and `Migration` custom resources | | VM outcome checks | Power state, CPU, memory, network mapping, storage mapping, PVC naming, guest agent, SSH connectivity, static IP preservation, node placement, labels, and affinity | Warm migration coverage is provider-aware. The warm test suite explicitly skips unsupported source types such as OpenStack, OpenShift, and OVA, so the test matrix follows the support rules encoded by the project itself. > **Tip:** Start with the `tier0` scenarios such as `test_sanity_cold_mtv_migration` or `test_sanity_warm_mtv_migration`. They exercise the same MTV lifecycle as the larger suites, but with a smaller and easier-to-debug scope. ## How a migration is validated A typical `mtv-api-tests` run follows the same five-step pattern used throughout the repository: 1. Load the selected source provider from `.providers.json`. 2. Create or connect the MTV `Provider` resources on OpenShift. 3. Build `StorageMap` and `NetworkMap` resources for the selected VMs. 4. Create a `Plan` and then a `Migration` custom resource. 5. Inspect the migrated VM on OpenShift and compare it to the source VM and expected plan settings. The core cold-migration test shows that pattern directly: ```python vms = [vm["name"] for vm in prepared_plan["virtual_machines"]] self.__class__.storage_map = get_storage_migration_map( fixture_store=fixture_store, source_provider=source_provider, destination_provider=destination_provider, source_provider_inventory=source_provider_inventory, ocp_admin_client=ocp_admin_client, target_namespace=target_namespace, vms=vms, ) assert self.storage_map, "StorageMap creation failed" vms = [vm["name"] for vm in prepared_plan["virtual_machines"]] self.__class__.network_map = get_network_migration_map( fixture_store=fixture_store, source_provider=source_provider, destination_provider=destination_provider, source_provider_inventory=source_provider_inventory, ocp_admin_client=ocp_admin_client, target_namespace=target_namespace, multus_network_name=multus_network_name, vms=vms, ) assert self.network_map, "NetworkMap creation failed" populate_vm_ids(prepared_plan, source_provider_inventory) self.__class__.plan_resource = create_plan_resource( ocp_admin_client=ocp_admin_client, fixture_store=fixture_store, source_provider=source_provider, destination_provider=destination_provider, storage_map=self.storage_map, network_map=self.network_map, virtual_machines_list=prepared_plan["virtual_machines"], target_namespace=target_namespace, warm_migration=prepared_plan.get("warm_migration", False), ) assert self.plan_resource, "Plan creation failed" execute_migration( ocp_admin_client=ocp_admin_client, fixture_store=fixture_store, plan=self.plan_resource, target_namespace=target_namespace, ) check_vms( plan=prepared_plan, source_provider=source_provider, destination_provider=destination_provider, network_map_resource=self.network_map, storage_map_resource=self.storage_map, source_provider_data=source_provider_data, source_vms_namespace=source_vms_namespace, source_provider_inventory=source_provider_inventory, vm_ssh_connections=vm_ssh_connections, ) ``` That same lifecycle appears across cold, warm, comprehensive, hook, remote, and copy-offload suites. What changes from test to test is the migration scenario and the validation expectations, not the basic MTV flow. Under the hood, that flow stays grounded in real platform state: - Source-provider adapters in `libs/providers/` connect to actual provider APIs. - The `prepared_plan` fixture can clone source VMs, power them on or off, create hooks, and prepare extra namespaces or networks before migration begins. - The `ForkliftInventory` helpers query the live `forklift-inventory` route and wait until provider, VM, storage, and network data are actually available before proceeding. ## How configuration works `mtv-api-tests` separates environment configuration from migration-scenario configuration. ### Provider and environment configuration Source-provider credentials and connection details are loaded from `.providers.json`. The example file shows the expected shape: ```jsonc { "vsphere": { "type": "vsphere", "version": "", "fqdn": "SERVER FQDN/IP", "api_url": "/sdk", "username": "USERNAME", "password": "PASSWORD", # pragma: allowlist secret "guest_vm_linux_user": "LINUX VMS USERNAME", "guest_vm_linux_password": "LINUX VMS PASSWORD", # pragma: allowlist secret "guest_vm_win_user": "WINDOWS VMS USERNAME", "guest_vm_win_password": "WINDOWS VMS PASSWORD", # pragma: allowlist secret "vddk_init_image": "" } } ``` The same example file also includes entries for `ovirt`, `openstack`, `openshift`, and `ova`, so the project can model more than one kind of source platform. For copy-offload scenarios, the example file adds a `copyoffload` section with storage-vendor and datastore settings. > **Note:** `.providers.json.example` contains inline comments for documentation and secret-scanning rules. Your real `.providers.json` must be valid JSON without those comments. Those provider entries do more than create MTV `Provider` resources. They also supply guest credentials used later for SSH-based validation of migrated VMs. ### Scenario configuration Individual migration scenarios live in `tests/tests_config/config.py`. That file is effectively the catalog of what the project knows how to validate. A single scenario can switch advanced MTV features on and off: ```python "test_warm_migration_comprehensive": { "virtual_machines": [ { "name": "mtv-win2022-ip-3disks", "source_vm_power": "on", "guest_agent": True, }, ], "warm_migration": True, "target_power_state": "on", "preserve_static_ips": True, "vm_target_namespace": "custom-vm-namespace", "multus_namespace": "default", "pvc_name_template": '{{ .FileName | trimSuffix ".vmdk" | replace "_" "-" }}-{{.DiskIndex}}', "pvc_name_template_use_generate_name": True, "target_labels": { "mtv-comprehensive-test": None, "static-label": "static-value", }, "target_affinity": { "podAffinity": { "preferredDuringSchedulingIgnoredDuringExecution": [ { "podAffinityTerm": { "labelSelector": {"matchLabels": {"app": "comprehensive-test"}}, "topologyKey": "kubernetes.io/hostname", }, "weight": 75, } ] } }, }, ``` This is a good example of what makes `mtv-api-tests` more than a smoke suite. A scenario can describe not only which VM to migrate, but also which migration mode to use and what should still be true afterward. The same configuration file also carries global test settings such as: - `mtv_namespace = "openshift-mtv"` - `target_namespace_prefix = "auto"` - `snapshots_interval = 2` - `plan_wait_timeout = 3600` Those defaults tell you a lot about the intended environment: MTV is expected to be present on the cluster, namespaces are created per test session, warm-migration precopy timing is tunable, and migrations are expected to run long enough to justify an explicit timeout. ## Why this is real migration validation A successful MTV `Plan` is not the same thing as a successful migration outcome. `mtv-api-tests` adds value because it keeps checking after the migration controller finishes. The `check_vms()` logic in `utilities/post_migration.py` shows the kind of user-visible validation the suite performs: ```python if vm_guest_agent: try: check_guest_agent(destination_vm=destination_vm) except Exception as exp: res[vm_name].append(f"check_guest_agent - {str(exp)}") # SSH connectivity check - only when destination VM is powered on if vm_ssh_connections and destination_vm.get("power_state") == "on": try: check_ssh_connectivity( vm_name=vm_name, vm_ssh_connections=vm_ssh_connections, source_provider_data=source_provider_data, source_vm_info=source_vm, ) except Exception as exp: res[vm_name].append(f"check_ssh_connectivity - {str(exp)}") # Static IP preservation check - only for Windows VMs with static IPs migrated from VSPHERE source_vm_data = plan.get("source_vms_data", {}).get(vm["name"], {}) if ( source_vm_data and source_vm_data.get("win_os") and source_provider.type == Provider.ProviderType.VSPHERE ): try: check_static_ip_preservation( vm_name=vm_name, vm_ssh_connections=vm_ssh_connections, source_vm_data=source_vm_data, source_provider_data=source_provider_data, ) except Exception as exp: res[vm_name].append(f"check_static_ip_preservation - {str(exp)}") # Check node placement if configured if plan.get("target_node_selector") and labeled_worker_node: try: check_vm_node_placement( destination_vm=destination_vm, expected_node=labeled_worker_node["node_name"], ) except Exception as exp: res[vm_name].append(f"check_vm_node_placement - {str(exp)}") # Check VM labels if configured if plan.get("target_labels") and target_vm_labels: try: check_vm_labels( destination_vm=destination_vm, expected_labels=target_vm_labels["vm_labels"], ) except Exception as exp: res[vm_name].append(f"check_vm_labels - {str(exp)}") # Check affinity if configured if plan.get("target_affinity"): try: check_vm_affinity( destination_vm=destination_vm, expected_affinity=plan["target_affinity"], ) except Exception as exp: res[vm_name].append(f"check_vm_affinity - {str(exp)}") ``` That means a run can fail for the reasons users actually care about: - The VM came up with the wrong power state - Guest connectivity never returned - Static IP preservation did not hold - The VM landed on the wrong node - Labels or affinity settings were not applied - Storage or network mappings did not produce the expected result The repository also includes feature-specific suites that go beyond basic migration success: - Copy-offload tests validate vSphere shared-storage migrations using `vsphere-xcopy-volume-populator` - Hook tests validate both expected success and expected failure paths for pre- and post-migration hooks - Comprehensive tests validate PVC naming, target namespaces, affinity, labels, and node selectors - Remote scenarios validate migrations where the destination is modeled as an explicit OpenShift provider ## Automation-friendly by design Although these are real-environment tests, the project is structured to run cleanly in automation. The repository-wide `pytest.ini` configuration makes that clear: ```ini addopts = -s -o log_cli=true -p no:logging --tc-file=tests/tests_config/config.py --tc-format=python --junit-xml=junit-report.xml --basetemp=/tmp/pytest --show-progress --strict-markers --jira --dist=loadscope ``` In practice, that means: - Scenario data is injected consistently from `tests/tests_config/config.py` - Results are emitted in JUnit format for downstream reporting - Marker usage is enforced - The suite is prepared for class-scoped parallel execution - Jira integration is part of the default test run The repository also ships a `Dockerfile` that installs the project with `uv` and provides a repeatable containerized execution environment. That makes it easier to run the same validation flow across teams, clusters, or lab environments without rebuilding the toolchain by hand. `mtv-api-tests` is best understood as a migration-confidence suite. If you need to know whether MTV can really move VMs from a supported source provider into OpenShift Virtualization, and whether the result still matches your expectations after the move, this project is built to answer that question. --- Source: project-structure.md # Project Structure `mtv-api-tests` is organized as an end-to-end `pytest` suite for Migration Toolkit for Virtualization (MTV). Instead of a conventional Python application layout such as `src/`, the repository is split into scenario files, shared fixtures, provider adapters, validation helpers, and repository automation/configuration files. > **Note:** The main entrypoint is `pytest`, not a packaged application. Most of the reusable behavior lives in `conftest.py`, `utilities/`, and `libs/`, while the files under `tests/` mostly define scenarios and expectations. ## At A Glance ```text mtv-api-tests/ ├── tests/ │ ├── tests_config/config.py │ ├── test_mtv_cold_migration.py │ ├── test_mtv_warm_migration.py │ ├── test_copyoffload_migration.py │ ├── test_cold_migration_comprehensive.py │ ├── test_warm_migration_comprehensive.py │ └── test_post_hook_retain_failed_vm.py ├── utilities/ ├── libs/ │ ├── base_provider.py │ ├── forklift_inventory.py │ └── providers/ ├── exceptions/ ├── docs/ │ └── copyoffload/ ├── tools/ ├── conftest.py ├── pyproject.toml ├── uv.lock ├── pytest.ini ├── tox.toml ├── Dockerfile ├── .providers.json.example ├── .pre-commit-config.yaml ├── renovate.json ├── .release-it.json ├── .coderabbit.yaml ├── .pr_agent.toml ├── .flake8 ├── .markdownlint.yaml ├── jira.cfg.example ├── OWNERS └── junit_report_example.xml ``` A useful way to read the repository is: 1. `pytest.ini` tells you how the suite is launched. 2. `tests/tests_config/config.py` tells you what each scenario wants to do. 3. `tests/*.py` tells you which migration flow is being exercised. 4. `conftest.py` shows how providers, namespaces, networks, hooks, and cleanup are prepared. 5. `libs/` shows how the suite talks to source and destination platforms. 6. `utilities/` shows how MTV resources are created and how migrated VMs are validated. `pytest.ini` makes that structure explicit by wiring `pytest` to the scenario config file, JUnit output, strict markers, Jira integration, and `xdist` distribution: ```1:25:pytest.ini [pytest] testpaths = tests addopts = -s -o log_cli=true -p no:logging --tc-file=tests/tests_config/config.py --tc-format=python --junit-xml=junit-report.xml --basetemp=/tmp/pytest --show-progress --strict-markers --jira --dist=loadscope markers = tier0: Core functionality tests (smoke tests) remote: Remote cluster migration tests warm: Warm migration tests copyoffload: Copy-offload (XCOPY) tests incremental: marks tests as incremental (xfail on previous failure) min_mtv_version: mark test to require minimum MTV version (e.g., @pytest.mark.min_mtv_version("2.6.0")) junit_logging = all ``` ## `tests/`: Scenario Suites The `tests/` directory contains the scenario definitions. These are mostly thin wrappers around shared fixtures and helpers. That keeps each suite readable while still allowing the repository to cover many migration variations. The main suites are: - `tests/test_mtv_cold_migration.py` covers the core cold migration path and remote OpenShift cold migration. - `tests/test_mtv_warm_migration.py` covers warm migration, cutover handling, and remote warm migration. - `tests/test_copyoffload_migration.py` is the largest suite and covers copy-offload/XCOPY behavior such as thin and thick disks, snapshots, multi-datastore scenarios, scale tests, naming edge cases, simultaneous plans, and mixed XCOPY/VDDK behavior. - `tests/test_cold_migration_comprehensive.py` covers higher-level cold migration features such as static IP preservation, PVC naming templates, node selection, labels, affinity, and custom target namespaces. - `tests/test_warm_migration_comprehensive.py` does the same for warm migration. - `tests/test_post_hook_retain_failed_vm.py` verifies hook-aware behavior when a migration is expected to fail after a post-hook but the migrated VM should still be retained. A representative test class from `tests/test_mtv_cold_migration.py` shows the standard pattern used across the suite: ```17:58:tests/test_mtv_cold_migration.py @pytest.mark.tier0 @pytest.mark.incremental @pytest.mark.parametrize( "class_plan_config", [ pytest.param( py_config["tests_params"]["test_sanity_cold_mtv_migration"], ) ], indirect=True, ids=["rhel8"], ) @pytest.mark.usefixtures("cleanup_migrated_vms") class TestSanityColdMtvMigration: """Cold migration test - sanity check.""" storage_map: StorageMap network_map: NetworkMap plan_resource: Plan def test_create_storagemap( self, prepared_plan, fixture_store, ocp_admin_client, source_provider, destination_provider, source_provider_inventory, target_namespace, ): """Create StorageMap resource for migration.""" vms = [vm["name"] for vm in prepared_plan["virtual_machines"]] self.__class__.storage_map = get_storage_migration_map( fixture_store=fixture_store, source_provider=source_provider, destination_provider=destination_provider, source_provider_inventory=source_provider_inventory, ocp_admin_client=ocp_admin_client, target_namespace=target_namespace, vms=vms, ) assert self.storage_map, "StorageMap creation failed" ``` That five-step flow repeats throughout the repository: 1. Create `StorageMap` 2. Create `NetworkMap` 3. Create `Plan` 4. Execute migration 5. Validate migrated VMs Scenario data lives in `tests/tests_config/config.py`. This file does much more than list VM names: it carries migration mode, target power state, hook behavior, PVC naming templates, labels, affinity rules, copy-offload flags, timeouts, and other per-scenario settings. For example, the comprehensive warm and cold scenarios define advanced feature coverage directly in configuration: ```434:516:tests/tests_config/config.py "test_warm_migration_comprehensive": { "virtual_machines": [ { "name": "mtv-win2022-ip-3disks", "source_vm_power": "on", "guest_agent": True, }, ], "warm_migration": True, "target_power_state": "on", "preserve_static_ips": True, "vm_target_namespace": "custom-vm-namespace", "multus_namespace": "default", # Cross-namespace NAD access "pvc_name_template": '{{ .FileName | trimSuffix ".vmdk" | replace "_" "-" }}-{{.DiskIndex}}', "pvc_name_template_use_generate_name": True, "target_labels": { "mtv-comprehensive-test": None, # None = auto-generate with session_uuid "static-label": "static-value", }, "target_affinity": { "podAffinity": { "preferredDuringSchedulingIgnoredDuringExecution": [ { "podAffinityTerm": { "labelSelector": {"matchLabels": {"app": "comprehensive-test"}}, "topologyKey": "kubernetes.io/hostname", }, "weight": 75, } ] } }, }, "test_cold_migration_comprehensive": { "virtual_machines": [ { "name": "mtv-win2019-3disks", "source_vm_power": "off", "guest_agent": True, }, ], "warm_migration": False, "target_power_state": "on", "preserve_static_ips": True, "pvc_name_template": "{{.VmName}}-disk-{{.DiskIndex}}", "pvc_name_template_use_generate_name": False, "target_node_selector": { "mtv-comprehensive-node": None, # None = auto-generate with session_uuid }, "target_labels": { "mtv-comprehensive-label": None, # None = auto-generate with session_uuid "test-type": "comprehensive", # Static value }, "target_affinity": { "podAffinity": { "preferredDuringSchedulingIgnoredDuringExecution": [ { "podAffinityTerm": { "labelSelector": {"matchLabels": {"app": "test"}}, "topologyKey": "kubernetes.io/hostname", }, "weight": 50, } ] } }, "vm_target_namespace": "mtv-comprehensive-vms", "multus_namespace": "default", # Cross-namespace NAD access }, "test_post_hook_retain_failed_vm": { "virtual_machines": [ { "name": "mtv-tests-rhel8", "source_vm_power": "on", "guest_agent": True, }, ], "warm_migration": False, "target_power_state": "off", "pre_hook": {"expected_result": "succeed"}, "post_hook": {"expected_result": "fail"}, "expected_migration_result": "fail", }, ``` > **Tip:** When you want to understand a scenario quickly, start with its entry in `tests/tests_config/config.py`, then read the matching class in `tests/`, and only then follow the shared helpers it imports. ## `conftest.py`: Shared Fixtures And Runtime Orchestration `conftest.py` is the operational heart of the repository. It is where session-wide and class-wide fixtures build the real test environment. Key responsibilities in `conftest.py` include: - Creating the target namespace with the right labels in `target_namespace` - Loading the requested provider entry from `.providers.json` in `source_provider_data` - Creating source and destination `Provider` CRs in `source_provider` and `destination_provider` - Resolving the right Forklift inventory implementation in `source_provider_inventory` - Downloading and caching `virtctl` for the current cluster in `virtctl_binary` - Creating class-scoped Multus `NetworkAttachmentDefinition` resources in `multus_network_name` - Preparing cloned VMs, custom namespaces, and optional hooks in `prepared_plan` - Managing post-migration SSH sessions in `vm_ssh_connections` - Cleaning up migrated VMs after each class in `cleanup_migrated_vms` A few practical details are worth calling out: - `prepared_plan` is where source VMs are cloned or otherwise prepared before the MTV `Plan` is created. - The fixture stores detailed source VM facts separately in `plan["source_vms_data"]`, so the serialized `virtual_machines` list stays clean for MTV resource creation. - Hook resources are created from plan configuration before the test methods start running. - `virtctl_binary` is cached by cluster version and protected by file locking, which makes parallel `pytest-xdist` runs safer. > **Note:** Reusable logic is intentionally centralized here. If a test file looks surprisingly small, that is usually by design. ## `libs/`: Provider Adapters And Inventory Clients The `libs/` directory is the platform abstraction layer. `libs/base_provider.py` defines the common interface the rest of the suite expects. Each provider implementation can connect, test availability, return a normalized VM description via `vm_dict()`, clone VMs where needed, delete VMs, and expose provider-specific network information through a shared contract. The provider implementations are: - `libs/providers/vmware.py` for VMware vSphere; this is the most feature-heavy adapter and includes cloning, guest info handling, datastore logic, snapshot handling, copy-offload support, and ESXi-related behavior. - `libs/providers/rhv.py` for RHV/oVirt. - `libs/providers/openstack.py` for OpenStack. - `libs/providers/openshift.py` for OpenShift Virtualization/KubeVirt; this adapter is especially important on the destination side because it inspects migrated `VirtualMachine` resources. - `libs/providers/ova.py` for OVA-based scenarios. `libs/forklift_inventory.py` is the second half of the abstraction. Instead of talking to source providers directly, some mapping logic needs the Forklift inventory service. This file wraps the `forklift-inventory` API and provides provider-specific inventory classes such as `VsphereForkliftInventory`, `OvirtForkliftInventory`, `OpenstackForliftinventory`, `OpenshiftForkliftInventory`, and `OvaForkliftInventory`. The fixture below shows how the right inventory client is selected at runtime: ```1193:1214:conftest.py @pytest.fixture(scope="session") def source_provider_inventory( ocp_admin_client: DynamicClient, mtv_namespace: str, source_provider: BaseProvider ) -> ForkliftInventory: if not source_provider.ocp_resource: raise ValueError("source_provider.ocp_resource is not set") providers = { Provider.ProviderType.OVA: OvaForkliftInventory, Provider.ProviderType.RHV: OvirtForkliftInventory, Provider.ProviderType.VSPHERE: VsphereForkliftInventory, Provider.ProviderType.OPENSHIFT: OpenshiftForkliftInventory, Provider.ProviderType.OPENSTACK: OpenstackForliftinventory, } provider_instance = providers.get(source_provider.type) if not provider_instance: raise ValueError(f"Provider {source_provider.type} not implemented") return provider_instance( # type: ignore client=ocp_admin_client, namespace=mtv_namespace, provider_name=source_provider.ocp_resource.name ) ``` This split is important when reading the codebase: - `libs/providers/*` talks to the source or destination platform itself. - `libs/forklift_inventory.py` talks to the MTV/Forklift inventory API that MTV uses for discovery and mapping. ## `utilities/`: Shared Orchestration, Validation, And Diagnostics The `utilities/` directory is where most of the suite’s real work happens. If `tests/` describes *what* to migrate, `utilities/` contains most of the code for *how* to set up, execute, verify, and clean up the migration. The major modules are: - `utilities/mtv_migration.py` builds `StorageMap`, `NetworkMap`, `Plan`, and `Migration` resources and waits for migration completion. - `utilities/post_migration.py` performs post-migration validation for CPU, memory, storage, networking, guest agent state, SSH, snapshots, static IP preservation, node placement, labels, and affinity. - `utilities/resources.py` centralizes resource creation and teardown tracking. - `utilities/utils.py` loads provider configuration, creates provider secrets and `Provider` CRs, fetches CA certificates, and contains general cluster helpers. - `utilities/hooks.py` creates MTV hook resources and validates expected hook-related failure behavior. - `utilities/migration_utils.py` handles cutover timing, plan archiving, migration cancelation, and cleanup checks for DVs, PVCs, and PVs. - `utilities/ssh_utils.py` provides post-migration SSH access to VMs through `virtctl port-forward`. - `utilities/virtctl.py` downloads the correct `virtctl` binary from the cluster for the current OS and architecture. - `utilities/worker_node_selection.py` picks worker nodes for placement-sensitive tests, using Prometheus metrics when available. - `utilities/copyoffload_migration.py`, `utilities/copyoffload_constants.py`, and `utilities/esxi.py` contain copy-offload-specific behavior and credentials handling. - `utilities/must_gather.py` collects diagnostics with `oc adm must-gather`. - `utilities/pytest_utils.py` handles dry-run behavior, resource collection, session teardown, and optional AI analysis wiring. - `utilities/naming.py` generates short unique resource names and sanitizes VM names for Kubernetes. - `utilities/logger.py` configures queue-based logging so parallel workers can write to a single stream cleanly. The shared resource creation helper is one of the simplest and most important building blocks in the repository: ```19:69:utilities/resources.py def create_and_store_resource( client: "DynamicClient", fixture_store: dict[str, Any], resource: type[Resource], test_name: str | None = None, **kwargs: Any, ) -> Any: kwargs["client"] = client _resource_name = kwargs.get("name") _resource_dict = kwargs.get("kind_dict", {}) _resource_yaml = kwargs.get("yaml_file") if _resource_yaml and _resource_dict: raise ValueError("Cannot specify both yaml_file and kind_dict") if not _resource_name: if _resource_yaml: with open(_resource_yaml) as fd: _resource_dict = yaml.safe_load(fd) _resource_name = _resource_dict.get("metadata", {}).get("name") if not _resource_name: _resource_name = generate_name_with_uuid(name=fixture_store["base_resource_name"]) if resource.kind in (Migration.kind, Plan.kind): _resource_name = f"{_resource_name}-{'warm' if kwargs.get('warm_migration') else 'cold'}" if len(_resource_name) > 63: LOGGER.warning(f"'{_resource_name=}' is too long ({len(_resource_name)} > 63). Truncating.") _resource_name = _resource_name[-63:] kwargs["name"] = _resource_name _resource = resource(**kwargs) try: _resource.deploy(wait=True) except ConflictError: LOGGER.warning(f"{_resource.kind} {_resource_name} already exists, reusing it.") _resource.wait() LOGGER.info(f"Storing {_resource.kind} {_resource.name} in fixture store") _resource_dict = {"name": _resource.name, "namespace": _resource.namespace, "module": _resource.__module__} if test_name: _resource_dict["test_name"] = test_name fixture_store["teardown"].setdefault(_resource.kind, []).append(_resource_dict) return _resource ``` That helper is used by higher-level orchestration code in `utilities/mtv_migration.py`. One good example is storage mapping, where the code branches between standard inventory-based mapping and copy-offload-specific mapping: ```445:505:utilities/mtv_migration.py if datastore_id and offload_plugin_config: # Copy-offload migration mode datastores_to_map = [datastore_id] if secondary_datastore_id: datastores_to_map.append(secondary_datastore_id) LOGGER.info(f"Creating copy-offload storage map for primary and secondary datastores: {datastores_to_map}") else: LOGGER.info(f"Creating copy-offload storage map for primary datastore: {datastore_id}") # Create a storage map entry for each XCOPY-capable datastore for ds_id in datastores_to_map: destination_config = { "storageClass": target_storage_class, } # Add copy-offload specific destination settings if access_mode: destination_config["accessMode"] = access_mode if volume_mode: destination_config["volumeMode"] = volume_mode storage_map_list.append({ "destination": destination_config, "source": {"id": ds_id}, "offloadPlugin": offload_plugin_config, }) LOGGER.info(f"Added storage map entry for datastore: {ds_id} with copy-offload") # Add non-XCOPY datastore mapping (with offload plugin for fallback) if non_xcopy_datastore_id: destination_config = {"storageClass": target_storage_class} if access_mode: destination_config["accessMode"] = access_mode if volume_mode: destination_config["volumeMode"] = volume_mode storage_map_list.append({ "destination": destination_config, "source": {"id": non_xcopy_datastore_id}, "offloadPlugin": offload_plugin_config, }) LOGGER.info(f"Added non-XCOPY datastore mapping for: {non_xcopy_datastore_id} (with xcopy fallback)") else: LOGGER.info(f"Creating standard storage map for VMs: {vms}") storage_migration_map = source_provider_inventory.vms_storages_mappings(vms=vms) for storage in storage_migration_map: storage_map_list.append({ "destination": {"storageClass": target_storage_class}, "source": storage, }) storage_map = create_and_store_resource( fixture_store=fixture_store, resource=StorageMap, client=ocp_admin_client, namespace=target_namespace, mapping=storage_map_list, source_provider_name=source_provider.ocp_resource.name, source_provider_namespace=source_provider.ocp_resource.namespace, destination_provider_name=destination_provider.ocp_resource.name, destination_provider_namespace=destination_provider.ocp_resource.namespace, ) ``` A few utility modules are especially helpful to know by name: - `utilities/post_migration.py` is where the deep VM checks happen. If a migrated VM has the wrong CPU, memory, disks, networks, PVC names, serial number, labels, or affinity, the logic is usually here. - `utilities/ssh_utils.py` reaches migrated VMs through `virtctl port-forward`, so validation does not depend on cluster nodes exposing guest SSH directly. - `utilities/hooks.py` supports both predefined success/failure hook playbooks and custom base64-encoded playbooks. - `utilities/must_gather.py` is what the suite uses when it needs richer failure diagnostics. - `utilities/worker_node_selection.py` exists because some scenarios validate placement-sensitive features rather than only migration success. > **Tip:** If you are tracing a failure after the MTV `Plan` is created, `utilities/mtv_migration.py` and `utilities/post_migration.py` are usually the next files to read. ## `exceptions/`: Centralized Domain Errors Custom exceptions are centralized in `exceptions/exceptions.py`. This keeps migration-specific failures easy to recognize and avoids scattering project-specific error types across multiple modules. Representative exceptions include: - `MigrationPlanExecError` for plan execution failure or timeout - `MigrationNotFoundError` and `MigrationStatusError` for missing or incomplete migration CR state - `VmPipelineError` and `VmMigrationStepMismatchError` for hook or pipeline analysis problems - `MissingProvidersFileError` for missing or empty `.providers.json` - `InvalidVMNameError`, `VmCloneError`, `VmBadDatastoreError`, and `VmNotFoundError` for provider-side and VM-side failures - `SessionTeardownError` for cleanup problems after the run This file is small, but it matters because the rest of the repository uses these names to make failures easier to understand during investigation. ## `docs/` And `tools/`: User Docs And Recovery Helpers The checked-in `docs/` tree is currently small and focused. Right now it contains `docs/copyoffload/how-to-run-copyoffload-tests.md`, which is a user-facing guide for setting up and running copy-offload scenarios. The `tools/` directory currently contains `tools/clean_cluster.py`, a recovery helper that reads a recorded resource list and calls `.clean_up()` on the matching objects. This is useful when a test run was interrupted and left resources behind. A few other support files are worth knowing about: - `junit_report_example.xml` shows the JUnit-style output shape emitted by `pytest` - `JOB_INSIGHT_PROMPT.md` contains instructions for automated job/failure analysis tooling - `OWNERS` lists repository approvers and reviewers > **Tip:** `tools/clean_cluster.py` pairs naturally with the resource tracking written by `utilities/pytest_utils.py`, which stores created resources in a JSON file when data collection is enabled. ## Configuration, Tooling, And Automation Files The repository root also contains the files that make the suite installable, configurable, lintable, and reviewable. The most important ones are: - `pyproject.toml` defines the Python project metadata and dependencies. It requires Python `>=3.12, <3.14` and includes `pytest`, `pytest-xdist`, `pytest-testconfig`, provider SDKs, `openshift-python-wrapper`, `openshift-python-utilities`, and other supporting libraries. - `uv.lock` locks the exact dependency set used by the repository. - `pytest.ini` configures test discovery, runtime options, markers, JUnit output, and `pytest-testconfig`. - `tests/tests_config/config.py` is the suite’s shared Python-based scenario configuration file. - `.providers.json.example` shows the structure expected by `load_source_providers()` and `source_provider_data()` for source providers and copy-offload settings. - `jira.cfg.example` shows the minimal format expected by the Jira integration enabled in `pytest.ini`. - `tox.toml` defines lightweight local automation tasks such as `pytest --setup-plan`, `pytest --collect-only`, and unused-code scanning. - `.pre-commit-config.yaml` configures local quality gates including `flake8`, `ruff`, `ruff-format`, `mypy`, `detect-secrets`, `gitleaks`, and `markdownlint-cli2`. - `Dockerfile` builds a Fedora-based test image, copies `utilities/`, `tests/`, `libs/`, and `exceptions/`, runs `uv sync --locked`, and defaults to `uv run pytest --collect-only`. - `.release-it.json` handles version bumping, tagging, pushing, and GitHub release creation. - `renovate.json` handles automated dependency update PRs and weekly lock file maintenance. - `.coderabbit.yaml` and `.pr_agent.toml` configure automated review behavior. - `.flake8` and `.markdownlint.yaml` hold narrower lint settings for Python and Markdown. The provider configuration example is especially important because most real test runs depend on it: ```16:44:.providers.json.example "vsphere-copy-offload": { "type": "vsphere", "version": "", "fqdn": "SERVER FQDN/IP", "api_url": "/sdk", "username": "USERNAME", "password": "PASSWORD", # pragma: allowlist secret "guest_vm_linux_user": "LINUX VMS USERNAME", "guest_vm_linux_password": "LINUX VMS PASSWORD", # pragma: allowlist secret "guest_vm_win_user": "WINDOWS VMS USERNAME", "guest_vm_win_password": "WINDOWS VMS PASSWORD", # pragma: allowlist secret "copyoffload": { # Supported storage_vendor_product values: # - "ontap" (NetApp ONTAP) # - "vantara" (Hitachi Vantara) # - "primera3par" (HPE Primera/3PAR) # - "pureFlashArray" (Pure Storage FlashArray) # - "powerflex" (Dell PowerFlex) # - "powermax" (Dell PowerMax) # - "powerstore" (Dell PowerStore) # - "infinibox" (Infinidat InfiniBox) # - "flashsystem" (IBM FlashSystem) "storage_vendor_product": "ontap", # Primary datastore for copy-offload operations (required) # This is the vSphere datastore ID (e.g., "datastore-12345") where VMs reside # Get via vSphere: Datacenter → Storage → Datastore → Summary → More Objects ID "datastore_id": "datastore-12345", ``` > **Warning:** `.providers.json.example` contains placeholder values and inline comments such as `# pragma: allowlist secret`. Those comments are useful inside this repository, but they are not valid JSON. Remove them when creating your real `.providers.json`. > **Warning:** This repository is code-complete, but many scenarios only make sense with live OpenShift, MTV, and source-provider environments. A local checkout without cluster access and provider credentials will let you read the structure, but not exercise the full migration stack. > **Note:** In this repository snapshot, there is no checked-in `.github/workflows` directory. CI/CD-adjacent behavior is expressed mostly through local tooling and repository-bot configuration files such as `tox.toml`, `.pre-commit-config.yaml`, `Dockerfile`, `.release-it.json`, `renovate.json`, `.coderabbit.yaml`, and `.pr_agent.toml`. > **Tip:** For the fastest mental model of the repository, read `pytest.ini`, then the matching scenario in `tests/tests_config/config.py`, then the test module in `tests/`, then `conftest.py`, and finally the helper modules imported from `utilities/` and `libs/`. --- Source: architecture-and-workflow.md # Architecture And Workflow `mtv-api-tests` is built around complete migration lifecycles, not isolated unit tests. A typical test run resolves source and destination providers, waits for Forklift inventory to discover what it should migrate, creates `StorageMap` and `NetworkMap` objects, creates an MTV `Plan`, runs a `Migration`, validates the migrated VM or VMs, and then removes both cluster-side and provider-side leftovers. > **Warning:** These are live integration tests. The repository’s built-in automation only does dry-run checks such as collection and setup planning. A real migration run needs a reachable OpenShift cluster with MTV installed, a valid source provider, credentials, storage, and networking. > **Tip:** If you want a concrete reference while reading this page, start with `tests/test_mtv_cold_migration.py`, `tests/test_cold_migration_comprehensive.py`, and `tests/test_warm_migration_comprehensive.py`. Together they show the normal workflow and most optional features. ## Runtime Inputs Two configuration layers drive everything: - `tests/tests_config/config.py` is loaded automatically by `pytest.ini` and holds cluster-wide settings plus per-test plan dictionaries under `tests_params`. - `.providers.json` selects the actual source provider profile. The repository includes `.providers.json.example` as the field reference for supported source types such as `vsphere`, `ovirt`, `openstack`, `openshift`, and `ova`, plus optional `copyoffload` settings. A representative plan config looks like this: ```467:502:tests/tests_config/config.py "test_cold_migration_comprehensive": { "virtual_machines": [ { "name": "mtv-win2019-3disks", "source_vm_power": "off", "guest_agent": True, }, ], "warm_migration": False, "target_power_state": "on", "preserve_static_ips": True, "pvc_name_template": "{{.VmName}}-disk-{{.DiskIndex}}", "pvc_name_template_use_generate_name": False, "target_node_selector": { "mtv-comprehensive-node": None, }, "target_labels": { "mtv-comprehensive-label": None, "test-type": "comprehensive", }, "target_affinity": { "podAffinity": { "preferredDuringSchedulingIgnoredDuringExecution": [ { "podAffinityTerm": { "labelSelector": {"matchLabels": {"app": "test"}}, "topologyKey": "kubernetes.io/hostname", }, "weight": 50, } ] } }, "vm_target_namespace": "mtv-comprehensive-vms", "multus_namespace": "default", }, ``` That single config entry already tells you a lot about the architecture. The same workflow can change behavior through plan fields such as `warm_migration`, `preserve_static_ips`, `pvc_name_template`, `target_labels`, `target_affinity`, `target_node_selector`, and `vm_target_namespace`. ## The Standard Test Shape Most migration tests follow the same five-step pattern: create storage map, create network map, create plan, execute migration, validate VMs. ```17:147:tests/test_mtv_cold_migration.py @pytest.mark.tier0 @pytest.mark.incremental @pytest.mark.parametrize( "class_plan_config", [ pytest.param( py_config["tests_params"]["test_sanity_cold_mtv_migration"], ) ], indirect=True, ids=["rhel8"], ) @pytest.mark.usefixtures("cleanup_migrated_vms") class TestSanityColdMtvMigration: storage_map: StorageMap network_map: NetworkMap plan_resource: Plan def test_create_storagemap(...): self.__class__.storage_map = get_storage_migration_map(...) assert self.storage_map, "StorageMap creation failed" def test_create_networkmap(...): self.__class__.network_map = get_network_migration_map(...) assert self.network_map, "NetworkMap creation failed" def test_create_plan(...): populate_vm_ids(prepared_plan, source_provider_inventory) self.__class__.plan_resource = create_plan_resource(...) assert self.plan_resource, "Plan creation failed" def test_migrate_vms(...): execute_migration(...) def test_check_vms(...): check_vms(...) ``` This class-based layout is intentional. Earlier methods create resources that later methods depend on, and `@pytest.mark.incremental` prevents the test from pretending the later stages are meaningful after an earlier failure. ## 1. Session Bootstrap And Provider Setup Before any migration-specific method runs, session fixtures do the shared setup work: - `pytest_sessionstart()` validates required settings such as `storage_class` and `source_provider`. - `target_namespace` creates a unique OpenShift namespace for the run. - `forklift_pods_state` waits until the Forklift pods are healthy. - `virtctl_binary` downloads and caches `virtctl`. - `source_provider_data` resolves the selected source provider from `.providers.json`. The source provider setup is centered around `utilities/utils.py:create_source_provider()`. It creates the source `Secret`, creates the source `Provider` custom resource, waits until that `Provider` is ready in Forklift, and only then opens the matching provider SDK wrapper. ```213:324:utilities/utils.py secret_string_data = { "url": source_provider_data_copy["api_url"], "insecureSkipVerify": "true" if insecure else "false", } provider_args = { "username": source_provider_data_copy["username"], "password": source_provider_data_copy["password"], "fixture_store": fixture_store, } if ocp_provider(provider_data=source_provider_data_copy): source_provider = OCPProvider source_provider_data_copy["api_url"] = ocp_admin_client.configuration.host source_provider_data_copy["type"] = Provider.ProviderType.OPENSHIFT source_provider_secret = destination_ocp_secret elif vmware_provider(provider_data=source_provider_data_copy): source_provider = VMWareProvider provider_args["host"] = source_provider_data_copy["fqdn"] secret_string_data["user"] = source_provider_data_copy["username"] secret_string_data["password"] = source_provider_data_copy["password"] if not insecure: _fetch_and_store_cacert(source_provider_data_copy, secret_string_data, tmp_dir, session_uuid) # ... RHV, OpenStack, and OVA branches omitted ... source_provider_secret = create_and_store_resource( fixture_store=fixture_store, resource=Secret, client=admin_client, namespace=namespace, string_data=secret_string_data, label=metadata_labels, ) ocp_resource_provider = create_and_store_resource( fixture_store=fixture_store, resource=Provider, client=admin_client, namespace=namespace, secret_name=source_provider_secret.name, secret_namespace=namespace, url=source_provider_data_copy["api_url"], provider_type=source_provider_data_copy["type"], vddk_init_image=source_provider_data_copy.get("vddk_init_image"), annotations=provider_annotations or None, ) ocp_resource_provider.wait_for_status(Provider.Status.READY, timeout=600, stop_status="ConnectionFailed") ``` A few important details come from this design: - The tests always create real Forklift `Provider` resources first. The provider SDK wrappers are helpers, not the system of record. - vSphere, RHV, OpenStack, OpenShift, and OVA all share the same outer workflow, even though their provider-specific secrets and connection logic differ. - Remote OpenShift destination tests reuse the same basic pattern, but switch from `destination_provider` to `destination_ocp_provider`. ## 2. Forklift Inventory Discovery And Plan Preparation Once the source `Provider` exists, the suite creates a `ForkliftInventory` adapter for that provider type in `conftest.py`. This is the layer that queries the `forklift-inventory` route and turns Forklift’s discovered objects into storage and network mappings. The class-scoped `prepared_plan` fixture is where the user-facing plan config becomes an actual migration input. It copies the config, creates custom namespaces if requested, clones or resolves source VMs, adjusts their source power state, stores source-side metadata, and then waits until Forklift inventory can see the exact VM it will migrate. ```840:952:conftest.py plan: dict[str, Any] = deepcopy(class_plan_config) virtual_machines: list[dict[str, Any]] = plan["virtual_machines"] warm_migration = plan.get("warm_migration", False) plan["source_vms_data"] = {} vm_target_namespace = plan.get("vm_target_namespace") if vm_target_namespace: get_or_create_namespace(...) plan["_vm_target_namespace"] = vm_target_namespace else: plan["_vm_target_namespace"] = target_namespace for vm in virtual_machines: clone_options = {**vm, "enable_ctk": warm_migration} provider_vm_api = source_provider.get_vm_by_name( query=vm["name"], vm_name_suffix=vm_name_suffix, clone_vm=True, session_uuid=fixture_store["session_uuid"], clone_options=clone_options, ) source_vm_power = vm.get("source_vm_power") if source_vm_power == "on": source_provider.start_vm(provider_vm_api) if source_provider.type == Provider.ProviderType.VSPHERE: source_provider.wait_for_vmware_guest_info(provider_vm_api, timeout=120) elif source_vm_power == "off": source_provider.stop_vm(provider_vm_api) source_vm_details = source_provider.vm_dict(...) vm["name"] = source_vm_details["name"] if source_provider.type != Provider.ProviderType.OVA: source_provider_inventory.wait_for_vm(name=vm["name"], timeout=300) plan["source_vms_data"][vm["name"]] = source_vm_details create_hook_if_configured(plan, "pre_hook", "pre", fixture_store, ocp_admin_client, target_namespace) create_hook_if_configured(plan, "post_hook", "post", fixture_store, ocp_admin_client, target_namespace) ``` This is one of the most important parts of the whole project. The test does not rush from “provider-side clone exists” to “create the Plan.” It waits until Forklift inventory has caught up. > **Note:** RHV/oVirt is the main exception to the normal “inventory first” story for networks. Those tests clone from templates, so early network discovery comes from the RHV template API rather than Forklift inventory. > **Note:** OpenStack inventory waits are stricter than a simple VM name lookup. The inventory adapter also waits until attached volumes and networks are queryable, because StorageMap and NetworkMap generation depend on that metadata. ## 3. StorageMap And NetworkMap After `prepared_plan` is ready, the suite creates the two map resources that make the rest of the workflow possible. For `StorageMap`, the standard path is: - Ask Forklift inventory which source storages the selected VM or VMs use. - Map each of those source storages to the configured destination `storage_class`. For `NetworkMap`, the rule is deterministic and simple: - The first source network maps to the destination pod network. - Every additional source network maps to a class-scoped Multus network attachment definition. - If the plan sets `multus_namespace`, those NADs can live outside the main test namespace. That behavior comes from `utilities/utils.py:gen_network_map_list()` and the `multus_network_name` fixture in `conftest.py`. A subtle but important detail happens right before Plan creation: `utilities/utils.py:populate_vm_ids()` injects Forklift inventory IDs into the VM list. The Plan is not built from names alone. > **Tip:** If a VM has only one NIC, no extra NADs are created. The Multus path only appears for the second and later source networks. ## 4. Plan Creation `utilities/mtv_migration.py:create_plan_resource()` is the assembly point where providers, maps, VM IDs, and optional plan features become an MTV `Plan` custom resource. ```200:259:utilities/mtv_migration.py plan_kwargs: dict[str, Any] = { "client": ocp_admin_client, "fixture_store": fixture_store, "resource": Plan, "namespace": target_namespace, "source_provider_name": source_provider.ocp_resource.name, "source_provider_namespace": source_provider.ocp_resource.namespace, "destination_provider_name": destination_provider.ocp_resource.name, "destination_provider_namespace": destination_provider.ocp_resource.namespace, "storage_map_name": storage_map.name, "storage_map_namespace": storage_map.namespace, "network_map_name": network_map.name, "network_map_namespace": network_map.namespace, "virtual_machines_list": virtual_machines_list, "target_namespace": vm_target_namespace or target_namespace, "warm_migration": warm_migration, "pre_hook_name": pre_hook_name, "pre_hook_namespace": pre_hook_namespace, "after_hook_name": after_hook_name, "after_hook_namespace": after_hook_namespace, "preserve_static_ips": preserve_static_ips, "pvc_name_template": pvc_name_template, "pvc_name_template_use_generate_name": pvc_name_template_use_generate_name, "target_power_state": target_power_state, } if target_node_selector: plan_kwargs["target_node_selector"] = target_node_selector if target_labels: plan_kwargs["target_labels"] = target_labels if target_affinity: plan_kwargs["target_affinity"] = target_affinity if copyoffload: plan_kwargs["pvc_name_template"] = "pvc" plan = create_and_store_resource(**plan_kwargs) plan.wait_for_condition(condition=Plan.Condition.READY, status=Plan.Condition.Status.TRUE, timeout=360) if copyoffload: wait_for_plan_secret(ocp_admin_client, target_namespace, plan.name) ``` A few things to notice: - The `Plan` resource itself lives in the test namespace, but the VMs can still target a separate `vm_target_namespace`. - Hooks, labels, affinity, target power state, PVC naming, and static IP preservation are all Plan-time features in this suite. - Copy-offload reuses the same overall path, but changes how the storage map is built and waits for a Forklift-created plan secret afterward. ## 5. Migration Execution Migration execution is intentionally small in the test code: the heavy lifting is delegated to MTV. `execute_migration()` creates a `Migration` CR that references the prepared `Plan`, then `wait_for_migration_complate()` polls the Plan status until it becomes `Succeeded` or `Failed`. Cold migrations create the `Migration` immediately. Warm migrations do two extra things: - They use `precopy_interval_forkliftcontroller` to patch the `ForkliftController` precopy interval. - They pass `get_cutover_value()` when creating the `Migration`, which schedules cutover using `mins_before_cutover` from `tests/tests_config/config.py`. > **Tip:** In this project, warm migration is not just `warm_migration=True` on the Plan. The test also sets a cutover timestamp on the `Migration` resource. ## 6. Post-Migration Validation `utilities/post_migration.py:check_vms()` is the main validator. It re-reads the source VM and the destination VM through the provider abstraction layer, runs a wide set of checks, collects all failures per VM, and only then fails the test. ```1151:1316:utilities/post_migration.py for vm in plan["virtual_machines"]: vm_name = vm["name"] source_vm = source_provider.vm_dict( name=vm_name, namespace=source_vms_namespace, source=True, source_provider_inventory=source_provider_inventory, ) destination_vm = destination_provider.vm_dict(...) check_vms_power_state(...) check_cpu(...) check_memory(...) if source_provider.type != Provider.ProviderType.OPENSHIFT: check_network(...) check_storage(...) if plan.get("pvc_name_template"): check_pvc_names(...) if source_provider.type == Provider.ProviderType.VSPHERE: check_snapshots(...) check_serial_preservation(...) if vm_guest_agent: check_guest_agent(...) if vm_ssh_connections and destination_vm.get("power_state") == "on": check_ssh_connectivity(...) if ...: check_static_ip_preservation(...) if plan.get("target_node_selector") and labeled_worker_node: check_vm_node_placement(...) if plan.get("target_labels") and target_vm_labels: check_vm_labels(...) if plan.get("target_affinity"): check_vm_affinity(...) ``` Depending on the plan and provider, validation can include: - Power state, CPU, and memory checks. - Network verification against the created `NetworkMap`. - Storage verification against the created `StorageMap`. - PVC naming checks for `pvc_name_template`. - VMware snapshot checks. - VMware serial preservation checks. - Guest agent verification. - SSH connectivity to the migrated guest. - Static IP preservation for Windows VMs migrated from vSphere. - Target node placement, labels, and affinity. - RHV-specific regression checks around unexpected source power-off behavior. This is why `prepared_plan["source_vms_data"]` matters: it preserves source-side facts that are needed later for snapshot, PVC-name, and static-IP comparisons. ## 7. Advanced Paths The normal workflow stays the same, but a few features add important branches. Warm migration changes execution timing. The Plan is warm, the source-side clone can have Change Block Tracking enabled, the Forklift precopy interval is patched, and the `Migration` is created with a cutover timestamp. Hooks are created during `prepared_plan` by `utilities/hooks.py:create_hook_if_configured()`. A plan can define `pre_hook` or `post_hook` with either a predefined success or failure playbook or a custom base64-encoded Ansible playbook. `tests/test_post_hook_retain_failed_vm.py` shows the intended behavior: a pre-hook failure can stop VM validation because the migration never really happened, while a post-hook failure can still leave migrated VMs behind and therefore still run `check_vms()`. Copy-offload keeps the same outer sequence but changes the storage side. In that mode: - The source provider is still vSphere. - The storage secret comes from the plan’s `copyoffload` configuration and optional environment-variable overrides. - `get_storage_migration_map()` uses datastore IDs and `offloadPlugin` entries instead of the standard inventory-derived storage list. - The workflow can expand to secondary or non-XCOPY datastores for multi-datastore and fallback scenarios. - If the provider requests SSH-based ESXi cloning, the setup fixtures install an SSH key before the migration and remove it afterward. Remote OpenShift destination tests also reuse the same pattern. The difference is mostly in which destination provider fixture they use, not in the rest of the migration flow. ## 8. Teardown And Failure Handling Cleanup is just as structured as setup. Almost every OpenShift-side resource is created through `utilities/resources.py:create_and_store_resource()`, which deploys the resource and records it in `fixture_store["teardown"]`. Class-level cleanup removes migrated VMs early, and session-level cleanup handles everything else. ```107:162:utilities/pytest_utils.py def session_teardown(session_store: dict[str, Any]) -> None: LOGGER.info("Running teardown to delete all created resources") ocp_client = get_cluster_client() if session_teardown_resources := session_store.get("teardown"): for migration_name in session_teardown_resources.get(Migration.kind, []): migration = Migration(name=migration_name["name"], namespace=migration_name["namespace"], client=ocp_client) cancel_migration(migration=migration) for plan_name in session_teardown_resources.get(Plan.kind, []): plan = Plan(name=plan_name["name"], namespace=plan_name["namespace"], client=ocp_client) archive_plan(plan=plan) leftovers = teardown_resources( session_store=session_store, ocp_client=ocp_client, target_namespace=session_store.get("target_namespace"), ) if leftovers: raise SessionTeardownError(f"Failed to clean up the following resources: {leftovers}") # inside teardown_resources(...) migrations = session_teardown_resources.get(Migration.kind, []) plans = session_teardown_resources.get(Plan.kind, []) providers = session_teardown_resources.get(Provider.kind, []) secrets = session_teardown_resources.get(Secret.kind, []) networkmaps = session_teardown_resources.get(NetworkMap.kind, []) storagemaps = session_teardown_resources.get(StorageMap.kind, []) virtual_machines = session_teardown_resources.get(VirtualMachine.kind, []) ``` Session teardown does more than delete a few CRs: - It cancels still-running migrations. - It archives Plans before deletion. - It deletes tracked `Provider`, `Secret`, `StorageMap`, `NetworkMap`, `Namespace`, `Migration`, and `VirtualMachine` resources. - It waits for `DataVolume`, `PVC`, and `PV` cleanup. - It reconnects to source providers such as vSphere, OpenStack, and RHV to delete cloned source-side VMs and snapshots that were created for the test. If `--skip-teardown` is set, the class and session cleanup paths intentionally leave resources behind. > **Note:** Failure handling is broader than cleanup. When data collection is enabled, the session writes created resources to `resources.json`, and failure paths can trigger `must-gather` collection so you can inspect what MTV and the cluster were doing at the time of failure. ## Automation And Dry Runs The repository includes local automation, but it is deliberately conservative. `pytest.ini` wires in `tests/tests_config/config.py`, enables strict markers, produces JUnit XML, and uses `loadscope` distribution. `tox.toml` does not try to run a real migration. Instead, it runs `pytest --setup-plan` and `pytest --collect-only`, which are useful for validating test discovery, parametrization, and fixture wiring without depending on a live MTV environment. That split is a good way to think about the project as a whole: - Configuration chooses the source provider and migration shape. - Fixtures turn that configuration into discoverable Forklift and OpenShift resources. - Tests create maps, Plans, and Migrations in a fixed order. - Validation compares source-side and destination-side reality. - Teardown removes everything the run created, both in the cluster and on the source side when needed. If you keep that control loop in mind, the rest of the repository becomes much easier to navigate. --- Source: supported-providers-and-scenarios.md # Supported Providers And Scenarios `mtv-api-tests` is an integration test suite for real MTV migrations into OpenShift Virtualization. The support matrix in this repository comes from three places: provider profiles in `.providers.json`, scenario definitions in `tests/tests_config/config.py`, and pytest markers in `pytest.ini`. ## Supported Source Providers The `source_provider` setting points to a named profile in `.providers.json`. That means you can keep multiple profiles for the same platform, such as different vSphere versions or labs, and select the one you want by profile name. ```2:27:.providers.json.example "vsphere": { "type": "vsphere", "version": "", "fqdn": "SERVER FQDN/IP", "api_url": "/sdk", "username": "USERNAME", "password": "PASSWORD", # pragma: allowlist secret "guest_vm_linux_user": "LINUX VMS USERNAME", "guest_vm_linux_password": "LINUX VMS PASSWORD", # pragma: allowlist secret "guest_vm_win_user": "WINDOWS VMS USERNAME", "guest_vm_win_password": "WINDOWS VMS PASSWORD", # pragma: allowlist secret "vddk_init_image": "" }, "vsphere-copy-offload": { "type": "vsphere", "version": "", "fqdn": "SERVER FQDN/IP", "api_url": "/sdk", "username": "USERNAME", "password": "PASSWORD", # pragma: allowlist secret // ... same guest credentials ... "copyoffload": { ``` The shipped example file also includes profiles for `ovirt`, `openstack`, `openshift`, and `ova`. | Source provider | `type` value | Cold migration | Warm migration | Copy-offload | Notes | | --- | --- | --- | --- | --- | --- | | VMware vSphere | `vsphere` | Yes | Yes | Yes | This is the broadest coverage area in the suite. Copy-offload is implemented as a vSphere profile with an extra `copyoffload` section. | | RHV / oVirt | `ovirt` | Yes | Yes, with extra suite gating | No | Standard MTV cold and warm scenarios exist in the codebase. | | OpenStack | `openstack` | Yes | No | No | Warm tests explicitly skip this provider family. | | OpenShift | `openshift` | Yes | No | No | Supported as a source provider, but not for warm scenarios. | | OVA | `ova` | Yes | No | No | Supported for cold-style scenarios. The built-in OVA path uses a fixed imported VM name during plan preparation. | > **Note:** `vsphere-copy-offload` is not a separate provider family. It is still `type: "vsphere"` with additional copy-offload settings. > **Warning:** Warm migration is not available for every source provider. The warm test modules explicitly skip `openstack`, `openshift`, and `ova`. ```21:27:tests/test_mtv_warm_migration.py pytestmark = [ pytest.mark.skipif( _SOURCE_PROVIDER_TYPE in (Provider.ProviderType.OPENSTACK, Provider.ProviderType.OPENSHIFT, Provider.ProviderType.OVA), reason=f"{_SOURCE_PROVIDER_TYPE} warm migration is not supported.", ), ] ``` ## Migration Modes This suite exercises three practical migration modes: - Cold migration: the default path, represented by scenarios where `warm_migration` is `False`. - Warm migration: the precopy and cutover path, represented by scenarios where `warm_migration` is `True`. - Copy-offload: the accelerated vSphere path, represented by scenarios where `copyoffload` is `True`. A simple warm scenario in the built-in test matrix looks like this: ```16:25:tests/tests_config/config.py "test_sanity_warm_mtv_migration": { "virtual_machines": [ { "name": "mtv-tests-rhel8", "source_vm_power": "on", "guest_agent": True, }, ], "warm_migration": True, }, ``` There is no separate `cold` marker. In this repository, “cold” is the normal case when `warm_migration` is not enabled. ### Copy-offload specifics Copy-offload is the most specialized mode in the suite. It is vSphere-only, and the repository includes both cold copy-offload coverage and a dedicated warm copy-offload scenario. The accepted `storage_vendor_product` values in the code are: - `ontap` - `vantara` - `primera3par` - `pureFlashArray` - `powerflex` - `powermax` - `powerstore` - `infinibox` - `flashsystem` Base copy-offload storage credentials are always required: - `storage_hostname` - `storage_username` - `storage_password` Some vendors also require extra fields: - `ontap`: `ontap_svm` - `vantara`: `vantara_storage_id`, `vantara_storage_port`, `vantara_hostgroup_id_list` - `pureFlashArray`: `pure_cluster_prefix` - `powerflex`: `powerflex_system_id` - `powermax`: `powermax_symmetrix_id` - `primera3par`, `powerstore`, `infinibox`, `flashsystem`: no extra vendor-specific fields beyond the base storage credentials If you use SSH-based ESXi cloning for copy-offload, the suite also expects `esxi_clone_method: "ssh"` plus `esxi_host`, `esxi_user`, and `esxi_password`. > **Tip:** Copy-offload credentials can come from `.providers.json` or from `COPYOFFLOAD_...` environment variables. Environment variables win, which is useful when you do not want secrets stored in files. ```21:45:utilities/copyoffload_migration.py def get_copyoffload_credential( credential_name: str, copyoffload_config: dict[str, Any], ) -> str | None: """ Get a copyoffload credential from environment variable or config file. Environment variables take precedence over config file values. Environment variable names are constructed as COPYOFFLOAD_{credential_name.upper()}. """ env_var_name = f"COPYOFFLOAD_{credential_name.upper()}" return os.getenv(env_var_name) or copyoffload_config.get(credential_name) ``` ## Pytest Markers The repository defines these markers in `pytest.ini`: ```17:23:pytest.ini markers = tier0: Core functionality tests (smoke tests) remote: Remote cluster migration tests warm: Warm migration tests copyoffload: Copy-offload (XCOPY) tests incremental: marks tests as incremental (xfail on previous failure) min_mtv_version: mark test to require minimum MTV version (e.g., @pytest.mark.min_mtv_version("2.6.0")) ``` In practice: - `tier0` is the smoke or core regression slice. - `warm`, `remote`, and `copyoffload` are the main user-facing selectors for scenario families. - `incremental` is execution behavior, not a functional feature area. These class-based tests move step by step through StorageMap, NetworkMap, Plan, migration execution, and validation. - `min_mtv_version` is available when a scenario needs a newer MTV version. > **Note:** `comprehensive` is a scenario family in this repository, not a pytest marker. You select it by file or class, not with `-m comprehensive`. > **Note:** Remote scenarios are opt-in. They are skipped unless `remote_ocp_cluster` is set. ## Major Scenario Families | Family | How you identify it | What it covers | | --- | --- | --- | | Tier0 | `@pytest.mark.tier0` | Core smoke coverage: sanity cold, sanity warm, comprehensive cold, comprehensive warm, and the post-hook retention failure scenario | | Warm | `@pytest.mark.warm` | Standard warm flows, a 2-disks/2-NICs case, remote warm, comprehensive warm, and warm copy-offload | | Remote | `@pytest.mark.remote` | Remote OpenShift destination scenarios for both cold and warm flows | | Comprehensive | Dedicated files and classes | Advanced plan options such as static IP preservation, custom VM namespaces, PVC naming templates, labels, affinity, and node placement | | Copy-offload | `@pytest.mark.copyoffload` | XCOPY/offload coverage across disk types, snapshots, datastores, naming behavior, scale, and concurrency | Scenario families are intentionally allowed to overlap. For example, comprehensive warm coverage is both `tier0` and `warm`, and warm copy-offload belongs to both `warm` and `copyoffload`. ### Tier0 coverage The `tier0` slice in this repository is broader than a single smoke test. It includes: - `TestSanityColdMtvMigration` - `TestSanityWarmMtvMigration` - `TestColdMigrationComprehensive` - `TestWarmMigrationComprehensive` - `TestPostHookRetainFailedVm` That last case matters because it covers a failure path: the migration is expected to fail in the post-hook stage while the migrated VMs are retained for verification. ### Remote coverage The built-in remote scenarios are: - `TestColdRemoteOcp` - `TestWarmRemoteOcp` These are gated by the `remote` marker and the `remote_ocp_cluster` setting. If you do not provide that setting, pytest skips them instead of failing later during migration setup. ### Comprehensive coverage The comprehensive tests are the best place to look when you want end-to-end coverage of plan options beyond “can the VM migrate.” A warm comprehensive scenario is configured like this: ```434:465:tests/tests_config/config.py "test_warm_migration_comprehensive": { "virtual_machines": [ { "name": "mtv-win2022-ip-3disks", "source_vm_power": "on", "guest_agent": True, }, ], "warm_migration": True, "target_power_state": "on", "preserve_static_ips": True, "vm_target_namespace": "custom-vm-namespace", "multus_namespace": "default", # Cross-namespace NAD access "pvc_name_template": '{{ .FileName | trimSuffix ".vmdk" | replace "_" "-" }}-{{.DiskIndex}}', "pvc_name_template_use_generate_name": True, "target_labels": { "mtv-comprehensive-test": None, # None = auto-generate with session_uuid "static-label": "static-value", }, "target_affinity": { "podAffinity": { "preferredDuringSchedulingIgnoredDuringExecution": [ { "podAffinityTerm": { "labelSelector": {"matchLabels": {"app": "comprehensive-test"}}, "topologyKey": "kubernetes.io/hostname", }, "weight": 75, } ] } }, }, ``` The cold comprehensive scenario follows the same idea, but adds cold-specific scheduling and naming checks such as `target_node_selector`, fixed PVC naming, and `warm_migration: False`. ### Copy-offload coverage `tests/test_copyoffload_migration.py` is the largest single scenario matrix in the repository. It covers: - Thin and thick-lazy disk migrations - Snapshot-based cases, including a 2 TB VM with snapshots - Multi-disk and multi-datastore layouts - Mixed XCOPY and non-XCOPY datastore behavior, including fallback paths - RDM virtual disks - Independent persistent and independent nonpersistent disks - Nonconforming source VM names - Warm copy-offload - Scale coverage with 5 VMs in one run - Simultaneous copy-offload plans - Concurrent XCOPY and VDDK plans in the same suite > **Warning:** Copy-offload is intentionally strict about prerequisites. The suite fails early if the source provider is not vSphere or if required copy-offload fields are missing. ## Practical Tip > **Tip:** Before running live migrations, use `uv run pytest --collect-only` to confirm what your current provider profile and markers will collect. The repository already uses that pattern in `tox.toml`, and the container image defaults to it. ```4:18:tox.toml [env.pytest-check] commands = [ [ "uv", "run", "pytest", "--setup-plan", ], [ "uv", "run", "pytest", "--collect-only", ], ] ``` If you only remember one rule for this page, make it this: vSphere has the broadest coverage, copy-offload is vSphere-only, warm migration is not supported for every provider, and `tier0`, `warm`, `remote`, and `copyoffload` are the markers you will use most often to slice the suite. --- Source: prerequisites.md # Prerequisites `mtv-api-tests` runs live end-to-end migrations. It talks to a real OpenShift cluster, a real MTV installation, and a real source provider. Before you run it, make sure the target cluster, source environment, storage, network, and RBAC are ready for destructive integration testing. > **Warning:** Run this suite only in a lab or other disposable environment. The tests create and delete `Namespace`, `Provider`, `StorageMap`, `NetworkMap`, `Plan`, `Migration`, `Hook`, `Secret`, `NetworkAttachmentDefinition`, `Pod`, and `VirtualMachine` resources. Some scenarios also clone source VMs, change source power state, add disks, and create snapshots. Example test definitions in `tests/tests_config/config.py` show that clearly: ```python "test_copyoffload_thin_snapshots_migration": { "virtual_machines": [ { "name": "xcopy-template-test", "source_vm_power": "off", "guest_agent": True, "clone": True, "disk_type": "thin", "snapshots": 2, }, ], "warm_migration": False, "copyoffload": True, }, ``` Choose source VMs that match the scenario you plan to run. Do not point the suite at production VMs or production-only datastores. ## Minimum environment Every run needs all of the following: - A reachable OpenShift API endpoint. - Valid OpenShift credentials for the test runner. - MTV installed and healthy in `openshift-mtv`, unless you intentionally override `mtv_namespace`. - OpenShift Virtualization installed on the destination cluster. - A usable destination `storage_class`. - A `source_provider` value that matches a key in `.providers.json`. The cluster client is created from these settings: ```python def get_cluster_client() -> DynamicClient: host = get_value_from_py_config("cluster_host") username = get_value_from_py_config("cluster_username") password = get_value_from_py_config("cluster_password") insecure_verify_skip = get_value_from_py_config("insecure_verify_skip") _client = get_client(host=host, username=username, password=password, verify_ssl=not insecure_verify_skip) ``` The built-in test config also defines the key global defaults: ```python insecure_verify_skip: str = "true" source_provider_insecure_skip_verify: str = "false" mtv_namespace: str = "openshift-mtv" remote_ocp_cluster: str = "" snapshots_interval: int = 2 plan_wait_timeout: int = 3600 ``` > **Note:** `insecure_verify_skip` controls TLS verification for the OpenShift API connection. `source_provider_insecure_skip_verify` is separate and defaults to `false`, so source-provider certificate verification is on unless you change it. Before tests start, the suite checks `forklift-*` pods in the MTV namespace and fails if the `forklift-controller` pod is missing or any `forklift` pod is not healthy. ## Source provider configuration The code supports these source provider types: | Source provider | Cold migration | Warm migration | Copy-offload | | --- | --- | --- | --- | | `vsphere` | Yes | Yes | Yes | | `ovirt` / RHV | Yes | Yes | No | | `openstack` | Yes | No | No | | `openshift` | Yes | No | No | | `ova` | Yes | No | No | Your `.providers.json` file must exist, must be non-empty, and must contain a key that exactly matches the `source_provider` value you pass at runtime. A trimmed example from `.providers.json.example`: ```json { "vsphere": { "type": "vsphere", "version": "", "fqdn": "SERVER FQDN/IP", "api_url": "/sdk", "username": "USERNAME", "password": "PASSWORD", "guest_vm_linux_user": "LINUX VMS USERNAME", "guest_vm_linux_password": "LINUX VMS PASSWORD", "guest_vm_win_user": "WINDOWS VMS USERNAME", "guest_vm_win_password": "WINDOWS VMS PASSWORD", "vddk_init_image": "" } } ``` > **Warning:** `.providers.json.example` is only an example. The real `.providers.json` is loaded with `json.loads()`, so it must be strict JSON. Remove example comments before you use it. Provider-specific expectations: - `vsphere` needs `fqdn`, `api_url`, username/password, and usually guest credentials for post-migration validation. If your MTV deployment expects VDDK, include `vddk_init_image`. - `ovirt` / RHV needs an API URL and username/password. - `openstack` needs extra auth fields: `project_name`, `user_domain_name`, `region_name`, `user_domain_id`, and `project_domain_id`. - `openshift` uses the local cluster as the source provider. - `ova` expects `api_url` to point at the OVA source location. Post-migration validation also uses guest credentials from `.providers.json`. Linux checks read `guest_vm_linux_user` and `guest_vm_linux_password`. Windows checks read `guest_vm_win_user` and `guest_vm_win_password`. > **Note:** With the default `source_provider_insecure_skip_verify: "false"`, the suite validates source-provider TLS. vSphere and OpenStack provider setup fetch CA data from `fqdn:443`, and RHV always pulls a CA certificate. Use insecure provider connections only in lab environments where that is acceptable. If you use OpenShift as the source provider, the suite expects OpenShift Virtualization template assets that match the hard-coded source VM fixture: ```python create_and_store_resource( resource=VirtualMachineFromInstanceType, fixture_store=fixture_store, name=f"{vm_dict['name']}{vm_name_suffix}", namespace=namespace, client=client, instancetype_name="u1.small", preference_name="rhel.9", datasource_name="rhel9", storage_size="30Gi", additional_networks=[network_name], ) ``` That means an OpenShift-source lab needs: - The `rhel9` `DataSource`. - The `u1.small` `VirtualMachineClusterInstancetype`. - The `rhel.9` `VirtualMachineClusterPreference`. ## Storage prerequisites The `storage_class` you choose must be a real, usable storage class for OpenShift Virtualization VM disks. The suite validates that migrated disks land on that exact storage class after migration. For standard cold and warm migration tests, that usually means: - the storage class can provision PVCs for KubeVirt, - the cluster can schedule VMs that use it, - the storage class is available in the target cluster where the tests run. Copy-offload adds stricter requirements: - the source provider must be `vSphere`, - the storage must be shared between vSphere and OpenShift, - the target storage class must be block-backed, - the environment must support `ReadWriteOnce` and `Block` volume mode for copy-offload mappings. The example provider file shows the copy-offload fields the suite expects: ```json { "vsphere-copy-offload": { "type": "vsphere", "version": "", "fqdn": "SERVER FQDN/IP", "api_url": "/sdk", "username": "USERNAME", "password": "PASSWORD", "copyoffload": { "storage_vendor_product": "ontap", "datastore_id": "datastore-12345", "secondary_datastore_id": "datastore-67890", "non_xcopy_datastore_id": "datastore-99999", "default_vm_name": "rhel9-template", "storage_hostname": "storage.example.com", "storage_username": "admin", "storage_password": "your-password-here", "ontap_svm": "vserver-name", "esxi_clone_method": "ssh", "esxi_host": "your-esxi-host.example.com", "esxi_user": "root", "esxi_password": "your-esxi-password", "rdm_lun_uuid": "naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" } } } ``` Supported copy-offload storage vendors are: - `ontap` - `vantara` - `primera3par` - `pureFlashArray` - `powerflex` - `powermax` - `powerstore` - `infinibox` - `flashsystem` Some copy-offload scenarios need extra fields: - `secondary_datastore_id` for multi-datastore tests - `non_xcopy_datastore_id` for mixed XCOPY/non-XCOPY tests - `rdm_lun_uuid` for RDM disk tests > **Warning:** Copy-offload is not an NFS scenario. The project’s copy-offload guide requires shared SAN/block storage and explicitly states that NFS is not supported for copy-offload. If you set `storage_class` to `nfs`, the suite patches the cluster `nfs` `StorageProfile` to `ReadWriteOnce` and `Filesystem` before running. > **Note:** That `StorageProfile` change is cluster-wide. Only use it in environments where changing the `nfs` profile is acceptable. ## Network prerequisites Source VMs must have at least one NIC. The suite fails immediately if it cannot discover any network interfaces for the selected source VMs. Network mapping works like this: - the first source NIC is mapped to the OpenShift pod network, - every additional source NIC is mapped to a `Multus` `NetworkAttachmentDefinition`. That behavior comes directly from the network mapping helper: ```python if pod_only or index == 0: _destination = {"type": "pod"} else: _destination = { "name": nad_name, "namespace": multus_namespace, "type": "multus", } ``` The default `Multus` CNI config created by the suite is a bridge called `cnv-bridge`: ```python bridge_type_and_name = "cnv-bridge" config = {"cniVersion": "0.3.1", "type": f"{bridge_type_and_name}", "bridge": f"{bridge_type_and_name}"} ``` In practice: - single-NIC migrations can run with the default pod-network mapping, - multi-NIC migrations need `Multus` to be installed and working, - the test account must be allowed to create `NetworkAttachmentDefinition` resources, - if you use a custom `multus_namespace`, the account must be allowed to create or reuse NADs there. ## Optional scenario prerequisites ### Warm migration Warm tests are written only for `vSphere` and `RHV`. They also update the `ForkliftController` precopy interval, so the test account must be able to patch `forklift-controller` in the MTV namespace. The comprehensive warm scenario is written around MTV 2.10+ features such as: - static IP preservation, - custom target VM namespace, - PVC name templates, - target labels, - target affinity. > **Note:** If you plan to run the comprehensive warm scenario, use an MTV version that already supports those features. ### Copy-offload Copy-offload tests are `vSphere`-only. They also need: - shared block storage visible from both vSphere and OpenShift, - storage credentials, - the copy-offload vendor-specific fields for your array, - an ESXi clone method: - `ssh`, with `esxi_host`, `esxi_user`, `esxi_password` - or the default `vib` path, if your ESXi environment allows the required community VIB install The copy-offload credential helper lets you supply sensitive values either in `.providers.json` or through environment variables, with environment variables taking precedence. > **Tip:** Useful copy-offload overrides include `COPYOFFLOAD_STORAGE_HOSTNAME`, `COPYOFFLOAD_STORAGE_USERNAME`, `COPYOFFLOAD_STORAGE_PASSWORD`, `COPYOFFLOAD_ESXI_HOST`, `COPYOFFLOAD_ESXI_USER`, and `COPYOFFLOAD_ESXI_PASSWORD`. For MTV versions earlier than 2.11, copy-offload must already be enabled on the `ForkliftController`: ```yaml spec: feature_copy_offload: 'true' ``` ### Remote scenarios Tests marked `remote` are skipped unless `remote_ocp_cluster` is set. In the current fixture implementation, that value is also validated against the connected cluster host string, so set it deliberately and make sure it matches your target environment naming. ### Scheduling and labeling scenarios The comprehensive cold scenario can: - label a worker node, - use `target_node_selector`, - apply `target_labels`, - apply `target_affinity`, - create or use a custom `vm_target_namespace`. Those scenarios need: - at least one worker node, - permission to patch `Node` resources, - permission to create resources in any custom target namespace. The node-selection helper prefers Prometheus metrics, but it falls back to the first worker node if monitoring access is not available. ## Permission prerequisites The easiest lab setup is `cluster-admin`. In shared environments, least-privilege RBAC is possible, but it still needs to cover the resources the suite actually creates, patches, reads, and deletes. At minimum, the account running the suite should be able to: - Create and delete `Namespace` resources. - Create, read, update, and delete `Provider`, `StorageMap`, `NetworkMap`, `Plan`, `Migration`, and `Hook` resources in the test namespace. - Create, read, and delete `Secret`, `NetworkAttachmentDefinition`, `Pod`, and `VirtualMachine` resources in the test namespace. - Read `Pod` resources in the MTV namespace so the suite can verify `forklift-*` health. - Patch `ForkliftController` in the MTV namespace for warm-migration scenarios. - Read `StorageClass` resources and patch `StorageProfile` if you use the `nfs` storage path. - Read `ClusterVersion`. - List and patch `Node` resources if you run the scheduling tests. - Create resources in any custom `vm_target_namespace` or `multus_namespace` you configure. Source-side permissions matter too. Depending on the provider and scenario, the suite may also need to: - read inventory, - clone source VMs, - power source VMs on or off, - delete cloned VMs during cleanup, - create or delete snapshots, - attach extra test disks. > **Warning:** Cleanup is best-effort, not a guarantee. If the run is interrupted, if you use `--skip-teardown`, or if the test account cannot clean up provider-side clones or snapshots, test artifacts can remain behind. ## Secure configuration handoff If you run the suite in-cluster as an OpenShift `Job`, the project already includes a working secret example that packages the core prerequisites: ```bash oc create namespace mtv-tests read -sp "Enter cluster password: " CLUSTER_PASSWORD && echo oc create secret generic mtv-test-config \ --from-file=providers.json=.providers.json \ --from-literal=cluster_host=https://api.your-cluster.com:6443 \ --from-literal=cluster_username=kubeadmin \ --from-literal=cluster_password="${CLUSTER_PASSWORD}" \ -n mtv-tests unset CLUSTER_PASSWORD ``` That example is useful even if you do not use `Job`s, because it shows the minimum configuration you need to have ready: - `.providers.json` - `cluster_host` - `cluster_username` - `cluster_password` You still need two more runtime values for an actual test run: - `source_provider` - `storage_class` If all of the prerequisites above are in place, the suite can create its migration resources, run real MTV workflows, validate the results, and clean up in the way the codebase expects. --- Source: installation-and-local-setup.md # Installation And Local Setup `mtv-api-tests` is a live `pytest` suite for Migration Toolkit for Virtualization (MTV). A real local setup needs more than a virtual environment: you also need access to an OpenShift cluster, MTV and OpenShift Virtualization installed on that cluster, and at least one source provider defined in a local `.providers.json` file. > **Warning:** A normal `pytest` run is not a mock or unit-test workflow. The suite talks to live providers and creates cluster resources such as namespaces, secrets, providers, plans, network maps, storage maps, and virtual machines. ## Python and `uv` The project metadata allows Python 3.12 through 3.13, and the project image pins Python 3.12. For the least surprising local experience, use Python 3.12 and install dependencies with `uv`. ```toml [project] requires-python = ">=3.12, <3.14" name = "mtv-api-tests" version = "2.8.3" description = "MTV API Tests" ``` The container build also makes the intended local setup clear: ```dockerfile ENV UV_PYTHON=python3.12 ENV UV_COMPILE_BYTECODE=1 ENV UV_NO_SYNC=1 ENV UV_CACHE_DIR=${APP_DIR}/.cache ``` From the repository root, install the locked environment with: ```bash uv sync --locked ``` `uv` is the source of truth here. The repository ships `pyproject.toml` and `uv.lock`; it does not use a `requirements.txt` workflow. > **Note:** If you plan to run the repository's pre-commit hooks as well, `.pre-commit-config.yaml` sets the hook interpreter to `python3.13`. The test environment itself still supports `>=3.12, <3.14`. ## Native packages on Linux If `uv sync` needs to build any dependencies locally, the container image shows the system packages the project expects on a Fedora-based system: ```dockerfile RUN dnf -y install \ libxml2-devel \ libcurl-devel \ openssl \ openssl-devel \ libcurl-devel \ gcc \ clang \ python3-devel \ ``` On other distributions, install the equivalent SSL, XML, compiler, and Python development packages. > **Tip:** You may not need all of these locally if `uv` can use prebuilt wheels, but they are the best reference for what the project image installs. ## Run from the repository root The provider loader looks for `.providers.json` as a relative path: ```python def load_source_providers() -> dict[str, dict[str, Any]]: """Load source providers from .providers.json. Returns: dict[str, dict[str, Any]]: Provider configurations keyed by provider name. """ providers_file = Path(".providers.json") ``` That means your working directory matters. > **Warning:** Run `uv` and `pytest` from the repository root. If you start from another directory, `.providers.json` can appear “missing” even when the file exists. ## How local configuration is loaded `pytest` is already wired to `pytest-testconfig`, and the default config file is part of the repo: ```ini addopts = -s -o log_cli=true -p no:logging --tc-file=tests/tests_config/config.py --tc-format=python ``` The default values in `tests/tests_config/config.py` are a starting point, not a complete local setup: ```python insecure_verify_skip: str = "true" source_provider_insecure_skip_verify: str = "false" number_of_vms: int = 1 check_vms_signals: bool = True target_namespace_prefix: str = "auto" mtv_namespace: str = "openshift-mtv" vm_name_search_pattern: str = "" remote_ocp_cluster: str = "" ``` In practice, the most important runtime values are: - `cluster_host`, `cluster_username`, and `cluster_password` for the OpenShift client - `source_provider` to select the provider entry from `.providers.json` - `storage_class` for the destination storage class - `mtv_namespace` if your MTV operator is not installed in `openshift-mtv` - `remote_ocp_cluster` only if you plan to run tests marked for remote-cluster scenarios The repository’s own invocation examples pass those values with `--tc=` overrides: ```bash uv run pytest -m copyoffload \ -v \ ${CLUSTER_HOST:+--tc=cluster_host:${CLUSTER_HOST}} \ ${CLUSTER_USERNAME:+--tc=cluster_username:${CLUSTER_USERNAME}} \ ${CLUSTER_PASSWORD:+--tc=cluster_password:${CLUSTER_PASSWORD}} \ --tc=source_provider:vsphere-8.0.3.00400 \ --tc=storage_class:my-block-storageclass ``` For local shell usage, pass the same `--tc=` keys directly or expand them from your shell environment. > **Tip:** The example above avoids typing the cluster password literally on the command line. Reusing that pattern is a good idea for local runs too. > **Note:** A `.env` file is only auto-loaded for the optional `--analyze-with-ai` path. Standard cluster and provider configuration still comes from `.providers.json` and `--tc=` values. ## Create `.providers.json` The suite expects a file named `.providers.json` in the repository root. Start from `.providers.json.example`, then replace the placeholders with real values. A typical vSphere entry looks like this in the example file: ```jsonc "vsphere": { "type": "vsphere", "version": "", "fqdn": "SERVER FQDN/IP", "api_url": "/sdk", "username": "USERNAME", "password": "PASSWORD", # pragma: allowlist secret "guest_vm_linux_user": "LINUX VMS USERNAME", "guest_vm_linux_password": "LINUX VMS PASSWORD", # pragma: allowlist secret "guest_vm_win_user": "WINDOWS VMS USERNAME", "guest_vm_win_password": "WINDOWS VMS PASSWORD", # pragma: allowlist secret "vddk_init_image": "" }, ``` The example file also includes provider templates for `ovirt`, `openstack`, `openshift`, and `ova`. A few important details matter here: - The top-level key is what `source_provider` selects. If you run with `--tc=source_provider:vsphere-copy-offload`, your `.providers.json` file must contain a top-level key with that exact name. - Guest OS credentials are not optional decoration. Post-migration checks read `guest_vm_linux_user` and `guest_vm_linux_password`, or the Windows equivalents, from the provider config. - The OpenShift provider example intentionally leaves connection fields blank. When the provider type is `openshift`, the code reuses the current cluster connection and cluster secret instead of building a completely separate provider secret. > **Note:** `.providers.json.example` contains comments such as `# pragma: allowlist secret`. Those comments are useful in the example file, but they are not valid JSON. Your real `.providers.json` must be valid JSON. > **Tip:** `.providers.json` is already listed in `.gitignore`, so keep real credentials there instead of checking them into the repository. ### Copy-offload overrides If you plan to run copy-offload tests, the helper code checks environment variables before reading the `copyoffload` block from `.providers.json`: ```python env_var_name = f"COPYOFFLOAD_{credential_name.upper()}" return os.getenv(env_var_name) or copyoffload_config.get(credential_name) ``` That means variables such as these override file values when present: - `COPYOFFLOAD_STORAGE_HOSTNAME` - `COPYOFFLOAD_STORAGE_USERNAME` - `COPYOFFLOAD_STORAGE_PASSWORD` - `COPYOFFLOAD_ESXI_HOST` - `COPYOFFLOAD_ESXI_USER` - `COPYOFFLOAD_ESXI_PASSWORD` > **Tip:** Using `COPYOFFLOAD_*` environment variables is a good way to keep storage-array and ESXi credentials out of `.providers.json`. ## Prepare the cluster and source VMs A working Python environment is only half of the setup. Before a real test run, your lab should also be ready: - MTV must already be installed, and its `forklift-*` pods must be running in the namespace configured by `mtv_namespace` (default: `openshift-mtv`) - Your OpenShift user must be able to create and clean up the resources the suite manages - Your source provider must actually contain VMs or templates that match the names used by the selected test plans - If a test expects `guest_agent: True`, the source VM should have a working guest agent The test plans are data-driven. For example, one of the built-in sanity plans looks like this: ```python tests_params: dict = { "test_sanity_warm_mtv_migration": { "virtual_machines": [ { "name": "mtv-tests-rhel8", "source_vm_power": "on", "guest_agent": True, }, ], "warm_migration": True, }, ``` That snippet tells you exactly what the lab needs for that scenario: - A source VM named `mtv-tests-rhel8` - The ability to power it on before migration - A guest agent available inside the VM More advanced plans in `tests/tests_config/config.py` add things like custom VM target namespaces, Multus networks, node selectors, labels, and copy-offload requirements. > **Tip:** For a fresh lab, start by aligning your environment with the simple sanity plans before trying comprehensive or copy-offload scenarios. > **Note:** Tests marked for remote-cluster scenarios are designed to use `remote_ocp_cluster`. Leave that value empty unless you actually have a remote-cluster setup. ## Validate the installation safely Before launching a real migration run, it is a good idea to do a dry validation first. `tox.toml` uses these commands for its lightweight pytest checks: ```bash uv run pytest --setup-plan uv run pytest --collect-only ``` These are useful first checks after `uv sync --locked` because they validate importability and test collection without running the live migrations themselves. > **Warning:** `uv run pytest` without a dry-run flag is a live infrastructure run. ## How `virtctl` is discovered or downloaded Most users do not need to install `virtctl` manually. The session-scoped setup code makes sure it is available. The first step is to reuse an existing binary if one is already present: ```python # Check if already available existing = _check_existing_virtctl(download_dir) if existing: add_to_path(str(existing.parent)) return existing LOGGER.info("virtctl not found, downloading from cluster...") # Get ConsoleCLIDownload resource console_cli_download = ConsoleCLIDownload( client=client, name="virtctl-clidownloads-kubevirt-hyperconverged", ensure_exists=True, ) ``` `virtctl` is needed because VM SSH access is implemented through `virtctl port-forward`: ```python cmd = [ virtctl_path, "port-forward", f"vm/{self.vm.name}", f"{local_port}:22", "--namespace", self.vm.namespace, "--address", "127.0.0.1", ] ``` In practice, the `virtctl` flow works like this: - If `virtctl` is already on `PATH`, the suite reuses it - If not, it checks a shared cache under the system temp directory - If there is no cached binary, it reads the cluster `ConsoleCLIDownload` named `virtctl-clidownloads-kubevirt-hyperconverged` - It picks the download URL that matches the local host OS and architecture - It downloads the archive, extracts the `virtctl` binary, makes it executable, and prepends its directory to `PATH` The auto-download logic currently covers: - Linux and macOS hosts - `x86_64`, `aarch64`, and `arm64` architectures The session fixture also caches `virtctl` by cluster version and guards the download with a file lock so parallel `pytest-xdist` workers do not all fetch the same binary at once. > **Warning:** Windows hosts are not covered by the current `virtctl` auto-detection logic. The downloader only maps `linux` and `darwin`. > **Tip:** If you want to force a fresh `virtctl` download, remove the cached `pytest-shared-virtctl/` directory under your system temp directory, or place a different `virtctl` earlier in `PATH`. ## Optional but useful local tools `oc` is not how the suite discovers `virtctl`, and it is not required just to create the Python environment. It is still a good tool to have locally because the default failure-data path can run `oc adm must-gather` when tests fail. > **Tip:** If you do not want automatic failure data collection, start pytest with `--skip-data-collector`. Otherwise the default behavior may try to collect `.data-collector` output and run must-gather on failures. --- Source: quickstart-first-run.md # Quickstart First Run `mtv-api-tests` uses `pytest` with `pytest-testconfig`. The shared test catalog is already wired into the repo, so a first run is mostly about three things: 1. Create `.providers.json` in the repository root. 2. Pass the environment-specific runtime values with `--tc=...`. 3. Start with `--collect-only` or `--setup-plan`, then run one small sanity class. This page assumes you are running from the repository root with project dependencies already available. ## How Configuration Works `pytest.ini` already tells pytest to load `tests/tests_config/config.py` through `pytest-testconfig`: ```ini [pytest] testpaths = tests addopts = -s -o log_cli=true -p no:logging --tc-file=tests/tests_config/config.py --tc-format=python --junit-xml=junit-report.xml --basetemp=/tmp/pytest --show-progress --strict-markers --jira --dist=loadscope ``` For real test execution, the suite requires two runtime keys: ```python def pytest_sessionstart(session): required_config = ("storage_class", "source_provider") ``` > **Note:** You normally do not need to pass `--tc-file` yourself. The repo already points pytest at `tests/tests_config/config.py`, so the usual workflow is to add runtime overrides with `--tc=key:value`. ## Create `.providers.json` The provider loader looks for `.providers.json` in the current working directory and parses it as JSON: ```python def load_source_providers() -> dict[str, dict[str, Any]]: providers_file = Path(".providers.json") if not providers_file.exists(): return {} with open(providers_file) as fd: content = fd.read() if not content.strip(): return {} return json.loads(content) ``` Create `.providers.json` in the repository root. Use `.providers.json.example` as the starting template. This is the vSphere block from that file: ```jsonc "vsphere": { "type": "vsphere", "version": "", "fqdn": "SERVER FQDN/IP", "api_url": "/sdk", "username": "USERNAME", "password": "PASSWORD", # pragma: allowlist secret "guest_vm_linux_user": "LINUX VMS USERNAME", "guest_vm_linux_password": "LINUX VMS PASSWORD", # pragma: allowlist secret "guest_vm_win_user": "WINDOWS VMS USERNAME", "guest_vm_win_password": "WINDOWS VMS PASSWORD", # pragma: allowlist secret "vddk_init_image": "" } ``` The same example file also includes templates for `ovirt`, `openstack`, `openshift`, and `ova`. > **Warning:** `.providers.json.example` is not valid JSON as-is. It contains comments and placeholder values. Your real `.providers.json` must be valid JSON, because the loader uses `json.loads(...)`. > **Note:** The top-level provider key is what you pass later as `--tc=source_provider:`. If your file uses `"vsphere"`, then your runtime flag must use `--tc=source_provider:vsphere`. > **Warning:** `.providers.json` contains credentials. Keep it local and treat it as sensitive. ## Know What the Built-In Plans Expect The shared defaults and named test plans live in `tests/tests_config/config.py`. These are the defaults most relevant to a first run: ```python insecure_verify_skip: str = "true" source_provider_insecure_skip_verify: str = "false" target_namespace_prefix: str = "auto" mtv_namespace: str = "openshift-mtv" remote_ocp_cluster: str = "" plan_wait_timeout: int = 3600 ``` For a first targeted migration, the smallest built-in cold plan is: ```python "test_sanity_cold_mtv_migration": { "virtual_machines": [ {"name": "mtv-tests-rhel8", "guest_agent": True}, ], "warm_migration": False, }, ``` The warm equivalent is: ```python "test_sanity_warm_mtv_migration": { "virtual_machines": [ { "name": "mtv-tests-rhel8", "source_vm_power": "on", "guest_agent": True, }, ], "warm_migration": True, }, ``` > **Tip:** If you want the smoothest first run, prepare a source VM named `mtv-tests-rhel8` and start with the cold sanity plan. If your lab uses different VM names, update the matching entry in `tests/tests_config/config.py` before your first real run. ## Pass Runtime Settings With `pytest-testconfig` For a first run, these are the settings you will usually care about: - `source_provider`: required for real execution; must match a top-level key in `.providers.json` - `storage_class`: required for real execution; destination OpenShift storage class - `cluster_host`, `cluster_username`, `cluster_password`: useful when you want to pass OpenShift access explicitly on the command line - `mtv_namespace`: optional; defaults to `openshift-mtv` - `target_namespace_prefix`: optional; defaults to `auto` - `insecure_verify_skip`: optional; controls OpenShift API TLS verification - `source_provider_insecure_skip_verify`: optional; controls source-provider TLS verification A typical first-run command shape is: ```bash uv run pytest -v tests/test_mtv_cold_migration.py::TestSanityColdMtvMigration \ --tc=source_provider:vsphere \ --tc=storage_class: \ --tc=cluster_host:https://api.:6443 \ --tc=cluster_username: \ --tc=cluster_password:${CLUSTER_PASSWORD} ``` If your provider key is not `vsphere`, replace it with the top-level key you actually used in `.providers.json`. If you need to relax source-provider TLS verification in a lab environment, add: ```bash --tc=source_provider_insecure_skip_verify:true ``` If your MTV operator is not installed in `openshift-mtv`, add: ```bash --tc=mtv_namespace: ``` ## Start With `--collect-only` Or `--setup-plan` This repo treats both `--collect-only` and `--setup-plan` as normal dry-run entry points. In fact, `tox.toml` uses both as a basic pytest check, and the container image in `Dockerfile` defaults to `uv run pytest --collect-only`. Use `--collect-only` when you want to confirm what pytest will select: ```bash uv run pytest --collect-only -q tests/test_mtv_cold_migration.py::TestSanityColdMtvMigration ``` Use `--setup-plan` when you want to see fixture/setup planning for the same target before a real run: ```bash uv run pytest --setup-plan tests/test_mtv_cold_migration.py::TestSanityColdMtvMigration \ --tc=source_provider:vsphere \ --tc=storage_class: ``` If you want to browse by marker first, the built-in markers are: - `tier0` - `warm` - `remote` - `copyoffload` For example, to list the smoke-suite tests: ```bash uv run pytest --collect-only -q -m tier0 ``` > **Tip:** `--collect-only` is the safest place to start when you want to confirm test names, markers, and node IDs before wiring in all runtime values. ## Run The First Targeted Test Class The cold sanity test is defined in `tests/test_mtv_cold_migration.py` like this: ```python @pytest.mark.tier0 @pytest.mark.incremental @pytest.mark.parametrize( "class_plan_config", [ pytest.param( py_config["tests_params"]["test_sanity_cold_mtv_migration"], ) ], indirect=True, ids=["rhel8"], ) @pytest.mark.usefixtures("cleanup_migrated_vms") class TestSanityColdMtvMigration: """Cold migration test - sanity check.""" ``` That class runs the migration as five dependent steps: - `test_create_storagemap` - `test_create_networkmap` - `test_create_plan` - `test_migrate_vms` - `test_check_vms` Run the whole class, not a single method: ```bash uv run pytest -v tests/test_mtv_cold_migration.py::TestSanityColdMtvMigration \ --tc=source_provider:vsphere \ --tc=storage_class: \ --tc=cluster_host:https://api.:6443 \ --tc=cluster_username: \ --tc=cluster_password:${CLUSTER_PASSWORD} ``` If you specifically want to try the warm sanity path afterward, use: ```bash uv run pytest -v tests/test_mtv_warm_migration.py::TestSanityWarmMtvMigration \ --tc=source_provider:vsphere \ --tc=storage_class: \ --tc=cluster_host:https://api.:6443 \ --tc=cluster_username: \ --tc=cluster_password:${CLUSTER_PASSWORD} ``` > **Warning:** Warm tests are explicitly skipped in this repo for `openstack`, `openshift`, and `ova` providers. For a first run, the cold sanity class is the safer starting point. ## Common First-Run Problems If the run fails early, check these first: - `.providers.json` is missing or empty. - The value passed in `--tc=source_provider:...` does not exactly match a top-level key in `.providers.json`. - `.providers.json` was copied from `.providers.json.example` without removing comments. - The built-in sanity plan expects a source VM named `mtv-tests-rhel8`, but that VM does not exist in your provider. - You targeted a single method such as `::test_create_plan` instead of the full class. - You chose a warm test on a provider type that the repo skips for warm migration. > **Tip:** By default, the suite writes `junit-report.xml` and tears down created resources after the run. If you need to inspect what was created after a failure, rerun with `--skip-teardown`. --- Source: container-and-openshift-job-execution.md # Container And OpenShift Job Execution Running `mtv-api-tests` from a container or an OpenShift `Job` gives you a repeatable environment for long migration runs, shared execution in CI-style workflows, and predictable artifact collection. The key is understanding what the image expects at runtime: - a `.providers.json` file in the container working directory - pytest testconfig values for the OpenShift cluster and test selection - an overridden container command, because the image defaults to collection only ## How The Image Starts The checked-in image is built to run from `/app`, and its default command only collects tests: ```dockerfile ARG APP_DIR=/app WORKDIR ${APP_DIR} CMD ["uv", "run", "pytest", "--collect-only"] ``` Pytest is also preconfigured in the repo: ```ini [pytest] addopts = -s -o log_cli=true -p no:logging --tc-file=tests/tests_config/config.py --tc-format=python --junit-xml=junit-report.xml --basetemp=/tmp/pytest --show-progress --strict-markers --jira --dist=loadscope ``` > **Note:** A plain `podman run ... ghcr.io/redhatqe/mtv-api-tests:latest` will not execute migrations. Override the command with `uv run pytest ...`. The suite reads provider definitions from `.providers.json` and builds the OpenShift client from pytest config values: ```python def load_source_providers() -> dict[str, dict[str, Any]]: providers_file = Path(".providers.json") if not providers_file.exists(): return {} def get_cluster_client() -> DynamicClient: host = get_value_from_py_config("cluster_host") username = get_value_from_py_config("cluster_username") password = get_value_from_py_config("cluster_password") insecure_verify_skip = get_value_from_py_config("insecure_verify_skip") _client = get_client(host=host, username=username, password=password, verify_ssl=not insecure_verify_skip) ``` In practice, every real run needs these values: - `.providers.json` - `source_provider` - `storage_class` - `cluster_host` - `cluster_username` - `cluster_password` > **Warning:** The suite does not currently create its OpenShift client from the pod service account. Even inside an OpenShift `Job`, you still need to provide `cluster_host`, `cluster_username`, and `cluster_password`. ## Provider And Test Config ### `.providers.json` The provider file lives at `.providers.json` in the working directory, so inside the image the simplest path is `/app/.providers.json`. A real example from `.providers.json.example`: ```json { "vsphere": { "type": "vsphere", "version": "", "fqdn": "SERVER FQDN/IP", "api_url": "/sdk", "username": "USERNAME", "password": "PASSWORD", "guest_vm_linux_user": "LINUX VMS USERNAME", "guest_vm_linux_password": "LINUX VMS PASSWORD", "guest_vm_win_user": "WINDOWS VMS USERNAME", "guest_vm_win_password": "WINDOWS VMS PASSWORD", "vddk_init_image": "" } } ``` The top-level key is the provider name you pass through pytest. If your file uses `"vsphere"`, then your run must include `--tc=source_provider:vsphere`. > **Tip:** The top-level key can be descriptive, versioned, or site-specific. What matters is that `source_provider` matches it exactly. ### `tests/tests_config/config.py` The repo already ships a Python testconfig file with global defaults and `tests_params` used by the test modules: ```python insecure_verify_skip: str = "true" source_provider_insecure_skip_verify: str = "false" target_namespace_prefix: str = "auto" mtv_namespace: str = "openshift-mtv" plan_wait_timeout: int = 3600 tests_params: dict = { "test_sanity_warm_mtv_migration": { "virtual_machines": [ { "name": "mtv-tests-rhel8", "source_vm_power": "on", "guest_agent": True, }, ], "warm_migration": True, }, ``` Most users should keep the checked-in `config.py` and supply runtime values with `--tc=...`. If you do mount your own replacement config file, start from `tests/tests_config/config.py` instead of creating an empty file. The tests rely on `tests_params` being present. ## Running From The Container Image Examples below use `podman`, but the same pattern works with Docker. A practical local run mounts `.providers.json`, exports cluster credentials into the container, and overrides the image command: ```bash export CLUSTER_HOST="https://api.example.com:6443" export CLUSTER_USERNAME="kubeadmin" export CLUSTER_PASSWORD="" podman run --rm \ -e CLUSTER_HOST \ -e CLUSTER_USERNAME \ -e CLUSTER_PASSWORD \ -v "$(pwd)/.providers.json:/app/.providers.json:ro" \ -v "$(pwd)/results:/app/results" \ ghcr.io/redhatqe/mtv-api-tests:latest \ /bin/bash -c 'uv run pytest -m tier0 \ -v \ ${CLUSTER_HOST:+--tc=cluster_host:${CLUSTER_HOST}} \ ${CLUSTER_USERNAME:+--tc=cluster_username:${CLUSTER_USERNAME}} \ ${CLUSTER_PASSWORD:+--tc=cluster_password:${CLUSTER_PASSWORD}} \ --tc=source_provider:vsphere \ --tc=storage_class:my-block-storageclass \ --junit-xml=/app/results/junit-report.xml \ --log-file=/app/results/pytest-tests.log \ --data-collector-path=/app/results/data' ``` Useful marker selections come directly from `pytest.ini` and the test modules: - `-m tier0` for smoke-style coverage - `-m warm` for warm migration coverage - `-m copyoffload` for copy-offload coverage - `-m remote` for remote-cluster scenarios when `remote_ocp_cluster` is configured > **Note:** `warm` is not universally supported for every provider type. If you select `-m warm` against a provider that the warm test module skips, pytest will report skips rather than failures. If you prefer a mounted config file instead of many `--tc=` flags, mount your own Python config and point pytest at it: ```bash uv run pytest --tc-file=/app/config.py --tc-format=python ... ``` That is only safe when `/app/config.py` is based on the repo’s existing `tests/tests_config/config.py`. ## Running Inside An OpenShift Job The repo already contains a checked-in `Job` pattern under `docs/copyoffload/`. It is written for copy-offload, but the wiring is the same for any suite: mount `.providers.json`, provide cluster credentials from a `Secret`, and expand them into `--tc=` arguments in the container command. ### 1. Create A Secret ```bash oc create namespace mtv-tests read -sp "Enter cluster password: " CLUSTER_PASSWORD && echo oc create secret generic mtv-test-config \ --from-file=providers.json=.providers.json \ --from-literal=cluster_host=https://api.your-cluster.com:6443 \ --from-literal=cluster_username=kubeadmin \ --from-literal=cluster_password="${CLUSTER_PASSWORD}" \ -n mtv-tests unset CLUSTER_PASSWORD ``` ### 2. Create The Job This example is copied from the checked-in documentation: ```yaml apiVersion: batch/v1 kind: Job metadata: name: mtv-copyoffload-tests namespace: mtv-tests spec: template: spec: restartPolicy: Never containers: - name: tests image: ghcr.io/redhatqe/mtv-api-tests:latest env: - name: CLUSTER_HOST valueFrom: secretKeyRef: name: mtv-test-config key: cluster_host optional: true - name: CLUSTER_USERNAME valueFrom: secretKeyRef: name: mtv-test-config key: cluster_username optional: true - name: CLUSTER_PASSWORD valueFrom: secretKeyRef: name: mtv-test-config key: cluster_password optional: true command: - /bin/bash - -c - | uv run pytest -m copyoffload \ -v \ ${CLUSTER_HOST:+--tc=cluster_host:${CLUSTER_HOST}} \ ${CLUSTER_USERNAME:+--tc=cluster_username:${CLUSTER_USERNAME}} \ ${CLUSTER_PASSWORD:+--tc=cluster_password:${CLUSTER_PASSWORD}} \ --tc=source_provider:vsphere-8.0.3.00400 \ --tc=storage_class:my-block-storageclass volumeMounts: - name: config mountPath: /app/.providers.json subPath: providers.json volumes: - name: config secret: secretName: mtv-test-config ``` To reuse this pattern for other suites: - change `-m copyoffload` to `-m tier0` for smoke tests - change `-m copyoffload` to `-m warm` for warm migration runs - replace `vsphere-8.0.3.00400` with your actual `.providers.json` key - replace `my-block-storageclass` with the storage class used by your target cluster - replace the image if you built and pushed your own copy > **Warning:** `CLUSTER_HOST`, `CLUSTER_USERNAME`, and `CLUSTER_PASSWORD` are not read directly by the test code. In this pattern they exist only so the shell can expand them into `--tc=` options. ## Copy-Offload Credentials Copy-offload is the one area where the suite can read credentials directly from environment variables as well as from `.providers.json`: ```python def get_copyoffload_credential( credential_name: str, copyoffload_config: dict[str, Any], ) -> str | None: env_var_name = f"COPYOFFLOAD_{credential_name.upper()}" return os.getenv(env_var_name) or copyoffload_config.get(credential_name) ``` That means a `Job` can inject copy-offload storage credentials from a `Secret` without hard-coding them in `.providers.json`. Common environment variable names come straight from the fixtures: - `COPYOFFLOAD_STORAGE_HOSTNAME` - `COPYOFFLOAD_STORAGE_USERNAME` - `COPYOFFLOAD_STORAGE_PASSWORD` - vendor-specific names such as `COPYOFFLOAD_ONTAP_SVM` - ESXi SSH values such as `COPYOFFLOAD_ESXI_HOST`, `COPYOFFLOAD_ESXI_USER`, and `COPYOFFLOAD_ESXI_PASSWORD` > **Note:** This environment-variable fallback applies to copy-offload storage credentials. It does not replace the normal `cluster_host`, `cluster_username`, `cluster_password`, or `source_provider` inputs. ## Collecting Results By default, a run produces these useful artifacts inside `/app`: - `junit-report.xml` - `pytest-tests.log` - `.data-collector/` The data collector path is configurable, and the suite uses it for resource tracking and failure collection: - default path: `.data-collector` - change it with `--data-collector-path=/some/path` - disable it with `--skip-data-collector` When data collection is enabled, the suite writes `resources.json` there and can also collect must-gather data on failures. For a finished OpenShift `Job`, the checked-in docs already show how to stream logs and copy the JUnit file out of the pod: ```bash oc logs -n mtv-tests job/mtv-copyoffload-tests -f POD_NAME=$(oc get pods -n mtv-tests -l job-name=mtv-copyoffload-tests -o jsonpath='{.items[0].metadata.name}') oc cp mtv-tests/$POD_NAME:/app/junit-report.xml ./junit-report.xml ``` You can collect the other artifacts the same way: ```bash oc cp mtv-tests/$POD_NAME:/app/pytest-tests.log ./pytest-tests.log oc cp mtv-tests/$POD_NAME:/app/.data-collector ./data-collector ``` For container runs, the easiest pattern is to mount a results directory and redirect outputs into it with: - `--junit-xml=/app/results/junit-report.xml` - `--log-file=/app/results/pytest-tests.log` - `--data-collector-path=/app/results/data` > **Tip:** For long-running OpenShift Jobs, mount a PVC and write all artifacts to that volume. That avoids depending on the pod filesystem after completion. ## Cleanup And Debugging The suite cleans up resources automatically unless you opt out. That includes created MTV resources and tracked VMs. If you want to keep resources around for investigation, add: ```bash --skip-teardown ``` If you skip teardown, keep the data collector output. The repo includes a cleanup helper that can use the saved `resources.json` file: ```bash uv run tools/clean_cluster.py .data-collector/resources.json ``` `resources.json` is only created when data collection is enabled. > **Warning:** If you pass `--skip-data-collector`, the suite will not write `.data-collector/resources.json`, and failed runs will not gather the extra collector output. > **Note:** The completed `Job` pod remains available until you delete the `Job`, so you can still fetch logs and artifacts after the test run finishes. --- Source: runtime-configuration.md # Runtime Configuration `mtv-api-tests` uses `pytest-testconfig` for suite settings and plain pytest flags for per-run behavior. In practice, you keep shared defaults in `tests/tests_config/config.py`, keep provider definitions in `.providers.json`, and pass environment-specific values such as `source_provider` and `storage_class` with `--tc=key:value`. Most users only need to remember three things: 1. `pytest.ini` already loads the default runtime config file for you. 2. A normal test run requires `source_provider` and `storage_class`. 3. The repo adds several custom pytest flags for cleanup, artifact collection, logging, and AI analysis. ## How Configuration Is Loaded The baseline pytest behavior is defined in `pytest.ini`: ```ini [pytest] testpaths = tests addopts = -s -o log_cli=true -p no:logging --tc-file=tests/tests_config/config.py --tc-format=python --junit-xml=junit-report.xml --basetemp=/tmp/pytest --show-progress --strict-markers --jira --dist=loadscope ``` What this means in day-to-day use: - Tests are collected from `tests/`. - `pytest-testconfig` automatically loads `tests/tests_config/config.py`. - JUnit XML is written by default to `junit-report.xml`. - The suite enables strict marker validation and xdist `loadscope` distribution. - Pytest's built-in logging plugin is disabled, and the suite configures its own logger. > **Note:** The default `addopts` also enable `--jira`. If you use JIRA-linked tests, `jira.cfg.example` shows the expected file format. ## Required Runtime Overrides For a normal run, the suite expects two runtime values even though they are not defined in the default config file: | Key | Required | Purpose | |---|---|---| | `source_provider` | Yes | Selects the source provider entry from `.providers.json` | | `storage_class` | Yes | Sets the target OpenShift storage class for migration | | `cluster_host` | Optional | Passed into cluster client creation when supplied | | `cluster_username` | Optional | Passed into cluster client creation when supplied | | `cluster_password` | Optional | Passed into cluster client creation when supplied | | `target_ocp_version` | Optional | Used only for generated VM suffix naming | The repository's own copy-offload job example uses `--tc=` overrides like this: ```bash uv run pytest -m copyoffload \ -v \ ${CLUSTER_HOST:+--tc=cluster_host:${CLUSTER_HOST}} \ ${CLUSTER_USERNAME:+--tc=cluster_username:${CLUSTER_USERNAME}} \ ${CLUSTER_PASSWORD:+--tc=cluster_password:${CLUSTER_PASSWORD}} \ --tc=source_provider:vsphere-8.0.3.00400 \ --tc=storage_class:my-block-storageclass ``` > **Warning:** `source_provider` is not the provider type. It must match a key in `.providers.json` exactly. The provider definitions themselves are loaded from `.providers.json` in the repository root. If that file is missing, the suite cannot resolve the requested provider. ## Global `pytest-testconfig` Values The default shared values live at the top of `tests/tests_config/config.py`: ```python global config insecure_verify_skip: str = "true" # SSL verification for OCP API connections source_provider_insecure_skip_verify: str = "false" # SSL verification for source provider (VMware, RHV, etc.) number_of_vms: int = 1 check_vms_signals: bool = True target_namespace_prefix: str = "auto" mtv_namespace: str = "openshift-mtv" vm_name_search_pattern: str = "" remote_ocp_cluster: str = "" snapshots_interval: int = 2 mins_before_cutover: int = 5 plan_wait_timeout: int = 3600 ``` ### Actively used global values | Key | Default | What it controls | |---|---|---| | `insecure_verify_skip` | `"true"` | OpenShift API SSL verification. The cluster client is created with `verify_ssl=not insecure_verify_skip`. | | `source_provider_insecure_skip_verify` | `"false"` | Source-provider SSL verification and the Provider secret's `insecureSkipVerify` value. | | `target_namespace_prefix` | `"auto"` | Base text used when generating the target namespace name for migrated resources. | | `mtv_namespace` | `"openshift-mtv"` | Namespace used for MTV resources, pod health checks, and must-gather collection. | | `remote_ocp_cluster` | `""` | Enables tests marked `remote` when set and validates the current cluster host against that value. | | `snapshots_interval` | `2` | Updates the `forklift-controller` precopy interval for warm migration tests. | | `mins_before_cutover` | `5` | Number of minutes added when calculating warm-migration cutover time. | | `plan_wait_timeout` | `3600` | Timeout used while waiting for migration plans to complete. | ### Defined in the default file but not currently used elsewhere A repository-wide search shows these keys are defined in `tests/tests_config/config.py` but are not referenced by the rest of the codebase today: | Key | Default | |---|---| | `number_of_vms` | `1` | | `check_vms_signals` | `True` | | `vm_name_search_pattern` | `""` | > **Warning:** For boolean-style CLI overrides such as `insecure_verify_skip` and `source_provider_insecure_skip_verify`, use lowercase string values like `true` and `false`. The default config file stores them as strings, and some code paths handle them that way. ## Named Test Plans In `tests_params` The same config file also contains `tests_params`, which is the catalog of named migration scenarios used by the test classes. These are not suite-wide defaults; they are per-scenario plan definitions. A real example from `tests/tests_config/config.py`: ```python "test_warm_migration_comprehensive": { "virtual_machines": [ { "name": "mtv-win2022-ip-3disks", "source_vm_power": "on", "guest_agent": True, }, ], "warm_migration": True, "target_power_state": "on", "preserve_static_ips": True, "vm_target_namespace": "custom-vm-namespace", "multus_namespace": "default", # Cross-namespace NAD access "pvc_name_template": '{{ .FileName | trimSuffix ".vmdk" | replace "_" "-" }}-{{.DiskIndex}}', "pvc_name_template_use_generate_name": True, "target_labels": { "mtv-comprehensive-test": None, # None = auto-generate with session_uuid "static-label": "static-value", }, "target_affinity": { "podAffinity": { "preferredDuringSchedulingIgnoredDuringExecution": [ { "podAffinityTerm": { "labelSelector": {"matchLabels": {"app": "comprehensive-test"}}, "topologyKey": "kubernetes.io/hostname", }, "weight": 75, } ] } }, }, ``` Those named plans are referenced directly by the tests. For example: ```python @pytest.mark.parametrize( "class_plan_config", [ pytest.param( py_config["tests_params"]["test_warm_migration_comprehensive"], ) ], indirect=True, ids=["comprehensive-warm"], ) ``` Common `tests_params` keys you will see in this repository include: - `virtual_machines` for the VM list and per-VM options such as `name`, `source_vm_power`, `guest_agent`, `clone`, and `disk_type`. - `warm_migration` and `copyoffload` for the main migration mode. - `target_power_state`, `preserve_static_ips`, `vm_target_namespace`, `multus_namespace`, `pvc_name_template`, `target_labels`, and `target_affinity` for plan behavior. - `pre_hook`, `post_hook`, and `expected_migration_result` for hook-driven scenarios. - `guest_agent_timeout` for scenarios that need a longer wait after migration. ## Custom Pytest Options The repository adds four user-facing runtime features on top of standard pytest options: artifact collection, teardown control, extra logging, and AI analysis. ### Data collection By default, the suite collects runtime artifacts for failed runs. - The base directory defaults to `.data-collector`. - On test failure, the suite attempts to run `oc adm must-gather` into a per-test subdirectory under that path. - At session finish, it writes a `resources.json` file with tracked resources. - If session teardown fails, it attempts an additional must-gather into the base collector directory. Use these flags to control that behavior: - `--skip-data-collector` disables artifact collection. - `--data-collector-path ` changes the output directory. > **Warning:** The suite deletes and recreates the base data-collector directory at session start. Use a dedicated path if you want to preserve older artifacts. > **Note:** The current help text for `--skip-data-collector` is misleading. In actual runtime behavior, the flag disables data collection. > **Tip:** When `resources.json` is available, the repository includes `tools/clean_cluster.py` to clean resources from that file. ### Teardown control By default, the suite cleans up the resources it created. That includes: - Session-level tracked resources such as plans, providers, namespaces, and migrated resources. - Class-level cleanup of migrated VMs through the `cleanup_migrated_vms` fixture. Use `--skip-teardown` when you want to keep resources around for debugging. The repository's own documentation shows it this way: ```bash uv run pytest -m copyoffload --skip-teardown \ -v \ ... ``` > **Warning:** `--skip-teardown` is for investigation and debugging. If you use it, you are responsible for cleaning up leftover resources afterward. ### Debug logging Logging is already enabled by default through `pytest.ini`: - `-s` keeps stdout/stderr visible. - `-o log_cli=true` enables live console logging. - The suite writes logs to `pytest-tests.log` unless you set a different `--log-file`. - The log level comes from pytest's `log_cli_level` option and falls back to `INFO`. There is also one custom debug flag: - `--openshift-python-wrapper-log-debug` sets `OPENSHIFT_PYTHON_WRAPPER_LOG_LEVEL=DEBUG`. In other words, the main knobs for troubleshooting are: - Standard pytest logging options such as `log_cli_level` and `log_file`. - The custom `--openshift-python-wrapper-log-debug` flag for wrapper internals. > **Tip:** For deeper troubleshooting, raise `log_cli_level`, write to a dedicated `--log-file`, and add `--openshift-python-wrapper-log-debug`. ### AI failure analysis The suite can enrich failed JUnit XML reports with AI-generated analysis. Enable it with: - `--analyze-with-ai` When enabled, the code does the following: - Calls `load_dotenv()`, so a local `.env` file can supply the settings. - Checks `JJI_SERVER_URL`. - Uses default values for provider and model if you did not set them. - After a failed run, reads the JUnit XML file and posts the raw XML to `${JJI_SERVER_URL}/analyze-failures`. - If enrichment succeeds, writes the enriched XML back to the same file. Environment variables used by this feature: | Variable | Required | Default | Purpose | |---|---|---|---| | `JJI_SERVER_URL` | Yes | None | URL of the analysis service | | `JJI_AI_PROVIDER` | No | `claude` | AI provider name sent to the service | | `JJI_AI_MODEL` | No | `claude-opus-4-6[1m]` | AI model name sent to the service | | `JJI_TIMEOUT` | No | `600` | Request timeout in seconds | Important behavior to know: - Successful runs skip AI enrichment. - `--collect-only` and `--setup-plan` disable AI analysis automatically. - If the JUnit XML file is missing, enrichment is skipped. - The default JUnit XML path is already configured as `junit-report.xml` in `pytest.ini`. > **Warning:** `--analyze-with-ai` sends the raw JUnit XML content to an external HTTP service. Review what your report contains before enabling this in shared or external environments. > **Note:** If enrichment succeeds, the original JUnit XML file is overwritten in place with the enriched version. ## Dry-Run Modes This repository treats `--collect-only` and `--setup-plan` as dry-run modes. That behavior is visible in `tox.toml`: ```toml [env.pytest-check] commands = [ [ "uv", "run", "pytest", "--setup-plan", ], [ "uv", "run", "pytest", "--collect-only", ], ] ``` And the container image defaults to collection-only mode: ```dockerfile CMD ["uv", "run", "pytest", "--collect-only"] ``` In dry-run mode: - Required runtime checks for `source_provider` and `storage_class` are skipped. - AI analysis is disabled. - Session-finish teardown and JUnit enrichment do not run. > **Tip:** If you start the published container image without overriding its command, it only performs test collection. To run real tests, supply your own `uv run pytest ...` command. --- Source: provider-config-file.md # Provider Config File `mtv-api-tests` does not ship a separate JSON Schema document for provider settings. The effective schema comes from `.providers.json.example` and the Python code that loads `./.providers.json`. In practice, that means the file is flexible, but specific fields become required when a provider path or validation step reads them. > **Warning:** `.providers.json.example` is an annotated template, not a ready-to-use `.providers.json`. The loader parses `.providers.json` with `json.loads(...)`, so your real file must be strict JSON: remove comments, keep valid quoting, and avoid trailing commas. > **Warning:** `.providers.json` usually contains provider passwords and guest OS passwords. Treat it as a secret file. > **Note:** The top-level key is the provider name you select with `source_provider`. It does not have to match `type`. For example, the example file has a key named `vsphere-copy-offload`, but its `"type"` is still `"vsphere"`. ## How the file is used - The test harness looks for `./.providers.json` in the directory where you run the tests. - The file can contain multiple provider entries in one JSON object. - `source_provider` must match one of the top-level keys in that file. - A missing or empty file fails fast. - `version` is used mainly in generated test resource names. It is not the field that decides how to connect. - There is no strict field whitelist. Extra keys are usually harmless until a specific provider or test path reads them. ## Common fields | Field | Meaning | Notes | | --- | --- | --- | | `type` | Provider implementation to use | Supported values from the example file are `vsphere`, `ovirt`, `openstack`, `openshift`, and `ova`. | | `version` | Provider version label | Used mainly in generated test resource names. Keep it populated for every entry. | | `fqdn` | Provider host name or IP | Important for VMware direct connections and for CA certificate download in secure VMware, RHV, and OpenStack flows. | | `api_url` | Provider API endpoint or share URL | Expected format depends on the provider: `/sdk` for vSphere, `/ovirt-engine/api` for RHV, `/v3` for OpenStack, and an NFS share URL for OVA. | | `username` / `password` | Provider login credentials | Required for VMware, RHV, and OpenStack. Kept as placeholders in the OpenShift and OVA examples. | | `guest_vm_linux_user` / `guest_vm_linux_password` | Linux guest login | Used for SSH-based post-migration validation, not for connecting to the source provider. | | `guest_vm_win_user` / `guest_vm_win_password` | Windows guest login | Also used for post-migration validation, not provider login. | | `vddk_init_image` | vSphere-specific provider field | Passed through to the MTV `Provider` resource when set. | | `copyoffload` | vSphere-only nested settings | Used by copy-offload tests to build storage secrets and storage-map plugin config. | ## Guest credentials Guest credentials are separate from provider credentials. The provider `username` and `password` fields log in to the source platform itself. The `guest_vm_*` fields are read later by post-migration SSH checks when the destination VM is powered on. If those checks run and the matching guest credentials are missing, validation fails. This matters even if the shipped example for a provider does not show guest credentials. The loader keeps extra keys, so it is fine to add `guest_vm_linux_*` and `guest_vm_win_*` to any provider entry when your selected tests need them. > **Tip:** Think of `guest_vm_linux_*` and `guest_vm_win_*` as per-guest test credentials, not part of the provider login. ## SSL behavior Source-provider SSL behavior is controlled in `tests/tests_config/config.py`, not inside `.providers.json`: ```python insecure_verify_skip: str = "true" # SSL verification for OCP API connections source_provider_insecure_skip_verify: str = "false" # SSL verification for source provider (VMware, RHV, etc.) ``` Key points: - `source_provider_insecure_skip_verify` controls the source provider secret created for VMware, RHV, OpenStack, and OVA. - `insecure_verify_skip` is for OpenShift API connections and does not control VMware/RHV/OpenStack provider validation. - These settings are stored as strings, so use `"true"` or `"false"`. - When `source_provider_insecure_skip_verify` is `"false"`, the harness fetches a CA certificate from `fqdn:443` and stores it in the provider secret for VMware and OpenStack. - RHV is special: the code always fetches the CA certificate, even when verification is skipped, because the ImageIO path still needs it. - OpenShift is also special: the source provider reuses the current cluster token secret, which is created with `insecureSkipVerify: "true"`. - OVA has no CA download step. > **Note:** Secure mode only works if `fqdn` points to a host that serves the provider certificate on port `443`. If the certificate fetch fails, provider creation fails. ## vSphere *Example from `.providers.json.example`:* ```jsonc "vsphere": { "type": "vsphere", "version": "", "fqdn": "SERVER FQDN/IP", "api_url": "/sdk", "username": "USERNAME", "password": "PASSWORD", # pragma: allowlist secret "guest_vm_linux_user": "LINUX VMS USERNAME", "guest_vm_linux_password": "LINUX VMS PASSWORD", # pragma: allowlist secret "guest_vm_win_user": "WINDOWS VMS USERNAME", "guest_vm_win_password": "WINDOWS VMS PASSWORD", # pragma: allowlist secret "vddk_init_image": "" } ``` What matters for vSphere: - `type` must be `vsphere`. - `fqdn` is used for the direct vSphere connection. - `api_url` becomes the MTV provider URL and should end with `/sdk`. - `username` and `password` are the vSphere credentials used by the harness. - The Linux and Windows guest credentials are used only for guest-level validation after migration. - `vddk_init_image` is passed to the MTV `Provider` resource when present. > **Tip:** Keep a separate vSphere entry for copy-offload, like the example’s `vsphere-copy-offload`. That makes it easy to switch between regular and copy-offload test runs by changing only `source_provider`. ## vSphere copy-offload The `copyoffload` section is only meaningful for vSphere entries. The code validates it before copy-offload tests run, uses it to build the storage secret, and then passes that secret into the `vsphereXcopyConfig` storage-map plugin configuration. *Core copy-offload fields from `.providers.json.example`:* ```jsonc "copyoffload": { # Supported storage_vendor_product values: # - "ontap" (NetApp ONTAP) # - "vantara" (Hitachi Vantara) # - "primera3par" (HPE Primera/3PAR) # - "pureFlashArray" (Pure Storage FlashArray) # - "powerflex" (Dell PowerFlex) # - "powermax" (Dell PowerMax) # - "powerstore" (Dell PowerStore) # - "infinibox" (Infinidat InfiniBox) # - "flashsystem" (IBM FlashSystem) "storage_vendor_product": "ontap", # Primary datastore for copy-offload operations (required) # This is the vSphere datastore ID (e.g., "datastore-12345") where VMs reside # Get via vSphere: Datacenter → Storage → Datastore → Summary → More Objects ID "datastore_id": "datastore-12345", # Optional: Secondary datastore for multi-datastore copy-offload tests # Only needed when testing VMs with disks spanning multiple datastores # When specified, tests can validate copy-offload with disks on different datastores "secondary_datastore_id": "datastore-67890", # Optional: Non-XCOPY datastore for mixed datastore tests # This should be a datastore that does NOT support XCOPY/VAAI primitives # Used for testing VMs with disks on both XCOPY and non-XCOPY datastores "non_xcopy_datastore_id": "datastore-99999", "default_vm_name": "rhel9-template", "storage_hostname": "storage.example.com", "storage_username": "admin", "storage_password": "your-password-here", # pragma: allowlist secret ``` Copy-offload field reference: | Field | Required when | Meaning | | --- | --- | --- | | `storage_vendor_product` | Always | Storage backend name. Must be one of the supported values listed above. | | `datastore_id` | Always | Primary vSphere datastore MoRef ID, such as `datastore-12345`. | | `storage_hostname` | Always, unless provided by environment variable | Storage system host used to build the copy-offload secret. | | `storage_username` | Always, unless provided by environment variable | Storage login name. | | `storage_password` | Always, unless provided by environment variable | Storage password. | | `secondary_datastore_id` | Only for multi-datastore tests | Second XCOPY-capable datastore. | | `non_xcopy_datastore_id` | Only for mixed/fallback tests | Datastore that does not support XCOPY/VAAI. | | `default_vm_name` | Optional | Overrides the source VM/template name for cloned copy-offload tests. | | `esxi_clone_method` | Optional | `vib` is the default. Set it to `ssh` to make the provider use SSH-based ESXi cloning. | | `esxi_host` / `esxi_user` / `esxi_password` | Required when `esxi_clone_method` is `ssh` | ESXi SSH connection settings. | | `rdm_lun_uuid` | Only for RDM tests | Required when running RDM disk tests. | Vendor-specific fields: | `storage_vendor_product` value | Additional fields | | --- | --- | | `ontap` | `ontap_svm` | | `vantara` | `vantara_storage_id`, `vantara_storage_port`, `vantara_hostgroup_id_list` | | `primera3par` | none | | `pureFlashArray` | `pure_cluster_prefix` | | `powerflex` | `powerflex_system_id` | | `powermax` | `powermax_symmetrix_id` | | `powerstore` | none | | `infinibox` | none | | `flashsystem` | none | > **Tip:** Every copy-offload credential can come from an environment variable instead of the file, and environment variables win. The code builds names as `COPYOFFLOAD_`, so examples include `COPYOFFLOAD_STORAGE_HOSTNAME`, `COPYOFFLOAD_STORAGE_USERNAME`, `COPYOFFLOAD_STORAGE_PASSWORD`, `COPYOFFLOAD_ONTAP_SVM`, `COPYOFFLOAD_ESXI_HOST`, `COPYOFFLOAD_ESXI_USER`, and `COPYOFFLOAD_ESXI_PASSWORD`. > **Warning:** The supported `storage_vendor_product` values are fixed in code. Use the exact spellings shown in the example and table above. ## RHV / oVirt The RHV source path uses `type: "ovirt"`. *Example from `.providers.json.example`:* ```jsonc "ovirt": { "type": "ovirt", "version": "", "fqdn": "SERVER FQDN/IP", "api_url": "/ovirt-engine/api", "username": "USERNAME", "password": "PASSWORD" # pragma: allowlist secret } ``` What matters for RHV: - Use `type: "ovirt"` even if you think of the source as RHV. - `api_url` should point to the engine API and end with `/ovirt-engine/api`. - `fqdn` should point to the engine host, because the CA certificate is fetched from `fqdn:443`. - `username` and `password` are required for the provider connection. - If your selected tests perform SSH-based guest validation, add `guest_vm_linux_*` and `guest_vm_win_*` to this entry even though the example does not show them. > **Note:** RHV is the one provider where the harness always downloads the CA certificate. In secure mode it is used for SDK validation; in insecure mode it is still carried because the ImageIO flow needs it. > **Note:** The RHV provider code also expects a data center named `MTV-CNV` to exist and be `up`. That is not configured in `.providers.json`, but it is enforced during connection. ## OpenStack *Example from `.providers.json.example`:* ```jsonc "openstack": { "type": "openstack", "version": "SERVER VERSION", "fqdn": "SERVER FQDN/IP", "api_url": ":/v3", "username": "USERNAME", "password": "PASSWORD", # pragma: allowlist secret "user_domain_name": "", "region_name": "", "project_name": "", "user_domain_id": "", "project_domain_id": "PROJECT DOMAIN ID", "guest_vm_linux_user": "LINUX VMS USERNAME", "guest_vm_linux_password": "LINUX VMS PASSWORD" # pragma: allowlist secret } ``` What matters for OpenStack: - `api_url` should be the Keystone v3 endpoint. - `project_name`, `user_domain_name`, `region_name`, `user_domain_id`, and `project_domain_id` are all read by the OpenStack provider code. Keep all of them populated. - `fqdn` still matters in secure mode because the harness fetches a CA certificate from `fqdn:443`. - The example includes Linux guest credentials because post-migration validation may SSH into powered-on Linux guests. - If your test selection includes powered-on Windows guests with guest-level validation, add `guest_vm_win_user` and `guest_vm_win_password` as well. ## OpenShift *Example from `.providers.json.example`:* ```jsonc "openshift": { "type": "openshift", "version": "", "fqdn": "", "api_url": "", "username": "", "password": "" # pragma: allowlist secret } ``` What matters for OpenShift: - Keep the placeholder shape from the example. - In this repo, the OpenShift source provider does not use `fqdn`, `api_url`, `username`, or `password` from `.providers.json` to log in. - Instead, the code rewrites the URL to the current cluster and reuses the current cluster token secret. - The blank values in the example are intentional. - If you run OpenShift-source scenarios that perform guest SSH validation, you can still add `guest_vm_linux_*` and `guest_vm_win_*` to this entry even though the example omits them. > **Note:** The reused OpenShift secret is created with `insecureSkipVerify: "true"`, so `source_provider_insecure_skip_verify` does not affect OpenShift the same way it affects VMware, RHV, or OpenStack. ## OVA *Example from `.providers.json.example`:* ```jsonc "ova": { "type": "ova", "version": "", # Can be anything, just placeholder "fqdn": "", "api_url": "", "username": "", "password": "" # pragma: allowlist secret } ``` What matters for OVA: - `api_url` is the NFS share URL. - The example already notes that `version` can be a placeholder. The code mainly uses it for naming, not for protocol negotiation. - The current OVA provider implementation only consumes `api_url`. - `username` and `password` stay in the example mostly to keep the provider entry shape consistent. - The OVA test path uses a fixed source VM name, `1nisim-rhel9-efi`, rather than selecting a source VM name from `.providers.json`. - There is no CA download step for OVA. ## Practical checklist - Start from `.providers.json.example`, then remove all comments before saving the real `.providers.json`. - Make sure your `source_provider` setting matches a top-level key in the file. - Keep `fqdn` accurate for VMware, RHV, and OpenStack, especially if SSL verification is enabled. - Add guest credentials for any provider entry whose powered-on guests will be validated over SSH. - For vSphere copy-offload, populate both the common storage credentials and the vendor-specific fields required by your chosen `storage_vendor_product`. --- Source: test-plan-configuration.md # Test Plan Configuration `tests_params` is the catalog of named migration plans used by this repository. Each entry defines which source VMs to use, which MTV plan features to enable, and which extra test behaviors should run before or after migration. The important thing to know is that the value you write in `tests/tests_config/config.py` is the **raw plan**. The test fixtures then turn that raw plan into a runtime `prepared_plan` before creating the MTV `Plan` custom resource. ## Where Plans Live The suite loads `tests/tests_config/config.py` as a Python config file, not as YAML or JSON: ```4:9:pytest.ini addopts = -s -o log_cli=true -p no:logging --tc-file=tests/tests_config/config.py --tc-format=python ``` Each test class then picks one named entry from `py_config["tests_params"]`: ```19:29:tests/test_mtv_cold_migration.py @pytest.mark.parametrize( "class_plan_config", [ pytest.param( py_config["tests_params"]["test_sanity_cold_mtv_migration"], ) ], indirect=True, ids=["rhel8"], ) ``` > **Note:** `tests/tests_config/config.py` contains both shared test-suite settings and the `tests_params` dictionary. Only entries inside `tests_params` are individual plan definitions. ## Basic Structure A minimal plan can be very small: ```46:50:tests/tests_config/config.py "test_sanity_cold_mtv_migration": { "virtual_machines": [ {"name": "mtv-tests-rhel8", "guest_agent": True}, ], "warm_migration": False, }, ``` A more advanced plan adds destination placement, PVC naming, labels, and affinity: ```434:466:tests/tests_config/config.py "test_warm_migration_comprehensive": { "virtual_machines": [ { "name": "mtv-win2022-ip-3disks", "source_vm_power": "on", "guest_agent": True, }, ], "warm_migration": True, "target_power_state": "on", "preserve_static_ips": True, "vm_target_namespace": "custom-vm-namespace", "multus_namespace": "default", # Cross-namespace NAD access "pvc_name_template": '{{ .FileName | trimSuffix ".vmdk" | replace "_" "-" }}-{{.DiskIndex}}', "pvc_name_template_use_generate_name": True, "target_labels": { "mtv-comprehensive-test": None, # None = auto-generate with session_uuid "static-label": "static-value", }, "target_affinity": { "podAffinity": { "preferredDuringSchedulingIgnoredDuringExecution": [ { "podAffinityTerm": { "labelSelector": {"matchLabels": {"app": "comprehensive-test"}}, "topologyKey": "kubernetes.io/hostname", }, "weight": 75, } ] } }, }, ``` In every entry: - `virtual_machines` is the per-VM section. - Everything else is plan-level behavior or test behavior. > **Note:** This file is plain Python. Use Python values such as `True`, `False`, and `None`, not JSON values like `true`, `false`, or `null`. ## Per-VM Settings Each item in `virtual_machines` describes one source VM or template. | Key | What it means | | --- | --- | | `name` | Source VM or template name. During preparation, this may be rewritten to the actual cloned runtime name. | | `source_vm_power` | Optional source power state before migration. `"on"` starts the VM, `"off"` stops it, and omitting it leaves the current state unchanged. | | `guest_agent` | Enables guest-agent-aware validation after migration. | | `clone` | Common in copy-offload and mutation-heavy scenarios. In this repository it is mostly a scenario hint, not the only thing that causes runtime cloning. | | `clone_name` | Overrides the default clone base name. | | `preserve_name_format` | Preserves uppercase letters and underscores in `clone_name` for name-format tests. | | `disk_type` | VMware disk provisioning mode for the cloned VM. | | `add_disks` | Adds extra disks to the cloned source VM before migration. | | `snapshots` | Number of source snapshots to create before migration in snapshot-focused tests. | | `target_datastore_id` | VMware target datastore for the cloned VM. | | `add_disks[*].datastore_id` | Optional datastore override for an individual added disk. | The VMware provider currently supports these `disk_type` values: ```77:80:libs/providers/vmware.py DISK_TYPE_MAP = { "thin": ("sparse", "Setting disk provisioning to 'thin' (sparse)."), "thick-lazy": ("flat", "Setting disk provisioning to 'thick-lazy' (flat)."), "thick-eager": ("eagerZeroedThick", "Setting disk provisioning to 'thick-eager' (eagerZeroedThick)."), } ``` A real multi-disk example looks like this: ```87:100:tests/tests_config/config.py "test_copyoffload_multi_disk_migration": { "virtual_machines": [ { "name": "xcopy-template-test", "source_vm_power": "off", "guest_agent": True, "clone": True, "add_disks": [ {"size_gb": 30, "disk_mode": "persistent", "provision_type": "thick-lazy"}, ], }, ], "warm_migration": False, "copyoffload": True, }, ``` For special naming tests, the config can override the clone name entirely: ```331:345:tests/tests_config/config.py "test_copyoffload_nonconforming_name_migration": { "virtual_machines": [ { "name": "xcopy-template-test", "clone_name": "XCopy_Test_VM_CAPS", # Non-conforming name for cloned VM "preserve_name_format": True, # Don't sanitize the name (keep capitals and underscores) "source_vm_power": "off", "guest_agent": True, "clone": True, "disk_type": "thin", }, ], "warm_migration": False, "copyoffload": True, }, ``` ### Snapshot-Driven Tests `snapshots` is a good example of a value that affects **test preparation**, not just MTV plan creation. Snapshot tests create the snapshots first, then store the pre-migration snapshot list back into the prepared plan for later validation: ```223:248:tests/test_copyoffload_migration.py vm_cfg = prepared_plan["virtual_machines"][0] provider_vm_api = prepared_plan["source_vms_data"][vm_cfg["name"]]["provider_vm_api"] # Ensure VM is powered on for snapshot creation source_provider.start_vm(provider_vm_api) source_provider.wait_for_vmware_guest_info(provider_vm_api, timeout=60) snapshots_to_create = int(vm_cfg["snapshots"]) snapshot_prefix = f"{vm_cfg['name']}-{fixture_store['session_uuid']}-snapshot" for idx in range(1, snapshots_to_create + 1): source_provider.create_snapshot( vm=provider_vm_api, name=f"{snapshot_prefix}-{idx}", description="mtv-api-tests copy-offload snapshots migration test", memory=False, quiesce=False, wait_timeout=60 * 10, ) # Refresh and store snapshots list for post-migration snapshot checks vm_cfg["snapshots_before_migration"] = source_provider.vm_dict(provider_vm_api=provider_vm_api)[ "snapshots_data" ] ``` > **Note:** In the class-based flow used by this repository, external-provider VMs are resolved with `clone_vm=True` during preparation. That means runtime VM names usually become per-test clone names even when the raw plan entry does not set `clone: true`. ## Plan-Level Settings ### Migration Behavior - `warm_migration`: Switches between warm and cold migration behavior. - `copyoffload`: Enables copy-offload-specific plan handling and validation. - `target_power_state`: Expected power state of the migrated VM after completion. - `preserve_static_ips`: Passes the static-IP preservation setting into the MTV plan and enables related validation in supported scenarios. - `guest_agent_timeout`: Overrides how long validation waits for the destination guest agent. If you omit `target_power_state`, post-migration validation falls back to the source VM power state: ```799:818:utilities/post_migration.py def check_vms_power_state( source_vm: dict[str, Any], destination_vm: dict[str, Any], source_power_before_migration: str | None, target_power_state: str | None = None, ) -> None: # If targetPowerState is specified, check that the destination VM matches it if target_power_state: actual_power_state = destination_vm["power_state"] LOGGER.info(f"Checking target power state: expected={target_power_state}, actual={actual_power_state}") assert actual_power_state == target_power_state, ( f"VM power state mismatch: expected {target_power_state}, got {actual_power_state}" ) elif source_power_before_migration: if source_power_before_migration not in ("on", "off"): raise ValueError(f"Invalid source_vm_power '{source_power_before_migration}'. Must be 'on' or 'off'") # Default behavior: destination VM should match source power state before migration assert destination_vm["power_state"] == source_power_before_migration ``` ### Placement, Naming, and Scheduling - `vm_target_namespace`: Custom namespace for migrated VMs. The preparation flow creates it if needed. - `multus_namespace`: Namespace where NADs are created for additional networks. - `pvc_name_template`: Forklift PVC naming template. - `pvc_name_template_use_generate_name`: When `True`, post-checks treat the rendered PVC name as a prefix because Kubernetes adds a generated suffix. - `target_node_selector`: Node label selector used for VM placement tests. - `target_labels`: Labels to apply to migrated VM metadata. - `target_affinity`: Affinity rules passed to the migrated VM template. `None` has a special meaning in `target_labels` and `target_node_selector`: the fixtures replace it with the current `session_uuid` so each test run gets a unique value. > **Tip:** `vm_target_namespace` and `multus_namespace` solve different problems. Use `vm_target_namespace` to choose where migrated VMs land, and `multus_namespace` to choose where the test-created NetworkAttachmentDefinitions live. ### Hooks and Failure Expectations The repository also supports hook-driven tests. A simple failure-path example looks like this: ```503:515:tests/tests_config/config.py "test_post_hook_retain_failed_vm": { "virtual_machines": [ { "name": "mtv-tests-rhel8", "source_vm_power": "on", "guest_agent": True, }, ], "warm_migration": False, "target_power_state": "off", "pre_hook": {"expected_result": "succeed"}, "post_hook": {"expected_result": "fail"}, "expected_migration_result": "fail", }, ``` In hook config: - `pre_hook` and `post_hook` are dictionaries. - The code accepts either `expected_result` (`"succeed"` or `"fail"`) or `playbook_base64` for a custom encoded Ansible playbook. - `expected_migration_result` tells the test whether a migration failure is expected. ## Copy-Offload-Specific Notes `copyoffload: true` is only the **plan-side** switch. The source provider also needs a `copyoffload` section in `.providers.json`. A real provider-side example from `.providers.json.example` looks like this: ```27:58:.providers.json.example "copyoffload": { # Supported storage_vendor_product values: # - "ontap" (NetApp ONTAP) # - "vantara" (Hitachi Vantara) # - "primera3par" (HPE Primera/3PAR) # - "pureFlashArray" (Pure Storage FlashArray) # - "powerflex" (Dell PowerFlex) # - "powermax" (Dell PowerMax) # - "powerstore" (Dell PowerStore) # - "infinibox" (Infinidat InfiniBox) # - "flashsystem" (IBM FlashSystem) "storage_vendor_product": "ontap", # Primary datastore for copy-offload operations (required) # This is the vSphere datastore ID (e.g., "datastore-12345") where VMs reside # Get via vSphere: Datacenter → Storage → Datastore → Summary → More Objects ID "datastore_id": "datastore-12345", # Optional: Secondary datastore for multi-datastore copy-offload tests # Only needed when testing VMs with disks spanning multiple datastores # When specified, tests can validate copy-offload with disks on different datastores "secondary_datastore_id": "datastore-67890", # Optional: Non-XCOPY datastore for mixed datastore tests # This should be a datastore that does NOT support XCOPY/VAAI primitives # Used for testing VMs with disks on both XCOPY and non-XCOPY datastores "non_xcopy_datastore_id": "datastore-99999", "default_vm_name": "rhel9-template", "storage_hostname": "storage.example.com", "storage_username": "admin", "storage_password": "your-password-here", # pragma: allowlist secret ``` > **Warning:** A raw plan with `copyoffload: true` is not enough by itself. The test session validates that the source provider is vSphere and that provider-side copy-offload settings are present, including at least `storage_vendor_product` and `datastore_id`. There is one more important implementation detail: when the helper creates an MTV plan with `copyoffload=True`, it forces `pvc_name_template` to `"pvc"`. ```238:243:utilities/mtv_migration.py # Add copy-offload specific parameters if enabled if copyoffload: # Set PVC naming template for copy-offload migrations # The volume populator framework requires this to generate consistent PVC names # Note: generateName is enabled by default, so Kubernetes adds random suffix automatically plan_kwargs["pvc_name_template"] = "pvc" ``` > **Warning:** Do not expect a custom `pvc_name_template` in `tests_params` to survive copy-offload plan creation. The helper currently overwrites it with `"pvc"`. ## How Raw Config Becomes `prepared_plan` The raw entry from `tests_params` is not used directly. The repository converts it into a runtime `prepared_plan` in several steps. ### 1. The Raw Plan Is Deep-Copied The fixture starts by copying the selected config and setting up runtime-only storage: ```840:859:conftest.py # Deep copy the plan config to avoid mutation plan: dict[str, Any] = deepcopy(class_plan_config) virtual_machines: list[dict[str, Any]] = plan["virtual_machines"] warm_migration = plan.get("warm_migration", False) # Initialize separate storage for source VM data (keeps virtual_machines clean for Plan CR serialization) plan["source_vms_data"] = {} # Handle custom VM target namespace vm_target_namespace = plan.get("vm_target_namespace") if vm_target_namespace: LOGGER.info(f"Using custom VM target namespace: {vm_target_namespace}") get_or_create_namespace( fixture_store=fixture_store, ocp_admin_client=ocp_admin_client, namespace_name=vm_target_namespace, ) plan["_vm_target_namespace"] = vm_target_namespace else: plan["_vm_target_namespace"] = target_namespace ``` ### 2. Each VM Is Resolved, Prepared, and Renamed The fixture then resolves a per-test VM, applies source power-state rules, rewrites the VM name to the actual runtime name, and stores provider details separately: ```903:951:conftest.py for vm in virtual_machines: # Get VM object first (without full vm_dict analysis) # Add enable_ctk flag for warm migrations clone_options = {**vm, "enable_ctk": warm_migration} provider_vm_api = source_provider.get_vm_by_name( query=vm["name"], vm_name_suffix=vm_name_suffix, clone_vm=True, session_uuid=fixture_store["session_uuid"], clone_options=clone_options, ) # Power state control: "on" = start VM, "off" = stop VM, not set = leave unchanged source_vm_power = vm.get("source_vm_power") # Optional - if not set, VM power state unchanged if source_vm_power == "on": source_provider.start_vm(provider_vm_api) # Wait for guest info to become available (VMware only) if source_provider.type == Provider.ProviderType.VSPHERE: source_provider.wait_for_vmware_guest_info(provider_vm_api, timeout=120) elif source_vm_power == "off": source_provider.stop_vm(provider_vm_api) # NOW call vm_dict() with VM in correct power state for guest info source_vm_details = source_provider.vm_dict( provider_vm_api=provider_vm_api, name=vm["name"], namespace=source_vms_namespace, clone=False, # Already cloned above vm_name_suffix=vm_name_suffix, session_uuid=fixture_store["session_uuid"], clone_options=vm, ) vm["name"] = source_vm_details["name"] # Wait for cloned VM to appear in Forklift inventory before proceeding # This is needed for external providers that Forklift needs to sync from # OVA is excluded because it doesn't clone VMs (uses pre-existing files) if source_provider.type != Provider.ProviderType.OVA: source_provider_inventory.wait_for_vm(name=vm["name"], timeout=300) provider_vm_api = source_vm_details["provider_vm_api"] vm["snapshots_before_migration"] = source_vm_details["snapshots_data"] # Store complete source VM data separately (keeps virtual_machines clean for Plan CR serialization) plan["source_vms_data"][vm["name"]] = source_vm_details # Create Hooks if configured create_hook_if_configured(plan, "pre_hook", "pre", fixture_store, ocp_admin_client, target_namespace) create_hook_if_configured(plan, "post_hook", "post", fixture_store, ocp_admin_client, target_namespace) ``` ### 3. The Test Adds VM IDs and Creates the MTV Plan Right before creating the MTV `Plan`, the tests add VM IDs from Forklift inventory and then pass the prepared values into `create_plan_resource()`: ```184:202:tests/test_warm_migration_comprehensive.py populate_vm_ids(plan=prepared_plan, inventory=source_provider_inventory) self.__class__.plan_resource = create_plan_resource( ocp_admin_client=ocp_admin_client, fixture_store=fixture_store, source_provider=source_provider, destination_provider=destination_provider, storage_map=self.storage_map, network_map=self.network_map, virtual_machines_list=prepared_plan["virtual_machines"], target_power_state=prepared_plan["target_power_state"], target_namespace=target_namespace, warm_migration=prepared_plan["warm_migration"], preserve_static_ips=prepared_plan["preserve_static_ips"], vm_target_namespace=prepared_plan["vm_target_namespace"], pvc_name_template=prepared_plan["pvc_name_template"], pvc_name_template_use_generate_name=prepared_plan["pvc_name_template_use_generate_name"], target_labels=target_vm_labels["vm_labels"], target_affinity=prepared_plan["target_affinity"], ) ``` ## Runtime Fields Added During Preparation These fields are derived at runtime. You do **not** write them yourself in `tests_params`. | Field | Added by | Purpose | | --- | --- | --- | | `source_vms_data` | `prepared_plan` | Stores rich source VM details for later validation without polluting `virtual_machines`. | | `_vm_target_namespace` | `prepared_plan` | Resolved namespace used by post-migration validation. | | `_pre_hook_name` / `_pre_hook_namespace` | Hook creation | Created Hook CR reference for plan creation. | | `_post_hook_name` / `_post_hook_namespace` | Hook creation | Created Hook CR reference for plan creation. | | `virtual_machines[*].name` | `prepared_plan` | Rewritten to the actual runtime VM name after provider lookup or cloning. | | `virtual_machines[*].snapshots_before_migration` | `prepared_plan` or snapshot test setup | Snapshot baseline used for post-migration checks. | | `virtual_machines[*].id` | `populate_vm_ids()` | Forklift VM ID required by plan creation. | A few raw config fields are also resolved by companion fixtures instead of `prepared_plan` itself: - `multus_namespace` is consumed by the network setup fixture that creates NADs. - `target_labels` is resolved by `target_vm_labels`. - `target_node_selector` is resolved by `labeled_worker_node`. > **Tip:** Keep `tests_params` declarative. Do not hand-write runtime fields such as `id`, `source_vms_data`, `_vm_target_namespace`, or `snapshots_before_migration`. Let the fixtures derive them. ## Practical Authoring Guidelines - Start with one VM and one or two plan-level flags, then add more only when the scenario really needs them. - Treat `virtual_machines` as source-side intent and `prepared_plan` as runtime state. They are not the same thing. - Use `None` intentionally in `target_labels` or `target_node_selector` when you want a unique value per run. - Expect VM names in `prepared_plan["virtual_machines"]` to differ from the raw `name` once preparation is complete. - For copy-offload scenarios, think in two layers: test plan config in `tests_params`, and provider/storage config in `.providers.json`. --- Source: optional-integrations-and-secrets.md # Optional Integrations And Secrets `mtv-api-tests` keeps its optional integrations outside the main test logic. In practice, that means you only add extra local files or environment variables when you want one of these features: - JIRA-aware test behavior through `pytest-jira` - AI-powered enrichment of failed JUnit reports - Copy-offload credential overrides for storage and ESXi access > **Note:** The repository already ignores the most important local-secret files: `.providers.json`, `jira.cfg`, `.env`, and `junit-report.xml`. ## JIRA Integration JIRA support is already wired into the test runner. The project depends on `pytest-jira`, and `pytest.ini` enables the plugin for normal test runs. Relevant excerpt from `pytest.ini`: ```ini addopts = -s -o log_cli=true -p no:logging --tc-file=tests/tests_config/config.py --tc-format=python --junit-xml=junit-report.xml --basetemp=/tmp/pytest --show-progress --strict-markers --jira --dist=loadscope ``` To connect that integration to your JIRA instance, use the shipped template in `jira.cfg.example` and create a local `jira.cfg` with the same shape: ```ini [DEFAULT] url = token = ``` The current codebase uses JIRA markers to gate specific tests around known issues. In both `tests/test_mtv_warm_migration.py` and `tests/test_warm_migration_comprehensive.py`, the RHV warm-migration path is annotated like this: ```python # Only apply Jira marker for RHV - skip if issue unresolved, run normally if resolved if _SOURCE_PROVIDER_TYPE == Provider.ProviderType.RHV: pytestmark.append(pytest.mark.jira("MTV-2846", run=False)) ``` That is the important user-facing behavior: JIRA is not just decorative metadata here. It is used to decide whether some tests should run. > **Tip:** Keep `jira.cfg` local, or generate it at job runtime from your CI secret store. The repo already ignores `jira.cfg`, so there is no reason to commit a token. ## AI Failure Analysis AI failure analysis is opt-in. It only activates when you pass `--analyze-with-ai`. From `conftest.py`: ```python analyze_with_ai_group = parser.getgroup(name="Analyze with AI") analyze_with_ai_group.addoption("--analyze-with-ai", action="store_true", help="Analyze test failures using AI") ``` When that flag is present, the suite loads `.env`, checks for a JJI server URL, and fills in defaults for the provider and model if you did not set them yourself. From `utilities/pytest_utils.py`: ```python load_dotenv() LOGGER.info("Setting up AI-powered test failure analysis") if not os.environ.get("JJI_SERVER_URL"): LOGGER.warning("JJI_SERVER_URL is not set. Analyze with AI features will be disabled.") session.config.option.analyze_with_ai = False else: if not os.environ.get("JJI_AI_PROVIDER"): os.environ["JJI_AI_PROVIDER"] = "claude" if not os.environ.get("JJI_AI_MODEL"): os.environ["JJI_AI_MODEL"] = "claude-opus-4-6[1m]" ``` The current environment variables are: | Variable | Required | Default in code | What it controls | | --- | --- | --- | --- | | `JJI_SERVER_URL` | Yes | none | Base URL of the Jenkins Job Insight service | | `JJI_AI_PROVIDER` | No | `claude` | Provider name sent to JJI | | `JJI_AI_MODEL` | No | `claude-opus-4-6[1m]` | Model name sent to JJI | | `JJI_TIMEOUT` | No | `600` | HTTP timeout in seconds for the analysis request | > **Note:** `claude-opus-4-6[1m]` above is the exact current default string in the code. The suite already writes a JUnit report by default because `pytest.ini` sets `--junit-xml=junit-report.xml`. When there are failures, the AI integration reads that XML, posts it to the JJI service, and writes the enriched XML back to the same file. From `utilities/pytest_utils.py`: ```python response = requests.post( f"{server_url.rstrip('/')}/analyze-failures", json={ "raw_xml": raw_xml, "ai_provider": ai_provider, "ai_model": ai_model, }, timeout=timeout_value, ) ``` A few practical details matter here: - If `JJI_SERVER_URL` is missing, the feature disables itself with a warning. - If `JJI_TIMEOUT` is invalid, the code falls back to `600`. - Dry-run modes such as `--collectonly` and `--setupplan` disable the feature. - Successful runs skip enrichment because there are no failures to analyze. - If enrichment fails, the original JUnit XML is preserved. > **Warning:** The AI path sends the raw JUnit XML to `JJI_SERVER_URL/analyze-failures`. That report can contain test names, failure messages, resource names, and any other details included in the report. Only enable this against a service you trust. > **Tip:** Because `.env` is gitignored and only loaded when `--analyze-with-ai` is enabled, it is a good place for local `JJI_*` settings. ## Copy-Offload Credential Overrides Copy-offload is the most secret-heavy optional path in the repository. It is also the strictest one: the fixtures fail early if required copy-offload configuration is missing. The base configuration lives under the source provider entry in `.providers.json`, which the suite loads from the repository root with `Path(".providers.json")` and `json.loads(...)`. A relevant excerpt from `.providers.json.example` shows the expected shape: ```jsonc "copyoffload": { # Supported storage_vendor_product values: # - "ontap" (NetApp ONTAP) # - "vantara" (Hitachi Vantara) # - "primera3par" (HPE Primera/3PAR) # - "pureFlashArray" (Pure Storage FlashArray) # - "powerflex" (Dell PowerFlex) # - "powermax" (Dell PowerMax) # - "powerstore" (Dell PowerStore) # - "infinibox" (Infinidat InfiniBox) # - "flashsystem" (IBM FlashSystem) "storage_vendor_product": "ontap", # Primary datastore for copy-offload operations (required) # This is the vSphere datastore ID (e.g., "datastore-12345") where VMs reside # Get via vSphere: Datacenter → Storage → Datastore → Summary → More Objects ID "datastore_id": "datastore-12345", # Optional: Secondary datastore for multi-datastore copy-offload tests # Only needed when testing VMs with disks spanning multiple datastores # When specified, tests can validate copy-offload with disks on different datastores "secondary_datastore_id": "datastore-67890", # Optional: Non-XCOPY datastore for mixed datastore tests # This should be a datastore that does NOT support XCOPY/VAAI primitives # Used for testing VMs with disks on both XCOPY and non-XCOPY datastores "non_xcopy_datastore_id": "datastore-99999", "default_vm_name": "rhel9-template", "storage_hostname": "storage.example.com", "storage_username": "admin", "storage_password": "your-password-here", # pragma: allowlist secret ``` And for SSH-based cloning, the same example file includes: ```jsonc # ESXi SSH configuration (optional, for SSH-based cloning): # Can be overridden via environment variables: COPYOFFLOAD_ESXI_HOST, COPYOFFLOAD_ESXI_USER, COPYOFFLOAD_ESXI_PASSWORD "esxi_clone_method": "ssh", # "vib" (default) or "ssh" "esxi_host": "your-esxi-host.example.com", # required for ssh method "esxi_user": "root", # required for ssh method "esxi_password": "your-esxi-password", # pragma: allowlist secret # required for ssh method ``` > **Warning:** The comments in `.providers.json.example` are not valid JSON. The real `.providers.json` file is parsed with `json.loads(...)`, so remove the `# pragma: allowlist secret` comments when creating your own file. ### How Environment Overrides Work The override rule is simple and explicit. For the fields that support overrides, environment variables win over values from `.providers.json`. From `utilities/copyoffload_migration.py`: ```python env_var_name = f"COPYOFFLOAD_{credential_name.upper()}" return os.getenv(env_var_name) or copyoffload_config.get(credential_name) ``` That helper is used for the credential-like copy-offload inputs. In the current code, the supported override names are: | `.providers.json` key | Environment variable | | --- | --- | | `storage_hostname` | `COPYOFFLOAD_STORAGE_HOSTNAME` | | `storage_username` | `COPYOFFLOAD_STORAGE_USERNAME` | | `storage_password` | `COPYOFFLOAD_STORAGE_PASSWORD` | | `ontap_svm` | `COPYOFFLOAD_ONTAP_SVM` | | `vantara_storage_id` | `COPYOFFLOAD_VANTARA_STORAGE_ID` | | `vantara_storage_port` | `COPYOFFLOAD_VANTARA_STORAGE_PORT` | | `vantara_hostgroup_id_list` | `COPYOFFLOAD_VANTARA_HOSTGROUP_ID_LIST` | | `pure_cluster_prefix` | `COPYOFFLOAD_PURE_CLUSTER_PREFIX` | | `powerflex_system_id` | `COPYOFFLOAD_POWERFLEX_SYSTEM_ID` | | `powermax_symmetrix_id` | `COPYOFFLOAD_POWERMAX_SYMMETRIX_ID` | | `esxi_host` | `COPYOFFLOAD_ESXI_HOST` | | `esxi_user` | `COPYOFFLOAD_ESXI_USER` | | `esxi_password` | `COPYOFFLOAD_ESXI_PASSWORD` | The code currently recognizes these `storage_vendor_product` values: `ontap`, `vantara`, `primera3par`, `pureFlashArray`, `powerflex`, `powermax`, `powerstore`, `infinibox`, and `flashsystem`. Only some vendors need extra vendor-specific secret values: - `ontap` requires `ontap_svm` - `vantara` requires `vantara_storage_id`, `vantara_storage_port`, and `vantara_hostgroup_id_list` - `pureFlashArray` requires `pure_cluster_prefix` - `powerflex` requires `powerflex_system_id` - `powermax` requires `powermax_symmetrix_id` - `primera3par`, `powerstore`, `infinibox`, and `flashsystem` use only the base storage credentials > **Warning:** Not every `copyoffload` key is overrideable. In the current code paths, `storage_vendor_product`, `datastore_id`, `secondary_datastore_id`, `non_xcopy_datastore_id`, `rdm_lun_uuid`, and `esxi_clone_method` are read directly from `.providers.json`, not through `COPYOFFLOAD_*` overrides. > **Tip:** A good working pattern is to keep stable, non-secret facts in `.providers.json` and move only the sensitive pieces, such as passwords and vendor credentials, into `COPYOFFLOAD_*` environment variables. ### How Those Values Become Runtime Secrets The copy-offload fixture converts the resolved values into a Kubernetes `Secret`. The base secret data is created like this: ```python secret_data = { "STORAGE_HOSTNAME": storage_hostname, "STORAGE_USERNAME": storage_username, "STORAGE_PASSWORD": storage_password, } ``` That secret is then referenced from the StorageMap config used by the copy-offload tests. From `tests/test_copyoffload_migration.py`: ```python offload_plugin_config = { "vsphereXcopyConfig": { "secretRef": copyoffload_storage_secret.name, "storageVendorProduct": storage_vendor_product, } } ``` This is why environment overrides are useful: they let you keep the StorageMap logic unchanged while changing only the secret material injected into the run. If you use SSH-based ESXi cloning, the vSphere provider also patches the provider setting to `esxiCloneMethod: ssh`. If you omit `esxi_clone_method` or leave it as `vib`, the code treats `vib` as the default and does not patch anything. ## Handling Sensitive Values In Practice The safest way to work with this repository is to separate stable configuration from secrets: - Keep stable settings in `.providers.json`: provider type, version, datastore IDs, vendor selection, VM names, and clone method. - Keep tokens and passwords in `jira.cfg`, `.env`, or `COPYOFFLOAD_*` environment variables. - In automation, create those files at runtime from your CI or cluster secret store instead of baking them into images or checking them into Git. - Treat `junit-report.xml` as sensitive if it may contain failure details you would not want to share broadly, especially when AI analysis is enabled. The existing OpenShift Job guidance in `docs/copyoffload/how-to-run-copyoffload-tests.md` already demonstrates a good secret-injection pattern for automation: ```bash read -sp "Enter cluster password: " CLUSTER_PASSWORD && echo oc create secret generic mtv-test-config \ --from-file=providers.json=.providers.json \ --from-literal=cluster_host=https://api.your-cluster.com:6443 \ --from-literal=cluster_username=kubeadmin \ --from-literal=cluster_password="${CLUSTER_PASSWORD}" \ -n mtv-tests unset CLUSTER_PASSWORD ``` That pattern is preferable to hardcoding credentials in manifests or committing local config files. There is also one explicit redaction path worth knowing about: the SSH helper masks the OpenShift token before logging the `virtctl` command. From `utilities/ssh_utils.py`: ```python cmd_str = " ".join(cmd) if self.ocp_token: cmd_str = cmd_str.replace(self.ocp_token, "[REDACTED]") LOGGER.info(f"Full virtctl command: {cmd_str}") ``` > **Note:** That masking is useful, but it is not a guarantee that every secret in every code path will be redacted automatically. > **Warning:** Copy the keys from `.providers.json.example`, not the comments. The `# pragma: allowlist secret` annotations are there for repository scanning and will break a real JSON file. > **Tip:** For day-to-day use, a practical split is: > keep `.providers.json` for non-secret structure, > keep `jira.cfg` and `.env` local, > and use `COPYOFFLOAD_*` or your CI secret manager for the values you would least want to store on disk. --- Source: cold-migrations.md # Cold Migrations Cold migration tests in `mtv-api-tests` follow a consistent pattern: prepare the source VM data, create a `StorageMap`, create a `NetworkMap`, create a `Plan`, execute a `Migration`, and then validate the migrated VM on the destination side. If you understand that flow, you understand the repository’s standard cold migration pattern. These are real integration tests, not unit tests. They create actual MTV and OpenShift resources, talk to a real source provider through Forklift inventory, and verify the migrated VM after the move finishes. ## Required Inputs A standard cold migration needs: - A plan entry in `tests/tests_config/config.py` - A source-provider entry in `.providers.json` - Session config that includes `source_provider` and `storage_class` The basic cold-migration test configuration is intentionally small: ```46:51:tests/tests_config/config.py "test_sanity_cold_mtv_migration": { "virtual_machines": [ {"name": "mtv-tests-rhel8", "guest_agent": True}, ], "warm_migration": False, }, ``` Provider details come from `.providers.json`. The example file shows the fields the suite expects, including guest credentials used for post-migration checks such as SSH validation: ```2:14:.providers.json.example "vsphere": { "type": "vsphere", "version": "", "fqdn": "SERVER FQDN/IP", "api_url": "/sdk", "username": "USERNAME", "password": "PASSWORD", # pragma: allowlist secret "guest_vm_linux_user": "LINUX VMS USERNAME", "guest_vm_linux_password": "LINUX VMS PASSWORD", # pragma: allowlist secret "guest_vm_win_user": "WINDOWS VMS USERNAME", "guest_vm_win_password": "WINDOWS VMS PASSWORD", # pragma: allowlist secret "vddk_init_image": "" }, ``` > **Note:** `.providers.json.example` contains inline comments for documentation. Remove those comments in your real `.providers.json`, because JSON does not support comments. At runtime, `conftest.py` also enforces `source_provider` and `storage_class` before the session starts, so a real cold-migration run needs both values in place. ## The Standard Flow In `tests/test_mtv_cold_migration.py`, the class-based cold test follows the same six stages every time: 1. `prepared_plan` builds the class-scoped migration input. 2. `test_create_storagemap` creates the `StorageMap`. 3. `test_create_networkmap` creates the `NetworkMap`. 4. `test_create_plan` creates the `Plan`. 5. `test_migrate_vms` creates the `Migration` CR and waits for completion. 6. `test_check_vms` validates the migrated VM. The class is marked `@pytest.mark.incremental`, so later stages only make sense after earlier ones succeed. It also uses `cleanup_migrated_vms`, which removes migrated VMs after the class finishes unless teardown is explicitly skipped. The same shape is reused for the remote-cluster cold migration class as well. The only meaningful difference there is the destination provider fixture. ## 1. Prepare The Plan Before any map is created, the `prepared_plan` fixture turns a small config entry into something the rest of the test class can use. It copies the config, decides where migrated VMs should land, prepares or clones the source VM, applies an optional power-state change, waits for Forklift inventory to see the VM, and stores full source VM details for later validation. ```840:947:conftest.py plan: dict[str, Any] = deepcopy(class_plan_config) virtual_machines: list[dict[str, Any]] = plan["virtual_machines"] warm_migration = plan.get("warm_migration", False) plan["source_vms_data"] = {} vm_target_namespace = plan.get("vm_target_namespace") if vm_target_namespace: LOGGER.info(f"Using custom VM target namespace: {vm_target_namespace}") get_or_create_namespace( fixture_store=fixture_store, ocp_admin_client=ocp_admin_client, namespace_name=vm_target_namespace, ) plan["_vm_target_namespace"] = vm_target_namespace else: plan["_vm_target_namespace"] = target_namespace # ... each VM is prepared before the Plan is created ... for vm in virtual_machines: clone_options = {**vm, "enable_ctk": warm_migration} provider_vm_api = source_provider.get_vm_by_name( query=vm["name"], vm_name_suffix=vm_name_suffix, clone_vm=True, session_uuid=fixture_store["session_uuid"], clone_options=clone_options, ) source_vm_power = vm.get("source_vm_power") if source_vm_power == "on": source_provider.start_vm(provider_vm_api) elif source_vm_power == "off": source_provider.stop_vm(provider_vm_api) source_vm_details = source_provider.vm_dict( provider_vm_api=provider_vm_api, name=vm["name"], namespace=source_vms_namespace, clone=False, vm_name_suffix=vm_name_suffix, session_uuid=fixture_store["session_uuid"], clone_options=vm, ) vm["name"] = source_vm_details["name"] source_provider_inventory.wait_for_vm(name=vm["name"], timeout=300) plan["source_vms_data"][vm["name"]] = source_vm_details ``` A few practical details matter here: - `source_vm_power` is optional. If you do not set it, the fixture leaves the source VM power state unchanged. - `source_vms_data` is where the suite keeps rich source-side details for later checks such as static IP and PVC-name validation. - `_vm_target_namespace` can be different from the namespace that holds the migration resources. That distinction matters in advanced cold-migration scenarios. > **Tip:** If you only need baseline cold-migration coverage, start with the minimal config shown above and let `prepared_plan` do the rest. ## 2. Create The StorageMap `test_create_storagemap()` takes the VM names from `prepared_plan` and passes them to `get_storage_migration_map()`. The helper does not hardcode source datastores or storage domains. Instead, it asks Forklift inventory which storages those VMs actually use, then maps each one to the selected OpenShift `storage_class`. ```429:505:utilities/mtv_migration.py target_storage_class: str = storage_class or py_config["storage_class"] storage_map_list: list[dict[str, Any]] = [] # ... copy-offload branch omitted ... LOGGER.info(f"Creating standard storage map for VMs: {vms}") storage_migration_map = source_provider_inventory.vms_storages_mappings(vms=vms) for storage in storage_migration_map: storage_map_list.append({ "destination": {"storageClass": target_storage_class}, "source": storage, }) storage_map = create_and_store_resource( fixture_store=fixture_store, resource=StorageMap, client=ocp_admin_client, namespace=target_namespace, mapping=storage_map_list, source_provider_name=source_provider.ocp_resource.name, source_provider_namespace=source_provider.ocp_resource.namespace, destination_provider_name=destination_provider.ocp_resource.name, destination_provider_namespace=destination_provider.ocp_resource.namespace, ) ``` This is why the cold tests stay fairly small at the test-method level. Storage discovery is delegated to the provider-specific inventory code in `libs/forklift_inventory.py`, so the test only has to name the VM and the target storage class. ## 3. Create The NetworkMap Network mapping follows the same inventory-driven idea. The suite asks Forklift inventory which source networks the chosen VM uses, then maps those networks to either the pod network or class-scoped Multus networks. The key rule is simple: the first network goes to the pod network, and every additional network is mapped to a Multus `NetworkAttachmentDefinition`. ```154:190:utilities/utils.py for index, network in enumerate(source_provider_inventory.vms_networks_mappings(vms=vms)): if pod_only or index == 0: _destination = {"type": "pod"} else: multus_network_name_str = multus_network_name["name"] multus_namespace = multus_network_name["namespace"] nad_name = f"{multus_network_name_str}-{multus_counter}" _destination = { "name": nad_name, "namespace": multus_namespace, "type": "multus", } multus_counter += 1 network_map_list.append({ "destination": _destination, "source": network, }) ``` That behavior is user-friendly in practice: - Single-NIC VMs usually need no special network setup beyond the default pod network. - Multi-NIC VMs automatically get additional Multus attachments. - If your config sets `multus_namespace`, the suite creates those NADs in that namespace instead of the default migration namespace. ## 4. Create The Plan And Execute The Migration Before creating the `Plan`, the cold tests call `populate_vm_ids()` so each VM in `virtual_machines` includes the Forklift inventory ID MTV expects. Then `create_plan_resource()` builds a `Plan` CR that ties together the source provider, destination provider, storage map, network map, VM list, and any optional plan features. The same helper also shows an important namespace detail: by default, migrated VMs land in the same `target_namespace`, but `vm_target_namespace` can override that when needed. ```200:295:utilities/mtv_migration.py plan_kwargs: dict[str, Any] = { "client": ocp_admin_client, "fixture_store": fixture_store, "resource": Plan, "namespace": target_namespace, "source_provider_name": source_provider.ocp_resource.name, "source_provider_namespace": source_provider.ocp_resource.namespace, "destination_provider_name": destination_provider.ocp_resource.name, "destination_provider_namespace": destination_provider.ocp_resource.namespace, "storage_map_name": storage_map.name, "storage_map_namespace": storage_map.namespace, "network_map_name": network_map.name, "network_map_namespace": network_map.namespace, "virtual_machines_list": virtual_machines_list, "target_namespace": vm_target_namespace or target_namespace, "warm_migration": warm_migration, "pre_hook_name": pre_hook_name, "pre_hook_namespace": pre_hook_namespace, "after_hook_name": after_hook_name, "after_hook_namespace": after_hook_namespace, "preserve_static_ips": preserve_static_ips, "pvc_name_template": pvc_name_template, "pvc_name_template_use_generate_name": pvc_name_template_use_generate_name, "target_power_state": target_power_state, } if target_node_selector: plan_kwargs["target_node_selector"] = target_node_selector if target_labels: plan_kwargs["target_labels"] = target_labels if target_affinity: plan_kwargs["target_affinity"] = target_affinity plan = create_and_store_resource(**plan_kwargs) plan.wait_for_condition(condition=Plan.Condition.READY, status=Plan.Condition.Status.TRUE, timeout=360) # ... later, execution creates the Migration CR ... create_and_store_resource( client=ocp_admin_client, fixture_store=fixture_store, resource=Migration, namespace=target_namespace, plan_name=plan.name, plan_namespace=plan.namespace, cut_over=cut_over, ) wait_for_migration_complate(plan=plan) ``` For standard cold migrations, the important switch is `warm_migration=False`. That means: - There is no warm-only precopy fixture. - There is no scheduled cutover time in the normal cold flow. - Execution goes directly from a ready `Plan` to a real `Migration` CR. By default, the config sets `plan_wait_timeout` to `3600` seconds in `tests/tests_config/config.py`, so long-running migrations have a wider execution window than the helper’s fallback default. ## 5. Validate The Migrated VM The post-migration phase is much more than “the VM exists.” The `check_vms()` helper fetches both the source VM and the destination VM, runs a series of validations, accumulates any mismatches per VM, and fails at the end if anything is wrong. In the standard cold flow, it checks: - Power state - CPU - Memory - Network mapping - Storage class and disk mapping When the plan config asks for more, it can also check: - PVC names - Guest-agent availability - SSH connectivity to the migrated VM - Static IP preservation - Target node placement - Target labels - Affinity rules Some checks are provider-specific: - Snapshot comparison and serial preservation are used for VMware-backed migrations. - False power-off validation is used for RHV-backed migrations. - Static IP preservation is currently implemented for Windows VMs migrated from vSphere. This validation step is where the cold migration pattern becomes genuinely useful. A migration can “finish” and still be wrong. The repository’s cold tests treat success as “the VM moved and still matches expectations,” not just “the Migration CR reached a terminal state.” > **Note:** `cleanup_migrated_vms` removes migrated VMs after the class finishes. If you run with `--skip-teardown`, those VMs are intentionally left behind for debugging. ## Advanced Cold Migration Features The comprehensive cold migration test shows how the same pattern scales up when you want to validate plan features, not just a successful move: ```467:502:tests/tests_config/config.py "test_cold_migration_comprehensive": { "virtual_machines": [ { "name": "mtv-win2019-3disks", "source_vm_power": "off", "guest_agent": True, }, ], "warm_migration": False, "target_power_state": "on", "preserve_static_ips": True, "pvc_name_template": "{{.VmName}}-disk-{{.DiskIndex}}", "pvc_name_template_use_generate_name": False, "target_node_selector": { "mtv-comprehensive-node": None, }, "target_labels": { "mtv-comprehensive-label": None, "test-type": "comprehensive", }, "target_affinity": { "podAffinity": { "preferredDuringSchedulingIgnoredDuringExecution": [ { "podAffinityTerm": { "labelSelector": {"matchLabels": {"app": "test"}}, "topologyKey": "kubernetes.io/hostname", }, "weight": 50, } ] } }, "vm_target_namespace": "mtv-comprehensive-vms", "multus_namespace": "default", }, ``` Each of those fields drives a real plan feature or a real validation path: - `source_vm_power: "off"` tells the preparation fixture to power the source VM off before migration. - `target_power_state: "on"` proves that the destination VM can come up powered on even if the source was prepared powered off. - `preserve_static_ips: True` enables static-IP verification. In the current codebase, that check is meant for Windows VMs coming from vSphere. - `pvc_name_template` and `pvc_name_template_use_generate_name` turn on PVC-name validation after migration. The helper also supports more advanced template behavior, including `generateName` handling. - `target_node_selector` causes a worker node to be labeled for the test, and post-migration validation checks that the migrated VM lands there. - `target_labels` adds expected labels to the migrated VM. When a value is `None`, the fixture replaces it with the current `session_uuid` so labels stay unique across runs. - `target_affinity` lets the test verify full affinity configuration on the destination VM. - `vm_target_namespace` separates the VM’s destination namespace from the namespace that holds the MTV resources. - `multus_namespace` lets the test create its NADs outside the default migration namespace. > **Tip:** Use `test_sanity_cold_mtv_migration` when you want baseline cold-migration coverage. Use `test_cold_migration_comprehensive` when you want to validate plan behavior such as target power state, PVC naming, labels, affinity, or custom VM namespaces. ## How Automation Treats Cold Tests The repository’s automation is careful about the difference between “this suite is structurally valid” and “a real cold migration succeeded.” Pytest is wired through `pytest.ini` to load `tests/tests_config/config.py` automatically and to use `--dist=loadscope`, which fits the class-based, incremental cold-migration pattern well. The included `tox` environment does not perform a live migration run. Instead, it uses dry-run style checks: ```4:18:tox.toml commands = [ [ "uv", "run", "pytest", "--setup-plan", ], [ "uv", "run", "pytest", "--collect-only", ], ] ``` The container image follows the same philosophy: its default command is `uv run pytest --collect-only`. > **Warning:** A successful `tox` run or container dry-run only proves that the suite can be collected and its fixtures can be planned. It does not prove that a cold migration will succeed against your real OpenShift cluster, source provider, network setup, or storage class. That split is important for users. The repository can validate test structure in automation, but real cold-migration confidence still comes from running the suite in a live environment with real provider and cluster access. --- Source: warm-migrations.md # Warm Migrations Warm migrations in `mtv-api-tests` are the powered-on migration path. The suite starts from a running source VM, lets MTV perform one or more precopy rounds, and then completes the migration at a scheduled cutover time. If you are setting up or tuning warm coverage, the main knobs are `warm_migration`, `snapshots_interval`, and `mins_before_cutover`. ## What You Need To Set A warm scenario in this repository is driven by a small set of plan fields: - `warm_migration: True` tells the suite to create a warm MTV plan. - `source_vm_power: "on"` makes the source VM run before migration starts. - `guest_agent: True` enables guest-agent validation after cutover. - `target_power_state` lets you enforce the expected destination VM power state. Example from `tests/tests_config/config.py`: ```python "test_sanity_warm_mtv_migration": { "virtual_machines": [ { "name": "mtv-tests-rhel8", "source_vm_power": "on", "guest_agent": True, }, ], "warm_migration": True, }, ``` In practice, warm tests in this repository also clone the source VM before migration. That makes repeated test runs safer and gives the suite a place to apply warm-specific preparation. ## How Warm Source Preparation Works For vSphere sources, the suite explicitly enables Change Block Tracking (CBT) on the cloned VM before warm migration starts: ```python # Enable Change Block Tracking (CBT) only for warm migrations enable_ctk = kwargs.get("enable_ctk", False) if enable_ctk: LOGGER.info("Enabling Change Block Tracking (CBT) for warm migration") cbt_option = vim.option.OptionValue() cbt_option.key = "ctkEnabled" cbt_option.value = "true" ``` That matters because warm migration depends on change tracking between precopy rounds. > **Tip:** Auto-generated Plan and Migration names in this suite get a `-warm` suffix for warm runs, which makes cluster-side debugging easier when you inspect resources during a test session. ## Precopy Interval Tuning Warm timing knobs live in `tests/tests_config/config.py`, which is the default test configuration file loaded by `pytest.ini`: ```python snapshots_interval: int = 2 mins_before_cutover: int = 5 plan_wait_timeout: int = 3600 ``` The `snapshots_interval` value is applied through the `precopy_interval_forkliftcontroller` fixture, which patches the live `ForkliftController` custom resource: ```python snapshots_interval = py_config["snapshots_interval"] forklift_controller.wait_for_condition( status=forklift_controller.Condition.Status.TRUE, condition=forklift_controller.Condition.Type.RUNNING, timeout=300, ) LOGGER.info( f"Updating forklift-controller ForkliftController CR with snapshots interval={snapshots_interval} seconds" ) with ResourceEditor( patches={ forklift_controller: { "spec": { "controller_precopy_interval": str(snapshots_interval), } } } ): ``` What these values do: - `snapshots_interval` controls how often Forklift schedules warm precopy snapshots. - `mins_before_cutover` controls how far in the future cutover is scheduled. - `plan_wait_timeout` controls how long the suite waits for the migration to finish. The defaults in this repository are a fast `2` second precopy interval and a `5` minute cutover delay. > **Warning:** `snapshots_interval` is not treated as a per-plan setting in the test suite. Warm tests patch the live `forklift-controller` `ForkliftController` resource while the fixture is active. ## Cutover Timing Warm tests do not cut over immediately by default. They compute the cutover timestamp in UTC from `mins_before_cutover`: ```python def get_cutover_value(current_cutover: bool = False) -> datetime: datetime_utc = datetime.now(pytz.utc) if current_cutover: return datetime_utc return datetime_utc + timedelta(minutes=int(py_config["mins_before_cutover"])) ``` That helper is passed into the migration execution step: ```python execute_migration( ocp_admin_client=ocp_admin_client, fixture_store=fixture_store, plan=self.plan_resource, target_namespace=target_namespace, cut_over=get_cutover_value(), ) ``` This gives the warm migration time to accumulate precopy work before the final switchover. Use these tuning rules: - Increase `mins_before_cutover` if you want a longer warm phase before downtime. - Decrease `mins_before_cutover` if you want the test to reach final cutover sooner. - Increase `plan_wait_timeout` if your environment is slow and the plan needs more time to finish. > **Tip:** The helper also supports immediate cutover with `current_cutover=True`, although the warm test classes in this repository use the delayed default. ## Supported Providers Warm provider support is enforced directly in the warm test modules. | Source provider | Warm status in this repo | Notes | | --- | --- | --- | | vSphere | Supported | Main warm path. Also used for warm copy-offload coverage. | | RHV | Implemented, but Jira-gated | The warm tests add a Jira marker for `MTV-2846`. | | OpenStack | Not supported | Explicitly skipped by the warm tests. | | OpenShift | Not supported as a warm source | Explicitly skipped by the warm tests. | | OVA | Not supported | Explicitly skipped by the warm tests. | The repository also includes a remote-destination warm scenario: - `TestWarmRemoteOcp` runs a warm migration to a remote OpenShift provider. - It requires `remote_ocp_cluster` to be configured. - It uses the `remote` marker rather than the `warm` marker. Marker definitions come from `pytest.ini`: ```ini markers = tier0: Core functionality tests (smoke tests) remote: Remote cluster migration tests warm: Warm migration tests copyoffload: Copy-offload (XCOPY) tests ``` > **Note:** Most warm suites are selected with the `warm` marker. The remote OpenShift warm scenario lives in `tests/test_mtv_warm_migration.py`, but it is selected through `remote`. ## Warm Scenarios Included In The Repository These are the main warm configurations currently defined in `tests/tests_config/config.py`: - `test_sanity_warm_mtv_migration`: basic warm migration of a powered-on RHEL VM - `test_mtv_migration_warm_2disks2nics`: warm migration with extra disk and NIC coverage - `test_warm_remote_ocp`: warm migration to a remote OpenShift destination - `test_warm_migration_comprehensive`: warm migration with static IP, PVC naming, target labels, affinity, and custom VM namespace coverage - `test_copyoffload_warm_migration`: warm migration through the vSphere copy-offload path ## What Warm Tests Validate Warm tests use the standard post-migration validation path in `check_vms()`. That means a warm run is not only checking whether MTV finishes, but also whether the migrated VM looks correct afterward. The standard validation path covers: - VM power state, including `target_power_state` when it is set - CPU and memory - network mapping - storage mapping - provider SSL configuration for VMware, RHV, and OpenStack providers - guest agent availability when the plan expects it - SSH connectivity when the destination VM is powered on - VMware snapshot tracking when snapshot data exists - VMware serial preservation on the destination VM This is especially useful for warm migrations because the destination VM is usually powered on after cutover, which allows the suite to verify guest-agent and SSH behavior immediately. > **Note:** SSH-based checks run only when the destination VM is powered on. If you set a powered-off target state, those checks are skipped. ## The Comprehensive Warm Validation Path The most feature-rich warm scenario in the repository is `test_warm_migration_comprehensive`: ```python "test_warm_migration_comprehensive": { "virtual_machines": [ { "name": "mtv-win2022-ip-3disks", "source_vm_power": "on", "guest_agent": True, }, ], "warm_migration": True, "target_power_state": "on", "preserve_static_ips": True, "vm_target_namespace": "custom-vm-namespace", "multus_namespace": "default", "pvc_name_template": '{{ .FileName | trimSuffix ".vmdk" | replace "_" "-" }}-{{.DiskIndex}}', "pvc_name_template_use_generate_name": True, "target_labels": { "mtv-comprehensive-test": None, "static-label": "static-value", }, "target_affinity": { "podAffinity": { "preferredDuringSchedulingIgnoredDuringExecution": [ { "podAffinityTerm": { "labelSelector": {"matchLabels": {"app": "comprehensive-test"}}, "topologyKey": "kubernetes.io/hostname", }, "weight": 75, } ] } }, }, ``` This configuration adds several warm-specific checks on top of the baseline `check_vms()` flow: - `preserve_static_ips` triggers static IP validation after cutover - `vm_target_namespace` moves the migrated VM into a custom namespace - `pvc_name_template` enables PVC name verification against the rendered Forklift template - `pvc_name_template_use_generate_name` switches PVC validation to prefix matching, because Kubernetes adds a random suffix - `target_labels` verifies labels on the migrated VM - `target_affinity` verifies the affinity block on the destination VM A few details are easy to miss: - In `target_labels`, a value of `None` means the suite replaces that value with the session UUID at runtime. - When `vm_target_namespace` is set, the fixture creates that namespace if needed before migration. - PVC template rendering supports Go template syntax and Sprig functions in the validation path. > **Note:** Static IP preservation is currently validated only for Windows VMs migrated from vSphere. > **Note:** The `{{.FileName}}` and `{{.DiskIndex}}` PVC template verification path is VMware-specific in the current code. ## Warm Copy-Offload: The Extra Validation Path The warm copy-offload scenario adds one more validation step after the standard warm checks. In `tests/test_copyoffload_migration.py`, the suite first runs the normal post-migration verification and then explicitly verifies disk count: ```python check_vms( plan=prepared_plan, source_provider=source_provider, destination_provider=destination_provider, network_map_resource=self.network_map, storage_map_resource=self.storage_map, source_provider_data=source_provider_data, source_vms_namespace=source_vms_namespace, source_provider_inventory=source_provider_inventory, vm_ssh_connections=vm_ssh_connections, ) verify_vm_disk_count( destination_provider=destination_provider, plan=prepared_plan, target_namespace=target_namespace ) ``` That extra assertion matters for the copy-offload path because it confirms that the migrated VM still has the expected disk inventory after the accelerated transfer flow. The corresponding warm test plan looks like this: ```python "test_copyoffload_warm_migration": { "virtual_machines": [ { "name": "xcopy-template-test", "source_vm_power": "on", "guest_agent": True, "clone": True, "disk_type": "thin", }, ], "warm_migration": True, "copyoffload": True, }, ``` This path is stricter about provider requirements than the standard warm path: - the source provider must be vSphere - the provider configuration must include a `copyoffload` section - the storage credentials and required copy-offload fields must be present > **Warning:** Warm copy-offload is not a generic warm migration path. The fixture fails early unless the source provider is vSphere and the required `copyoffload` settings are configured. ## Provider Configuration For Full Warm Validation If you want the full warm validation path, especially guest-agent and SSH-based checks, your provider entry needs guest VM credentials. The vSphere example in `.providers.json.example` includes them: ```jsonc "vsphere": { "type": "vsphere", "version": "", "fqdn": "SERVER FQDN/IP", "api_url": "/sdk", "username": "USERNAME", "password": "PASSWORD", # pragma: allowlist secret "guest_vm_linux_user": "LINUX VMS USERNAME", "guest_vm_linux_password": "LINUX VMS PASSWORD", # pragma: allowlist secret "guest_vm_win_user": "WINDOWS VMS USERNAME", "guest_vm_win_password": "WINDOWS VMS PASSWORD", # pragma: allowlist secret "vddk_init_image": "" } ``` The warm copy-offload path expects a `copyoffload` subsection under the vSphere provider entry. The example file includes fields such as `storage_vendor_product`, `datastore_id`, `storage_hostname`, `storage_username`, and `storage_password`. > **Note:** The example provider file uses comments for secret-scanning exceptions. Those comments are fine in the example file, but they are not valid in strict JSON. ## Practical Guidance - Start with `test_sanity_warm_mtv_migration` when you want to prove your environment can complete a basic warm cycle. - Move to `test_mtv_migration_warm_2disks2nics` when you want more disk and network coverage without turning on every advanced feature. - Use `test_warm_migration_comprehensive` when you specifically need to validate static IP preservation, PVC naming, target labels, affinity, and custom VM namespace placement. - Use `test_copyoffload_warm_migration` only when your environment is already prepared for vSphere copy-offload. - Increase `mins_before_cutover` if your goal is to observe more precopy work before final downtime. - Keep `target_power_state: "on"` when you want SSH-based post-migration validation to run. --- Source: copy-offload-migrations.md # Copy-Offload Migrations Copy-offload migrations in `mtv-api-tests` cover the VMware-to-OpenShift path where MTV can let the storage array move VM disk data instead of using the standard VDDK copy path. In practice, that means a vSphere source provider, shared storage between vSphere and OpenShift, a `StorageMap` with `offloadPlugin.vsphereXcopyConfig`, and the right storage credentials. This project does more than validate a single happy path. The existing copy-offload coverage includes thin and thick disks, snapshots, RDM, multi-datastore layouts, warm migration, non-XCOPY fallback, VM naming edge cases, scale, and concurrent XCOPY/VDDK execution. ## What You Need - A vSphere source provider. The copy-offload validation in this project fails fast for non-VMware sources. - Shared storage between vSphere and OpenShift. - A block-backed OpenShift storage class. The copy-offload tests map the destination as `ReadWriteOnce` with `Block` volume mode. - MTV installed and healthy. - A cloneable test VM or template with working guest access. Several scenarios power VMs on, wait for guest info, create snapshots, or validate guest connectivity after migration. - A clone method: `vib` or `ssh`. > **Warning:** The repository's copy-offload guidance assumes SAN or block-backed storage. The existing project docs explicitly call out NFS as unsupported for copy-offload scenarios. For older MTV environments, the project docs include this feature-gate example: ```yaml spec: feature_copy_offload: 'true' ``` > **Note:** `mtv-api-tests` does not toggle that feature gate for you. If your MTV version requires it, enable it before running copy-offload scenarios. ## Copy-Offload Provider Config The copy-offload settings live under the VMware provider entry in `.providers.json`. This is the actual `copyoffload` block from `.providers.json.example`: ```jsonc "copyoffload": { # Supported storage_vendor_product values: # - "ontap" (NetApp ONTAP) # - "vantara" (Hitachi Vantara) # - "primera3par" (HPE Primera/3PAR) # - "pureFlashArray" (Pure Storage FlashArray) # - "powerflex" (Dell PowerFlex) # - "powermax" (Dell PowerMax) # - "powerstore" (Dell PowerStore) # - "infinibox" (Infinidat InfiniBox) # - "flashsystem" (IBM FlashSystem) "storage_vendor_product": "ontap", # Primary datastore for copy-offload operations (required) # This is the vSphere datastore ID (e.g., "datastore-12345") where VMs reside # Get via vSphere: Datacenter → Storage → Datastore → Summary → More Objects ID "datastore_id": "datastore-12345", # Optional: Secondary datastore for multi-datastore copy-offload tests # Only needed when testing VMs with disks spanning multiple datastores # When specified, tests can validate copy-offload with disks on different datastores "secondary_datastore_id": "datastore-67890", # Optional: Non-XCOPY datastore for mixed datastore tests # This should be a datastore that does NOT support XCOPY/VAAI primitives # Used for testing VMs with disks on both XCOPY and non-XCOPY datastores "non_xcopy_datastore_id": "datastore-99999", "default_vm_name": "rhel9-template", "storage_hostname": "storage.example.com", "storage_username": "admin", "storage_password": "your-password-here", # pragma: allowlist secret # Vendor-specific fields (configure based on your storage_vendor_product): # IMPORTANT: Only configure the fields for your selected storage_vendor_product. # For example, if storage_vendor_product == "ontap", only configure ontap_svm. # You may leave other vendor-specific fields blank or remove them from your config. # Note: Both datastore_id and secondary_datastore_id (if used) must be on the # same storage array and support XCOPY/VAAI primitives for copy-offload to work. # See forklift vsphere-xcopy-volume-populator code/README for details # NetApp ONTAP (required for "ontap"): "ontap_svm": "vserver-name", # Hitachi Vantara (required for "vantara"): "vantara_storage_id": "123456789", # Storage array serial number "vantara_storage_port": "443", # Storage API port "vantara_hostgroup_id_list": "CL1-A,1:CL2-B,2:CL4-A,1:CL6-A,1", # IO ports and host group IDs # Pure Storage FlashArray (required for "pureFlashArray"): # Get with: printf "px_%.8s" $(oc get storagecluster -A -o=jsonpath='{.items[?(@.spec.cloudStorage.provider=="pure")].status.clusterUid}') "pure_cluster_prefix": "px_a1b2c3d4", # Dell PowerFlex (required for "powerflex"): # Get from vxflexos-config ConfigMap in vxflexos or openshift-operators namespace "powerflex_system_id": "system-id", # Dell PowerMax (required for "powermax"): # Get from ConfigMap in powermax namespace used by CSI driver "powermax_symmetrix_id": "000123456789", # HPE Primera/3PAR, Dell PowerStore, Infinidat InfiniBox, IBM FlashSystem: # No additional vendor-specific fields required - use only the common fields above # ESXi SSH configuration (optional, for SSH-based cloning): # Can be overridden via environment variables: COPYOFFLOAD_ESXI_HOST, COPYOFFLOAD_ESXI_USER, COPYOFFLOAD_ESXI_PASSWORD "esxi_clone_method": "ssh", # "vib" (default) or "ssh" "esxi_host": "your-esxi-host.example.com", # required for ssh method "esxi_user": "root", # required for ssh method "esxi_password": "your-esxi-password", # pragma: allowlist secret # required for ssh method # RDM testing (optional, for RDM disk tests): # Note: datastore_id must be a VMFS datastore for RDM disk support "rdm_lun_uuid": "naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" } ``` The minimum fields that this project validates before running are: - `storage_vendor_product` - `datastore_id` - `storage_hostname` - `storage_username` - `storage_password` Add these when you want advanced scenarios: - `secondary_datastore_id` for multi-datastore tests - `non_xcopy_datastore_id` for mixed and fallback tests - `rdm_lun_uuid` for RDM tests - `default_vm_name` if your copy-offload-ready template differs from the default test data > **Note:** The `# pragma: allowlist secret` comments in `.providers.json.example` are there for repository tooling. They are not valid JSON and must be removed from your real `.providers.json`. `default_vm_name` is especially useful when your environment has a single known-good copy-offload template. The suite applies that override to cloned VM scenarios so you do not have to change every test entry by hand. ## Environment Variable Overrides The repository lets you override any copy-offload credential from the environment, and environment values always win over `.providers.json`: ```python env_var_name = f"COPYOFFLOAD_{credential_name.upper()}" return os.getenv(env_var_name) or copyoffload_config.get(credential_name) ``` That pattern works for the common storage credentials: - `COPYOFFLOAD_STORAGE_HOSTNAME` - `COPYOFFLOAD_STORAGE_USERNAME` - `COPYOFFLOAD_STORAGE_PASSWORD` It also works for vendor-specific and ESXi-specific values such as: - `COPYOFFLOAD_ONTAP_SVM` - `COPYOFFLOAD_VANTARA_HOSTGROUP_ID_LIST` - `COPYOFFLOAD_ESXI_HOST` - `COPYOFFLOAD_ESXI_USER` - `COPYOFFLOAD_ESXI_PASSWORD` > **Tip:** A good pattern is to keep the structural values in `.providers.json` and inject the sensitive values through environment variables at runtime. ## Supported Storage Vendors Use these exact `storage_vendor_product` values. They come directly from the repository's copy-offload constants and secret-mapping logic. | `storage_vendor_product` | Storage platform | Extra required fields | | --- | --- | --- | | `ontap` | NetApp ONTAP | `ontap_svm` | | `vantara` | Hitachi Vantara | `vantara_storage_id`, `vantara_storage_port`, `vantara_hostgroup_id_list` | | `pureFlashArray` | Pure Storage FlashArray | `pure_cluster_prefix` | | `powerflex` | Dell PowerFlex | `powerflex_system_id` | | `powermax` | Dell PowerMax | `powermax_symmetrix_id` | | `powerstore` | Dell PowerStore | None beyond base storage credentials | | `primera3par` | HPE Primera / 3PAR | None beyond base storage credentials | | `infinibox` | Infinidat InfiniBox | None beyond base storage credentials | | `flashsystem` | IBM FlashSystem | None beyond base storage credentials | ## Storage Secrets In `mtv-api-tests`, you usually do not create the copy-offload storage secret by hand. The suite creates it automatically from the VMware provider's `copyoffload` block. That matters because: - The secret values can come from `.providers.json` or environment variables. - The secret is created in the same target namespace where the suite creates the `StorageMap` and `Plan`. - The `offloadPlugin` can reference the secret by name, without extra manual wiring. The suite always creates these base secret keys: - `STORAGE_HOSTNAME` - `STORAGE_USERNAME` - `STORAGE_PASSWORD` It then adds vendor-specific keys such as `ONTAP_SVM`, `STORAGE_ID`, `HOSTGROUP_ID_LIST`, `PURE_CLUSTER_PREFIX`, `POWERFLEX_SYSTEM_ID`, or `POWERMAX_SYMMETRIX_ID`, depending on `storage_vendor_product`. After the plan is created, the suite also waits for Forklift to create the plan-specific secret used during copy-offload. If that secret never appears, the run continues long enough to fail with a clearer migration error. > **Note:** This automatic secret handling is specific to how `mtv-api-tests` drives copy-offload. It removes a lot of manual setup from the test workflow. ## StorageMap and Plan Behavior The core of copy-offload in this project is the `offloadPlugin` block. The tests build it like this: ```python offload_plugin_config = { "vsphereXcopyConfig": { "secretRef": copyoffload_storage_secret.name, "storageVendorProduct": storage_vendor_product, } } ``` The storage map entries then attach that plugin to the source datastore mapping and set the destination for block-backed PVCs: ```python storage_map_list.append({ "destination": { "storageClass": target_storage_class, "accessMode": "ReadWriteOnce", "volumeMode": "Block", }, "source": {"id": ds_id}, "offloadPlugin": offload_plugin_config, }) ``` The project also changes plan behavior for copy-offload runs: ```python if copyoffload: plan_kwargs["pvc_name_template"] = "pvc" plan = create_and_store_resource(**plan_kwargs) if copyoffload: wait_for_plan_secret(ocp_admin_client, target_namespace, plan.name) ``` And when the suite creates the VMware `Provider` for copy-offload, it adds this annotation: ```python provider_annotations["forklift.konveyor.io/empty-vddk-init-image"] = "yes" ``` That is the repository's way of steering the provider toward the copy-offload path instead of a VDDK-only setup. The difference is explicit in the concurrent XCOPY/VDDK scenario: the XCOPY `StorageMap` must contain `offloadPlugin`, and the VDDK `StorageMap` must not. > **Tip:** If you want a fast sanity check that your environment is wired correctly, start with `test_copyoffload_thin_migration`. It uses the same `offloadPlugin` structure as the advanced scenarios, but with fewer moving parts. ## Clone Methods The suite supports both copy-offload clone methods exposed by the populator: `vib` and `ssh`. ### VIB `vib` is the default. If you omit `esxi_clone_method`, the repository leaves the provider's clone method alone and relies on the default VIB behavior. Use `vib` when: - Your ESXi environment allows community-level VIB installation. - You do not want the suite to manage ESXi SSH credentials. > **Note:** The repository does not perform extra VIB-specific setup. It assumes the populator and ESXi host permissions are already ready for the VIB path. ### SSH If you set `esxi_clone_method` to `ssh`, the suite patches the VMware `Provider` so MTV uses SSH-based cloning: ```python patch = {"spec": {"settings": {"esxiCloneMethod": clone_method}}} ResourceEditor(patches={self.ocp_resource: patch}).update() ``` It then retrieves the provider-generated public key from the `offload-ssh-keys--public` secret and installs a restricted key on the ESXi host. The restricted command is taken directly from the ESXi helper: ```python command_template = ( 'command="python /vmfs/volumes/{datastore_name}/secure-vmkfstools-wrapper.py",' "no-port-forwarding,no-agent-forwarding,no-X11-forwarding {public_key}" ) ``` For SSH mode, you must provide: - `esxi_host` - `esxi_user` - `esxi_password` > **Warning:** SSH mode temporarily updates `/etc/ssh/keys-root/authorized_keys` on the ESXi host. The suite removes the key during teardown, but it is still a real host-side change. > **Tip:** SSH mode is a good choice when you want a fully test-managed setup path. The suite handles the provider patch, key installation, and cleanup for you. ## Fallback Modes Copy-offload is not all-or-nothing in this repository. The existing tests explicitly cover cases where some or all disks live on a datastore that does not support XCOPY/VAAI. There are two main fallback patterns: - Mixed-datastore fallback: one disk uses an XCOPY-capable datastore and another disk lives on `non_xcopy_datastore_id`. - Full non-XCOPY fallback: the VM is relocated to `non_xcopy_datastore_id`, and added disks are placed there too. The storage-map helper keeps the `offloadPlugin` on the non-XCOPY mapping so Forklift can exercise fallback behavior: ```python storage_map_list.append({ "destination": destination_config, "source": {"id": non_xcopy_datastore_id}, "offloadPlugin": offload_plugin_config, }) ``` The full fallback case is modeled directly in the repository like this: ```python "test_copyoffload_fallback_large_migration": { "virtual_machines": [ { "name": "xcopy-template-test", "source_vm_power": "off", "guest_agent": True, "clone": True, "target_datastore_id": "non_xcopy_datastore_id", "disk_type": "thin", "add_disks": [ { "size_gb": 100, "provision_type": "thin", "datastore_id": "non_xcopy_datastore_id", }, ], }, ], "warm_migration": False, "copyoffload": True, } ``` > **Warning:** `non_xcopy_datastore_id` must point to a real datastore that does not support XCOPY/VAAI. If it is missing, the mixed and fallback scenarios fail fast before migration starts. ## Advanced Copy-Offload Scenarios The copy-offload test matrix in this repository is broader than the basic thin-disk path. | Scenario | What it validates | Key test name(s) | | --- | --- | --- | | Basic provisioning | Thin and thick-lazy disk copy-offload | `test_copyoffload_thin_migration`, `test_copyoffload_thick_lazy_migration` | | Multi-disk layouts | Additional disks on the same datastore | `test_copyoffload_multi_disk_migration` | | Custom datastore paths | Extra disks placed under a custom folder like `shared_disks` | `test_copyoffload_multi_disk_different_path_migration` | | Multi-datastore | A VM with disks spanning primary and secondary XCOPY datastores | `test_copyoffload_multi_datastore_migration` | | Mixed XCOPY / non-XCOPY | Some disks accelerate while others fall back | `test_copyoffload_mixed_datastore_migration` | | Full fallback on non-XCOPY | Large VM and added disk entirely on a non-XCOPY datastore | `test_copyoffload_fallback_large_migration` | | RDM | RDM virtual-disk migration using `rdm_lun_uuid` | `test_copyoffload_rdm_virtual_disk_migration` | | Snapshots | Source snapshots before migration, including a 2 TB case | `test_copyoffload_thin_snapshots_migration`, `test_copyoffload_2tb_vm_snapshots_migration` | | Disk modes | Independent persistent and independent nonpersistent disks | `test_copyoffload_independent_persistent_disk_migration`, `test_copyoffload_independent_nonpersistent_disk_migration` | | Large and dense VMs | 1 TB VM and 10 mixed thin/thick disks | `test_copyoffload_large_vm_migration`, `test_copyoffload_10_mixed_disks_migration` | | Warm copy-offload | Warm migration with cutover | `test_copyoffload_warm_migration` | | Name edge cases | Non-Kubernetes VM names with uppercase letters and underscores | `test_copyoffload_nonconforming_name_migration` | | Scale | Five copy-offload VMs in one run | `test_copyoffload_scale_migration` | | Concurrency | Two copy-offload plans at once | `test_simultaneous_copyoffload_migrations` | | XCOPY vs VDDK | One copy-offload plan and one standard VDDK plan running together | `test_concurrent_xcopy_vddk_migration` | The repository's multi-datastore scenario uses a symbolic secondary datastore key instead of hardcoding the MoID into every disk entry: ```python "test_copyoffload_multi_datastore_migration": { "virtual_machines": [ { "name": "xcopy-template-test", "source_vm_power": "off", "guest_agent": True, "clone": True, "disk_type": "thin", "add_disks": [ { "size_gb": 30, "disk_mode": "persistent", "provision_type": "thin", "datastore_id": "secondary_datastore_id", }, ], }, ], "warm_migration": False, "copyoffload": True, } ``` The non-conforming-name scenario is also deliberate. Its source config preserves uppercase letters and underscores in the cloned VMware name, and the suite then verifies that MTV sanitizes the destination VM name to a Kubernetes-safe value. > **Note:** The project docs currently call out RDM copy-offload support only for Pure Storage, and they require `datastore_id` to be a VMFS datastore for RDM scenarios. ## What the Suite Automates for You When you run copy-offload scenarios through `mtv-api-tests`, the project handles several setup steps automatically: - It validates that the source provider is vSphere and that the `copyoffload` section exists. - It creates the copy-offload storage secret from `.providers.json` or environment variables. - It creates `StorageMap` entries with `offloadPlugin.vsphereXcopyConfig`. - It annotates the VMware provider with `forklift.konveyor.io/empty-vddk-init-image: "yes"`. - It patches `esxiCloneMethod` to `ssh` when you choose SSH cloning. - It waits for Forklift to create the plan-specific copy-offload secret. - It forces `pvc_name_template` to `pvc` for copy-offload plans. That is why the user-facing setup is mostly about getting the provider config, datastore IDs, storage array credentials, and clone method right. ## Running the Copy-Offload Marker The repository's copy-offload guide uses the `copyoffload` pytest marker. This command is taken directly from the existing Job example: ```bash uv run pytest -m copyoffload \ -v \ ${CLUSTER_HOST:+--tc=cluster_host:${CLUSTER_HOST}} \ ${CLUSTER_USERNAME:+--tc=cluster_username:${CLUSTER_USERNAME}} \ ${CLUSTER_PASSWORD:+--tc=cluster_password:${CLUSTER_PASSWORD}} \ --tc=source_provider:vsphere-8.0.3.00400 \ --tc=storage_class:my-block-storageclass ``` Replace the provider key with the exact key from your `.providers.json`, and replace the storage class with the block-backed class that maps to the same storage array as your vSphere datastores. > **Tip:** Start with `test_copyoffload_thin_migration`, then move to `test_copyoffload_multi_datastore_migration`, `test_copyoffload_mixed_datastore_migration`, or `test_concurrent_xcopy_vddk_migration` once the base path is working. --- Source: remote-openshift-migrations.md # Remote OpenShift Migrations Remote OpenShift migrations in `mtv-api-tests` use the same MTV lifecycle as the default OpenShift destination flow. The suite still creates a `StorageMap`, creates a `NetworkMap`, creates a `Plan`, executes the migration, and validates the migrated VMs. If you already know the default local-destination flow, the easiest way to understand the remote path is this: the test flow stays the same, but the destination OpenShift provider changes from the implicit local form to an explicit provider that includes an API URL and token-backed secret. ## What "remote" means in this repository The repository exposes remote scenarios as dedicated pytest classes marked with `remote`. Those classes live in `tests/test_mtv_warm_migration.py` and `tests/test_mtv_cold_migration.py`, and they are only enabled when `remote_ocp_cluster` is set. ```python @pytest.mark.remote @pytest.mark.incremental @pytest.mark.parametrize( "class_plan_config", [ pytest.param( py_config["tests_params"]["test_warm_remote_ocp"], ) ], indirect=True, ids=["MTV-394"], ) @pytest.mark.skipif(not get_value_from_py_config("remote_ocp_cluster"), reason="No remote OCP cluster provided") @pytest.mark.usefixtures("precopy_interval_forkliftcontroller", "cleanup_migrated_vms") class TestWarmRemoteOcp: """Warm remote OCP migration test.""" ``` The repository’s `pytest.ini` registers the `remote` marker and loads default config from `tests/tests_config/config.py`. > **Note:** In this codebase, remote scenarios do not introduce a second checked-in cluster credential set. The active OpenShift connection still comes from the standard cluster settings. ## Destination Provider Handling ### Default local destination flow The default destination fixture in `conftest.py` creates an OpenShift `Provider` with an empty `url` and an empty `secret` block. That is the local, in-cluster destination path used by the non-remote tests. ```python @pytest.fixture(scope="session") def destination_provider(session_uuid, ocp_admin_client, target_namespace, fixture_store): kind_dict = { "apiVersion": "forklift.konveyor.io/v1beta1", "kind": "Provider", "metadata": {"name": f"{session_uuid}-local-ocp-provider", "namespace": target_namespace}, "spec": {"secret": {}, "type": "openshift", "url": ""}, } provider = create_and_store_resource( fixture_store=fixture_store, resource=Provider, kind_dict=kind_dict, client=ocp_admin_client, ) return OCPProvider(ocp_resource=provider, fixture_store=fixture_store) ``` ### Remote destination flow The remote path creates a secret from the active OpenShift API token and then creates an explicit OpenShift `Provider` with a real API URL. ```python @pytest.fixture(scope="session") def destination_ocp_secret(fixture_store, ocp_admin_client, target_namespace): api_key: str = ocp_admin_client.configuration.api_key.get("authorization") if not api_key: raise ValueError("API key not found in configuration") secret = create_and_store_resource( client=ocp_admin_client, fixture_store=fixture_store, resource=Secret, namespace=target_namespace, # API key format: 'Bearer sha256~', split it to get token. string_data={"token": api_key.split()[-1], "insecureSkipVerify": "true"}, ) yield secret @pytest.fixture(scope="session") def destination_ocp_provider(fixture_store, destination_ocp_secret, ocp_admin_client, session_uuid, target_namespace): provider = create_and_store_resource( client=ocp_admin_client, fixture_store=fixture_store, resource=Provider, name=f"{session_uuid}-destination-ocp-provider", namespace=target_namespace, secret_name=destination_ocp_secret.name, secret_namespace=destination_ocp_secret.namespace, url=ocp_admin_client.configuration.host, provider_type=Provider.ProviderType.OPENSHIFT, ) yield OCPProvider(ocp_resource=provider, fixture_store=fixture_store) ``` For users, the practical difference is straightforward: - Local destination tests use the implicit local OpenShift provider form. - Remote destination tests use an explicit OpenShift provider with a token and API URL. - The destination provider is created automatically at runtime. You do not define it in `.providers.json`. > **Note:** The destination secret hardcodes `insecureSkipVerify: "true"` for the remote OpenShift provider. There is no separate destination-side SSL verification toggle in `tests/tests_config/config.py`. ## Remote-Specific Configuration The shared config file already contains the remote toggle and the runtime knobs that matter to remote runs: ```python insecure_verify_skip: str = "true" # SSL verification for OCP API connections source_provider_insecure_skip_verify: str = "false" # SSL verification for source provider (VMware, RHV, etc.) number_of_vms: int = 1 check_vms_signals: bool = True target_namespace_prefix: str = "auto" mtv_namespace: str = "openshift-mtv" vm_name_search_pattern: str = "" remote_ocp_cluster: str = "" snapshots_interval: int = 2 mins_before_cutover: int = 5 plan_wait_timeout: int = 3600 ``` > **Note:** `remote_ocp_cluster` is empty by default. If you do not override it, the remote classes are skipped. Remote runs still use the same cluster connection inputs as the default flow. The OpenShift client is built from `cluster_host`, `cluster_username`, and `cluster_password` in `utilities/utils.py`. ```python def get_cluster_client() -> DynamicClient: """Get a DynamicClient for the cluster. Returns: DynamicClient: The cluster client. Raises: ValueError: If the client cannot be created. """ host = get_value_from_py_config("cluster_host") username = get_value_from_py_config("cluster_username") password = get_value_from_py_config("cluster_password") insecure_verify_skip = get_value_from_py_config("insecure_verify_skip") _client = get_client(host=host, username=username, password=password, verify_ssl=not insecure_verify_skip) if isinstance(_client, DynamicClient): return _client raise ValueError("Failed to get client for cluster") ``` Treat the inputs like this: - `cluster_host`, `cluster_username`, `cluster_password`: identify and authenticate to the OpenShift cluster used by the test session. - `remote_ocp_cluster`: enables the remote test classes and checks that `cluster_host` points at the expected cluster. - `source_provider`: still selects the source side from `.providers.json`. - `storage_class`: still controls where migrated VM disks land. - `snapshots_interval`, `mins_before_cutover`, and `plan_wait_timeout`: still apply because remote runs share the same warm/cold migration helpers. > **Warning:** `remote_ocp_cluster` is both a gate and a sanity check. In `conftest.py`, the session fails early if the configured value does not appear in the connected API host. ## The Migration Path After Provider Selection Once the suite has picked the destination provider, the rest of the path is shared. The same helper code in `utilities/mtv_migration.py` threads the selected destination provider into the `Plan`. ```python plan_kwargs: dict[str, Any] = { "client": ocp_admin_client, "fixture_store": fixture_store, "resource": Plan, "namespace": target_namespace, "source_provider_name": source_provider.ocp_resource.name, "source_provider_namespace": source_provider.ocp_resource.namespace, "destination_provider_name": destination_provider.ocp_resource.name, "destination_provider_namespace": destination_provider.ocp_resource.namespace, "storage_map_name": storage_map.name, "storage_map_namespace": storage_map.namespace, "network_map_name": network_map.name, "network_map_namespace": network_map.namespace, "virtual_machines_list": virtual_machines_list, "target_namespace": vm_target_namespace or target_namespace, "warm_migration": warm_migration, ``` That is why remote scenarios behave so much like local ones: - The same helper creates the `Plan`. - The same `execute_migration()` function creates the `Migration` CR and waits for completion. - The same `check_vms()` validation path is used afterward. This is the most useful mental model for users: in `mtv-api-tests`, remote OpenShift migration is mainly a destination-provider swap, not a separate migration architecture. ## Warm and Cold Remote Scenarios The checked-in remote coverage is currently split into two classes: - `TestWarmRemoteOcp` - `TestColdRemoteOcp` The warm remote path keeps the normal warm-migration behavior: - It uses `precopy_interval_forkliftcontroller`. - It uses `get_cutover_value()`, which is driven by `mins_before_cutover`. - It uses the same post-migration validation flow as the local warm path. > **Warning:** Remote warm migration does not bypass the repository’s normal warm-migration limits. In `tests/test_mtv_warm_migration.py`, warm tests are still skipped for `openstack`, `openshift`, and `ova` source providers. The cold remote path also stays close to the default flow: it creates the same resources, executes the same migration helper, and runs the same validation logic after completion. ## How Remote Differs From the Default Local Destination Flow | Area | Default local destination flow | Remote OpenShift flow | | --- | --- | --- | | Test selection | Standard classes | `@pytest.mark.remote` classes | | Config gate | None | `remote_ocp_cluster` must be set | | Destination provider | Local OpenShift provider with empty `url` and empty `secret` | Explicit OpenShift provider with API `url` and token-backed `Secret` | | Cluster connection | `cluster_host`, `cluster_username`, `cluster_password` | Same values | | Source provider config | `.providers.json` | Same `.providers.json` source config | | Plan / StorageMap / NetworkMap helpers | Shared | Same shared helpers | | Warm behavior | Shared warm logic | Same shared warm logic | > **Tip:** When you troubleshoot or document a remote scenario, start with provider creation and config values first. Most of the remote-specific behavior is in destination provider setup, not in the migration execution steps. ## Automation and Job Patterns This repository does not include a dedicated remote-specific pipeline, workflow, or job file. The closest checked-in automation example is the OpenShift Job pattern in `docs/copyoffload/how-to-run-copyoffload-tests.md`, which shows how the project passes cluster settings into `pytest`. ```bash uv run pytest -m copyoffload \ -v \ ${CLUSTER_HOST:+--tc=cluster_host:${CLUSTER_HOST}} \ ${CLUSTER_USERNAME:+--tc=cluster_username:${CLUSTER_USERNAME}} \ ${CLUSTER_PASSWORD:+--tc=cluster_password:${CLUSTER_PASSWORD}} \ --tc=source_provider:vsphere-8.0.3.00400 \ --tc=storage_class:my-block-storageclass ``` That example is not remote-specific, but it shows the exact configuration pattern the project uses: - runtime values are injected with `--tc=...` - source-side selection still happens with `source_provider` - cluster-side selection still happens with `cluster_host`, `cluster_username`, and `cluster_password` For remote scenarios, the extra remote-specific value is `remote_ocp_cluster`, and test selection happens through the `remote` marker rather than a special runner script. ## Practical Guidance - Keep `.providers.json` focused on the source provider. The destination OpenShift provider is created by fixtures in `conftest.py`. - Treat `remote_ocp_cluster` as both an enable switch and a hostname sanity check. - Expect remote warm and remote cold runs to behave like their local equivalents after the destination provider has been created. - If you need to automate remote runs, follow the repository’s existing `--tc=` pattern rather than looking for a separate remote-only pipeline in this checkout. In short, the remote OpenShift path in `mtv-api-tests` keeps the normal MTV test lifecycle but replaces the implicit local destination provider with an explicit OpenShift provider built from the active cluster’s API endpoint and token. --- Source: hooks-and-expected-failures.md # Hooks And Expected Failures Hooks let you run Ansible logic before or after an MTV migration. In this repository, you do not hand-craft `Hook` resources yourself. You describe the hook in the plan configuration, and the test suite creates the `Hook` resource, attaches it to the `Plan`, and validates the outcome. The project supports two hook modes: - predefined playbooks selected with `expected_result` - custom playbooks supplied through `playbook_base64` When you intentionally test a failure, the suite validates more than “migration failed.” It checks whether the failure happened in `PreHook` or `PostHook`, and it changes later VM validation based on that result. ## Where Hook Configuration Lives Hook settings live in `tests/tests_config/config.py`. The repository’s end-to-end hook example looks like this: ```503:516:tests/tests_config/config.py "test_post_hook_retain_failed_vm": { "virtual_machines": [ { "name": "mtv-tests-rhel8", "source_vm_power": "on", "guest_agent": True, }, ], "warm_migration": False, "target_power_state": "off", "pre_hook": {"expected_result": "succeed"}, "post_hook": {"expected_result": "fail"}, "expected_migration_result": "fail", }, ``` This config uses two different “expected” keys: | Key | What it controls | | --- | --- | | `pre_hook.expected_result` / `post_hook.expected_result` | Chooses a built-in hook playbook: `succeed` or `fail`. | | `expected_migration_result` | Tells the test whether the overall migration should raise `MigrationPlanExecError`. | Use `pre_hook` when you want to affect the migration before VM work begins. Use `post_hook` when you want the migration to reach the end of VM processing and then test what happens after that. > **Note:** `expected_result` and `expected_migration_result` are not interchangeable. The first controls hook behavior. The second controls the expected result of the entire migration test. ## How Hooks Get Attached To A Plan During `prepared_plan`, the suite checks the plan for `pre_hook` and `post_hook`. If either exists, it creates the corresponding `Hook` resource and stores the generated name and namespace back into the plan. ```306:334:utilities/hooks.py def create_hook_if_configured( plan: dict[str, Any], hook_key: str, hook_type: str, fixture_store: dict[str, Any], ocp_admin_client: "DynamicClient", target_namespace: str, ) -> None: """Create hook if configured in plan and store references. ... """ hook_config = plan.get(hook_key) if hook_config: hook_name, hook_namespace = create_hook_for_plan( hook_config=hook_config, hook_type=hook_type, fixture_store=fixture_store, ocp_admin_client=ocp_admin_client, target_namespace=target_namespace, ) plan[f"_{hook_type}_hook_name"] = hook_name plan[f"_{hook_type}_hook_namespace"] = hook_namespace ``` When the suite creates the MTV `Plan`, it passes those hook references into the Plan helper. Pre-hooks use `pre_hook_name` and `pre_hook_namespace`. Post-hooks are passed through the helper as `after_hook_name` and `after_hook_namespace`. ```200:223:utilities/mtv_migration.py plan_kwargs: dict[str, Any] = { "client": ocp_admin_client, "fixture_store": fixture_store, "resource": Plan, "namespace": target_namespace, "source_provider_name": source_provider.ocp_resource.name, "source_provider_namespace": source_provider.ocp_resource.namespace, "destination_provider_name": destination_provider.ocp_resource.name, "destination_provider_namespace": destination_provider.ocp_resource.namespace, "storage_map_name": storage_map.name, "storage_map_namespace": storage_map.namespace, "network_map_name": network_map.name, "network_map_namespace": network_map.namespace, "virtual_machines_list": virtual_machines_list, "target_namespace": vm_target_namespace or target_namespace, "warm_migration": warm_migration, "pre_hook_name": pre_hook_name, "pre_hook_namespace": pre_hook_namespace, "after_hook_name": after_hook_name, "after_hook_namespace": after_hook_namespace, "preserve_static_ips": preserve_static_ips, "pvc_name_template": pvc_name_template, "pvc_name_template_use_generate_name": pvc_name_template_use_generate_name, "target_power_state": target_power_state, } ``` > **Note:** You do not configure hook resource names manually in the test config. The suite generates them and stores them as `_pre_hook_name`, `_pre_hook_namespace`, `_post_hook_name`, and `_post_hook_namespace`. > **Note:** Hook resources are created in the migration namespace passed as `target_namespace`. If you also use `vm_target_namespace`, that changes where migrated VMs land, not where the hook CR itself is created. ## Predefined Playbooks If you set `expected_result`, the suite chooses one of two built-in Ansible playbooks stored as base64 strings in `utilities/hooks.py`. The file includes their decoded content in comments: ```29:43:utilities/hooks.py # HOOK_PLAYBOOK_SUCCESS decodes to: # - name: Successful-hook # hosts: localhost # tasks: # - name: Success task # debug: # msg: "Hook executed successfully" # # HOOK_PLAYBOOK_FAIL decodes to: # - name: Failing-post-migration # hosts: localhost # tasks: # - name: Task that will fail # fail: # msg: "This hook is designed to fail for testing purposes" ``` In other words: - `expected_result: succeed` uses a simple playbook that logs a debug message. - `expected_result: fail` uses a playbook that fails on purpose. This makes predefined mode the easiest way to test hook behavior without having to build and base64-encode your own playbook. > **Tip:** If your goal is to verify that a migration fails specifically in `PreHook` or `PostHook`, predefined playbooks are the clearest option because the test can compare the actual failed step against a declared expectation. ## Custom Playbooks If the built-in success and failure playbooks are not enough, you can provide your own playbook in `playbook_base64`. Before the suite creates the hook, it enforces several validation rules: ```74:130:utilities/hooks.py expected_result = hook_config.get("expected_result") custom_playbook = hook_config.get("playbook_base64") # Validate mutual exclusivity if expected_result is not None and custom_playbook is not None: raise ValueError( f"Invalid {hook_type} hook config: 'expected_result' and 'playbook_base64' are " f"mutually exclusive. Use 'expected_result' for predefined playbooks, or " f"'playbook_base64' for custom playbooks." ) if expected_result is None and custom_playbook is None: raise ValueError( f"Invalid {hook_type} hook config: must specify either 'expected_result' or 'playbook_base64'." ) # Reject empty strings for both expected_result and custom_playbook if isinstance(expected_result, str) and expected_result.strip() == "": raise ValueError(f"Invalid {hook_type} hook config: 'expected_result' cannot be empty or whitespace-only.") if isinstance(custom_playbook, str) and custom_playbook.strip() == "": raise ValueError(f"Invalid {hook_type} hook config: 'playbook_base64' cannot be empty or whitespace-only.") ... # Validate base64 encoding try: decoded = base64.b64decode(playbook_base64, validate=True) except binascii.Error as e: raise ValueError(f"Invalid {hook_type} hook playbook_base64: not valid base64 encoding. Error: {e}") from e ... # Validate Ansible playbook structure (must be a non-empty list) if not isinstance(playbook_data, list) or not playbook_data: raise ValueError( f"Invalid {hook_type} hook playbook_base64: Ansible playbook must be a non-empty list of plays" ) ``` A custom hook payload must therefore be: - valid base64 - valid UTF-8 after decoding - valid YAML - a non-empty list of plays > **Warning:** `expected_result` and `playbook_base64` are mutually exclusive. You must supply exactly one of them for each hook. > **Warning:** Passing validation only proves the payload is structurally valid. It does not guarantee the playbook will succeed at runtime. > **Note:** This repository has an end-to-end example for predefined hooks, but it does not currently include a dedicated sample test case that uses `playbook_base64`. ## How Expected Failures Are Validated A correct hook-failure scenario does not show up as a broken test in this suite. The migration itself fails, but the pytest test passes because that failure was expected and was validated. The existing hook test does that explicitly. If `expected_migration_result` is `fail`, it expects `execute_migration()` to raise `MigrationPlanExecError`, and then it asks the hook utility whether VM checks should still run. ```195:246:tests/test_post_hook_retain_failed_vm.py expected_result = prepared_plan["expected_migration_result"] if expected_result == "fail": with pytest.raises(MigrationPlanExecError): execute_migration( ocp_admin_client=ocp_admin_client, fixture_store=fixture_store, plan=self.plan_resource, target_namespace=target_namespace, ) self.__class__.should_check_vms = validate_hook_failure_and_check_vms(self.plan_resource, prepared_plan) else: execute_migration( ocp_admin_client=ocp_admin_client, fixture_store=fixture_store, plan=self.plan_resource, target_namespace=target_namespace, ) self.__class__.should_check_vms = True ... # Runtime skip needed - decision based on previous test's migration execution result if not self.__class__.should_check_vms: pytest.skip("Skipping VM checks - hook failed before VM migration") ``` The suite then looks deeper than the high-level `Plan` status. It locates the related `Migration` resource and scans each VM’s pipeline for the first step with an error. ```55:99:utilities/mtv_migration.py def _get_failed_migration_step(plan: Plan, vm_name: str) -> str: """Get step where VM migration failed. Examines the Migration status (not Plan) to find which pipeline step failed. The Migration CR contains the detailed VM pipeline execution status. """ migration = _find_migration_for_plan(plan) if not hasattr(migration.instance, "status") or not migration.instance.status: raise MigrationStatusError(migration_name=migration.name) vms_status = getattr(migration.instance.status, "vms", None) if not vms_status: raise MigrationStatusError(migration_name=migration.name) for vm_status in vms_status: vm_id = getattr(vm_status, "id", "") vm_status_name = getattr(vm_status, "name", "") if vm_name not in (vm_id, vm_status_name): continue pipeline = getattr(vm_status, "pipeline", None) if not pipeline: raise VmPipelineError(vm_name=vm_name) for step in pipeline: step_error = getattr(step, "error", None) if step_error: step_name = step.name LOGGER.info(f"VM {vm_name} failed at step '{step_name}': {step_error}") return step_name ``` Once it knows the actual failed step, the hook utility validates it against the configured hook and decides what to do next: ```223:303:utilities/hooks.py def validate_expected_hook_failure( actual_failed_step: str, plan_config: dict[str, Any], ) -> None: """ Validate the actual failed step matches expected (predefined mode only). For custom playbook mode (no expected_result set), this is a no-op. """ # Extract hook configs with type validation pre_hook_config = plan_config.get("pre_hook") if pre_hook_config is not None and not isinstance(pre_hook_config, dict): raise TypeError(f"pre_hook must be a dict, got {type(pre_hook_config).__name__}") pre_hook_expected = pre_hook_config.get("expected_result") if pre_hook_config else None post_hook_config = plan_config.get("post_hook") if post_hook_config is not None and not isinstance(post_hook_config, dict): raise TypeError(f"post_hook must be a dict, got {type(post_hook_config).__name__}") post_hook_expected = post_hook_config.get("expected_result") if post_hook_config else None # PreHook runs before PostHook, so check PreHook first if pre_hook_expected == "fail": expected_step = "PreHook" elif post_hook_expected == "fail": expected_step = "PostHook" else: LOGGER.info("No expected_result specified - skipping step validation") return if actual_failed_step != expected_step: raise AssertionError( f"Migration failed at step '{actual_failed_step}' but expected to fail at '{expected_step}'" ) LOGGER.info("Migration correctly failed at expected step '%s'", expected_step) def validate_hook_failure_and_check_vms( plan_resource: "Plan", prepared_plan: dict[str, Any], ) -> bool: ... if actual_failed_step == "PostHook": return True elif actual_failed_step == "PreHook": return False else: raise ValueError(f"Unexpected failure step: {actual_failed_step}") ``` That leads to a simple rule set: - If the actual failed step matches the expected hook, the expected-failure test passes. - If the actual failed step is wrong, the test fails. - If the failure happened in `PreHook`, VM checks are skipped because migration stopped too early. - If the failure happened in `PostHook`, VM checks still run because the VMs should already exist. > **Warning:** For multi-VM plans, the suite expects all VMs to fail in the same step. If different VMs fail in different pipeline steps, validation fails with `VmMigrationStepMismatchError`. > **Note:** In custom-playbook mode, the suite still determines whether the failure happened in `PreHook` or `PostHook`. What it skips is the comparison against a declared expected step, because custom mode does not use `expected_result`. ## Pytest And Reporting Behavior These hook tests use the repository’s standard incremental class pattern. If an earlier step in the class fails unexpectedly, later steps are marked `xfail` instead of running anyway. ```123:180:conftest.py # Incremental test support - track failures for class-based tests if "incremental" in item.keywords and rep.when == "call" and rep.failed: item.parent._previousfailed = item ... def pytest_runtest_setup(item): # Incremental test support - xfail if previous test in class failed if "incremental" in item.keywords: previousfailed = getattr(item.parent, "_previousfailed", None) if previousfailed is not None: pytest.xfail(f"previous test failed ({previousfailed.name})") ``` This is different from hook expected-failure validation: - `pytest.xfail()` is used here to stop later class steps after an unexpected earlier failure. - Hook expected failures are validated explicitly with `pytest.raises(MigrationPlanExecError)` plus step checking. > **Tip:** If a hook failure is part of the test’s intended behavior, model it with `expected_migration_result: fail` and hook-step validation, not with `pytest.xfail()`. The test runner is also configured to write `junit-report.xml`, so correctly validated hook-failure scenarios appear as normal passed or skipped test steps in standard reporting. The migration failed, but the test did exactly what it was supposed to do. --- Source: advanced-plan-features.md # Advanced Plan Features Advanced plan options let you control how MTV creates the destination VM after the basic provider, storage, and network mappings are in place. In this project, you define those options under `tests_params` in `tests/tests_config/config.py`, then pass them into `create_plan_resource()` from your test class. | Key | What it controls | Example | | --- | --- | --- | | `target_power_state` | Final destination VM power state | `"on"` | | `preserve_static_ips` | Preserve guest static IP configuration | `True` | | `pvc_name_template` | Destination PVC naming pattern | `"{{.VmName}}-disk-{{.DiskIndex}}"` | | `pvc_name_template_use_generate_name` | Let Kubernetes append a random suffix | `True` | | `vm_target_namespace` | Namespace where migrated VMs are created | `"custom-vm-namespace"` | | `target_node_selector` | Node labels the VM should be scheduled onto | `{"mtv-comprehensive-node": None}` | | `target_labels` | Labels added to the migrated VM template | `{"static-label": "static-value"}` | | `target_affinity` | Affinity rules applied to the migrated VM | `{"podAffinity": {...}}` | | `multus_namespace` | Namespace where extra NADs are created | `"default"` | ## Typical usage A full example already exists in `tests/tests_config/config.py`: ```python "test_warm_migration_comprehensive": { "virtual_machines": [ { "name": "mtv-win2022-ip-3disks", "source_vm_power": "on", "guest_agent": True, }, ], "warm_migration": True, "target_power_state": "on", "preserve_static_ips": True, "vm_target_namespace": "custom-vm-namespace", "multus_namespace": "default", # Cross-namespace NAD access "pvc_name_template": '{{ .FileName | trimSuffix ".vmdk" | replace "_" "-" }}-{{.DiskIndex}}', "pvc_name_template_use_generate_name": True, "target_labels": { "mtv-comprehensive-test": None, # None = auto-generate with session_uuid "static-label": "static-value", }, "target_affinity": { "podAffinity": { "preferredDuringSchedulingIgnoredDuringExecution": [ { "podAffinityTerm": { "labelSelector": {"matchLabels": {"app": "comprehensive-test"}}, "topologyKey": "kubernetes.io/hostname", }, "weight": 75, } ] } }, }, ``` That configuration is then passed into the Plan helper in `tests/test_warm_migration_comprehensive.py`: ```python self.__class__.plan_resource = create_plan_resource( ocp_admin_client=ocp_admin_client, fixture_store=fixture_store, source_provider=source_provider, destination_provider=destination_provider, storage_map=self.storage_map, network_map=self.network_map, virtual_machines_list=prepared_plan["virtual_machines"], target_power_state=prepared_plan["target_power_state"], target_namespace=target_namespace, warm_migration=prepared_plan["warm_migration"], preserve_static_ips=prepared_plan["preserve_static_ips"], vm_target_namespace=prepared_plan["vm_target_namespace"], pvc_name_template=prepared_plan["pvc_name_template"], pvc_name_template_use_generate_name=prepared_plan["pvc_name_template_use_generate_name"], target_labels=target_vm_labels["vm_labels"], target_affinity=prepared_plan["target_affinity"], ) ``` ## Target power state Use `target_power_state` when you want the migrated VM to end in a known power state, regardless of how the source VM started out. The repository examples use both `"on"` and `"off"`. If you omit `target_power_state`, the post-migration check falls back to the VM's pre-migration source power state. > **Note:** `source_vm_power` and `target_power_state` are different. `source_vm_power` controls how the source VM is prepared before migration. `target_power_state` controls the expected state of the destination VM after migration. Practical examples from `tests/tests_config/config.py` include: - `"target_power_state": "on"` in both comprehensive migration tests - `"target_power_state": "off"` in `test_post_hook_retain_failed_vm` ## Static IP preservation Set `preserve_static_ips` to `True` when you want MTV to preserve guest static IP settings. Both comprehensive migration configs enable it. For vSphere sources, the project records guest IP origin and treats `origin == "manual"` as a static IP: ```python if hasattr(ip_info, "origin"): ip_config["ip_origin"] = ip_info.origin ip_config["is_static_ip"] = ip_info.origin == "manual" LOGGER.info( f"VM {vm.name} NIC {device.deviceInfo.label}: IPv4={ip_info.ipAddress}" f" Origin={ip_info.origin} Static={ip_info.origin == 'manual'}", ) ``` After migration, the verifier connects to the destination VM over SSH and checks the guest configuration. > **Warning:** In this repository, static IP verification is currently implemented only for Windows guests migrated from vSphere. The verification path uses `ipconfig /all` inside the guest and does not support non-Windows guests yet. > **Note:** If no static interfaces are found on the source VM, the check is skipped rather than failed. ## PVC naming templates Use `pvc_name_template` when you want stable, readable PVC names. The repository covers both exact names and `generateName`-style prefixes. Examples from `tests/tests_config/config.py`: ```python "pvc_name_template": '{{ .FileName | trimSuffix ".vmdk" | replace "_" "-" }}-{{.DiskIndex}}', "pvc_name_template_use_generate_name": True, ``` ```python "pvc_name_template": "{{.VmName}}-disk-{{.DiskIndex}}", "pvc_name_template_use_generate_name": False, ``` The validator in `utilities/post_migration.py` explicitly supports: - `{{.VmName}}` - `{{.DiskIndex}}` - `{{.FileName}}` - Sprig functions such as `trimSuffix`, `replace`, `lower`, and `upper` When `pvc_name_template_use_generate_name` is `True`, the project expects Kubernetes to append a random suffix and validates the PVC by prefix match. When it is `False`, the validator expects an exact name match. > **Note:** The validator also accounts for Kubernetes name-length limits by truncating the rendered template before comparing it with the actual PVC name. > **Warning:** In this repository's post-migration validation, templates using `{{.FileName}}` and `{{.DiskIndex}}` are effectively vSphere-oriented. If the source provider is not vSphere, the PVC-name verifier logs a warning and skips that validation path. There is one important exception for copy-offload migrations. In `utilities/mtv_migration.py`, `copyoffload=True` overrides any custom template: ```python if copyoffload: plan_kwargs["pvc_name_template"] = "pvc" ``` > **Warning:** If you are running copy-offload tests, do not expect your custom `pvc_name_template` to be used. The helper forces it to `"pvc"` so the volume populator framework gets predictable PVC prefixes. ## Custom VM namespaces Use `vm_target_namespace` when you want the migrated VM to land in a namespace that is different from the namespace where the Plan and mapping resources are created. The comprehensive tests show two examples: - `"vm_target_namespace": "custom-vm-namespace"` - `"vm_target_namespace": "mtv-comprehensive-vms"` In this project: - The Plan, `StorageMap`, and `NetworkMap` are still created in the regular `target_namespace` - The migrated VM itself is created in `vm_target_namespace` - The `prepared_plan` fixture creates that namespace automatically if it does not already exist - The post-migration checks also look up the migrated VM in that custom namespace > **Note:** This is useful when you want one namespace for migration resources and another namespace for the resulting VMs. ## Node selectors, labels, and affinity These options control where the migrated VM runs and what metadata MTV applies to it. A cold-migration example from `tests/test_cold_migration_comprehensive.py` shows the scheduling-related arguments passed into the Plan helper: ```python target_node_selector={labeled_worker_node["label_key"]: labeled_worker_node["label_value"]}, target_labels=target_vm_labels["vm_labels"], target_affinity=prepared_plan["target_affinity"], vm_target_namespace=prepared_plan["vm_target_namespace"], ``` ### Node selectors Use `target_node_selector` to place the destination VM on nodes with a matching label. Example from `tests/tests_config/config.py`: ```python "target_node_selector": { "mtv-comprehensive-node": None, }, ``` In the comprehensive cold test, the `labeled_worker_node` fixture picks a worker node, applies the label, and passes the resolved selector into `create_plan_resource()`. After migration, the project verifies that the VM actually landed on that node. > **Tip:** In this project, setting the selector value to `None` tells the fixture to replace it with the current `session_uuid`. That makes the label unique for each run and helps avoid collisions in parallel execution. ### Labels Use `target_labels` to stamp metadata onto the migrated VM template. Examples from `tests/tests_config/config.py`: - `"static-label": "static-value"` - `"mtv-comprehensive-test": None` - `"test-type": "comprehensive"` The OpenShift provider reads labels back from the VM template metadata and the post-migration verifier checks that every expected label is present with the expected value. > **Tip:** As with node selectors, a label value of `None` is replaced by the current `session_uuid`, which is useful when you need unique per-run labels. ### Affinity Use `target_affinity` when you need Kubernetes scheduling preferences or constraints on the migrated VM. The repository examples use `podAffinity` with `preferredDuringSchedulingIgnoredDuringExecution`, for example matching VMs near pods with a specific label and topology key. Because the verifier deep-compares the resulting affinity block, it is best to keep the structure in your test config exactly as MTV should apply it. > **Note:** In this repository, labels are validated from the VM template metadata and affinity is validated from the VM template spec. Node placement is validated from the running VMI's assigned node. ## Multus setup Multus support is handled as part of network-map preparation, not as a direct Plan field. The default Multus CNI configuration comes from `conftest.py`: ```python @pytest.fixture(scope="session") def multus_cni_config() -> str: bridge_type_and_name = "cnv-bridge" config = {"cniVersion": "0.3.1", "type": f"{bridge_type_and_name}", "bridge": f"{bridge_type_and_name}"} return json.dumps(config) ``` When the source VM has multiple NICs, `utilities/utils.py` maps the first source network to the pod network and every additional network to a Multus NAD: ```python for index, network in enumerate(source_provider_inventory.vms_networks_mappings(vms=vms)): if pod_only or index == 0: # First network or pod_only mode → pod network _destination = _destination_pod else: # Extract base name and namespace from multus_network_name (only when needed) multus_network_name_str = multus_network_name["name"] multus_namespace = multus_network_name["namespace"] # Generate unique NAD name for each additional network # Use consistent naming: {base_name}-1, {base_name}-2, etc. # Where base_name includes unique test identifier (e.g., cnv-bridge-abc12345) nad_name = f"{multus_network_name_str}-{multus_counter}" _destination = { "name": nad_name, "namespace": multus_namespace, "type": "multus", } multus_counter += 1 # Increment for next NAD ``` What this means in practice: - The first NIC stays on the pod network - Each additional NIC gets its own `NetworkAttachmentDefinition` - NADs are named from a short base name plus `-1`, `-2`, and so on - `multus_namespace` decides where those NADs are created Both comprehensive configs use: ```python "multus_namespace": "default", ``` That enables cross-namespace NAD access by creating or reusing the NADs in `default`. > **Note:** A single-NIC VM does not need Multus. The fixture calculates the number of NADs as `max(0, len(networks) - 1)`, so only additional NICs get Multus attachments. > **Tip:** If you want to keep migration resources in one namespace but share NADs from another namespace, set `multus_namespace` explicitly, as the comprehensive tests do. --- Source: pytest-options-and-markers.md # Pytest Options And Markers This suite comes with an opinionated `pytest` setup. If you run `pytest` or `uv run pytest` from the repository root, it already knows where tests live, which config file to load, how to write JUnit XML, and how to behave when you enable xdist. > **Warning:** A real test run requires `source_provider` and `storage_class`. The session start hook exits early if either is missing. In practice, you usually pass them with `--tc=...`. ```bash uv run pytest -m copyoffload \ -v \ ${CLUSTER_HOST:+--tc=cluster_host:${CLUSTER_HOST}} \ ${CLUSTER_USERNAME:+--tc=cluster_username:${CLUSTER_USERNAME}} \ ${CLUSTER_PASSWORD:+--tc=cluster_password:${CLUSTER_PASSWORD}} \ --tc=source_provider:vsphere-8.0.3.00400 \ --tc=storage_class:my-block-storageclass ``` ## Default pytest behavior The repo-level defaults live in `pytest.ini`: ```ini [pytest] testpaths = tests addopts = -s -o log_cli=true -p no:logging --tc-file=tests/tests_config/config.py --tc-format=python --junit-xml=junit-report.xml --basetemp=/tmp/pytest --show-progress --strict-markers --jira --dist=loadscope markers = tier0: Core functionality tests (smoke tests) remote: Remote cluster migration tests warm: Warm migration tests copyoffload: Copy-offload (XCOPY) tests incremental: marks tests as incremental (xfail on previous failure) min_mtv_version: mark test to require minimum MTV version (e.g., @pytest.mark.min_mtv_version("2.6.0")) junit_logging = all ``` What that means in day-to-day use: - `testpaths = tests` limits default discovery to the `tests/` directory. - `-s` disables output capture, so test output is shown live. - `-p no:logging` disables pytest’s built-in logging plugin. The suite sets up its own console and file logging in `conftest.py`. - `--tc-file=tests/tests_config/config.py` and `--tc-format=python` load defaults through `pytest-testconfig`. - `--junit-xml=junit-report.xml` always writes a JUnit report in the repo working directory. - `junit_logging = all` means logs are included in the JUnit output. - `--basetemp=/tmp/pytest` gives pytest a fixed temp root for the run. - `--show-progress` enables progress output from `pytest-progress`. - `--strict-markers` turns marker typos into immediate errors instead of silently ignoring them. - `--jira` enables `pytest-jira`. - `--dist=loadscope` preconfigures xdist scheduling, but it does not start parallel workers by itself. You still need to add `-n` if you want xdist. The suite also rewrites collected item names so reports are easier to read: ```python for item in items: item.name = f"{item.name}-{py_config.get('source_provider')}-{py_config.get('storage_class')}" ``` > **Note:** Because collected names are rewritten, terminal output and JUnit entries include the selected `source_provider` and `storage_class`, not just the raw test function name. ## Marker reference The suite registers these project markers: | Marker | What it means | Typical use in this repo | | --- | --- | --- | | `tier0` | Core smoke and sanity coverage | Basic cold/warm migration flows and comprehensive smoke-style scenarios | | `warm` | Warm migration scenarios | Warm migration tests, including one copy-offload warm case | | `remote` | Remote OpenShift destination scenarios | Tests that require `remote_ocp_cluster` to be configured | | `copyoffload` | Copy-offload/XCOPY scenarios | The large copy-offload suite in `tests/test_copyoffload_migration.py` | | `incremental` | Sequential class semantics | Multi-step migration classes where later steps depend on earlier ones | | `min_mtv_version` | MTV version gate | Used with `mtv_version_checker` to skip below a required MTV version | A typical class combines multiple markers and environment gates: ```python pytestmark = [ pytest.mark.skipif( _SOURCE_PROVIDER_TYPE in (Provider.ProviderType.OPENSTACK, Provider.ProviderType.OPENSHIFT, Provider.ProviderType.OVA), reason=f"{_SOURCE_PROVIDER_TYPE} warm migration is not supported.", ), ] if _SOURCE_PROVIDER_TYPE == Provider.ProviderType.RHV: pytestmark.append(pytest.mark.jira("MTV-2846", run=False)) @pytest.mark.tier0 @pytest.mark.warm @pytest.mark.incremental @pytest.mark.parametrize( "class_plan_config", [ pytest.param( py_config["tests_params"]["test_sanity_warm_mtv_migration"], ) ], indirect=True, ids=["rhel8"], ) @pytest.mark.usefixtures("precopy_interval_forkliftcontroller", "cleanup_migrated_vms") class TestSanityWarmMtvMigration: ``` A remote-only class is gated explicitly: ```python @pytest.mark.remote @pytest.mark.incremental @pytest.mark.skipif(not get_value_from_py_config("remote_ocp_cluster"), reason="No remote OCP cluster provided") ``` The repo also supports version-gated tests through `min_mtv_version`, but the marker only has effect when the checker fixture is active: ```python @pytest.mark.usefixtures("mtv_version_checker") @pytest.mark.min_mtv_version("2.10.0") def test_something(...): # Test runs only if MTV >= 2.10.0 ``` > **Note:** Some warm tests add `pytest.mark.jira("MTV-2846", run=False)` for RHV. That behavior comes from `pytest-jira`, which is enabled by default via `--jira`. The repo includes a `jira.cfg.example` template for that plugin. ## Selection and dry-run modes This suite supports the standard pytest selection tools, and they map cleanly to how the tests are organized. ### Marker selection with `-m` Use project markers to slice the suite by scenario type. The checked-in docs already use this pattern: ```bash uv run pytest -m copyoffload \ -v \ ${CLUSTER_HOST:+--tc=cluster_host:${CLUSTER_HOST}} \ ${CLUSTER_USERNAME:+--tc=cluster_username:${CLUSTER_USERNAME}} \ ${CLUSTER_PASSWORD:+--tc=cluster_password:${CLUSTER_PASSWORD}} \ --tc=source_provider:vsphere-8.0.3.00400 \ --tc=storage_class:my-block-storageclass ``` In the same way, you can select other registered markers such as `tier0`, `warm`, or `remote`. ### Keyword selection with `-k` `-k` works well because test names are descriptive. The repository’s own docs call this out directly: - Add `-k test_name` after `-m copyoffload`. - Example: `-m copyoffload -k test_copyoffload_thin_migration` The repo also lists concrete test names you can target with `-k`: - `test_copyoffload_thin_migration` - `test_copyoffload_thick_lazy_migration` - `test_copyoffload_multi_disk_migration` - `test_copyoffload_multi_disk_different_path_migration` - `test_copyoffload_rdm_virtual_disk_migration` ### Standard pytest path and node selection This repo does not replace pytest’s normal file, class, or node selection. If you prefer selecting by file or specific test node, standard pytest syntax still applies. ### Supported dry-run modes The suite explicitly treats `--collect-only` and `--setup-plan` as dry-run modes: ```python def is_dry_run(config: pytest.Config) -> bool: """Check if pytest was invoked in dry-run mode (collectonly or setupplan).""" return config.option.setupplan or config.option.collectonly ``` Those dry-run modes are used in repository automation too: ```toml commands = [ [ "uv", "run", "pytest", "--setup-plan", ], [ "uv", "run", "pytest", "--collect-only", ], ] ``` The container image also defaults to dry-run discovery: ```dockerfile CMD ["uv", "run", "pytest", "--collect-only"] ``` `--collect-only` is the safest way to preview what your `-m` and `-k` expression will match. The checked-in copy-offload docs recommend it directly: ```bash pytest --collect-only -m copyoffload ``` `--setup-plan` is useful when you want pytest to show fixture setup planning without running the tests themselves. > **Note:** In this repository, dry-run mode is more than “just don’t execute tests.” When `--collect-only` or `--setup-plan` is active, the suite skips runtime-only validation, teardown, failure data collection, must-gather capture, and AI failure analysis. > **Warning:** Dry-run does not validate a migration path. It only validates collection and, for `--setup-plan`, setup planning. It does not exercise MTV, providers, or cluster-side migration behavior. ## Incremental semantics Most test classes in this repository are structured as a five-step workflow: 1. Create `StorageMap` 2. Create `NetworkMap` 3. Create `Plan` 4. Execute migration 5. Validate migrated VMs That is why `incremental` matters so much here. The suite implements its own incremental behavior in `conftest.py`: ```python # Incremental test support - track failures for class-based tests if "incremental" in item.keywords and rep.when == "call" and rep.failed: item.parent._previousfailed = item ``` ```python # Incremental test support - xfail if previous test in class failed if "incremental" in item.keywords: previousfailed = getattr(item.parent, "_previousfailed", None) if previousfailed is not None: pytest.xfail(f"previous test failed ({previousfailed.name})") ``` In practice, that means: - The first real failure in an incremental class is the one you should focus on. - Later tests in the same class are converted to `xfail` with a message like `previous test failed (...)`. - This prevents a broken early step from creating a long tail of noisy follow-up failures. There is one important nuance: this implementation only records failures from the `call` phase. A setup or teardown error does not set `_previousfailed` the same way a call-phase failure does. > **Tip:** When an incremental class fails, start with the earliest failing step in the class. Later `xfail` results are usually downstream effects, not new root causes. ## xdist behavior `pytest-xdist` is installed and the suite is xdist-aware, but parallel execution is opt-in. Nothing in `pytest.ini` sets a worker count, so runs stay single-process until you add `-n`. What is preconfigured is the distribution strategy: - `--dist=loadscope` is enabled by default. - That is a good fit for this repository because tests are heavily class-based, use shared class attributes, and often rely on `incremental` semantics. - Keeping related tests together on one worker reduces the chance of splitting a multi-step class across workers. The suite also includes explicit worker-side handling for `pytest-harvest` state: ```python def pytest_harvest_xdist_worker_dump(worker_id, session_items, fixture_store): # persist session_items and fixture_store in the file system with open(RESULTS_PATH / (f"{worker_id}.pkl"), "wb") as f: try: pickle.dump((session_items, fixture_store), f) except Exception as exp: LOGGER.warning(f"Error while pickling worker {worker_id}'s harvested results: [{exp.__class__}] {exp}") ``` And it protects worker-shared setup where needed. For example, the `virtctl_binary` fixture uses a file lock and a shared cache directory specifically for xdist-safe downloads. > **Tip:** If you enable parallelism with `-n`, keep the default `--dist=loadscope`. It matches the suite’s class-based design much better than fine-grained distribution. ## Suite-specific pytest options Beyond standard pytest options, `conftest.py` adds a small set of suite-specific flags: ```python analyze_with_ai_group.addoption("--analyze-with-ai", action="store_true", help="Analyze test failures using AI") data_collector_group.addoption("--skip-data-collector", action="store_true", help="Collect data for failed tests") data_collector_group.addoption( "--data-collector-path", help="Path to store collected data for failed tests", default=".data-collector" ) teardown_group.addoption( "--skip-teardown", action="store_true", help="Do not teardown resource created by the tests" ) openshift_python_wrapper_group.addoption( "--openshift-python-wrapper-log-debug", action="store_true", help="Enable debug logging in the openshift-python-wrapper module", ) ``` Here is what those flags actually do: | Option | Behavior | | --- | --- | | `--skip-teardown` | Preserves resources after the run instead of deleting them | | `--skip-data-collector` | Disables failure data collection and must-gather capture | | `--data-collector-path` | Changes where collector output is written; default is `.data-collector` | | `--analyze-with-ai` | Enriches failure reporting through the JUnit XML path after the run | | `--openshift-python-wrapper-log-debug` | Sets `OPENSHIFT_PYTHON_WRAPPER_LOG_LEVEL=DEBUG` during session startup | The real-run guard for required config lives here too: ```python required_config = ("storage_class", "source_provider") if not is_dry_run(session.config): missing_configs: list[str] = [] for _req in required_config: if not py_config.get(_req): missing_configs.append(_req) if missing_configs: pytest.exit(reason=f"Some required config is missing {required_config=} - {missing_configs=}", returncode=1) ``` And teardown/data collection behavior is handled here: ```python if not session.config.getoption("skip_data_collector"): collect_created_resources(session_store=_session_store, data_collector_path=_data_collector_path) if session.config.getoption("skip_teardown"): LOGGER.warning("User requested to skip teardown of resources") else: session_teardown(session_store=_session_store) ``` > **Warning:** `--skip-teardown` is a debugging tool, not a normal operating mode. If you use it, expect to clean up VMs, Plans, Providers, namespaces, and any source-side cloned resources yourself. > **Note:** The current help text for `--skip-data-collector` is misleading. The implementation uses it as a true skip flag: when it is set, the suite does not collect resource metadata or run must-gather on failures. > **Tip:** If you are building a complex selection expression, use `--collect-only` first. Once the collected set looks right, rerun the same command without dry-run mode. --- Source: plan-and-resource-reference.md # Plan And Resource Reference The suite builds the same MTV resource chain you would create by hand: `Provider`, `StorageMap`, `NetworkMap`, `Plan`, and `Migration`. For specific scenarios it also creates `Hook`, `Secret`, `Namespace`, and `NetworkAttachmentDefinition` resources so the plan can run end to end. > **Note:** The dictionaries in `tests/tests_config/config.py` are not raw CR YAML. Some keys become CR fields, while others only control setup or validation. For example, `clone`, `guest_agent`, and `source_vm_power` affect fixture behavior, not the final `Plan` spec. ## Lifecycle Overview | Resource | Why the suite creates it | Default placement | Extra readiness rule | | --- | --- | --- | --- | | `Namespace` | Isolate each run and any optional VM/NAD namespaces | Per-run `target_namespace`, plus optional custom namespaces | Waits for `Active` | | `Secret` | Hold provider credentials, OCP token, and copy-offload storage credentials | Usually the per-run `target_namespace` | Created with deploy wait only | | `Provider` | Represent source and destination endpoints for MTV | Per-run `target_namespace` | Source provider waits for `Ready`; VMware SSH copy-offload also waits for `Validated=True` after patching | | `StorageMap` | Map source storage to the destination storage class or offload plugin | Per-run `target_namespace` | Created with deploy wait only | | `NetworkMap` | Map source NICs to pod or Multus networks | Per-run `target_namespace` | Created with deploy wait only | | `Hook` | Run pre- or post-migration Ansible playbooks | Per-run `target_namespace` | Config is validated before creation | | `Plan` | Tie providers, mappings, VMs, and optional plan settings together | Per-run `target_namespace` | Waits for `Ready=True` | | `Migration` | Execute a `Plan` | Per-run `target_namespace` | The suite watches the `Plan` until it becomes `Succeeded` or `Failed` | ## Where Resources Live | Namespace | What the suite puts there | | --- | --- | | `target_namespace` | `Provider`, `StorageMap`, `NetworkMap`, `Plan`, `Migration`, `Hook`, provider `Secret`s, and copy-offload storage `Secret`s | | `vm_target_namespace` | Migrated VMs only, when you set `vm_target_namespace` | | `multus_namespace` | Extra `NetworkAttachmentDefinition`s, when you set `multus_namespace` | | `${session_uuid}-source-vms` | Temporary source CNV VMs for OpenShift source-provider tests | > **Note:** The suite uses `mtv_namespace` for Forklift operator objects such as the inventory route and `ForkliftController`. By default that namespace is `openshift-mtv`. The test-owned migration CRs themselves are created in the per-run `target_namespace`. ## Shared Naming Rules Every test-owned OpenShift resource goes through `create_and_store_resource()` in `utilities/resources.py`. That helper is the source of most naming, waiting, and cleanup behavior. ```python if not _resource_name: _resource_name = generate_name_with_uuid(name=fixture_store["base_resource_name"]) if resource.kind in (Migration.kind, Plan.kind): _resource_name = f"{_resource_name}-{'warm' if kwargs.get('warm_migration') else 'cold'}" if len(_resource_name) > 63: LOGGER.warning(f"'{_resource_name=}' is too long ({len(_resource_name)} > 63). Truncating.") _resource_name = _resource_name[-63:] kwargs["name"] = _resource_name _resource = resource(**kwargs) try: _resource.deploy(wait=True) except ConflictError: LOGGER.warning(f"{_resource.kind} {_resource_name} already exists, reusing it.") _resource.wait() LOGGER.info(f"Storing {_resource.kind} {_resource.name} in fixture store") _resource_dict = {"name": _resource.name, "namespace": _resource.namespace, "module": _resource.__module__} fixture_store["teardown"].setdefault(_resource.kind, []).append(_resource_dict) ``` A few practical rules fall out of that helper: - Auto-generated names start from `base_resource_name`, which is built as `{session_uuid}-source-{provider_type}-{version}`. - Copy-offload source providers add `-xcopy` to that base name. - Auto-generated `Plan` and `Migration` names also add `-warm` or `-cold`. - If a name is longer than 63 characters, the helper keeps the last 63 characters so the unique suffix survives. - If you pass an explicit `name`, or a `kind_dict`/`yaml_file` that already contains one, that name wins. - Every created resource is tracked for teardown. The suite also applies related naming rules outside CR creation: - `session_uuid` comes from `generate_name_with_uuid("auto")`, so generated run identifiers look like `auto-xxxx`. - `target_namespace_prefix` defaults to `auto`; the fixture strips the literal `auto` before appending it to `session_uuid` so default runs do not produce doubled prefixes. - Destination VM lookups on OpenShift are sanitized to DNS-1123 format, so names with uppercase letters, `_`, or `.` are converted before lookup. > **Tip:** When you are debugging a run, search by the session UUID first. Even when names are truncated, the suite preserves the unique suffix rather than the human-friendly prefix. ## Provider Source provider definitions come from the repo-root `.providers.json`. The loader in `utilities/utils.py` reads that file, and the `source_provider` pytest config value chooses one top-level entry by name. From `.providers.json.example`: ```text "vsphere": { "type": "vsphere", "version": "", "fqdn": "SERVER FQDN/IP", "api_url": "/sdk", "username": "USERNAME", "password": "PASSWORD", # pragma: allowlist secret "guest_vm_linux_user": "LINUX VMS USERNAME", "guest_vm_linux_password": "LINUX VMS PASSWORD", # pragma: allowlist secret "guest_vm_win_user": "WINDOWS VMS USERNAME", "guest_vm_win_password": "WINDOWS VMS PASSWORD", # pragma: allowlist secret "vddk_init_image": "" }, ``` The example file supports these provider types: - `vsphere` - `ovirt` - `openstack` - `openshift` - `ova` > **Note:** `.providers.json.example` is a template, not literal JSON. It contains inline comments, so you need to clean those up before using it as a real `.providers.json`. > **Warning:** The suite fails early if `.providers.json` is missing, empty, or if `source_provider` does not exactly match one of the top-level keys in that file. Provider creation behavior is slightly different depending on the role: - Source providers usually follow a two-step flow: create a `Secret`, then create a `Provider` CR that references it. - The default destination provider is a local OpenShift provider named like `${session_uuid}-local-ocp-provider`. - Remote-cluster tests create a destination OCP token `Secret` and a provider named like `${session_uuid}-destination-ocp-provider`. Provider readiness rules are stricter than most other resources: - The source `Provider` must reach `Ready` within 600 seconds. - The wait stops early if the provider reports `ConnectionFailed`. - After the `Provider` exists, Forklift inventory must expose it before the suite can build storage and network mappings. - For VMware copy-offload with `esxi_clone_method: "ssh"`, the suite patches the provider with `spec.settings.esxiCloneMethod: ssh` and then waits for `Validated=True`. ## Mapping Source Identifiers The suite does not hard-code one universal `source` shape for mappings. Instead, it asks provider-specific inventory adapters in `libs/forklift_inventory.py` for the right source identifiers. | Source provider type | `StorageMap` source shape | `NetworkMap` source shape | | --- | --- | --- | | `vsphere` | datastore `name` | network `name` | | `ovirt` | storage domain `name` | network `path` | | `openstack` | volume type `name` | network `id` and `name` | | `openshift` | storage class `name` | `{"type": "pod"}` or Multus `networkName` | | `ova` | storage `id` | network `name` | > **Note:** OpenStack inventory is treated a little more carefully than the others. The suite waits not only for the VM to appear in Forklift inventory, but also for its attached volumes and networks to become queryable before it builds mappings. ## StorageMap A standard `StorageMap` is inventory-driven: the suite asks Forklift which source storages the chosen VMs use, then maps each one to the destination `storage_class`. > **Warning:** `storage_class` is not defined in `tests/tests_config/config.py`. The suite expects it from pytest config, and `get_storage_migration_map()` falls back to `py_config["storage_class"]`. For copy-offload scenarios, the `StorageMap` carries an offload plugin configuration instead of relying only on source inventory. From `tests/test_copyoffload_migration.py`: ```python offload_plugin_config = { "vsphereXcopyConfig": { "secretRef": copyoffload_storage_secret.name, "storageVendorProduct": storage_vendor_product, } } self.__class__.storage_map = get_storage_migration_map( fixture_store=fixture_store, target_namespace=target_namespace, source_provider=source_provider, destination_provider=destination_provider, ocp_admin_client=ocp_admin_client, source_provider_inventory=source_provider_inventory, vms=vms_names, storage_class=storage_class, datastore_id=datastore_id, offload_plugin_config=offload_plugin_config, access_mode="ReadWriteOnce", volume_mode="Block", ) ``` In practice, that means: - Standard mode maps each source storage to `{"destination": {"storageClass": }, "source": }`. - Copy-offload mode maps datastores by ID and adds `offloadPlugin`, `accessMode`, and `volumeMode`. - Mixed and multi-datastore copy-offload are supported through `secondary_datastore_id` and `non_xcopy_datastore_id`. > **Warning:** `secondary_datastore_id` and `non_xcopy_datastore_id` are only valid when `datastore_id` is also set. The helper rejects those combinations otherwise. The copy-offload storage secret is also test-owned. Its credentials can come from environment variables or from the provider’s `copyoffload` section in `.providers.json`, and vendor-specific keys are validated before the secret is created. ## NetworkMap `NetworkMap` creation is intentionally simple and predictable: - The first source network always maps to the pod network. - Every additional source network maps to a Multus `NetworkAttachmentDefinition`. - Extra NADs are named from a short class hash so parallel tests do not collide. From `utilities/utils.py`: ```python network_map_list: list[dict[str, dict[str, str]]] = [] _destination_pod: dict[str, str] = {"type": "pod"} multus_counter = 1 for index, network in enumerate(source_provider_inventory.vms_networks_mappings(vms=vms)): if pod_only or index == 0: _destination = _destination_pod else: multus_network_name_str = multus_network_name["name"] multus_namespace = multus_network_name["namespace"] nad_name = f"{multus_network_name_str}-{multus_counter}" _destination = { "name": nad_name, "namespace": multus_namespace, "type": "multus", } multus_counter += 1 network_map_list.append({ "destination": _destination, "source": network, }) ``` That logic ties directly to the class-scoped NAD fixture in `conftest.py`: - The base NAD name is `cb-<6-char-sha256>`. - Additional NADs become `cb--1`, `cb--2`, and so on. - The names are kept short to stay under Linux bridge interface limits. - If you set `multus_namespace`, the suite creates or reuses that namespace and puts the NADs there. Otherwise they go into the main `target_namespace`. > **Tip:** A one-NIC VM still gets a `NetworkMap`, but it does not need any extra `NetworkAttachmentDefinition` objects because the first network always maps to the pod network. ## Hook Hooks are optional, but when you configure them the suite creates real `Hook` CRs and wires their generated names into the `Plan`. From `tests/tests_config/config.py`: ```python "test_post_hook_retain_failed_vm": { "virtual_machines": [ { "name": "mtv-tests-rhel8", "source_vm_power": "on", "guest_agent": True, }, ], "warm_migration": False, "target_power_state": "off", "pre_hook": {"expected_result": "succeed"}, "post_hook": {"expected_result": "fail"}, "expected_migration_result": "fail", }, ``` Hook configuration supports two modes: - Predefined mode: set `expected_result` to `succeed` or `fail`, and the suite chooses one of the built-in base64-encoded Ansible playbooks from `utilities/hooks.py`. - Custom mode: set `playbook_base64` to your own base64-encoded playbook. Validation rules are strict: - `expected_result` and `playbook_base64` are mutually exclusive. - You must set one of them. - Empty strings are rejected. - Custom playbooks must be valid base64, valid UTF-8, valid YAML, and a non-empty Ansible play list. > **Note:** The suite reads detailed hook failure steps from the owned `Migration` CR, not from the `Plan`, because the VM pipeline status lives there. A failing `PostHook` still leads to VM validation, while a failing `PreHook` skips VM checks because the migration never reached the VM validation stage. ## Plan The `Plan` is where all of the pieces come together. Before the suite creates it, `prepared_plan` may clone VMs, adjust source power state, wait for cloned VMs to appear in Forklift inventory, and call `populate_vm_ids()` so each VM entry has the inventory ID Forklift expects. The most feature-rich plan-style config in the repo looks like this. From `tests/tests_config/config.py`: ```python "test_cold_migration_comprehensive": { "virtual_machines": [ { "name": "mtv-win2019-3disks", "source_vm_power": "off", "guest_agent": True, }, ], "warm_migration": False, "target_power_state": "on", "preserve_static_ips": True, "pvc_name_template": "{{.VmName}}-disk-{{.DiskIndex}}", "pvc_name_template_use_generate_name": False, "target_node_selector": { "mtv-comprehensive-node": None, # None = auto-generate with session_uuid }, "target_labels": { "mtv-comprehensive-label": None, # None = auto-generate with session_uuid "test-type": "comprehensive", # Static value }, "target_affinity": { "podAffinity": { "preferredDuringSchedulingIgnoredDuringExecution": [ { "podAffinityTerm": { "labelSelector": {"matchLabels": {"app": "test"}}, "topologyKey": "kubernetes.io/hostname", }, "weight": 50, } ] } }, "vm_target_namespace": "mtv-comprehensive-vms", "multus_namespace": "default", # Cross-namespace NAD access }, ``` Those settings break down into a few practical groups: - `warm_migration` controls whether the plan is warm or cold and also changes the auto-generated `Plan` and `Migration` name suffixes. - `target_power_state`, `preserve_static_ips`, `pvc_name_template`, `pvc_name_template_use_generate_name`, `target_node_selector`, `target_labels`, `target_affinity`, and `vm_target_namespace` are passed into `create_plan_resource()`. - `multus_namespace` is not a `Plan` field. It tells the NAD fixture where to create extra Multus networks before the `Plan` is built. - `target_node_selector` and `target_labels` treat `None` specially. The fixtures replace `None` with the current `session_uuid`, which gives you unique labels and selectors without hard-coding a value. `pvc_name_template` is especially useful when you want predictable PVC names: - The validation helper supports `{{.VmName}}`, `{{.DiskIndex}}`, and VMware-only `{{.FileName}}`. - It also supports Sprig functions, because validation uses a Go template renderer. - Long generated names are truncated during validation to match Kubernetes limits. - When `pvc_name_template_use_generate_name` is `True`, the suite checks only the generated prefix because Kubernetes adds its own random suffix. > **Tip:** Use `None` in `target_node_selector` and `target_labels` when you want uniqueness without inventing a new value yourself. The suite will replace it with the run’s `session_uuid`. After creation, the plan has its own readiness rules: - The suite waits for `Plan.Condition.READY=True` with a 360-second timeout. - On timeout, it logs the `Plan` plus both source and destination provider objects to make provider-side issues easier to debug. - For copy-offload plans, it also waits up to 60 seconds for Forklift to create a plan-specific secret whose name starts with `-`. > **Note:** The `Plan` CR itself stays in the main `target_namespace` even when `vm_target_namespace` sends migrated VMs somewhere else. Only the migrated VMs move. > **Note:** The copy-offload secret wait is best-effort. If the `-*` secret is late, the suite continues anyway because the later migration failure is usually more actionable than a generic “secret missing” timeout. ## Migration `execute_migration()` creates a separate `Migration` CR that points at the `Plan`. For cold migrations it passes no cutover time. For warm migrations it computes one from the current UTC time and the configured offset. From `utilities/migration_utils.py`: ```python def get_cutover_value(current_cutover: bool = False) -> datetime: datetime_utc = datetime.now(pytz.utc) if current_cutover: return datetime_utc return datetime_utc + timedelta(minutes=int(py_config["mins_before_cutover"])) ``` The runtime settings that matter most for migration readiness come from `tests/tests_config/config.py`: | Key | Default | Why it matters | | --- | --- | --- | | `mtv_namespace` | `openshift-mtv` | Where the suite expects Forklift operator objects | | `snapshots_interval` | `2` | Patched into `ForkliftController.spec.controller_precopy_interval` for warm tests | | `mins_before_cutover` | `5` | Offset used by `get_cutover_value()` for warm migrations | | `plan_wait_timeout` | `3600` | Timeout for waiting on migration completion via `Plan` status | | `target_namespace_prefix` | `auto` | Prefix material for the per-run namespace name | A few implementation details are worth knowing when you debug a `Migration`: - The suite creates the `Migration` with `plan_name`, `plan_namespace`, and optional `cut_over`. - It does not use the `Migration` object alone for final success and failure. Instead, it watches the `Plan`. - The helper treats a plan as `Executing` as soon as it finds an owned `Migration` CR for that `Plan`. - Final `Succeeded` and `Failed` states come from advisory conditions on the `Plan`. - Detailed per-VM failure steps such as `PreHook`, `PostHook`, or `DiskTransfer` come from `migration.status.vms[].pipeline[]`. > **Note:** Before the suite starts creating plans and migrations, it waits up to five minutes for all `forklift-*` pods in `mtv_namespace` to be `Running` or `Succeeded`, and it requires a controller pod to exist. > **Note:** Warm migration coverage is not universal. `tests/test_mtv_warm_migration.py` explicitly skips warm tests for `openstack`, `openshift`, and `ova` source providers. ## Related Resources and Cleanup A few supporting resources show up often enough that they are worth calling out directly: - The main `target_namespace` is labeled with restricted pod-security settings and `mutatevirtualmachines.kubemacpool.io=ignore`. - Custom namespaces created through `get_or_create_namespace()` are created with the same standard labels and waited to `Active`. - OpenShift source-provider tests create a separate `${session_uuid}-source-vms` namespace for source-side CNV VMs. - Copy-offload tests create a storage credential `Secret` in the run namespace and may also rely on the plan-specific secret Forklift creates later. - Every object created through `create_and_store_resource()` is registered in `fixture_store["teardown"]`. Cleanup happens in two layers: - `cleanup_migrated_vms` removes migrated VMs at class teardown, using `vm_target_namespace` when you set one. - Session teardown removes `Migration`, `Plan`, `Provider`, `Secret`, `StorageMap`, `NetworkMap`, `NetworkAttachmentDefinition`, namespaces, migrated VMs, and even source-side cloned VMs for providers such as VMware, OpenStack, and RHV. > **Warning:** `--skip-teardown` leaves those resources behind on purpose. Use it only when you want leftover `Plan`, `Migration`, `Provider`, PVC, VM, and namespace objects for debugging. --- Source: provider-and-inventory-reference.md # Provider And Inventory Reference `mtv-api-tests` uses two sources of truth for source-side data: - direct provider classes for actions such as connect, clone, power control, snapshots, and VM inspection - Forklift inventory adapters for the names, IDs, networks, and storages that MTV itself consumes That split explains most of the repository's behavior. If a problem is about cloning, power state, guest information, or direct SDK access, start with the provider class. If a problem is about `NetworkMap`, `StorageMap`, or a VM ID inside a `Plan`, start with Forklift inventory. ## End-To-End Flow 1. The suite loads `.providers.json` from the repository root. 2. `py_config["source_provider"]` selects one top-level provider entry from that file. 3. `create_source_provider()` creates the source `Secret` and source `Provider` CR, then instantiates the matching `BaseProvider` subclass. 4. `source_provider_inventory` chooses the matching `ForkliftInventory` adapter for that provider type. 5. `prepared_plan` clones or creates source VMs, normalizes them through `vm_dict()`, rewrites the VM name if needed, and waits for Forklift inventory to see the VM. 6. `get_network_migration_map()` and `get_storage_migration_map()` turn inventory data into `NetworkMap` and `StorageMap` CR payloads. 7. `populate_vm_ids()` copies Forklift VM IDs into the `Plan` payload just before `create_plan_resource()` runs. The synchronization step is explicit in `conftest.py`: ```python for vm in virtual_machines: # Get VM object first (without full vm_dict analysis) # Add enable_ctk flag for warm migrations clone_options = {**vm, "enable_ctk": warm_migration} provider_vm_api = source_provider.get_vm_by_name( query=vm["name"], vm_name_suffix=vm_name_suffix, clone_vm=True, session_uuid=fixture_store["session_uuid"], clone_options=clone_options, ) source_vm_details = source_provider.vm_dict( provider_vm_api=provider_vm_api, name=vm["name"], namespace=source_vms_namespace, clone=False, # Already cloned above vm_name_suffix=vm_name_suffix, session_uuid=fixture_store["session_uuid"], clone_options=vm, ) vm["name"] = source_vm_details["name"] if source_provider.type != Provider.ProviderType.OVA: source_provider_inventory.wait_for_vm(name=vm["name"], timeout=300) ``` > **Note:** A map-generation failure often means Forklift inventory has not finished syncing the VM yet. The suite intentionally waits for inventory before creating maps, because MTV consumes inventory objects, not the provider SDK's in-memory objects. ## Provider Configuration And Selection The selection logic is intentionally simple: `.providers.json` is loaded, and `py_config["source_provider"]` picks one named entry. ```python @pytest.fixture(scope="session") def source_provider_data(source_providers: dict[str, dict[str, Any]], fixture_store: dict[str, Any]) -> dict[str, Any]: """Resolve source provider configuration from .providers.json.""" if not source_providers: raise MissingProvidersFileError() requested_provider = py_config["source_provider"] if requested_provider not in source_providers: raise ValueError( f"Source provider '{requested_provider}' not found in '.providers.json'. " f"Available providers: {sorted(source_providers.keys())}" ) _source_provider = source_providers[requested_provider] fixture_store["source_provider_data"] = _source_provider return _source_provider ``` The repository does not commit a sample `.providers.json`, so the code is the best reference for required fields. | Field | Why the suite reads it | | --- | --- | | `type` | Selects the provider class and the Forklift inventory adapter | | `version` | Used in generated resource names by `base_resource_name` | | `api_url` | Written into the source provider secret and the `Provider` CR | | `username`, `password` | Read before provider-specific branching | | `fqdn` | Used by the CA certificate helper when certificate download is needed | The factory in `utilities/utils.py` reads the common connection fields before it knows which provider type it is handling: ```python secret_string_data = { "url": source_provider_data_copy["api_url"], "insecureSkipVerify": "true" if insecure else "false", } provider_args = { "username": source_provider_data_copy["username"], "password": source_provider_data_copy["password"], "fixture_store": fixture_store, } ``` Provider-specific extras are then added on top: | Source type | Extra fields read by code | | --- | --- | | `vsphere` | optional `copyoffload`, optional `vddk_init_image` | | `rhv` | no extra SDK login fields, but the current code still expects `fqdn` for CA cert download | | `openstack` | `project_name`, `user_domain_name`, `region_name`, `user_domain_id`, `project_domain_id` | | `openshift` | no extra provider-specific fields; the source uses the destination cluster secret | | `ova` | no extra provider-specific fields; direct provider logic is intentionally minimal | > **Warning:** `create_source_provider()` reads `api_url`, `username`, and `password` before provider-specific branching, so every provider entry needs those keys. `version` is also needed for generated resource names. When certificate download is involved, `fqdn` matters too, because the certificate helper always connects to `:443`. ## The BaseProvider Contract `BaseProvider` is the common surface that keeps the rest of the suite provider-agnostic. Each concrete provider can use its own SDK, but it must expose the same core operations to the test code. The normalized VM shape is defined in `libs/base_provider.py`: ```python VIRTUAL_MACHINE_TEMPLATE: dict[str, Any] = { "id": "", "name": "", "provider_type": "", # "ovirt" / "vsphere" / "openstack" "provider_vm_api": None, "network_interfaces": [], "disks": [], "cpu": {}, "memory_in_mb": 0, "snapshots_data": [], "power_state": "", } ``` The most important abstraction for mapping logic is `get_vm_or_template_networks()`: ```python def get_vm_or_template_networks( self, names: list[str], inventory: ForkliftInventory, ) -> list[dict[str, str]]: """Get network mappings for VMs or templates (before cloning). This method handles provider-specific differences: - RHV: Queries template networks directly (templates don't exist in inventory yet) - VMware/OpenStack/OVA/OpenShift: Queries VM networks from Forklift inventory """ ``` In practice, `BaseProvider` gives the suite five important guarantees: - `connect()` and `disconnect()` manage the direct SDK session. - `test` is used immediately after provider creation to fail fast when the source is not reachable. - `vm_dict()` returns a normalized view of the source or destination VM. - `clone_vm()` and `delete_vm()` let the suite prepare and clean up source-side test VMs without special-case code in every test. - `get_vm_or_template_networks()` lets the suite size destination networking even before cloned VMs have finished syncing into inventory. ## Provider-Specific Behavior ### vSphere `VMWareProvider` in `libs/providers/vmware.py` is the most feature-rich source implementation in the repo. - It uses `pyVmomi` for clone, power, disk, and guest-information operations. - Clone-time options cover disk provisioning (`thin`, `thick-lazy`, `thick-eager`), extra disks, datastore overrides, ESXi host overrides, MAC regeneration, and Change Block Tracking for warm migrations. - If `copyoffload.esxi_clone_method` is set to `ssh`, the provider patches the MTV `Provider` CR to set `esxiCloneMethod`. - When copy-offload is configured, the source `Provider` CR also gets the annotation `forklift.konveyor.io/empty-vddk-init-image: yes`. - Standard network and storage mappings come from `VsphereForkliftInventory`. Copy-offload storage mappings can bypass inventory and use explicit datastore IDs instead. > **Note:** The test-side vSphere SDK connection uses `disableSslCertValidation=True`, while the Forklift `Provider` CR still honors `source_provider_insecure_skip_verify` and can include `cacert`. That means direct provider access and MTV-side validation are related, but not identical, code paths. ### RHV / oVirt `OvirtProvider` in `libs/providers/rhv.py` has two important behaviors that are easy to miss. - It refuses to connect unless the RHV datacenter named `MTV-CNV` exists and is `up`. - `clone_vm()` clones from a template, not from a running VM. In other words, `source_vm_name` is treated as a template name in RHV flows. - `get_vm_or_template_networks()` ignores inventory during pre-clone network discovery and queries template NICs directly. - Later, once cloned VMs exist and Forklift has synced them, final `NetworkMap` and `StorageMap` generation still comes from inventory. - RHV always fetches a CA certificate in `create_source_provider()`, even when insecure mode is enabled, because the code comments call out imageio as a dependency for that certificate. ### OpenStack `OpenStackProvider` in `libs/providers/openstack.py` clones more like an image pipeline than a simple VM copy. - It snapshots the source server. - It creates new volumes from the resulting snapshots. - It preserves boot order across the recreated volumes. - It boots the cloned server from those new volumes. OpenStack is also the strictest inventory-sync path. The suite does not treat “VM exists in inventory” as enough. It also waits for attached volumes and networks to become queryable: ```python if self.provider_type == Provider.ProviderType.OPENSTACK: if not ( self._check_openstack_volumes_synced(vm, name) and self._check_openstack_networks_synced(vm, name) ): return None ``` That matters because OpenStack storage mapping is based on volume type, and network mapping is based on inventory network objects matched against VM address data. If either side is late, map generation will be wrong. ### OpenShift `OCPProvider` is both the destination provider for migrations and a supported source provider for source-side CNV test setups. - If OpenShift is the source, `prepared_plan` creates source CNV VMs and a source-side `NetworkAttachmentDefinition` automatically. - `OpenshiftForkliftInventory` resolves storage by following the VM's data volumes to PVCs and then reading `storageClassName`. - It resolves networks from the VM template: `multus.networkName` becomes a named source network, and a pod network becomes `{"type": "pod"}`. - On destination lookups, `OCPProvider.vm_dict()` sanitizes VM names to Kubernetes-safe resource names before querying the cluster. ### OVA `OVAProvider` is intentionally thin. - `connect()` is effectively a no-op. - `clone_vm()` and `delete_vm()` are not implemented. - `vm_dict()` only fills the normalized template with the provider type and `power_state="off"`. - In the current `prepared_plan` implementation, OVA sources are rewritten to the fixed VM name `1nisim-rhel9-efi`. - In practice, OVA mapping behavior depends much more on Forklift inventory than on direct provider logic. > **Note:** The warm-migration test suites currently skip OpenStack, OpenShift, and OVA sources. RHV warm coverage is gated by a Jira marker. In this repository, vSphere is the fully exercised warm-migration source. ## Forklift Inventory Adapters Forklift inventory lives behind the `forklift-inventory` route in the MTV namespace. Each adapter subclasses `ForkliftInventory` and knows how to translate that provider's inventory model into the source-side names and IDs that MTV map CRs expect. The adapter selection is defined in `conftest.py`: ```python providers = { Provider.ProviderType.OVA: OvaForkliftInventory, Provider.ProviderType.RHV: OvirtForkliftInventory, Provider.ProviderType.VSPHERE: VsphereForkliftInventory, Provider.ProviderType.OPENSHIFT: OpenshiftForkliftInventory, Provider.ProviderType.OPENSTACK: OpenstackForliftinventory, } provider_instance = providers.get(source_provider.type) ``` The adapters all use the same base route plumbing in `libs/forklift_inventory.py`: they discover the provider ID, build a provider-specific URL path, and then query VM, network, and storage endpoints through the inventory service. | Source type | Adapter | Storage resolution | Network resolution | | --- | --- | --- | --- | | vSphere | `VsphereForkliftInventory` | Each disk's `datastore.id` is matched to an inventory datastore name | Each inventory VM network `id` is matched to an inventory network name | | RHV | `OvirtForkliftInventory` | Disk attachment -> disk -> `storageDomain` -> storage domain name | NIC profile -> network ID -> inventory network `path` | | OpenStack | `OpenstackForliftinventory` | Attached volume -> volume details -> `volumeType` | VM `addresses` keys are matched to inventory network names | | OpenShift | `OpenshiftForkliftInventory` | VM data volumes -> PVC -> `storageClassName` | VM template networks become either named multus networks or `{"type": "pod"}` | | OVA | `OvaForkliftInventory` | Inventory storage entries whose name contains the VM name; mapping returns storage `id` | Inventory VM network `ID` is matched to an inventory network name | Two adapter details are especially practical: - RHV uses provider-side template queries for pre-clone network discovery, but inventory for final map generation. - OpenShift storage resolution is not just “whatever inventory says.” The adapter actually follows data volumes to PVCs and reads the live `storageClassName`. ## How Source Network Mappings Are Resolved Network mapping is a two-step process. First, the suite decides how many destination networks it needs. It does that in the `multus_network_name` fixture by calling `source_provider.get_vm_or_template_networks()`. This is why RHV can use template networks before clones exist. Second, once the cloned or prepared source VM has been synced into Forklift inventory, the actual `NetworkMap` payload is built from inventory data. The core rule lives in `utilities/utils.py`: ```python for index, network in enumerate(source_provider_inventory.vms_networks_mappings(vms=vms)): if pod_only or index == 0: _destination = _destination_pod else: multus_network_name_str = multus_network_name["name"] multus_namespace = multus_network_name["namespace"] nad_name = f"{multus_network_name_str}-{multus_counter}" _destination = { "name": nad_name, "namespace": multus_namespace, "type": "multus", } multus_counter += 1 network_map_list.append({ "destination": _destination, "source": network, }) ``` What that means in plain language: 1. The first source network is always mapped to the destination pod network. 2. Every additional source network is mapped to a generated Multus NAD. 3. Those NADs are named from a base name plus a numeric suffix: `{base}-1`, `{base}-2`, and so on. 4. If the plan config sets `multus_namespace`, the NADs are created there instead of in the main target namespace. > **Warning:** Network mapping is order-based. The first source network returned by inventory becomes the pod network, so inventory ordering matters for multi-NIC VMs. > **Tip:** If a source VM has `N` networks, the suite creates `N - 1` NADs, because the first network is reserved for the destination pod network. ## How Source Storage Mappings Are Resolved The storage path is simpler than the network path in standard migrations: the helper trusts the selected inventory adapter to tell it which source storage objects matter, then it adds the destination storage class. The standard branch of `get_storage_migration_map()` looks like this: ```python storage_migration_map = source_provider_inventory.vms_storages_mappings(vms=vms) for storage in storage_migration_map: storage_map_list.append({ "destination": {"storageClass": target_storage_class}, "source": storage, }) ``` A few practical consequences follow from that: - The destination storage class is `py_config["storage_class"]` unless the caller passes an explicit `storage_class` argument. - The meaning of the source side is provider-specific. On vSphere it is a datastore. On RHV it is a storage domain. On OpenStack it is a volume type. On OpenShift it is a storage class. On OVA it is a storage ID. - If the adapter returns the wrong source identifier, the `StorageMap` will be wrong even if the direct provider SDK can see the disks just fine. ### vSphere Copy-Offload Copy-offload is the main exception to inventory-derived storage mapping. In copy-offload mode, `get_storage_migration_map()` can build the source side from explicit datastore IDs instead of asking inventory. ```python if datastore_id and offload_plugin_config: datastores_to_map = [datastore_id] if secondary_datastore_id: datastores_to_map.append(secondary_datastore_id) for ds_id in datastores_to_map: destination_config = { "storageClass": target_storage_class, } if access_mode: destination_config["accessMode"] = access_mode if volume_mode: destination_config["volumeMode"] = volume_mode storage_map_list.append({ "destination": destination_config, "source": {"id": ds_id}, "offloadPlugin": offload_plugin_config, }) else: storage_migration_map = source_provider_inventory.vms_storages_mappings(vms=vms) for storage in storage_migration_map: storage_map_list.append({ "destination": {"storageClass": target_storage_class}, "source": storage, }) ``` The repo validates copy-offload prerequisites in `conftest.py`: ```python required_credentials = ["storage_hostname", "storage_username", "storage_password"] required_params = ["storage_vendor_product", "datastore_id"] ``` In practice, the copy-offload flow expects: - `storage_vendor_product` - `datastore_id` - `storage_hostname` - `storage_username` - `storage_password` Optional extensions used by the code include: - `secondary_datastore_id` - `non_xcopy_datastore_id` - `esxi_clone_method` - `esxi_host` - `esxi_user` - `esxi_password` The storage secret builder also knows vendor-specific extras. Depending on `storage_vendor_product`, the repo may look for keys such as `ontap_svm`, `vantara_storage_id`, `vantara_storage_port`, `vantara_hostgroup_id_list`, `pure_cluster_prefix`, `powerflex_system_id`, or `powermax_symmetrix_id`. The test suite builds the offload plugin config like this: ```python offload_plugin_config = { "vsphereXcopyConfig": { "secretRef": copyoffload_storage_secret.name, "storageVendorProduct": storage_vendor_product, } } ``` When `copyoffload=True`, `create_plan_resource()` also forces `pvc_name_template` to `"pvc"` because the volume-populator path expects predictable PVC naming. > **Warning:** Copy-offload in this repository is a vSphere-specific path. The validation fixture explicitly fails if the selected source provider is not vSphere. > **Tip:** Copy-offload credentials can come from environment variables as well as `.providers.json`. The code looks for names such as `COPYOFFLOAD_STORAGE_HOSTNAME`, `COPYOFFLOAD_STORAGE_USERNAME`, and `COPYOFFLOAD_STORAGE_PASSWORD`, plus vendor-specific `COPYOFFLOAD_` overrides. ## Inventory IDs And PVC Naming Forklift inventory does two more important jobs after maps are built. First, it provides the VM IDs that go into the migration `Plan`: ```python def populate_vm_ids(plan: dict[str, Any], inventory: ForkliftInventory) -> None: if not isinstance(plan, dict) or not isinstance(plan.get("virtual_machines"), list): raise ValueError("plan must contain 'virtual_machines' list") for vm in plan["virtual_machines"]: vm_name = vm["name"] vm_data = inventory.get_vm(vm_name) vm["id"] = vm_data["id"] ``` Second, it can supply disk filenames for PVC-template validation. In `utilities/post_migration.py`, the `{{.FileName}}` wildcard is resolved from Forklift inventory disk file paths, not from the direct provider SDK. That validation path is only enabled for vSphere sources. > **Tip:** If you use `pvc_name_template` with `{{.FileName}}`, you are depending on inventory disk metadata. The suite sorts vSphere source disks by `(controller_key, unit_number)` before it renders expected PVC names. ## Real Test Config Examples These are exact snippets from `tests/tests_config/config.py`. They are useful because they show the real shapes the repository already exercises. ### Comprehensive Warm Migration Example This example combines custom VM namespace placement, cross-namespace Multus, static IP preservation, and a PVC naming template based on inventory file names. ```python "test_warm_migration_comprehensive": { "virtual_machines": [ { "name": "mtv-win2022-ip-3disks", "source_vm_power": "on", "guest_agent": True, }, ], "warm_migration": True, "target_power_state": "on", "preserve_static_ips": True, "vm_target_namespace": "custom-vm-namespace", "multus_namespace": "default", # Cross-namespace NAD access "pvc_name_template": '{{ .FileName | trimSuffix ".vmdk" | replace "_" "-" }}-{{.DiskIndex}}', "pvc_name_template_use_generate_name": True, "target_labels": { "mtv-comprehensive-test": None, # None = auto-generate with session_uuid "static-label": "static-value", }, "target_affinity": { "podAffinity": { "preferredDuringSchedulingIgnoredDuringExecution": [ { "podAffinityTerm": { "labelSelector": {"matchLabels": {"app": "comprehensive-test"}}, "topologyKey": "kubernetes.io/hostname", }, "weight": 75, } ] } }, }, ``` ### Mixed-Datastore Copy-Offload Example This example shows a vSphere copy-offload plan where an added disk is intentionally placed on a non-XCOPY datastore. ```python "test_copyoffload_mixed_datastore_migration": { "virtual_machines": [ { "name": "xcopy-template-test", "source_vm_power": "off", "guest_agent": True, "clone": True, "disk_type": "thin", "add_disks": [ { "size_gb": 30, "provision_type": "thin", "datastore_id": "non_xcopy_datastore_id", }, ], }, ], "warm_migration": False, "copyoffload": True, }, ``` > **Warning:** Symbolic datastore values such as `secondary_datastore_id` and `non_xcopy_datastore_id` are not global magic strings. The vSphere provider resolves them from the selected provider's `copyoffload` configuration. Once you know which data comes from the direct provider class and which data comes from Forklift inventory, the rest of the repository becomes much easier to predict. Provider classes explain how source VMs are created and inspected. Inventory adapters explain how MTV sees those VMs. The map helpers then turn that inventory view into the exact `StorageMap`, `NetworkMap`, and `Plan` payloads that the migration uses. --- Source: python-module-reference.md # Python Module Reference The modules in `utilities/` are the shared plumbing behind the MTV API test suite. They create and track MTV/OpenShift resources, prepare providers, manage `virtctl`, open SSH sessions to migrated VMs, collect diagnostics, and run the post-migration checks that most tests depend on. Most users do not call every helper directly. In practice, you reach them through fixtures in `conftest.py`, especially `target_namespace`, `source_provider`, `prepared_plan`, `virtctl_binary`, `vm_ssh_connections`, and `cleanup_migrated_vms`. > **Note:** These modules are designed for live OpenShift and MTV environments. Repository automation only validates collection and setup: `tox.toml` runs `uv run pytest --setup-plan` and `uv run pytest --collect-only`, and the container image defaults to `uv run pytest --collect-only`. ## At a Glance | Module | What it handles | Main entry points | | --- | --- | --- | | `utilities.utils` | cluster client setup, provider loading, provider CR creation | `get_cluster_client()`, `load_source_providers()`, `create_source_provider()` | | `utilities.resources` | tracked creation of OpenShift and MTV resources | `create_and_store_resource()`, `get_or_create_namespace()` | | `utilities.mtv_migration` | storage maps, network maps, plans, migrations | `get_storage_migration_map()`, `get_network_migration_map()`, `create_plan_resource()`, `execute_migration()` | | `utilities.virtctl` | getting the right `virtctl` binary onto the test host | `download_virtctl_from_cluster()`, `add_to_path()` | | `utilities.ssh_utils` | SSH access to migrated VMs over `virtctl port-forward` | `VMSSHConnection`, `SSHConnectionManager` | | `utilities.post_migration` | end-to-end validation of migrated VMs | `check_vms()` and the focused `check_*` helpers | | `utilities.hooks` | hook creation and hook-failure validation | `create_hook_if_configured()`, `validate_hook_failure_and_check_vms()` | | `utilities.must_gather` | targeted MTV must-gather collection | `run_must_gather()` | | `utilities.pytest_utils` | failure-time data collection and session cleanup | `collect_created_resources()`, `session_teardown()` | | `utilities.migration_utils` | cancel/archive flows and storage cleanup | `cancel_migration()`, `archive_plan()`, `check_dv_pvc_pv_deleted()` | ## Core Setup ### `utilities.utils` `utilities.utils` is where the suite turns configuration into live connections. `load_source_providers()` reads `.providers.json`, `get_cluster_client()` builds the OpenShift `DynamicClient`, and `get_value_from_py_config()` converts string booleans such as `"true"` and `"false"` into real Python booleans so the rest of the code can treat settings consistently. This is also the module that creates source-side provider resources. `create_source_provider()` handles the provider-specific differences for VMware, RHV, OpenStack, OVA, and OpenShift, including creating the right `Secret` and `Provider` CRs, fetching CA certificates when SSL verification is enabled, and passing copy-offload settings through when present. If you are writing tests rather than extending framework code, you usually consume this module indirectly through the `ocp_admin_client`, `source_provider_data`, and `source_provider` fixtures instead of importing it directly. ### `utilities.resources` `utilities.resources` is the resource lifecycle foundation of the repository. Its core helper, `create_and_store_resource()`, does more than create a resource: - It fills in the client automatically. - It chooses a name from `name`, `kind_dict`, `yaml_file`, or generates one from the session base name. - It appends `-warm` or `-cold` to `Plan` and `Migration` names. - It truncates names to Kubernetes-safe length. - It deploys and waits. - It records the resource in `fixture_store["teardown"]` so later cleanup and diagnostics know it exists. ```19:68:utilities/resources.py def create_and_store_resource( client: "DynamicClient", fixture_store: dict[str, Any], resource: type[Resource], test_name: str | None = None, **kwargs: Any, ) -> Any: kwargs["client"] = client _resource_name = kwargs.get("name") _resource_dict = kwargs.get("kind_dict", {}) _resource_yaml = kwargs.get("yaml_file") if not _resource_name: if _resource_yaml: with open(_resource_yaml) as fd: _resource_dict = yaml.safe_load(fd) _resource_name = _resource_dict.get("metadata", {}).get("name") if not _resource_name: _resource_name = generate_name_with_uuid(name=fixture_store["base_resource_name"]) if resource.kind in (Migration.kind, Plan.kind): _resource_name = f"{_resource_name}-{'warm' if kwargs.get('warm_migration') else 'cold'}" if len(_resource_name) > 63: LOGGER.warning(f"'{_resource_name=}' is too long ({len(_resource_name)} > 63). Truncating.") _resource_name = _resource_name[-63:] kwargs["name"] = _resource_name _resource = resource(**kwargs) try: _resource.deploy(wait=True) except ConflictError: LOGGER.warning(f"{_resource.kind} {_resource_name} already exists, reusing it.") _resource.wait() LOGGER.info(f"Storing {_resource.kind} {_resource.name} in fixture store") _resource_dict = {"name": _resource.name, "namespace": _resource.namespace, "module": _resource.__module__} if test_name: _resource_dict["test_name"] = test_name fixture_store["teardown"].setdefault(_resource.kind, []).append(_resource_dict) return _resource ``` `get_or_create_namespace()` builds on that helper. It reuses an existing namespace when possible, but when it creates one itself it applies the standard labels used across this suite, including `pod-security.kubernetes.io/enforce=restricted`. > **Tip:** Use `create_and_store_resource()` for anything that creates a cluster object during a test. That is what makes later teardown, `resources.json`, and leftover detection work. ## Migration Orchestration ### `utilities.mtv_migration` `utilities.mtv_migration` is the module most test authors reuse first. It owns the standard flow for creating `StorageMap`, `NetworkMap`, `Plan`, and `Migration` resources and waiting for them to reach the right state. Most tests follow the same pattern: ```96:147:tests/test_mtv_cold_migration.py populate_vm_ids(prepared_plan, source_provider_inventory) self.__class__.plan_resource = create_plan_resource( ocp_admin_client=ocp_admin_client, fixture_store=fixture_store, source_provider=source_provider, destination_provider=destination_provider, storage_map=self.storage_map, network_map=self.network_map, virtual_machines_list=prepared_plan["virtual_machines"], target_namespace=target_namespace, warm_migration=prepared_plan.get("warm_migration", False), ) assert self.plan_resource, "Plan creation failed" execute_migration( ocp_admin_client=ocp_admin_client, fixture_store=fixture_store, plan=self.plan_resource, target_namespace=target_namespace, ) check_vms( plan=prepared_plan, source_provider=source_provider, destination_provider=destination_provider, network_map_resource=self.network_map, storage_map_resource=self.storage_map, source_provider_data=source_provider_data, source_vms_namespace=source_vms_namespace, source_provider_inventory=source_provider_inventory, vm_ssh_connections=vm_ssh_connections, ) ``` The most important entry points are: - `get_storage_migration_map()`: Creates a `StorageMap`. In the normal case it derives mappings from provider inventory and uses `py_config["storage_class"]` unless you override it. - `get_network_migration_map()`: Creates a `NetworkMap`. The first source network maps to the pod network, and additional networks map to generated Multus NADs. - `create_plan_resource()`: Creates the `Plan` CR and waits for `Plan.Condition.READY=True`. - `execute_migration()`: Creates the `Migration` CR and waits for the plan to finish. - `wait_for_migration_complate()`: Polls the plan until it reaches `Succeeded` or `Failed`. - `verify_vm_disk_count()` and `wait_for_concurrent_migration_execution()`: Specialized helpers used by copy-offload and multi-plan scenarios. `conftest.py` does a lot of prep work before these helpers run. In particular, `prepared_plan` deep-copies the class config, clones or discovers source VMs, stores source-side facts in `source_vms_data`, creates configured hooks, and resolves `_vm_target_namespace` so later validation looks in the right namespace. The plan config can drive much more than just warm versus cold migration. This warm migration example from `tests/tests_config/config.py` enables custom VM namespace placement, static IP preservation, PVC naming, labels, and affinity: ```434:466:tests/tests_config/config.py "test_warm_migration_comprehensive": { "virtual_machines": [ { "name": "mtv-win2022-ip-3disks", "source_vm_power": "on", "guest_agent": True, }, ], "warm_migration": True, "target_power_state": "on", "preserve_static_ips": True, "vm_target_namespace": "custom-vm-namespace", "multus_namespace": "default", "pvc_name_template": '{{ .FileName | trimSuffix ".vmdk" | replace "_" "-" }}-{{.DiskIndex}}', "pvc_name_template_use_generate_name": True, "target_labels": { "mtv-comprehensive-test": None, "static-label": "static-value", }, "target_affinity": { "podAffinity": { "preferredDuringSchedulingIgnoredDuringExecution": [ { "podAffinityTerm": { "labelSelector": {"matchLabels": {"app": "comprehensive-test"}}, "topologyKey": "kubernetes.io/hostname", }, "weight": 75, } ] } }, }, ``` In this repository, `None` under keys like `target_labels` or `target_node_selector` is a placeholder for “fill this with the current `session_uuid`,” which keeps parallel test runs from colliding. The same module also supports copy-offload storage maps. When `datastore_id` and `offload_plugin_config` are passed, `get_storage_migration_map()` switches from inventory-derived mappings to explicit XCOPY-capable datastore mappings: ```84:112:tests/test_copyoffload_migration.py copyoffload_config_data = source_provider_data["copyoffload"] storage_vendor_product = copyoffload_config_data["storage_vendor_product"] datastore_id = copyoffload_config_data["datastore_id"] storage_class = py_config["storage_class"] vms_names = [vm["name"] for vm in prepared_plan["virtual_machines"]] offload_plugin_config = { "vsphereXcopyConfig": { "secretRef": copyoffload_storage_secret.name, "storageVendorProduct": storage_vendor_product, } } self.__class__.storage_map = get_storage_migration_map( fixture_store=fixture_store, target_namespace=target_namespace, source_provider=source_provider, destination_provider=destination_provider, ocp_admin_client=ocp_admin_client, source_provider_inventory=source_provider_inventory, vms=vms_names, storage_class=storage_class, datastore_id=datastore_id, offload_plugin_config=offload_plugin_config, access_mode="ReadWriteOnce", volume_mode="Block", ) ``` For warm migrations, `execute_migration()` is often paired with `get_cutover_value()` from `utilities.migration_utils`, which computes the cutover time from `mins_before_cutover`. ### `utilities.hooks` `utilities.hooks` lets plan configuration create pre- and post-migration Hook CRs without hand-writing YAML in every test. It supports two modes: - `expected_result`: Use one of the built-in playbooks for a hook that should succeed or fail. - `playbook_base64`: Supply your own base64-encoded Ansible playbook. The module validates that you set exactly one of those options, and it rejects invalid base64, invalid UTF-8, invalid YAML, or playbooks that are not valid Ansible play lists. A test scenario that intentionally keeps the migrated VM after a failing post hook is configured like this: ```503:515:tests/tests_config/config.py "test_post_hook_retain_failed_vm": { "virtual_machines": [ { "name": "mtv-tests-rhel8", "source_vm_power": "on", "guest_agent": True, }, ], "warm_migration": False, "target_power_state": "off", "pre_hook": {"expected_result": "succeed"}, "post_hook": {"expected_result": "fail"}, "expected_migration_result": "fail", }, ``` `create_hook_if_configured()` stores the generated hook name and namespace back into the prepared plan as `_pre_hook_name`, `_pre_hook_namespace`, `_post_hook_name`, and `_post_hook_namespace`, so `create_plan_resource()` can pass them into the `Plan` CR. `validate_hook_failure_and_check_vms()` is the helper that makes expected failures practical: - If the migration failed in `PreHook`, it returns `False`, because the VM was never migrated. - If the migration failed in `PostHook`, it returns `True`, because the VM may already exist and should still be validated. > **Tip:** Use `expected_result` when you only need to exercise hook success or failure behavior. Switch to `playbook_base64` when the hook needs custom logic. ## Access and Validation ### `utilities.virtctl` and `utilities.ssh_utils` This repository does not rely on node IP access for guest validation. Instead, it uses `virtctl port-forward` to reach a KubeVirt VM locally, then hands that tunnel to `python-rrmngmnt` for SSH operations. `utilities.virtctl` makes that possible by locating or downloading a matching `virtctl` binary. It first checks whether `virtctl` is already in `PATH`, then checks for a previously downloaded copy, and only then falls back to downloading from the cluster’s `ConsoleCLIDownload` resource. The downloader knows how to match Linux and macOS builds and `x86_64` or `arm64` architectures. `conftest.py` exposes this through the `virtctl_binary` fixture, which caches the binary in a cluster-versioned shared temp directory and uses a file lock so parallel `pytest-xdist` workers do not all download it at once. The actual SSH tunnel command is built in `VMSSHConnection.setup_port_forward()`: ```98:141:utilities/ssh_utils.py virtctl_path = shutil.which("virtctl") if not virtctl_path: raise RuntimeError( "virtctl command not found in PATH. " "Please install virtctl before running the test suite. " "See README.md for installation instructions." ) if local_port is None: with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: s.bind(("", 0)) local_port = s.getsockname()[1] cmd = [ virtctl_path, "port-forward", f"vm/{self.vm.name}", f"{local_port}:22", "--namespace", self.vm.namespace, "--address", "127.0.0.1", ] if self.ocp_api_server: cmd.extend(["--server", self.ocp_api_server]) if self.ocp_token: cmd.extend(["--token", self.ocp_token]) if self.ocp_insecure: cmd.append("--insecure-skip-tls-verify") cmd.extend(["-v", "3"]) cmd_str = " ".join(cmd) if self.ocp_token: cmd_str = cmd_str.replace(self.ocp_token, "[REDACTED]") LOGGER.info(f"Full virtctl command: {cmd_str}") ``` `SSHConnectionManager` is the higher-level wrapper most tests use. It creates VM connections through the destination provider, extracts the OpenShift API token on demand, and keeps track of all open connections so fixture teardown can close them cleanly. > **Note:** This port-forward approach means SSH validation works even when worker nodes do not have public IPs. The actual guest credentials come from `.providers.json`: - `guest_vm_linux_user` and `guest_vm_linux_password` for Linux guests - `guest_vm_win_user` and `guest_vm_win_password` for Windows guests > **Warning:** `check_vms()` only tries SSH when the destination VM is powered on. ### `utilities.post_migration` `utilities.post_migration` is the high-level “did the migration really work?” module. Its main entry point, `check_vms()`, looks up the source and destination VM objects, runs a broad set of focused validators, aggregates failures per VM, and only fails at the end. That gives you a much fuller error picture than stopping at the first failed assertion. Depending on provider type and plan options, `check_vms()` can verify: - Power state, CPU, and memory - Network and storage mappings - PVC names from `pvcNameTemplate` - Snapshot preservation for vSphere - BIOS serial preservation for vSphere, including the OCP 4.20+ format change - Guest agent availability - SSH connectivity - Static IP preservation - Node placement - VM labels - Affinity - Provider secret SSL settings versus `source_provider_insecure_skip_verify` A few behaviors matter in practice: - `check_vms()` uses `plan["_vm_target_namespace"]`, so custom `vm_target_namespace` settings work automatically. - `check_pvc_names()` understands Go-template-style `pvcNameTemplate` values, including `{{.VmName}}`, `{{.DiskIndex}}`, `{{.FileName}}`, and Sprig functions. If `pvc_name_template_use_generate_name` is `True`, it switches from exact matching to prefix matching. - `check_ssl_configuration()` does a useful safety check: it compares the global source-provider SSL setting with the actual `insecureSkipVerify` value stored in the Provider secret. > **Warning:** `check_static_ip_preservation()` is currently implemented only for Windows guests migrated from vSphere. > **Tip:** Use `check_vms()` when you want the repository’s full standard validation bundle. If a test only cares about one behavior, calling a focused helper such as `check_vm_labels()` or `check_serial_preservation()` is often cleaner. ## Diagnostics and Teardown ### `utilities.must_gather` `utilities.must_gather` is the repository’s failure-time diagnostic collector. It does not hardcode a must-gather image. Instead, it looks up the installed MTV `ClusterServiceVersion`, resolves the matching image digest mirror set, and builds the final image reference from the installed SHA. That keeps must-gather aligned with the cluster’s actual operator version. When a plan is known, it runs targeted collection: ```166:181:utilities/must_gather.py must_gather_image = _resolve_must_gather_image( ocp_admin_client=ocp_admin_client, mtv_subs=mtv_subs, mtv_csv=mtv_csv, ) _must_gather_base_cmd = f"oc adm must-gather --image={must_gather_image} --dest-dir={data_collector_path}" if plan: plan_name = plan["name"] plan_namespace = plan["namespace"] run_command( shlex.split(f"{_must_gather_base_cmd} -- NS={plan_namespace} PLAN={plan_name} /usr/bin/targeted") ) else: run_command(shlex.split(f"{_must_gather_base_cmd} -- -- NS={mtv_namespace}")) ``` That targeted mode is especially useful when a single migration plan failed and you want operator-side data for that specific plan instead of a much broader dump. Errors in `run_must_gather()` are logged, but the helper does not crash the whole test run just because diagnostics collection failed. ### `utilities.pytest_utils` and `utilities.migration_utils` These two modules are the cleanup and safety-net layer. `utilities.pytest_utils.session_teardown()` is the top-level cleanup entry point. It cancels running migrations, archives plans, then hands off to the deeper resource deletion logic: ```107:128:utilities/pytest_utils.py def session_teardown(session_store: dict[str, Any]) -> None: LOGGER.info("Running teardown to delete all created resources") ocp_client = get_cluster_client() # When running in parallel (-n auto) `session_store` can be empty. if session_teardown_resources := session_store.get("teardown"): for migration_name in session_teardown_resources.get(Migration.kind, []): migration = Migration(name=migration_name["name"], namespace=migration_name["namespace"], client=ocp_client) cancel_migration(migration=migration) for plan_name in session_teardown_resources.get(Plan.kind, []): plan = Plan(name=plan_name["name"], namespace=plan_name["namespace"], client=ocp_client) archive_plan(plan=plan) leftovers = teardown_resources( session_store=session_store, ocp_client=ocp_client, target_namespace=session_store.get("target_namespace"), ) if leftovers: raise SessionTeardownError(f"Failed to clean up the following resources: {leftovers}") ``` From there, the cleanup path does a few important things: - `collect_created_resources()` writes the tracked resource list to `resources.json` under the data collector path. - `teardown_resources()` deletes tracked `Migration`, `Plan`, `Provider`, `Secret`, `StorageMap`, `NetworkMap`, `Namespace`, and other resources. - `cancel_migration()` cancels only migrations that are still running. - `archive_plan()` marks plans as archived and waits for plan-owned pods to disappear. - `check_dv_pvc_pv_deleted()` waits for `DataVolume`, `PersistentVolumeClaim`, and `PersistentVolume` cleanup in parallel. - `pytest_exception_interact` and session-finish hooks call `run_must_gather()` when data collection is enabled and failures or leftovers justify it. There is also a nearer, class-scoped cleanup path in `conftest.py`: `cleanup_migrated_vms` deletes migrated VMs after each test class finishes, including cases where VMs were intentionally migrated into a custom namespace. Session teardown is the backstop if anything survives beyond that. > **Warning:** `--skip-teardown` is a debugging tool, not a normal operating mode. It leaves migrated VMs and tracked resources behind on purpose. ## Configuration Notes A few configuration points affect these modules over and over: - Global session settings in `tests/tests_config/config.py` include `insecure_verify_skip`, `source_provider_insecure_skip_verify`, `snapshots_interval`, `mins_before_cutover`, and `plan_wait_timeout`. - Per-test entries in `tests/tests_config/config.py` control feature behavior with keys such as `warm_migration`, `target_power_state`, `preserve_static_ips`, `vm_target_namespace`, `pvc_name_template`, `pvc_name_template_use_generate_name`, `target_labels`, `target_affinity`, `target_node_selector`, `pre_hook`, and `post_hook`. - Provider-specific connection details, guest OS credentials, and copy-offload settings come from `.providers.json`. > **Tip:** If you are adding a new migration scenario, start by reusing `get_storage_migration_map()`, `get_network_migration_map()`, `create_plan_resource()`, `execute_migration()`, and `check_vms()`. That path matches the rest of the repository and gives you automatic teardown, SSH validation, and diagnostics with very little extra code. --- Source: troubleshooting-and-diagnostics.md # Troubleshooting And Diagnostics When a migration test fails, the fastest path is usually: 1. Read `pytest-tests.log` to find the first real failure. 2. Open `junit-report.xml` to see the CI-friendly result and embedded logs. 3. Inspect `.data-collector/` for tracked resources and any must-gather output. 4. Follow the resource names into the cluster: `Plan`, `Migration`, `Provider`, target namespace pods, and events. This repository already generates most of those artifacts for you by default. ## Start Here The repo enables JUnit reporting for every run and includes pytest logging in the XML: ```4:25:pytest.ini addopts = -s -o log_cli=true -p no:logging --tc-file=tests/tests_config/config.py --tc-format=python --junit-xml=junit-report.xml --basetemp=/tmp/pytest --show-progress --strict-markers --jira --dist=loadscope markers = tier0: Core functionality tests (smoke tests) remote: Remote cluster migration tests warm: Warm migration tests copyoffload: Copy-offload (XCOPY) tests incremental: marks tests as incremental (xfail on previous failure) min_mtv_version: mark test to require minimum MTV version (e.g., @pytest.mark.min_mtv_version("2.6.0")) junit_logging = all ``` That means a normal run gives you: - `pytest-tests.log`: the easiest way to see fixture setup, `SETUP` / `CALL` / `TEARDOWN`, and final `PASSED` / `FAILED` / `ERROR` status. - `junit-report.xml`: the structured result file that CI systems can ingest. - Console output with the same high-level status markers. If you run tests inside a container or an OpenShift Job, the default working directory is `/app`, so the JUnit file usually ends up at `/app/junit-report.xml`. > **Tip:** Test names are rewritten during collection to include the selected `source_provider` and `storage_class`, so those suffixes are useful when you are matching a JUnit failure back to the exact environment that ran. ## Diagnostic Flags That Matter A few pytest options change what diagnostics you get back: - `--analyze-with-ai` enables post-failure AI enrichment of the JUnit XML. - `--skip-data-collector` disables automatic artifact collection under `.data-collector/`. - `--data-collector-path` changes the artifact directory from the default `.data-collector`. - `--skip-teardown` leaves created resources in place so you can inspect them live. - `--openshift-python-wrapper-log-debug` turns on deeper wrapper-level logging, which is useful when CR creation or wait logic is failing. > **Warning:** `--skip-data-collector` disables both `resources.json` tracking and automatic must-gather collection. > **Warning:** `--skip-teardown` is very useful for debugging, but it also means cleanup becomes your responsibility. > **Tip:** If the failure is happening before MTV even starts a migration, `--openshift-python-wrapper-log-debug` often gives the most useful extra detail. ## JUnit Output `junit-report.xml` is the main machine-readable artifact. It is the best file to archive from local runs, OpenShift Jobs, or any external automation. Because `junit_logging = all` is enabled, the report is more useful than a bare pass/fail summary. It includes the logs that explain whether the failure happened during: - fixture setup - plan creation - migration execution - post-migration validation - teardown A passing example in the repo is very small: ```1:1:junit_report_example.xml ``` In real failure runs, the XML is much more informative because it includes logging and failure details from pytest. ## AI Enrichment When you pass `--analyze-with-ai`, the suite can send the raw JUnit XML to an analysis service and write the enriched XML back to the same file. ```423:479:utilities/pytest_utils.py xml_path_raw = getattr(session.config.option, "xmlpath", None) if not xml_path_raw: LOGGER.warning("xunit file not found; pass --junitxml. Skipping AI analysis enrichment") return xml_path = Path(xml_path_raw) if not xml_path.exists(): LOGGER.warning( "xunit file not found under %s. Skipping AI analysis enrichment", xml_path_raw, ) return # ... provider/model validation omitted ... response = requests.post( f"{server_url.rstrip('/')}/analyze-failures", json={ "raw_xml": raw_xml, "ai_provider": ai_provider, "ai_model": ai_model, }, timeout=timeout_value, ) response.raise_for_status() result = response.json() if enriched_xml := result.get("enriched_xml"): xml_path.write_text(enriched_xml) LOGGER.info("JUnit XML enriched with AI analysis: %s", xml_path) ``` Important behavior: - AI enrichment only runs when the session exits with failures. - It requires a JUnit XML path to exist. - It requires `JJI_SERVER_URL`. - If `JJI_AI_PROVIDER` and `JJI_AI_MODEL` are not set, the code defaults them to `claude` and `claude-opus-4-6[1m]`. - `JJI_TIMEOUT` controls the request timeout and defaults to `600` seconds. - If enrichment fails, the original JUnit file is preserved. > **Note:** If `JJI_SERVER_URL` is not set, the suite logs a warning and disables AI analysis instead of failing the run. > **Warning:** AI enrichment rewrites the existing `junit-report.xml` in place. ## Collected Resource Artifacts Unless you use `--skip-data-collector`, the suite prepares a clean `.data-collector/` directory at session start and writes run artifacts there. The most important files are: - `.data-collector/resources.json`: a dump of tracked resources created during the run. - `.data-collector//...`: per-failure must-gather output when the failure hook runs. - `.data-collector/...` at the root: session-level must-gather output when teardown cleanup fails. The tracked resource file is especially useful after a partial or messy failure because it tells you exactly which names and namespaces were created. The cleanup helper in `tools/clean_cluster.py` is built to consume that file. The suite stores resource metadata as it creates objects, including `name`, `namespace`, `module`, and sometimes `test_name`. That gives you a practical bridge from “the test failed” to “which exact `Plan`, `Provider`, `StorageMap`, or `NetworkMap` should I inspect?” > **Tip:** If you rerun the test suite, the base data collector path is recreated. Move or archive old artifacts first if you want to keep them. ## Must-Gather Support The suite has built-in must-gather support. When it can match a failure to a specific plan, it can run a targeted gather. Otherwise it falls back to an MTV-namespace gather. ```166:181:utilities/must_gather.py must_gather_image = _resolve_must_gather_image( ocp_admin_client=ocp_admin_client, mtv_subs=mtv_subs, mtv_csv=mtv_csv, ) _must_gather_base_cmd = f"oc adm must-gather --image={must_gather_image} --dest-dir={data_collector_path}" if plan: plan_name = plan["name"] plan_namespace = plan["namespace"] run_command( shlex.split(f"{_must_gather_base_cmd} -- NS={plan_namespace} PLAN={plan_name} /usr/bin/targeted") ) else: run_command(shlex.split(f"{_must_gather_base_cmd} -- -- NS={mtv_namespace}")) ``` What this means in practice: - If the failure can be tied back to a specific plan, the gather can be scoped to that plan. - If the failure happens earlier, or there is no plan match, the gather is scoped to the MTV namespace instead. - If teardown leaves leftovers behind, a session-level must-gather is collected as a fallback. > **Note:** The must-gather image is not hardcoded. The code resolves it from the installed MTV operator CSV and ImageDigestMirrorSet, so the gather matches the cluster’s installed MTV build. > **Tip:** A per-test must-gather directory is often the quickest way to answer “what did the cluster look like at the moment this test failed?” ## Common Failure Points Most failures in this repo fall into one of these buckets. ### Before Migration Starts - Missing required config such as `storage_class` or `source_provider`. - Missing or empty `.providers.json`. - A provider key passed with `--tc=source_provider:...` that does not exist in `.providers.json`. - Wrong cluster credentials from `cluster_host`, `cluster_username`, or `cluster_password`. - SSL verification mismatches between your config and the created provider secret. - `forklift-*` pods not being healthy before tests begin. These failures usually show up in fixtures or session startup, before you ever get a `Migration` CR. ### During Provider, Map, or Plan Setup - The source `Provider` CR never becomes ready. - The source provider endpoint is unreachable or credentials are wrong. - The source VM is missing from inventory. - The source VM has no networks, so `NetworkMap` generation fails. - `StorageMap` points to the wrong storage class or invalid copy-offload datastores. - A remote-cluster configuration mismatch prevents the OpenShift client setup from proceeding. These issues usually show up as setup failures, plan readiness timeouts, or early MTV resource errors. ### During Migration Execution - The `Plan` becomes ready, but the migration never reaches `Succeeded`. - The migration reaches `Failed`. - The run times out waiting for migration completion. - Hook execution fails at `PreHook` or `PostHook`. - Warm migration timing is wrong for the environment. The repo’s default timeout config is important here: - `plan_wait_timeout` defaults to `3600` seconds. - `mins_before_cutover` defaults to `5`. If a migration is stuck, the most important resources are the `Plan`, the `Migration`, and the relevant MTV controller or conversion pod logs. ### After Migration Completes A migration can succeed and the test can still fail later. The post-migration validation is broad and covers things like: - VM power state - CPU and memory - network mapping - storage mapping and storage class - PVC naming - guest agent - SSH access - static IP preservation - labels - affinity - node placement - VMware snapshot and serial preservation - RHV-specific power-off behavior So “migration failed” and “test failed” are not always the same thing. > **Tip:** If the test only fails in `test_check_vms`, the migration itself may already be complete. At that point, spend less time in the initial `Plan` conditions and more time on the migrated VM, its PVCs, its launcher/VMI state, and guest-level checks. ## Copy-Offload-Specific Checks Copy-offload adds extra prerequisites, so it also adds extra failure modes. ```27:58:.providers.json.example "copyoffload": { # Supported storage_vendor_product values: # - "ontap" (NetApp ONTAP) # - "vantara" (Hitachi Vantara) # - "primera3par" (HPE Primera/3PAR) # - "pureFlashArray" (Pure Storage FlashArray) # - "powerflex" (Dell PowerFlex) # - "powermax" (Dell PowerMax) # - "powerstore" (Dell PowerStore) # - "infinibox" (Infinidat InfiniBox) # - "flashsystem" (IBM FlashSystem) "storage_vendor_product": "ontap", # Primary datastore for copy-offload operations (required) "datastore_id": "datastore-12345", # Optional: Secondary datastore for multi-datastore copy-offload tests "secondary_datastore_id": "datastore-67890", # Optional: Non-XCOPY datastore for mixed datastore tests "non_xcopy_datastore_id": "datastore-99999", "default_vm_name": "rhel9-template", "storage_hostname": "storage.example.com", "storage_username": "admin", "storage_password": "your-password-here", # pragma: allowlist secret ``` Check these first for copy-offload failures: - `storage_vendor_product` is correct for your storage backend. - `datastore_id` is valid and matches where the source VM disks actually live. - `secondary_datastore_id` and `non_xcopy_datastore_id` are only used when the test really needs them. - Storage credentials are present either in `.providers.json` or the matching `COPYOFFLOAD_*` environment variables. - Vendor-specific fields are set when your selected storage vendor requires them. - If you use SSH cloning, the ESXi host, user, and password are correct. - If the offload path itself is the problem, inspect `vsphere-xcopy-volume-populator` logs in `openshift-mtv` in addition to the normal MTV controller logs. > **Warning:** Copy-offload failures are often configuration issues first, product issues second. Always verify the storage backend, datastore IDs, and vendor-specific fields before assuming the migration code is at fault. ## Expected Failures and Hook Tests Not every `MigrationPlanExecError` means something is wrong. This repo contains tests that intentionally expect migration failure and then validate how that failure happened. A good example is `test_post_hook_retain_failed_vm`, which expects the migration to fail and then checks whether the failure happened at the expected hook step: ```195:205:tests/test_post_hook_retain_failed_vm.py expected_result = prepared_plan["expected_migration_result"] if expected_result == "fail": with pytest.raises(MigrationPlanExecError): execute_migration( ocp_admin_client=ocp_admin_client, fixture_store=fixture_store, plan=self.plan_resource, target_namespace=target_namespace, ) self.__class__.should_check_vms = validate_hook_failure_and_check_vms(self.plan_resource, prepared_plan) ``` Why this matters: - Some tests deliberately expect `PreHook` or `PostHook` failure. - A `PostHook` failure can still leave a migrated VM behind, and that VM is worth inspecting. - The real question is often not “did it fail?” but “did it fail at the expected step?” > **Tip:** In class-based incremental tests, focus on the first real failure. Later tests in the same class may be marked `xfail` because the earlier step already failed. ## Best Places To Inspect When Migrations Fail | Inspect This | What It Tells You | Best For | | --- | --- | --- | | `pytest-tests.log` | First failing phase, fixture flow, high-level status | Early setup failures and quick triage | | `junit-report.xml` | Structured results, embedded logs, optional AI analysis | CI artifacts and cross-run comparisons | | `.data-collector/resources.json` | Exact resource names and namespaces created during the run | Finding or cleaning up leftovers | | `.data-collector//` | Must-gather output captured near the failure | Deep cluster-side diagnosis | | `Plan` CR | Readiness, mappings, target namespace, conditions | Plan creation and migration start issues | | `Migration` CR | Per-VM pipeline details in `status.vms[].pipeline[]` | Mid-migration failures and hook step debugging | | `Provider` CR | Connection and readiness status | Source or destination connectivity problems | | `forklift-controller` logs | MTV orchestration and controller-side errors | Plan or migration logic failures | | `forklift-inventory` logs | Inventory sync and source discovery issues | VM lookup and provider inventory problems | | Target namespace events and pods | `virt-v2v`, DV/PVC/PV, VM/VMI, launcher behavior | Transfer, boot, and runtime issues | | Source provider logs | vCenter, RHV, or OpenStack-side errors | External provider problems | A practical first command set is: - `oc logs -n openshift-mtv deployment/forklift-controller` - `oc get migration -n -o yaml` - `oc get plan -n -o yaml` - `oc get provider -n -o yaml` - `oc get vm -n -o yaml` - `oc get events -n --sort-by='.lastTimestamp'` For copy-offload runs, also inspect the volume populator logs in `openshift-mtv`. ## A Good Debugging Order If you want one repeatable process, use this: 1. Open `pytest-tests.log` and identify the first failing test and phase. 2. Check whether the failure is an expected one, especially in hook tests or incremental classes. 3. Read `junit-report.xml` for the same test case and capture the resource names involved. 4. Open `.data-collector/resources.json` and any must-gather output under `.data-collector/`. 5. Inspect the `Plan`, `Migration`, and `Provider` CRs. 6. Check `forklift-controller`, `forklift-inventory`, and target-namespace pod logs. 7. If the migration succeeded but the test still failed, inspect the migrated VM, its PVCs, its VMI/launcher state, and guest-level connectivity instead of only looking at the controller. That order lines up with how this repository itself reports and classifies failures, and it usually gets you to the root cause faster than starting from cluster logs alone. --- Source: cleanup-and-teardown.md # Cleanup And Teardown `mtv-api-tests` cleans up automatically in normal runs. The project uses two cleanup layers: class-level cleanup for migrated VMs, and session-level cleanup for the rest of the resources the run created. If you need to stop that behavior for debugging, `--skip-teardown` preserves the environment so you can inspect it manually. ## Default Behavior A standard run does all of the following: - Tracks created resources as they are created. - Removes migrated `VirtualMachine` objects when each test class finishes. - Runs a broader session teardown at the end of pytest. - Writes a resource inventory to `.data-collector/resources.json` unless you disable the data collector. Resource tracking starts in `utilities/resources.py`: ```python LOGGER.info(f"Storing {_resource.kind} {_resource.name} in fixture store") _resource_dict = {"name": _resource.name, "namespace": _resource.namespace, "module": _resource.__module__} if test_name: _resource_dict["test_name"] = test_name fixture_store["teardown"].setdefault(_resource.kind, []).append(_resource_dict) ``` Anything created through `create_and_store_resource()` is registered automatically, which is why teardown can later find it again without guessing names or namespaces. Not all cleanup waits until session end. Some fixture-scoped helpers clean up immediately after use. For example, SSH test connections are closed with `cleanup_all()`, copy-offload SSH keys are removed after the fixture yields, and temporary cluster edits made with `ResourceEditor` are reverted automatically when their context exits. ## Automatic Cleanup Flow ### Per-class VM cleanup The standard class-based tests opt into VM cleanup explicitly. From `tests/test_mtv_cold_migration.py`: ```python @pytest.mark.usefixtures("cleanup_migrated_vms") class TestSanityColdMtvMigration: """Cold migration test - sanity check.""" ``` That fixture runs after the class completes. From `conftest.py`: ```python yield if request.config.getoption("skip_teardown"): LOGGER.info("Skipping VM cleanup due to --skip-teardown flag") return vm_namespace = prepared_plan.get("_vm_target_namespace", target_namespace) for vm in prepared_plan["virtual_machines"]: vm_name = vm["name"] vm_obj = VirtualMachine( client=ocp_admin_client, name=vm_name, namespace=vm_namespace, ) if vm_obj.exists: LOGGER.info(f"Cleaning up migrated VM: {vm_name} from namespace: {vm_namespace}") vm_obj.clean_up() ``` A few practical details come from that logic: - `cleanup_migrated_vms` is teardown-only. It does not set anything up; it just removes migrated VMs after the class. - The same `--skip-teardown` flag disables this VM cleanup too. - If the plan migrated into a custom `vm_target_namespace`, that namespace is used automatically. ### Session teardown At the end of the session, `conftest.py` writes the resource inventory and then decides whether to run teardown: ```python if not session.config.getoption("skip_data_collector"): collect_created_resources(session_store=_session_store, data_collector_path=_data_collector_path) if session.config.getoption("skip_teardown"): LOGGER.warning("User requested to skip teardown of resources") else: try: session_teardown(session_store=_session_store) except Exception as exp: LOGGER.error(f"the following resources was left after tests are finished: {exp}") if not session.config.getoption("skip_data_collector"): run_must_gather(data_collector_path=_data_collector_path) ``` The session teardown in `utilities/pytest_utils.py` starts by cancelling active migrations and archiving plans: ```python if session_teardown_resources := session_store.get("teardown"): for migration_name in session_teardown_resources.get(Migration.kind, []): migration = Migration(name=migration_name["name"], namespace=migration_name["namespace"], client=ocp_client) cancel_migration(migration=migration) for plan_name in session_teardown_resources.get(Plan.kind, []): plan = Plan(name=plan_name["name"], namespace=plan_name["namespace"], client=ocp_client) archive_plan(plan=plan) leftovers = teardown_resources( session_store=session_store, ocp_client=ocp_client, target_namespace=session_store.get("target_namespace"), ) if leftovers: raise SessionTeardownError(f"Failed to clean up the following resources: {leftovers}") ``` From there, `teardown_resources()` works through the rest of the inventory. In practice, the session-level sweep covers: - `Migration` and `Plan` resources. - `Provider`, `Secret`, and `Host` resources. - `NetworkAttachmentDefinition`, `StorageMap`, and `NetworkMap` resources. - Tracked `VirtualMachine` and `Pod` resources. - Namespaces created during the run. - Source-side cloned VMs for VMware, OpenStack, and RHV. - OpenStack volume snapshots that were recorded during clone preparation. It also performs extra cleanup and verification in the target namespace by: - Deleting any remaining VMs with `delete_all_vms()`. - Waiting for matching pods to disappear. - Waiting for matching `DataVolume`, `PersistentVolumeClaim`, and `PersistentVolume` objects to be deleted. > **Note:** `.data-collector/resources.json` is written before session teardown runs. That means the file is available both when you use `--skip-teardown` and when teardown later reports a problem. ### Leftover detection Teardown is more than a best-effort delete loop. The code explicitly tracks leftovers and raises `SessionTeardownError` if resources are still present after cleanup attempts. That leftover detection is especially important for migration side effects such as pods, PVCs, and PVs. The session code looks for objects tied to the current run’s session UUID and records anything that did not disappear cleanly. If the data collector is enabled and teardown hits a problem, the session then runs MTV `must-gather` to capture diagnostics in the same collector path. > **Warning:** Leftover teardown problems are currently surfaced through session-finish logging. `pytest_sessionfinish()` logs the teardown exception and can trigger `must-gather`, but it does not re-raise that exception after logging it. Always check the end-of-run output, not just the individual test results. ## Debugging With `--skip-teardown` The user-facing flag lives in `conftest.py`: ```python teardown_group.addoption( "--skip-teardown", action="store_true", help="Do not teardown resource created by the tests" ) ``` The data-collector path is configurable too: ```python data_collector_group.addoption( "--data-collector-path", help="Path to store collected data for failed tests", default=".data-collector" ) ``` A repository example from `docs/copyoffload/how-to-run-copyoffload-tests.md` shows the intended usage inside a job command: ```yaml # In the Job command section, add --skip-teardown: uv run pytest -m copyoffload --skip-teardown \ -v \ ... ``` Use `--skip-teardown` when you want to inspect the environment after a run, for example: - The migrated VMs that were created in the target namespace. - The `Plan`, `Migration`, `StorageMap`, and `NetworkMap` objects. - The pods, PVCs, DataVolumes, and provider-side clones that would normally be removed automatically. > **Warning:** `--skip-teardown` disables both cleanup layers. The class-level `cleanup_migrated_vms` fixture returns early, and the end-of-session `session_teardown()` call is skipped entirely. > **Tip:** If you keep resources for debugging, do not also use `--skip-data-collector` unless you truly want no tracking artifacts. Leaving the data collector enabled gives you `.data-collector/resources.json`, which is the easiest input for follow-up cleanup. ## Manual Cleanup Helpers ### Use the recorded resource inventory When the data collector is enabled, the run writes a JSON inventory of the resources it created. From `utilities/pytest_utils.py`: ```python if resources: try: LOGGER.info(f"Write created resources data to {data_collector_path}/resources.json") with open(data_collector_path / "resources.json", "w") as fd: json.dump(session_store["teardown"], fd) ``` The repository includes a standalone helper script for replaying that inventory. From `tools/clean_cluster.py`: ```python def clean_cluster_by_resources_file(resources_file: str) -> None: with open(resources_file, "r") as fd: data: dict[str, list[dict[str, str]]] = json.load(fd) for _resource_kind, _resources_list in data.items(): for _resource in _resources_list: _resource_module = importlib.import_module(_resource["module"]) _resource_class = getattr(_resource_module, _resource_kind) _kwargs = {"name": _resource["name"]} if _resource.get("namespace"): _kwargs["namespace"] = _resource["namespace"] _resource_class(**_kwargs).clean_up() ``` The CLI entrypoint shows the expected usage format: ```python if __name__ == "__main__": if len(sys.argv) != 2: print("Usage: python clean_cluster.py ") sys.exit(1) clean_cluster_by_resources_file(resources_file=sys.argv[1]) ``` With the default collector path, the input file is `.data-collector/resources.json`. If you used `--data-collector-path`, point the script at that directory’s `resources.json` instead. If you ran with `--skip-data-collector`, the helper has no inventory file to consume, and teardown failures will not trigger `must-gather`. > **Note:** `resources.json` is a record of what the session created, not a leftovers-only report. In a successful run, some or all of those resources may already be gone by the time you inspect the file. > **Warning:** `tools/clean_cluster.py` is best suited to OpenShift-side resources recorded through `create_and_store_resource()`, because it recreates objects from the stored `module` and resource kind. Provider-side clone cleanup for VMware/OpenStack/RHV and OpenStack volume snapshots is handled by the full session teardown logic in `utilities/pytest_utils.py`, not by this standalone helper. ### Use the session name to find leftovers When you need to clean up manually in OpenShift or in the source provider UI/CLI, the run-specific session UUID is your most useful search key. From `conftest.py`: ```python def session_uuid(fixture_store): _session_uuid = generate_name_with_uuid(name="auto") fixture_store["session_uuid"] = _session_uuid return _session_uuid ``` The main target namespace is built from that same session value: ```python unique_namespace_name = f"{session_uuid}{_target_namespace}"[:63] fixture_store["target_namespace"] = unique_namespace_name ``` Source-provider clones are named the same way. From `libs/base_provider.py`: ```python clone_vm_name = generate_name_with_uuid(f"{session_uuid}-{base_name}") ``` That means a single session prefix, typically something like `auto-ab12`, often shows up in all of these places: - The auto-created target namespace. - Auto-generated OpenShift resource names. - Source-provider clone names. > **Tip:** If you skipped teardown, start by finding the session prefix from the run logs or from `.data-collector/resources.json`, then search OpenShift and the source provider for that same prefix. > **Note:** Some tests use a custom `vm_target_namespace`. In those cases, manual cleanup needs to check that namespace too, because migrated VMs and their storage objects may live there instead of the default session target namespace. OpenShift-source runs can also create a `source_vms_namespace` named `-source-vms`, so check that namespace as well when cleaning manually. --- Source: extending-the-suite.md # Extending The Suite Most new coverage in `mtv-api-tests` follows the same recipe: 1. Add a scenario to `tests/tests_config/config.py`. 2. Point a class at that scenario with `class_plan_config`. 3. Keep the standard five-step migration flow. 4. Reuse the shared fixtures for setup, cleanup, and validation. 5. Extend provider or validation helpers only when the existing abstractions stop being enough. > **Note:** This suite is intentionally class-based. In most cases, adding a new test means adding one config entry and one new class, not building a brand-new setup stack. ## Add A Test Config `pytest.ini` wires pytest-testconfig to `tests/tests_config/config.py`, so new scenarios start there. A minimal cold-migration entry can be very small: ```python "test_sanity_cold_mtv_migration": { "virtual_machines": [ {"name": "mtv-tests-rhel8", "guest_agent": True}, ], "warm_migration": False, }, ``` When you need more coverage, keep the same shape and add the keys the suite already understands: ```python "test_cold_migration_comprehensive": { "virtual_machines": [ { "name": "mtv-win2019-3disks", "source_vm_power": "off", "guest_agent": True, }, ], "warm_migration": False, "target_power_state": "on", "preserve_static_ips": True, "pvc_name_template": "{{.VmName}}-disk-{{.DiskIndex}}", "pvc_name_template_use_generate_name": False, "target_node_selector": { "mtv-comprehensive-node": None, }, "target_labels": { "mtv-comprehensive-label": None, "test-type": "comprehensive", }, "target_affinity": { "podAffinity": { "preferredDuringSchedulingIgnoredDuringExecution": [ { "podAffinityTerm": { "labelSelector": {"matchLabels": {"app": "test"}}, "topologyKey": "kubernetes.io/hostname", }, "weight": 50, } ] } }, "vm_target_namespace": "mtv-comprehensive-vms", "multus_namespace": "default", # Cross-namespace NAD access }, ``` A few patterns are worth knowing up front: - `virtual_machines` is always the center of the scenario. - `warm_migration` controls whether the flow is warm or cold. - VM-level keys such as `source_vm_power`, `guest_agent`, `clone`, `disk_type`, `add_disks`, `snapshots`, and `clone_name` are already used by existing tests. - Plan-level keys such as `target_power_state`, `preserve_static_ips`, `pvc_name_template`, `vm_target_namespace`, `target_node_selector`, `target_labels`, `target_affinity`, `pre_hook`, `post_hook`, and `copyoffload` are already supported by the shared helpers. > **Tip:** In `target_node_selector` and `target_labels`, a value of `None` does not mean “missing”. The fixtures replace it with the current `session_uuid`, which makes it easy to create unique labels safely. > **Note:** The runtime plan is not the raw config entry. `prepared_plan` deep-copies the config, clones VMs when needed, updates VM names, creates hooks, and stores extra source VM metadata. In test methods, always work from `prepared_plan`, not the literal values from `config.py`. ## Follow The Five-Step Class Pattern The standard migration classes all use the same shape. The cold sanity test is the simplest example: ```python @pytest.mark.tier0 @pytest.mark.incremental @pytest.mark.parametrize( "class_plan_config", [ pytest.param( py_config["tests_params"]["test_sanity_cold_mtv_migration"], ) ], indirect=True, ids=["rhel8"], ) @pytest.mark.usefixtures("cleanup_migrated_vms") class TestSanityColdMtvMigration: """Cold migration test - sanity check.""" storage_map: StorageMap network_map: NetworkMap plan_resource: Plan ``` From there, the class follows the same five steps every time: 1. `test_create_storagemap()` builds the `StorageMap` with `get_storage_migration_map()`. 2. `test_create_networkmap()` builds the `NetworkMap` with `get_network_migration_map()`. 3. `test_create_plan()` populates VM IDs and creates the MTV `Plan` with `create_plan_resource()`. 4. `test_migrate_vms()` starts the migration with `execute_migration()`. 5. `test_check_vms()` validates the result with `check_vms()`. That pattern is consistent across: - `tests/test_mtv_cold_migration.py` - `tests/test_mtv_warm_migration.py` - `tests/test_cold_migration_comprehensive.py` - `tests/test_warm_migration_comprehensive.py` - `tests/test_copyoffload_migration.py` - `tests/test_post_hook_retain_failed_vm.py` The shared state also stays consistent: classes store `storage_map`, `network_map`, and `plan_resource` on the class itself so later steps can reuse them. > **Warning:** Keep `@pytest.mark.incremental` on these classes. The steps depend on each other, and the suite is written to stop later steps cleanly when an earlier one fails. When choosing markers, reuse the ones already declared in `pytest.ini`: - `tier0` for core migration coverage - `warm` for warm migration coverage - `remote` for remote-cluster destination coverage - `copyoffload` for XCOPY/copy-offload coverage - `incremental` for dependent class flows Warm classes also use `precopy_interval_forkliftcontroller`, and remote-destination classes switch from `destination_provider` to `destination_ocp_provider`. ## Reuse Fixtures Instead Of Rebuilding Setup Most of the hard work is already in `conftest.py` and the utility modules. Reuse that layer first. - `prepared_plan` is the main runtime plan fixture. It deep-copies the class config, prepares cloned VMs, tracks source VM metadata in `source_vms_data`, creates hooks when configured, and sets `_vm_target_namespace`. - `target_namespace` creates a unique namespace for migration resources and stores it for cleanup. - `source_provider` and `destination_provider` give you provider objects instead of raw credentials. - `source_provider_inventory` gives you the Forklift inventory view that the mapping helpers use. - `multus_network_name` automatically creates as many NetworkAttachmentDefinitions as the source VMs need and returns the base name and namespace that `get_network_migration_map()` expects. - `cleanup_migrated_vms` deletes migrated VMs after the class finishes and automatically uses the custom VM namespace if your plan sets `vm_target_namespace`. - `precopy_interval_forkliftcontroller` patches the `ForkliftController` for warm-migration snapshot timing, so warm tests should keep using it rather than patching the controller themselves. - `labeled_worker_node` and `target_vm_labels` are the fixtures to use when your config includes `target_node_selector` or `target_labels`. - `vm_ssh_connections` gives post-migration validation a reusable SSH connection manager. - `copyoffload_config`, `copyoffload_storage_secret`, `setup_copyoffload_ssh`, and `mixed_datastore_config` are the copy-offload-specific fixtures already used by the XCOPY tests. - `prepared_plan_1` and `prepared_plan_2` split a multi-VM plan into two independent plans for simultaneous migration coverage. If you need to create an extra OpenShift resource for a new scenario, use `create_and_store_resource()` instead of deploying it directly. That helper generates a safe name when needed, deploys the resource, and registers it in the fixture store for teardown. > **Tip:** `target_namespace` and `vm_target_namespace` are different things. `target_namespace` is where the migration resources live. `vm_target_namespace` is an optional plan setting that tells MTV to place the migrated VMs in a different namespace. ## Extend Provider Coverage Most test classes are already provider-neutral because they work through `source_provider`, `destination_provider`, and `source_provider_inventory`. In practice, extending provider coverage usually means keeping the same five-step class and passing a few extra provider-specific arguments. The copy-offload tests are a good example. They still use `get_storage_migration_map()`, but add provider-specific storage plugin data instead of rewriting the whole flow: ```python offload_plugin_config = { "vsphereXcopyConfig": { "secretRef": copyoffload_storage_secret.name, "storageVendorProduct": storage_vendor_product, } } self.__class__.storage_map = get_storage_migration_map( fixture_store=fixture_store, target_namespace=target_namespace, source_provider=source_provider, destination_provider=destination_provider, ocp_admin_client=ocp_admin_client, source_provider_inventory=source_provider_inventory, vms=vms_names, storage_class=storage_class, datastore_id=datastore_id, offload_plugin_config=offload_plugin_config, access_mode="ReadWriteOnce", volume_mode="Block", ) ``` That is the pattern to follow when you want to add provider-specific behavior: - Keep the class structure the same. - Keep using the shared map and plan helpers. - Add only the extra provider inputs the helper already supports. A few existing provider-specific patterns are already in the suite: - Warm migration tests gate unsupported source providers at module level with `pytest.mark.skipif(...)`. - Remote destination tests use `destination_ocp_provider` and skip when `remote_ocp_cluster` is not configured. - Copy-offload tests layer extra fixtures on top of the standard class flow rather than creating a separate framework. ### Adding A New Provider Backend If you need a brand-new provider type, there are two places where the provider/inventory pairing is wired together. One of them is `source_provider_inventory` in `conftest.py`: ```python providers = { Provider.ProviderType.OVA: OvaForkliftInventory, Provider.ProviderType.RHV: OvirtForkliftInventory, Provider.ProviderType.VSPHERE: VsphereForkliftInventory, Provider.ProviderType.OPENSHIFT: OpenshiftForkliftInventory, Provider.ProviderType.OPENSTACK: OpenstackForliftinventory, } ``` A new provider type needs all of the following: 1. A concrete `BaseProvider` implementation under `libs/providers/`. 2. A matching `ForkliftInventory` implementation in `libs/forklift_inventory.py`. 3. Registration in `utilities/utils.py:create_source_provider()` so the fixture layer can construct the provider from `.providers.json`. 4. Registration in `conftest.py:source_provider_inventory()` so the mapping helpers know how to query storage and network data. 5. A `vm_dict()` implementation that fills the fields the validators already expect, including CPU, memory, NICs, disks, power state, and any provider-specific metadata your checks need. The active source provider is selected from `.providers.json` through `load_source_providers()`, so provider coverage should usually be added by configuration first. Only add a new provider implementation when the suite genuinely needs a new backend, not just a new scenario. ## Extend Validation Coverage For most new test scenarios, the best place to add coverage is `utilities/post_migration.py`, not the `test_check_vms()` method itself. `check_vms()` is the central post-migration validator. It already covers: - power state - CPU and memory - network mapping - storage mapping - PVC naming templates - snapshots - serial preservation - guest agent state - SSH connectivity - static IP preservation - node placement - VM labels - VM affinity - RHV-specific power-off behavior The existing label, node-placement, and affinity checks show the pattern clearly: ```python if plan.get("target_node_selector") and labeled_worker_node: try: check_vm_node_placement( destination_vm=destination_vm, expected_node=labeled_worker_node["node_name"], ) except Exception as exp: res[vm_name].append(f"check_vm_node_placement - {str(exp)}") if plan.get("target_labels") and target_vm_labels: try: check_vm_labels( destination_vm=destination_vm, expected_labels=target_vm_labels["vm_labels"], ) except Exception as exp: res[vm_name].append(f"check_vm_labels - {str(exp)}") if plan.get("target_affinity"): try: check_vm_affinity( destination_vm=destination_vm, expected_affinity=plan["target_affinity"], ) except Exception as exp: res[vm_name].append(f"check_vm_affinity - {str(exp)}") ``` When you want to add a new validation, the usual path is: 1. Add a plan key to `tests/tests_config/config.py` if the validation is scenario-driven. 2. Collect any setup-time data in `prepared_plan` or a dedicated fixture. 3. Pass any plan-level MTV fields through `create_plan_resource()` if the validation depends on plan configuration. 4. Add a focused helper such as `check_vm_labels()` or `check_pvc_names()` to `utilities/post_migration.py`. 5. Call that helper from `check_vms()` behind an `if plan.get("your_key"):` guard. This keeps the test classes simple. The class still ends with `check_vms()`, and the validation logic stays in one place. > **Tip:** Negative-path tests should still keep the five-step flow. `tests/test_post_hook_retain_failed_vm.py` shows the pattern: wrap `execute_migration()` in `pytest.raises(MigrationPlanExecError)` when failure is expected, then decide whether `check_vms()` should still run based on where the failure happened. ## Validate And Collect Your New Tests The repository does not include a checked-in GitHub Actions or GitLab pipeline file. The validation path that is checked into the repo is visible in `pytest.ini`, `tox.toml`, `Dockerfile`, and `.pre-commit-config.yaml`. `tox.toml` already defines the first validation pass for new tests: ```toml [env.pytest-check] commands = [ [ "uv", "run", "pytest", "--setup-plan", ], [ "uv", "run", "pytest", "--collect-only", ], ] ``` That leads to a practical workflow for new suite extensions: - Run `uv run pytest --collect-only` first. It is also the default `CMD` in the `Dockerfile`, which makes test discovery a first-class check in this repo. - Run `uv run pytest --setup-plan` or `tox -e pytest-check` to catch setup and collection issues before trying a full migration run. - Run `pre-commit run --all-files` before you send changes out. The repo’s hooks include `flake8`, `ruff`, `ruff-format`, `mypy`, `detect-secrets`, `gitleaks`, and `markdownlint-cli2`. - Keep using the existing markers unless you truly need a new one. > **Warning:** `pytest.ini` enables `--strict-markers`. If you introduce a new marker and do not add it to `pytest.ini`, collection will fail. > **Tip:** Start with collection and setup validation before a live run. This suite depends on real clusters, real providers, and real credentials, so the fastest feedback loop is usually `--collect-only`, `--setup-plan`, and pre-commit. --- Source: development-workflow.md # Development Workflow This repository does not revolve around a single `make` target. The practical local workflow is a short loop built around `uv`, `pre-commit`, `tox`, and the checked-in `Dockerfile`. For most day-to-day work, the fastest feedback loop is: ```bash uv sync pre-commit run --all-files tox -e pytest-check tox -e unused-code podman build -f Dockerfile -t mtv-api-tests . ``` ## Prerequisites The project targets Python `>=3.12, <3.14`, and it keeps a small `dev` dependency group for interactive tooling. ```toml [project] requires-python = ">=3.12, <3.14" [dependency-groups] dev = ["ipdb>=0.13.13", "ipython>=8.12.3", "python-jenkins>=1.8.2"] ``` > **Note:** `pre-commit` and `tox` are configured in this repository, but they are not declared as project dependencies in `pyproject.toml`. If those commands are not already available on your machine, install them with the Python CLI tool manager you normally use. ## Sync Dependencies Run `uv sync` from the repository root to create or refresh the local environment from `uv.lock`. That is the right starting point after cloning the repo or switching to a branch with dependency changes. When you want the strictest possible sync, use `uv sync --locked`. That is the exact mode used by the container build: ```dockerfile RUN uv sync --locked\ && if [ -n "${OPENSHIFT_PYTHON_WRAPPER_COMMIT}" ]; then uv pip install git+https://github.com/RedHatQE/openshift-python-wrapper.git@$OPENSHIFT_PYTHON_WRAPPER_COMMIT; fi \ && if [ -n "${OPENSHIFT_PYTHON_UTILITIES_COMMIT}" ]; then uv pip install git+https://github.com/RedHatQE/openshift-python-utilities.git@$OPENSHIFT_PYTHON_UTILITIES_COMMIT; fi \ && find ${APP_DIR}/ -type d -name "__pycache__" -print0 | xargs -0 rm -rfv \ && rm -rf ${APP_DIR}/.cache ``` > **Tip:** If you want to reproduce the container's dependency resolution locally, run `uv sync --locked` before troubleshooting. ## Pre-commit Checks `pre-commit run --all-files` is the main local quality gate. It brings together repository hygiene checks, secret scanning, Python linting and formatting, typing, and Markdown linting. Key hooks from `.pre-commit-config.yaml`: ```yaml default_language_version: python: python3.13 repos: - repo: https://github.com/PyCQA/flake8 rev: 7.3.0 hooks: - id: flake8 args: [--config=.flake8] additional_dependencies: [flake8-mutable] - repo: https://github.com/astral-sh/ruff-pre-commit rev: v0.15.4 hooks: - id: ruff - id: ruff-format - repo: https://github.com/pre-commit/mirrors-mypy rev: v1.19.1 hooks: - id: mypy additional_dependencies: [ "types-pyvmomi", "types-requests", "types-six", "types-pytz", "types-PyYAML", "types-paramiko", ] - repo: https://github.com/DavidAnson/markdownlint-cli2 rev: v0.21.0 hooks: - id: markdownlint-cli2 args: ["--fix"] ``` The full hook set also includes `check-added-large-files`, `detect-private-key`, `detect-secrets`, `gitleaks`, `mixed-line-ending`, `trailing-whitespace`, and other small safety checks from `pre-commit-hooks`. Use the full suite when you want the closest thing to a local CI gate: ```bash pre-commit run --all-files ``` Or rerun a single hook while iterating on one kind of issue: ```bash pre-commit run ruff --all-files pre-commit run ruff-format --all-files pre-commit run mypy --all-files pre-commit run flake8 --all-files pre-commit run markdownlint-cli2 --all-files ``` > **Warning:** The hook environments default to `python3.13`. If that interpreter is missing on your workstation, `pre-commit` can fail while creating hook environments even though the project itself supports Python 3.12. > **Note:** Secret scanning is part of the default local workflow. If you add examples that look like credentials, expect `detect-secrets` and `gitleaks` to review them. ## Linting, Typing, and Docs Rules The Python tool settings live in `pyproject.toml`: ```toml [tool.ruff] preview = true line-length = 120 fix = true output-format = "grouped" [tool.ruff.lint] select = ["PLC0415"] [tool.mypy] disallow_incomplete_defs = true no_implicit_optional = true show_error_codes = true warn_unused_ignores = true ``` In practice, that means: - Ruff is configured to auto-fix where possible, so it is usually the first thing to rerun after Python edits. - Mypy is part of the default quality gate, including third-party type stubs for common dependencies used in this repository. - Flake8 is intentionally narrow here. It is focused on the `flake8-mutable` rule rather than acting as a second full Python linter. ```ini [flake8] select=M511 exclude = doc, .tox, .git, .yml, Pipfile.*, docs/*, .cache/* ``` If you are editing documentation, `markdownlint-cli2` is already part of the same workflow. Its repo-level config allows fairly wide lines and a few inline HTML elements often used in docs: ```yaml MD013: line_length: 180 MD033: allowed_elements: - details - summary - strong ``` > **Tip:** For docs-only changes, `pre-commit run markdownlint-cli2 --all-files` is usually the fastest targeted check. ## Tox Targets Unlike some Python projects, tox is not the main entry point for every local check here. This repository defines two focused tox environments in `tox.toml`: ```toml skipsdist = true env_list = ["pytest-check", "unused-code"] [env.pytest-check] commands = [ ["uv", "run", "pytest", "--setup-plan"], ["uv", "run", "pytest", "--collect-only"], ] description = "Run pytest collect-only and setup-plan" deps = ["uv"] [env.unused-code] description = "Find unused code" deps = ["python-utility-scripts"] commands = [["pyutils-unusedcode", "--exclude-function-prefixes", "pytest_"]] ``` Run them with: ```bash tox -e pytest-check tox -e unused-code ``` These environments do two different jobs: - `pytest-check` validates pytest structure without executing real migrations. - `unused-code` runs `pyutils-unusedcode` and ignores pytest hook-style function names with `--exclude-function-prefixes pytest_`. `pytest-check` is intentionally safe because the repository treats both `--setup-plan` and `--collect-only` as dry-run modes: ```python def is_dry_run(config: pytest.Config) -> bool: return config.option.setupplan or config.option.collectonly ``` > **Tip:** Run `tox -e pytest-check` after changing fixtures, parametrization, markers, or imports. It is a fast way to catch collection breakage before you try a real environment-backed run. ## Real Pytest Runs A plain `pytest` invocation already picks up repository defaults from `pytest.ini`. Test discovery is limited to `tests/`, and the default options load `tests/tests_config/config.py`, write JUnit XML, enforce strict markers, and enable `loadscope` distribution. ```ini [pytest] testpaths = tests addopts = -s -o log_cli=true -p no:logging --tc-file=tests/tests_config/config.py --tc-format=python --junit-xml=junit-report.xml --basetemp=/tmp/pytest --show-progress --strict-markers --jira --dist=loadscope ``` Real test execution is not a unit-test-only workflow. In `conftest.py`, the session requires both `storage_class` and `source_provider` unless pytest is running in dry-run mode: ```python required_config = ("storage_class", "source_provider") if not is_dry_run(session.config): BASIC_LOGGER.info(f"{separator(symbol_='-', val='SESSION START')}") missing_configs: list[str] = [] for _req in required_config: if not py_config.get(_req): missing_configs.append(_req) if missing_configs: pytest.exit(reason=f"Some required config is missing {required_config=} - {missing_configs=}", returncode=1) ``` An actual checked-in example of a real test command appears in the copy-offload documentation: ```bash uv run pytest -m copyoffload \ -v \ ${CLUSTER_HOST:+--tc=cluster_host:${CLUSTER_HOST}} \ ${CLUSTER_USERNAME:+--tc=cluster_username:${CLUSTER_USERNAME}} \ ${CLUSTER_PASSWORD:+--tc=cluster_password:${CLUSTER_PASSWORD}} \ --tc=source_provider:vsphere-8.0.3.00400 \ --tc=storage_class:my-block-storageclass ``` > **Warning:** Use real `pytest` runs only when you have access to a live OpenShift/MTV environment and valid provider configuration. For routine local verification, `pre-commit` and `tox -e pytest-check` are the safer defaults. > **Tip:** The repo includes `.providers.json.example` and ignores `.providers.json` in `.gitignore`. Use the example as a reference, but make sure your real `.providers.json` is valid JSON, because the loader reads it with `json.loads()` and the example file contains inline comments for documentation purposes. ## Container Builds Use the checked-in `Dockerfile` when you want a clean, reproducible runtime that matches the repository's containerized execution path. The image is based on Fedora 41, installs dependencies with `uv sync --locked`, and defaults to a safe collection-only pytest command. ```dockerfile FROM quay.io/fedora/fedora:41 ENV UV_PYTHON=python3.12 ENV UV_COMPILE_BYTECODE=1 ENV UV_NO_SYNC=1 ARG OPENSHIFT_PYTHON_WRAPPER_COMMIT='' ARG OPENSHIFT_PYTHON_UTILITIES_COMMIT='' CMD ["uv", "run", "pytest", "--collect-only"] ``` Build locally with either Podman or Docker: ```bash podman build -f Dockerfile -t mtv-api-tests . docker build -f Dockerfile -t mtv-api-tests . ``` If you need to validate unreleased dependency changes in `openshift-python-wrapper` or `openshift-python-utilities`, the `Dockerfile` already exposes build arguments for that: ```bash podman build \ -f Dockerfile \ -t mtv-api-tests \ --build-arg OPENSHIFT_PYTHON_WRAPPER_COMMIT= \ --build-arg OPENSHIFT_PYTHON_UTILITIES_COMMIT= \ . ``` Because the default container command is `uv run pytest --collect-only`, you can smoke-test the image without starting a real migration run: ```bash podman run --rm mtv-api-tests ``` That makes the container build a good final check when you touch packaging, dependency resolution, or anything that could behave differently outside your local shell. --- Source: automation-and-release.md # Automation And Release Most of the automation in `mtv-api-tests` is configuration-driven. The repository tells tools what to check, how to build the test image, how to cut a release, how dependencies should be updated, and how pull requests should be reviewed. What it does not include is the CI/CD pipeline definition that decides when those things run. > **Note:** No GitHub Actions workflows, `Jenkinsfile`, Tekton definitions, or `.gitlab-ci.yml` files are present in this repository. Treat the repo as the source of automation policy, not as the orchestration layer. ## What Is Automated Here - `pre-commit` enforces repository hygiene, Python linting and formatting, type checking, markdown linting, and secret scanning. - `release-it` handles version bumping, commit and tag creation, pushing, changelog generation, and GitHub release creation. - Renovate manages dependency update PRs and weekly lock-file maintenance. - The `Dockerfile` defines a repeatable container build for the test suite. - CodeRabbit and Qodo Merge/PR-Agent automate pull request review. - `pytest` and `tox` expose CI-friendly entry points and artifacts such as JUnit XML. ## Pre-commit Quality Gates The first automation layer is `.pre-commit-config.yaml`. It combines generic repository safety checks with Python tooling and security scanning, so a single `pre-commit` run catches a lot of problems early. ```1:67:.pre-commit-config.yaml --- default_language_version: python: python3.13 repos: - repo: https://github.com/pre-commit/pre-commit-hooks rev: v6.0.0 hooks: - id: check-added-large-files - id: check-docstring-first - id: check-executables-have-shebangs - id: check-merge-conflict - id: check-symlinks - id: detect-private-key - id: mixed-line-ending - id: debug-statements - id: trailing-whitespace args: [--markdown-linebreak-ext=md] # Do not process Markdown files. - id: end-of-file-fixer - id: check-ast - id: check-builtin-literals - id: check-docstring-first - id: check-toml - repo: https://github.com/PyCQA/flake8 rev: 7.3.0 hooks: - id: flake8 args: [--config=.flake8] additional_dependencies: [flake8-mutable] - repo: https://github.com/Yelp/detect-secrets rev: v1.5.0 hooks: - id: detect-secrets - repo: https://github.com/astral-sh/ruff-pre-commit rev: v0.15.4 hooks: - id: ruff - id: ruff-format - repo: https://github.com/gitleaks/gitleaks rev: v8.30.0 hooks: - id: gitleaks - repo: https://github.com/pre-commit/mirrors-mypy rev: v1.19.1 hooks: - id: mypy additional_dependencies: [ "types-pyvmomi", "types-requests", "types-six", "types-pytz", "types-PyYAML", "types-paramiko", ] - repo: https://github.com/DavidAnson/markdownlint-cli2 rev: v0.21.0 hooks: - id: markdownlint-cli2 args: ["--fix"] ``` That hook list tells you what the repo cares about: - repository safety: large files, merge conflicts, broken symlinks, stray debug statements, invalid TOML, and missing EOF newlines - Python quality: `flake8`, `ruff`, `ruff-format`, and `mypy` - secret prevention: `detect-private-key`, `detect-secrets`, and `gitleaks` - docs hygiene: `markdownlint-cli2 --fix` The repo-specific behavior for `ruff` and `mypy` lives in `pyproject.toml`, so the hooks follow local rules rather than generic defaults. ```1:18:pyproject.toml [tool.ruff] preview = true line-length = 120 fix = true output-format = "grouped" [tool.ruff.format] exclude = [".git", ".venv", ".mypy_cache", ".tox", "__pycache__"] [tool.ruff.lint] select = ["PLC0415"] [tool.mypy] disallow_any_generics = false disallow_incomplete_defs = true no_implicit_optional = true show_error_codes = true warn_unused_ignores = true ``` A few practical takeaways matter for contributors and CI maintainers: - `ruff` is allowed to auto-fix code. - `mypy` is configured to reject incomplete function definitions and implicit optional types. - `flake8` is intentionally narrow here: `.flake8` selects `M511` and loads `flake8-mutable`. - Markdown formatting can be auto-corrected during a hook run instead of being fixed manually. Because the repo runs both `detect-secrets` and `gitleaks`, example files can contain `# pragma: allowlist secret` comments to keep fake credentials from being flagged. That is why files such as `.providers.json.example` contain secret-looking placeholders with allowlist annotations. > **Warning:** Pre-commit hook environments are pinned to `python3.13`, while the project itself allows `>=3.12,<3.14` and the container image sets `UV_PYTHON=python3.12`. Make sure your local machine or CI runner can provide the hook interpreter. > **Tip:** If you copy snippets out of example JSON-like files, remove `# pragma: allowlist secret` comments before using them as real JSON. Those comments exist for secret scanners, not for JSON parsers. ## Release Automation With `release-it` Release configuration lives in `.release-it.json`. This repository uses `release-it` for Git and GitHub release operations, not for publishing an npm package. ```1:48:.release-it.json { "npm": { "publish": false }, "git": { "requireCleanWorkingDir": true, "requireBranch": false, "requireUpstream": true, "requireCommits": false, "addUntrackedFiles": false, "commit": true, "commitMessage": "Release ${version}", "commitArgs": [], "tag": true, "tagName": null, "tagMatch": null, "tagAnnotation": "Release ${version}", "tagArgs": [], "push": true, "pushArgs": ["--follow-tags"], "pushRepo": "", "changelog": "git log --no-merges --pretty=format:\"* %s (%h) by %an on %as\" ${from}...${to}" }, "github": { "release": true, "releaseName": "Release ${version}", "releaseNotes": null, "autoGenerate": false, "preRelease": false, "draft": false, "tokenRef": "GITHUB_TOKEN", "assets": null, "host": null, "timeout": 0, "proxy": null, "skipChecks": false, "web": false }, "plugins": { "@release-it/bumper": { "in": "pyproject.toml", "out": { "file": "pyproject.toml", "path": "project.version" } } }, "hooks": { "after:bump": "uv sync" } } ``` Here is what that means in practice: - the release job must start from a clean working tree - the branch must have an upstream remote - `release-it` creates a release commit and tag, then pushes both with `--follow-tags` - GitHub release creation is enabled, using `GITHUB_TOKEN` - GitHub's auto-generated release notes are disabled - changelog text is built from `git log --no-merges ... ${from}...${to}` The version source is the Python project metadata in `pyproject.toml`: ```27:30:pyproject.toml [project] requires-python = ">=3.12, <3.14" name = "mtv-api-tests" version = "2.8.3" ``` That is important because the release flow is Python-package-centric even though the release tool comes from the Node ecosystem. The `@release-it/bumper` plugin updates `project.version`, and the `after:bump` hook runs `uv sync` so the environment and `uv.lock` stay aligned with the new release. Two details are easy to miss: - `requireBranch` is `false`, so the repo itself does not enforce a release branch policy - `requireCommits` is `false`, so the repo itself does not require new commits before a release If you want stricter rules, enforce them in your external release pipeline. > **Warning:** The repository contains `.release-it.json`, but it does not contain `package.json`, `package-lock.json`, `yarn.lock`, or `pnpm-lock.yaml`. Your release runner must provide `release-it`, `@release-it/bumper`, and `GITHUB_TOKEN` from outside the repo. ## Dependency Updates With Renovate Renovate behavior is defined in `renovate.json`: ```1:20:renovate.json { "$schema": "https://docs.renovatebot.com/renovate-schema.json", "extends": [ ":dependencyDashboard", ":maintainLockFilesWeekly", ":prHourlyLimitNone", ":semanticCommitTypeAll(ci )" ], "prConcurrentLimit": 0, "recreateWhen": "never", "lockFileMaintenance": { "enabled": true }, "packageRules": [ { "matchPackagePatterns": ["*"], "groupName": "python-deps" } ] } ``` This gives the repo a clear dependency-update strategy: - a dependency dashboard is enabled - lock-file maintenance runs weekly - Renovate is not throttled by an hourly PR cap - concurrent PRs are unlimited - closed PRs are not automatically recreated - matched dependencies are grouped under `python-deps` Because this project uses `uv` and checks in `uv.lock`, Renovate is not just bumping top-level requirements. It is also part of keeping the lock file fresh. > **Tip:** Grouping everything under `python-deps` reduces PR noise, but it also means update PRs can be broader than a one-package-at-a-time workflow. ## Container Image Build Inputs The container image is built from `Dockerfile`, and the file is very explicit about what goes into the image: ```1:47:Dockerfile FROM quay.io/fedora/fedora:41 ARG APP_DIR=/app ENV JUNITFILE=${APP_DIR}/output/ ENV UV_PYTHON=python3.12 ENV UV_COMPILE_BYTECODE=1 ENV UV_NO_SYNC=1 ENV UV_CACHE_DIR=${APP_DIR}/.cache RUN dnf -y install \ libxml2-devel \ libcurl-devel \ openssl \ openssl-devel \ libcurl-devel \ gcc \ clang \ python3-devel \ && dnf clean all \ && rm -rf /var/cache/dnf \ && rm -rf /var/lib/dnf \ && truncate -s0 /var/log/*.log && rm -rf /var/cache/yum WORKDIR ${APP_DIR} RUN mkdir -p ${APP_DIR}/output COPY utilities utilities COPY tests tests COPY libs libs COPY exceptions exceptions COPY README.md pyproject.toml uv.lock conftest.py pytest.ini ./ COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /usr/local/bin/ ARG OPENSHIFT_PYTHON_WRAPPER_COMMIT='' ARG OPENSHIFT_PYTHON_UTILITIES_COMMIT='' RUN uv sync --locked\ && if [ -n "${OPENSHIFT_PYTHON_WRAPPER_COMMIT}" ]; then uv pip install git+https://github.com/RedHatQE/openshift-python-wrapper.git@$OPENSHIFT_PYTHON_WRAPPER_COMMIT; fi \ && if [ -n "${OPENSHIFT_PYTHON_UTILITIES_COMMIT}" ]; then uv pip install git+https://github.com/RedHatQE/openshift-python-utilities.git@$OPENSHIFT_PYTHON_UTILITIES_COMMIT; fi \ && find ${APP_DIR}/ -type d -name "__pycache__" -print0 | xargs -0 rm -rfv \ && rm -rf ${APP_DIR}/.cache CMD ["uv", "run", "pytest", "--collect-only"] ``` This build has a few important characteristics: - the base image is `quay.io/fedora/fedora:41` - Python dependency installation is driven by `uv sync --locked`, so `uv.lock` matters - only a specific subset of the repository is copied into the image - the build can optionally swap in development commits of `openshift-python-wrapper` and `openshift-python-utilities` - the default container command only performs test collection That last point is intentional: a plain container run validates the test suite can be discovered, but it does not launch a real migration test job. The repository's `.dockerignore` is minimal, so the effective build-input boundary is the `COPY` list in `Dockerfile`, not the ignore file. In other words, the image is controlled more by what is explicitly copied than by what is excluded. > **Tip:** `OPENSHIFT_PYTHON_WRAPPER_COMMIT` and `OPENSHIFT_PYTHON_UTILITIES_COMMIT` are practical escape hatches when you need to validate this test suite against unreleased helper-library commits. > **Warning:** The default container command is `uv run pytest --collect-only`. If your CI job should run real tests, it must override `CMD` or pass an explicit command. ## AI Review Bots This repository configures two PR review bots: CodeRabbit and Qodo Merge/PR-Agent. They both automate review, but they are wired differently. ### CodeRabbit CodeRabbit is configured for assertive automatic review on non-draft PRs targeting `main`, and it can request changes. ```14:72:.coderabbit.yaml reviews: # Review profile: assertive for strict enforcement profile: assertive # Request changes for critical violations request_changes_workflow: true # Review display settings high_level_summary: true poem: false review_status: true collapse_walkthrough: false # Abort review if PR is closed abort_on_close: true # Auto-review configuration auto_review: auto_incremental_review: true ignore_title_keywords: - "WIP" enabled: true drafts: false base_branches: - main # Enable relevant tools for Python/JavaScript project tools: # Python linting ruff: enabled: true pylint: enabled: true # JavaScript linting eslint: enabled: true # Shell script checking shellcheck: enabled: true # YAML validation yamllint: enabled: true # Security scanning gitleaks: enabled: true semgrep: enabled: true # GitHub Actions workflow validation actionlint: enabled: true # Dockerfile linting hadolint: enabled: true ``` That configuration tells users a lot about expected review behavior: - reviews are direct rather than gentle - draft PRs are skipped - PRs with `WIP` in the title are ignored for auto-review - the bot can ask for changes - review coverage is broader than just Python style, including security, YAML, shell, and Dockerfile checks CodeRabbit also points its knowledge-base and guideline logic at `CLAUDE.md`, so it is expected to review against repository-specific rules instead of only generic style advice. ### Qodo Merge / PR-Agent Qodo Merge is configured in `.pr_agent.toml`: ```4:47:.pr_agent.toml [config] response_language = "en-US" add_repo_metadata = true add_repo_metadata_file_list = ["CLAUDE.md"] ignore_pr_title = ["^\\[WIP\\]", "^WIP:", "^Draft:"] ignore_pr_labels = ["wip", "work-in-progress"] [github_app] handle_pr_actions = ["opened", "reopened", "ready_for_review"] pr_commands = ["/describe", "/review", "/improve"] feedback_on_draft_pr = false handle_push_trigger = true push_commands = ["/review", "/improve"] [pr_reviewer] extra_instructions = """ Review Style: - Be direct and specific. Explain WHY rules exist. - Classify each finding by severity: * CRITICAL: Security vulnerabilities, blocking issues - must fix before merge * HIGH: Type errors, defensive programming issues - should fix * MEDIUM: Style/code quality issues - nice to fix * LOW: Suggestions/optional enhancements Focus Areas: - Python code quality (type annotations, exception handling) - Security vulnerabilities (injection, credential exposure) - YAML syntax validation - Follow CLAUDE.md guidelines for project-specific standards """ require_security_review = true require_tests_review = true require_estimate_effort_to_review = true require_score_review = false enable_review_labels_security = true enable_review_labels_effort = true num_max_findings = 50 persistent_comment = false enable_help_text = true [pr_code_suggestions] extra_instructions = "Focus on Python best practices, security, and maintainability. Follow CLAUDE.md standards." focus_only_on_problems = false suggestions_score_threshold = 5 ``` Compared to CodeRabbit, this config emphasizes command-driven interaction: - GitHub App events trigger reviews on open, reopen, ready-for-review, and push - reviewers can explicitly ask for `/describe`, `/review`, or `/improve` - security review and tests review are required focus areas - WIP and draft states are intentionally ignored > **Note:** Both bot configs pull repository-specific guidance from `CLAUDE.md`. They are meant to reinforce the repo's own standards, not replace human ownership or branch policy. There is also a separate, opt-in AI feature inside the pytest plugin. `conftest.py` adds `--analyze-with-ai`, and `utilities/pytest_utils.py` uses `JJI_SERVER_URL`, `JJI_AI_PROVIDER`, and `JJI_AI_MODEL` to enrich JUnit XML through an external `/analyze-failures` service. That is test-report post-processing, not a pull-request review bot. ## What External CI/CD Must Do Because orchestration lives outside this repository, your CI/CD platform needs to call the repo's entry points explicitly. The repo already provides a good integration contract for that. `pytest.ini` makes test output CI-friendly by default, especially through JUnit XML generation: ```1:25:pytest.ini [pytest] testpaths = tests addopts = -s -o log_cli=true -p no:logging --tc-file=tests/tests_config/config.py --tc-format=python --junit-xml=junit-report.xml --basetemp=/tmp/pytest --show-progress --strict-markers --jira --dist=loadscope markers = tier0: Core functionality tests (smoke tests) remote: Remote cluster migration tests warm: Warm migration tests copyoffload: Copy-offload (XCOPY) tests incremental: marks tests as incremental (xfail on previous failure) min_mtv_version: mark test to require minimum MTV version (e.g., @pytest.mark.min_mtv_version("2.6.0")) junit_logging = all ``` That default configuration is designed for automation consumers: - JUnit XML is always generated as `junit-report.xml` - marker handling is strict - xdist is configured with `--dist=loadscope` - logging is enabled in a CI-readable way The repo also ships a small `tox` surface for automation: ```1:26:tox.toml skipsdist = true env_list = ["pytest-check", "unused-code"] [env.pytest-check] commands = [ [ "uv", "run", "pytest", "--setup-plan", ], [ "uv", "run", "pytest", "--collect-only", ], ] description = "Run pytest collect-only and setup-plan" deps = ["uv"] [env.unused-code] description = "Find unused code" deps = ["python-utility-scripts"] commands = [["pyutils-unusedcode", "--exclude-function-prefixes", "pytest_"]] ``` That means an external pipeline can use the repository in layers: 1. run `pre-commit run --all-files` 2. run `tox -e pytest-check` or `uv run pytest --collect-only` as a fast wiring check 3. build the image from `Dockerfile` 4. run real, cluster-backed test jobs with the required provider credentials and OpenShift access 5. collect `junit-report.xml` as an artifact 6. optionally enable `--analyze-with-ai` if you operate the external analysis service 7. run `release-it` in a dedicated release job when you are ready to cut a version The repo also adds extra pytest switches in `conftest.py`, including `--skip-data-collector`, `--skip-teardown`, `--openshift-python-wrapper-log-debug`, and `--analyze-with-ai`. Those are useful knobs for external jobs, but they are not pipeline definitions by themselves. > **Tip:** The cleanest mental model is: this repository defines automation rules, while your CI/CD platform supplies the runner, credentials, scheduling, and environment needed to execute them. ---