Container And OpenShift Job Execution

Running mtv-api-tests from a container or an OpenShift Job gives you a repeatable environment for long migration runs, shared execution in CI-style workflows, and predictable artifact collection. The key is understanding what the image expects at runtime:

  • a .providers.json file in the container working directory
  • pytest testconfig values for the OpenShift cluster and test selection
  • an overridden container command, because the image defaults to collection only

Build And Run Locally

A pre-built image is published at ghcr.io/redhatqe/mtv-api-tests:latest, so you can use it directly. To build your own instead:

podman build -t quay.io/<your-registry>/mtv-tests:latest .
podman push quay.io/<your-registry>/mtv-tests:latest

Docker works the same way. On RHEL and Fedora with SELinux enforcing, add ,z to volume mounts.

How The Image Starts

The Dockerfile is a two-stage build. A builder stage installs the toolchain and runs uv sync --locked, and a runtime stage on ubi-minimal copies the finished /app, sets JUNITFILE, and runs as UID 1001.

Because the container runs as a non-root user, read-only mounts (:ro) are unaffected, but any writable bind mount must already be accessible to UID 1001.

The image is built to run from /app, and its default command only collects tests:

ARG APP_DIR=/app
WORKDIR ${APP_DIR}

CMD ["uv", "run", "pytest", "--collect-only"]

Pytest is also preconfigured in the repo:

[pytest]
addopts =
  -s
  -o log_cli=true
  -p no:logging
  --tc-file=tests/tests_config/config.py
  --tc-format=python
  --junit-xml=junit-report.xml
  --show-progress
  --strict-markers
  --dist=loadscope

Note: A plain podman run ... ghcr.io/redhatqe/mtv-api-tests:latest will not execute migrations. Override the command with uv run pytest ....

The suite reads provider definitions from .providers.json and builds the OpenShift client from pytest config values, falling back to CLUSTER_HOST, CLUSTER_USERNAME, and CLUSTER_PASSWORD environment variables:

def get_cluster_client() -> DynamicClient:
    _validate_cluster_credentials()
    host = _resolve_cluster_credential("cluster_host")
    username = _resolve_cluster_credential("cluster_username")
    password = _resolve_cluster_credential("cluster_password")
    ...
    client = get_client(host=host, username=username, password=password, verify_ssl=not insecure_verify_skip)

Each credential is resolved by _resolve_cluster_credential(): the --tc value first, then the matching CLUSTER_* environment variable. _validate_cluster_credentials() requires all three resolved values to be nonblank unless all three keys are absent from both pytest config and the environment. Otherwise, it fails fast with a ValueError naming each absent, empty, or whitespace-only value, for example Missing cluster credentials: set 'cluster_username' in pytest config or CLUSTER_USERNAME. Omitting all three is legitimate: the client library then authenticates from the ambient kubeconfig, which must be available to the test process; containers do not automatically inherit the host's kubeconfig. A direct pytest run takes this path whenever a kubeconfig is available to the test process. mtv-api-tests run takes it only when no OpenShift provider is selected from .providers.json: the CLI checks for an existing oc token (oc whoami -t) only then, and with an OpenShift provider entry present — as in the example configuration — it instead uses that provider's host, username, and password.

Every real run needs:

  • .providers.json
  • source_provider
  • storage_class

The explicit-credential container and generated Job examples below also require nonblank cluster_host, cluster_username, and cluster_password values. Direct pytest runs using a kubeconfig available to the test process may omit all three keys from both pytest config and the environment; mtv-api-tests run omits them only when it finds no OpenShift provider in .providers.json to supply cluster credentials and an existing oc token is available.

cluster_username and cluster_password can come from the environment instead of the command line. That is why the generated Job manifest wires CLUSTER_HOST, CLUSTER_USERNAME, CLUSTER_PASSWORD, and CLUSTER_VERIFY_SSL in from a Secret.

Warning: The generated Job uses explicit credentials, so provide nonblank cluster_host, cluster_username, and cluster_password values for that example. It does not mount the host's kubeconfig or configure the test client to use the pod service account. A token-only run requires a kubeconfig available inside the container; an existing oc login on the host alone is not enough.

Provider And Test Config

.providers.json

The provider file lives at .providers.json in the working directory, so inside the image the simplest path is /app/.providers.json.

A real example from .providers.json.example:

{
  "vsphere": {
    "type": "vsphere",
    "version": "<SERVER VERSION>",
    "fqdn": "SERVER FQDN/IP",
    "api_url": "<SERVER FQDN/IP>/sdk",
    "username": "USERNAME",
    "password": "PASSWORD",
    "guest_vm_linux_user": "LINUX VMS USERNAME",
    "guest_vm_linux_password": "LINUX VMS PASSWORD",
    "guest_vm_win_user": "WINDOWS VMS USERNAME",
    "guest_vm_win_password": "WINDOWS VMS PASSWORD",
    "luks_passphrase": "LUKS DISK ENCRYPTION PASSPHRASE",
    "vddk_init_image": "<PATH TO VDDK INIT IMAGE>",
    "endpoint_type": "vcenter"
  }
}

The top-level key is the provider name you pass through pytest. If your file uses "vsphere", then your run must include --tc=source_provider:vsphere.

Note: Inside the container you can also pass --providers-json /some/other/path.json, or set PROVIDERS_JSON_PATH. The mount path below is a convention, not a hard requirement.

Tip: The top-level key can be descriptive, versioned, or site-specific. What matters is that source_provider matches it exactly.

tests/tests_config/config.py

The repo already ships a Python testconfig file with global defaults and tests_params used by the test modules:

insecure_verify_skip: str = "true"
source_provider_insecure_skip_verify: str = "false"
target_namespace_prefix: str = "auto"
mtv_namespace: str = "openshift-mtv"
plan_wait_timeout: int = 3600

tests_params: dict = {
    "test_sanity_warm_mtv_migration": {
        "virtual_machines": [
            {
                "name": "mtv-tests-rhel8",
                "source_vm_power": "on",
                "guest_agent": True,
            },
        ],
        "warm_migration": True,
        "preserve_static_ips": True,
    },

Most users should keep the checked-in config.py and supply runtime values with --tc=....

If you do mount your own replacement config file, start from tests/tests_config/config.py instead of creating an empty file. The tests rely on tests_params being present.

Running From The Container Image

Examples below use podman, but the same pattern works with Docker.

A practical local run mounts .providers.json, exports cluster credentials into the container, and overrides the image command:

mkdir -p results

export CLUSTER_HOST="https://api.example.com:6443"
export CLUSTER_USERNAME="kubeadmin"
export CLUSTER_PASSWORD="<password>"

podman run --rm \
  -e CLUSTER_HOST \
  -e CLUSTER_USERNAME \
  -e CLUSTER_PASSWORD \
  -v "$(pwd)/.providers.json:/app/.providers.json:ro" \
  -v "$(pwd)/results:/app/results" \
  ghcr.io/redhatqe/mtv-api-tests:latest \
  /bin/bash -c 'uv run pytest -m tier0 \
    -v \
    --tc=source_provider:vsphere \
    --tc=storage_class:my-block-storageclass \
    --junit-xml=/app/results/junit-report.xml \
    --log-file=/app/results/pytest-tests.log \
    --data-collector-path=/app/results/data'

No --tc=cluster_* flags are needed here. get_cluster_client() falls back to CLUSTER_HOST, CLUSTER_USERNAME, and CLUSTER_PASSWORD from the environment, which keeps the credentials out of the process arguments.

Warning: The container runs as UID 1001. A results directory created by your own user is usually not writable by that UID, so grant write access first (chmod 777 results, or create it owned by UID 1001). The ,z suffix on an SELinux-enforcing host only relabels the mount for SELinux; it does not grant filesystem write permission, so on its own it leaves the bind-mounted results path unwritable and the documented JUnit and log outputs cannot be written.

Useful marker selections come directly from pytest.ini and the test modules:

  • -m tier0 for smoke-style coverage
  • -m tier1 for the extended LUKS, XFS, and CA-cert scenarios
  • -m warm for warm migration coverage
  • -m copyoffload for the full copy-offload coverage
  • -m copyoffload_sanity for the core copy-offload scenarios when you want a quicker pass
  • -m copyoffload_snapshots for the vSphere copy-offload snapshot scenarios
  • -m upgrade for the MTV operator upgrade scenario, which needs its own config keys
  • -m remote for remote-cluster scenarios when remote_ocp_cluster is configured

Note: warm is not universally supported for every provider type. If you select -m warm against a provider that the warm test module skips, pytest will report skips rather than failures.

If you prefer a mounted config file instead of many --tc= flags, mount your own Python config and point pytest at it:

uv run pytest --tc-file=/app/config.py --tc-format=python ...

That is only safe when /app/config.py is based on the repo’s existing tests/tests_config/config.py.

Running Inside An OpenShift Job

The supported way to get a Job manifest is to let the CLI generate one. Do not hand-write it:

uv run mtv-api-tests generate
uv run mtv-api-tests run --mode job

generate writes mtv-api-tests-manifests.yaml, a single self-contained file with a Namespace, a Secret, and a Job. The namespace name carries a random suffix, so parallel runs do not collide. run --mode job applies it with oc apply -f and prints the namespace and job name to follow.

The manifest looks roughly like this. The Secret supplies the credentials as environment variables, and the Job command maps one of them into a --tc= flag: CLUSTER_HOST becomes --tc=cluster_host (cli/mtv_api_tests/common.py:939). CLUSTER_USERNAME and CLUSTER_PASSWORD are deliberately left as environment variables so the credentials never appear in the process arguments. --tc=insecure_verify_skip is not read from an environment variable — the wizard resolves the value when it writes the manifest and _build_pytest_command() embeds that literal (cli/mtv_api_tests/common.py:1063,940), while CLUSTER_VERIFY_SSL is mounted only so get_cluster_client() can honour it when the container reads its own environment.

apiVersion: batch/v1
kind: Job
metadata:
  name: mtv-tests
  namespace: mtv-tests-a1b2c3
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: tests
        image: ghcr.io/redhatqe/mtv-api-tests:latest
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
              - ALL
        env:
        - name: CLUSTER_HOST
          valueFrom:
            secretKeyRef:
              name: mtv-tests-a1b2c3-config
              key: cluster_host
        - name: CLUSTER_USERNAME
          valueFrom:
            secretKeyRef:
              name: mtv-tests-a1b2c3-config
              key: cluster_username
        - name: CLUSTER_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mtv-tests-a1b2c3-config
              key: cluster_password
        - name: CLUSTER_VERIFY_SSL
          valueFrom:
            secretKeyRef:
              name: mtv-tests-a1b2c3-config
              key: cluster_verify_ssl
        command:
          - /bin/sh
          - -c
          - |
            uv run pytest \
                      -v \
                      --tc=source_provider:vsphere-8.0.3.00400 \
                      --tc=storage_class:my-block-storageclass \
                      --tc=cluster_host:"$CLUSTER_HOST" \
                      --tc=insecure_verify_skip:false
        volumeMounts:
        - name: config
          mountPath: /app/.providers.json
          subPath: providers.json
      volumes:
      - name: config
        secret:
          secretName: mtv-tests-a1b2c3-config

To reuse this pattern for other suites:

  • pass a category to generate, for example uv run mtv-api-tests generate --category tier0 or --category warm (all, copyoffload, tier0, tier1, warm, remote); without it the wizard asks interactively
  • replace the provider key vsphere-8.0.3.00400 with your actual .providers.json key
  • replace my-block-storageclass with the storage class used by your target cluster
  • pass --image if you built and pushed your own copy

Warning: CLUSTER_HOST, CLUSTER_USERNAME, CLUSTER_PASSWORD, and CLUSTER_VERIFY_SSL are not read by the pytest fixtures directly. They exist so the shell can expand them, and so get_cluster_client() can pick them up from the environment.

Warning: A completed Job will not re-run. run --mode job detects the "unchanged" response from oc apply and tells you to delete it first with oc delete -f mtv-api-tests-manifests.yaml.

Building The Secret By Hand

If you would rather assemble the Secret yourself, this is the same minimum set the generator produces:

oc create namespace mtv-tests

read -sp "Enter cluster password: " CLUSTER_PASSWORD && echo
oc create secret generic mtv-tests-config \
  --from-file=providers.json=.providers.json \
  --from-literal=cluster_host=https://api.your-cluster.com:6443 \
  --from-literal=cluster_username=kubeadmin \
  --from-literal=cluster_password="${CLUSTER_PASSWORD}" \
  --from-literal=cluster_verify_ssl=true \
  -n mtv-tests
unset CLUSTER_PASSWORD

Warning: The generated Job mounts its config through a non-optional secretKeyRef, so all four cluster_* keys must exist or the container never starts. The Secret name is <namespace>-config — mtv-tests-config for the namespace above — because the generator derives it from the namespace, not from the Secret you create by hand. Omitting cluster_verify_ssl or naming the Secret anything other than <namespace>-config produces a Job stuck in CreateContainerConfigError.

Note: generate also stores a cluster_ca_bundle key, but the test runner does not consume it yet. CA bundle injection into the container is still unimplemented.

Copy-Offload Credentials

Copy-offload is the one area where the suite can read credentials directly from environment variables as well as from .providers.json:

def get_copyoffload_credential(
    credential_name: str,
    copyoffload_config: dict[str, Any],
) -> str | None:
    env_var_name = f"COPYOFFLOAD_{credential_name.upper()}"
    return os.getenv(env_var_name) or copyoffload_config.get(credential_name)

That means a Job can inject copy-offload storage credentials from a Secret without hard-coding them in .providers.json.

Common environment variable names come straight from the fixtures:

  • COPYOFFLOAD_STORAGE_HOSTNAME
  • COPYOFFLOAD_STORAGE_USERNAME
  • COPYOFFLOAD_STORAGE_PASSWORD
  • vendor-specific names such as COPYOFFLOAD_ONTAP_SVM or COPYOFFLOAD_VANTARA_HOSTGROUP_ID_LIST
  • ESXi SSH values such as COPYOFFLOAD_ESXI_HOST, COPYOFFLOAD_ESXI_USER, and COPYOFFLOAD_ESXI_PASSWORD
  • COPYOFFLOAD_STORAGE_SECRET_EXTRA, a JSON object of extra Secret stringData keys that override storage_secret_extra from the file

Note: This environment-variable fallback applies to copy-offload storage credentials. It does not replace the normal cluster_host, cluster_username, cluster_password, or source_provider inputs.

Collecting Results

By default, a run produces these useful artifacts inside /app:

  • junit-report.xml
  • pytest-tests.log
  • .data-collector/

The data collector path is configurable, and the suite uses it for resource tracking and failure collection:

  • default path: .data-collector
  • change it with --data-collector-path=/some/path
  • disable it with --skip-data-collector

When data collection is enabled, the suite writes resources.json there and can also collect must-gather data on failures.

For a finished OpenShift Job, stream the logs and copy the JUnit file out of the pod:

oc logs -f -n mtv-tests-a1b2c3 job/mtv-tests

POD_NAME=$(oc get pods -n mtv-tests-a1b2c3 -l job-name=mtv-tests -o jsonpath='{.items[0].metadata.name}')
oc cp mtv-tests-a1b2c3/$POD_NAME:/app/junit-report.xml ./junit-report.xml

You can collect the other artifacts the same way:

oc cp mtv-tests-a1b2c3/$POD_NAME:/app/pytest-tests.log ./pytest-tests.log
oc cp mtv-tests-a1b2c3/$POD_NAME:/app/.data-collector ./data-collector

For container runs, the easiest pattern is to mount a results directory and redirect outputs into it with:

  • --junit-xml=/app/results/junit-report.xml
  • --log-file=/app/results/pytest-tests.log
  • --data-collector-path=/app/results/data

Tip: For long-running OpenShift Jobs, mount a PVC and write all artifacts to that volume. That avoids depending on the pod filesystem after completion.

Cleanup And Debugging

The suite cleans up resources automatically unless you opt out. That includes created MTV resources and tracked VMs.

If you want to keep resources around for investigation, add:

--skip-teardown

If you skip teardown, keep the data collector output. tools/clean_cluster.py reads that saved resources.json and calls clean_up() on each recorded resource:

uv run tools/clean_cluster.py .data-collector/resources.json

Warning: If you pass --skip-data-collector, the suite will not write .data-collector/resources.json, and failed runs will not gather the extra collector output.

Note: The completed Job pod remains available until you delete the Job, so you can still fetch logs and artifacts after the test run finishes.