Peter Maydell [Fri, 6 Sep 2024 11:33:07 +0000 (12:33 +0100)]
Merge tag 'migration-20240904-pull-request' of https://gitlab.com/farosas/qemu into staging
Migration pull request
- Steve's cleanup of unused variable
- Peter Maydell's fixes for several leaks in migration-test
- Fabiano's flexibilization of multifd data structures for device
state migration
- Arman Nabiev's fix for ppc e500 migration
- Thomas' fix for migration-test vs. --without-default-devices
# -----BEGIN PGP SIGNATURE-----
#
# iQJEBAABCAAuFiEEqhtIsKIjJqWkw2TPx5jcdBvsMZ0FAmbYVXwQHGZhcm9zYXNA
# c3VzZS5kZQAKCRDHmNx0G+wxnRucEAC1vo046UGdUmbb4PaF5vKAg97io6RB2nrH
# HMz56Yc0AcAKRUGwe2Z80e2jY8B6zi8Ha8b9l7cVsej095eGCF+tINIL4wRX4lHm
# alDY/LkhuqjE5g5c/DaeTztyBOFLvdWHPU5eJyDOC9r7kSlnUcL1gAslH23b8uL0
# xvhPVKaTWjGIzNL1q/XfBr1WgRGqfD6dYb32HJDTq85yOnUT5sEr55aoEEu0euKh
# MYbXPmi5AMbrp8nP21kzUopX8iYERRdoKwhF0ZssciGi/qJVevH70tNdbDEQSxyp
# +vtP54TnL3LrzD4uY5Snng9zT9h0QrZujY79OEcxu20U0s29OQaudWkIjp7yLLUv
# UnPZHS+bIyaS53DdpV94GKGGBX1wrjGC/sn8eGYzmb2yMlMjLTBoE8L5r9cadshX
# XTeF4MtKGqaS3xDM2fIgACHHFl6qr/l0nENspv0raFzpf9Jx/WbpekghvTuWN6/B
# pZHnoOTNiAqXS/Rnyy829vsQ0Pw4hi6wx79Z73RP+35ubZTgTmOsQx9f2FjuEh6k
# JS+q9k4VJ+nntUWsYn4GS1Jlt+FXJ2hfzNj1NNFN4xLT1oioc6pCHsQyV7SBArB1
# ml2zYyfKCTC3riIRhcv/ew6OcKbhHcPFOpd/v0y40LO3mx8S0LZnUWXkcrl3XIZS
# Mj5CBdlFgA==
# =SRN4
# -----END PGP SIGNATURE-----
# gpg: Signature made Wed 04 Sep 2024 13:41:32 BST
# gpg: using RSA key AA1B48B0A22326A5A4C364CFC798DC741BEC319D
# gpg: issuer "farosas@suse.de"
# gpg: Good signature from "Fabiano Rosas <farosas@suse.de>" [unknown]
# gpg: aka "Fabiano Almeida Rosas <fabiano.rosas@suse.com>" [unknown]
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg: There is no indication that the signature belongs to the owner.
# Primary key fingerprint: AA1B 48B0 A223 26A5 A4C3 64CF C798 DC74 1BEC 319D
* tag 'migration-20240904-pull-request' of https://gitlab.com/farosas/qemu: (34 commits)
tests/qtest/migration: Add a check for the availability of the "pc" machine
target/ppc: Fix migration of CPUs with TLB_EMB TLB type
migration/multifd: Add documentation for multifd methods
migration/multifd: Add a couple of asserts for p->iov
migration/multifd: Fix p->iov leak in multifd-uadk.c
migration/multifd: Stop changing the packet on recv side
migration/multifd: Make MultiFDMethods const
migration/multifd: Move nocomp code into multifd-nocomp.c
migration/multifd: Register nocomp ops dynamically
migration/multifd: Standardize on multifd ops names
migration/multifd: Allow multifd sync without flush
migration/multifd: Replace multifd_send_state->pages with client data
migration/multifd: Don't send ram data during SYNC
migration/multifd: Isolate ram pages packet data
migration/multifd: Remove total pages tracing
migration/multifd: Move pages accounting into multifd_send_zero_page_detect()
migration/multifd: Replace p->pages with an union pointer
migration/multifd: Make MultiFDPages_t:offset a flexible array member
migration/multifd: Introduce MultiFDSendData
migration/multifd: Pass in MultiFDPages_t to file_write_ramblock_iov
...
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Peter Maydell [Thu, 5 Sep 2024 17:01:51 +0000 (18:01 +0100)]
Merge tag 'pull-request-2024-09-04' of https://gitlab.com/thuth/qemu into staging
* Bump Avocado to version 103
* Introduce new functional test framework for Python-based tests
* Convert many Avocado tests to the new functional test framework
* tag 'pull-request-2024-09-04' of https://gitlab.com/thuth/qemu: (42 commits)
docs/devel/testing: Add documentation for functional tests
docs/devel/testing: Rename avocado_qemu.Test class
docs/devel/testing: Split the Avocado documentation into a separate file
docs/devel: Split testing docs from the build docs and move to separate folder
gitlab-ci: Add "check-functional" to the build tests
tests/avocado: Remove unused QemuUserTest class
tests/functional: Convert ARM bFLT linux-user avocado test
tests/functional: Add QemuUserTest class
tests/functional: Convert mips64el Fuloong2e avocado test (1/2)
tests/functional: Convert Aarch64 Virt machine avocado tests
tests/functional: Convert Aarch64 SBSA-Ref avocado tests
tests/functional: Convert ARM Integrator/CP avocado tests
tests/functional: Convert the linux_initrd avocado test into a standalone test
tests/functional: Convert the rx_gdbsim avocado test into a standalone test
tests/functional: Convert the acpi-bits test into a standalone test
tests/functional: Convert the m68k nextcube test with tesseract
tests/functional: Convert the ppc_hv avocado test into a standalone test
tests/functional: Convert the ppc_amiga avocado test into a standalone test
tests/functional: Convert most ppc avocado tests into standalone tests
tests/functional: Convert the virtio_gpu avocado test into a standalone test
...
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
* tag 'edgar/xen-queue-2024-09-04.for-upstream' of https://gitlab.com/edgar.iglesias/qemu:
docs/system/i386: xenpvh: Add a basic description
hw/i386/xen: Add a Xen PVH x86 machine
hw/xen: pvh-common: Add support for creating PCIe/GPEX
hw/arm: xenpvh: Reverse virtio-mmio creation order
hw/arm: xenpvh: Rename xen_arm.c -> xen-pvh.c
hw/arm: xenpvh: Break out a common PVH machine
hw/arm: xenpvh: Move stubbed functions to xen-stubs.c
hw/arm: xenpvh: Remove double-negation in warning
hw/arm: xenpvh: Add support for SMP guests
hw/arm: xenpvh: Tweak machine description
hw/arm: xenpvh: Update file header to use SPDX
MAINTAINERS: Add docs/system/arm/xenpvh.rst
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Edgar E. Iglesias [Fri, 9 Aug 2024 13:39:04 +0000 (15:39 +0200)]
hw/arm: xenpvh: Reverse virtio-mmio creation order
We've been creating the virtio-mmio devices in forwards order
but since the qbus lists prepend (rather than append) entries,
the virtio busses end up with decreasing base address order.
Xen enables virtio-mmio nodes in forwards order so there's been
a missmatch. So far, we've been working around this with an
out-of-tree patch to Xen.
This reverses the order making sure the virtio busses end up
ordered with increasing base addresses avoiding the need to
patch Xen.
Signed-off-by: Edgar E. Iglesias <edgar.iglesias@amd.com> Acked-by: Stefano Stabellini <sstabellini@kernel.org>
Edgar E. Iglesias [Fri, 16 Aug 2024 13:54:19 +0000 (15:54 +0200)]
hw/arm: xenpvh: Rename xen_arm.c -> xen-pvh.c
Rename xen_arm.c -> xen-pvh.c to better express that this
is a PVH machine and to align with x86 HVM and future PVH
machine filenames:
hw/i386/xen/xen-hvm.c
hw/i386/xen/xen-pvh.c (in preparation)
No functional changes.
Signed-off-by: Edgar E. Iglesias <edgar.iglesias@amd.com> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Edgar E. Iglesias [Thu, 8 Aug 2024 11:30:12 +0000 (13:30 +0200)]
hw/arm: xenpvh: Add support for SMP guests
Add SMP support for Xen PVH ARM guests.
Create ms->smp.max_cpus ioreq servers to handle hotplug.
Note that ms->smp.max_cpus will be passed to us by the
user (Xen tools) set to the guests maxvcpus.
The value in mc->max_cpus is an absolute maximum for the
-smp option and won't be used to setup ioreq servers unless
the user explicitly specifies it with -smp.
If the user doesn't pass -smp on the command-line, smp.cpus
and smp.max_cpus will default to 1.
Signed-off-by: Edgar E. Iglesias <edgar.iglesias@amd.com> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
Peter Maydell [Wed, 4 Sep 2024 12:20:17 +0000 (13:20 +0100)]
Merge tag 'hppa-v9.1-fixes-pull-request' of https://github.com/hdeller/qemu-hppa into staging
hppa target fixes
Two important patches for the hppa target which missed qemu-v9.1:
- One fix for random linux-user crashes
- One fix for random issues due to loosing the division V-bit
during delivery of hardware interrupts. This triggers all sorts
of random faults when running in system mode.
Helge
# -----BEGIN PGP SIGNATURE-----
#
# iHUEABYKAB0WIQS86RI+GtKfB8BJu973ErUQojoPXwUCZtgy9AAKCRD3ErUQojoP
# X068AQCDCwbzsoQpFX/6Kx2lm3UsW4wrESh/CYcP+epd+X59dAD+LxNwN0ol5Gvl
# kPzM+7QdKC7geIXvBxIG6UuggTscJgM=
# =9roy
# -----END PGP SIGNATURE-----
# gpg: Signature made Wed 04 Sep 2024 11:14:12 BST
# gpg: using EDDSA key BCE9123E1AD29F07C049BBDEF712B510A23A0F5F
# gpg: Good signature from "Helge Deller <deller@gmx.de>" [unknown]
# gpg: aka "Helge Deller <deller@kernel.org>" [unknown]
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg: There is no indication that the signature belongs to the owner.
# Primary key fingerprint: 4544 8228 2CD9 10DB EF3D 25F8 3E5F 3D04 A7A2 4603
# Subkey fingerprint: BCE9 123E 1AD2 9F07 C049 BBDE F712 B510 A23A 0F5F
* tag 'hppa-v9.1-fixes-pull-request' of https://github.com/hdeller/qemu-hppa:
target/hppa: Fix random 32-bit linux-user crashes
target/hppa: Fix PSW V-bit packaging in cpu_hppa_get for hppa64
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Thomas Huth [Fri, 30 Aug 2024 13:38:38 +0000 (15:38 +0200)]
docs/devel/testing: Add documentation for functional tests
Document the new functional testing framework. The text is originally
based on the Avocado documentation, but heavily modified to match the
new framework.
Message-ID: <20240830133841.142644-45-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Thomas Huth [Fri, 30 Aug 2024 13:38:37 +0000 (15:38 +0200)]
docs/devel/testing: Rename avocado_qemu.Test class
The avocado_qemu.Test class has been renamed a while back in commit 2283b627bc ("tests/avocado: Rename avocado_qemu.Test -> QemuSystemTest"),
so we should reflect this now in the documentation, too.
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20240830133841.142644-44-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Thomas Huth [Fri, 30 Aug 2024 13:38:36 +0000 (15:38 +0200)]
docs/devel/testing: Split the Avocado documentation into a separate file
The main testing documentation file got very overloaded already.
Thus let's split the Avocado information into a separate file.
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20240830133841.142644-43-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Thomas Huth [Fri, 30 Aug 2024 13:38:34 +0000 (15:38 +0200)]
gitlab-ci: Add "check-functional" to the build tests
Now that we converted many tests from the "check-avocado" test suite
to the "check-functional" test suite, we should make sure that these
also get tested in the CI.
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20240830133841.142644-41-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Philippe Mathieu-Daudé [Fri, 30 Aug 2024 13:38:32 +0000 (15:38 +0200)]
tests/functional: Convert ARM bFLT linux-user avocado test
Straight forward conversion. Update the SHA1 hashes to
SHA256 hashes since SHA1 should not be used anymore nowadays.
Expose cpio_extract() in qemu_test.utils for possible reuse.
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20240822104238.75045-3-philmd@linaro.org>
[thuth: Add test to meson.build]
Message-ID: <20240830133841.142644-39-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Philippe Mathieu-Daudé [Fri, 30 Aug 2024 13:38:31 +0000 (15:38 +0200)]
tests/functional: Add QemuUserTest class
Per commit 5334df4822 ("tests/avocado: Introduce
QemuUserTest base class"):
Similarly to the 'System' Test base class with methods
for testing system emulation, the QemuUserTest class
contains methods useful to test user-mode emulation.
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20240822104238.75045-2-philmd@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-ID: <20240830133841.142644-38-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Straight forward conversion. Since SBSA_FLASH files are not
tarballs, use lzma_uncompress() method.
Avocado used to set a timeout of 11 tests * 180s = 1980s.
Hopefully 600s should be sufficient.
Running on macOS Sonoma / Apple silicon M1:
$ QEMU_TEST_TIMEOUT_EXPECTED=1 make check-functional-aarch64 V=1
▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_alpine_linux_max OK
▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_alpine_linux_max_pauth_impdef OK
▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_alpine_linux_max_pauth_off OK
▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_alpine_linux_neoverse_n1 OK
▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_edk2_firmware OK
▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_openbsd73_cortex_a57 OK
▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_openbsd73_max OK
▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_openbsd73_max_pauth_impdef OK
▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_openbsd73_max_pauth_off OK
▶ 1/5 test_aarch64_sbsaref.Aarch64SbsarefMachine.test_sbsaref_openbsd73_neoverse_n1 OK
1/5 qemu:func-thorough+func-aarch64-thorough+thorough / func-aarch64-aarch64_sbsaref OK 241.79s 11 subtests passed
...
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20240823131614.10269-1-philmd@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
Message-ID: <20240830133841.142644-35-thuth@redhat.com>
[thuth: remove remaining Avocado tags] Signed-off-by: Thomas Huth <thuth@redhat.com>
Thomas Huth [Fri, 30 Aug 2024 13:38:24 +0000 (15:38 +0200)]
tests/functional: Convert the rx_gdbsim avocado test into a standalone test
Provide a "gzip_uncompress" function based on the standard "gzip" module
to avoid the usage of avocado.utils here.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20240830133841.142644-31-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Thomas Huth [Fri, 30 Aug 2024 13:38:23 +0000 (15:38 +0200)]
tests/functional: Convert the acpi-bits test into a standalone test
Mostly a straight-forward conversion. Looks like we can simply drop
the avocado datadrainer stuff when not using the avocado framework
anymore.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20240830133841.142644-30-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Thomas Huth [Fri, 30 Aug 2024 13:38:22 +0000 (15:38 +0200)]
tests/functional: Convert the m68k nextcube test with tesseract
The code that handles running of tesseract needs to be tweaked a little
bit to be able to run without the functions from avocado.utils, and
while we're at it, drop some legacy stuff that was still there due to
Tesseract 3 support that we already dropped a while ago.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20240830133841.142644-29-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Thomas Huth [Fri, 30 Aug 2024 13:38:21 +0000 (15:38 +0200)]
tests/functional: Convert the ppc_hv avocado test into a standalone test
A straight forward conversion, we just also have to remove the decorator
@skipUnless(os.getenv('SPEED')) since all non-trivial functional tests
are running in SPEED=thorough mode now. Also make sure that the extracted
assets are writable, so that the test does not fail if it gets re-run
and there are stale read-only files already around.
Message-ID: <20240830133841.142644-28-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Thomas Huth [Fri, 30 Aug 2024 13:38:20 +0000 (15:38 +0200)]
tests/functional: Convert the ppc_amiga avocado test into a standalone test
Use the Python standard zipfile module instead of avocado.utils for
extracting the ZIP file that we download here, and use the standard
subprocess module for running the "tail" command.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20240830133841.142644-27-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Thomas Huth [Fri, 30 Aug 2024 13:38:19 +0000 (15:38 +0200)]
tests/functional: Convert most ppc avocado tests into standalone tests
Nothing thrilling in here, just straight forward conversions.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20240830133841.142644-26-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Thomas Huth [Fri, 30 Aug 2024 13:38:18 +0000 (15:38 +0200)]
tests/functional: Convert the virtio_gpu avocado test into a standalone test
Nothing thrilling in here, it's just a straight forward conversion.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20240830133841.142644-25-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Thomas Huth [Fri, 30 Aug 2024 13:38:16 +0000 (15:38 +0200)]
tests/functional: Convert the microblaze avocado tests into standalone tests
The machine_microblaze.py file contained two tests, one for each
endianness. Since we only support one QEMU target binary per file
in the new functional test environment, we have to split this file
up into two files now.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20240830133841.142644-23-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Thomas Huth [Fri, 30 Aug 2024 13:38:15 +0000 (15:38 +0200)]
tests/functional: Convert the x86_cpu_model_versions test
Nothing thrilling in here, it's just a straight forward conversion.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20240830133841.142644-22-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Thomas Huth [Fri, 30 Aug 2024 13:38:14 +0000 (15:38 +0200)]
tests/functional: Convert the s390x avocado tests into standalone tests
These tests use archive.lzma_uncompress() from the Avocado utils,
so provide a small helper function for this, based on the
standard lzma module from Python instead.
And while we're at it, replace the MD5 hashes in the topology test
with proper SHA256 hashes, since MD5 should not be used anymore
nowadays.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20240830133841.142644-21-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Thomas Huth [Fri, 30 Aug 2024 13:38:13 +0000 (15:38 +0200)]
tests/functional: Convert some avocado tests that needed avocado.utils.archive
Instead of using the "archive" module from avocado.utils, switch
these tests to use the new wrapper function that is based on the
"tarfile" module instead.
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20240830133841.142644-20-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Thomas Huth [Fri, 30 Aug 2024 13:38:12 +0000 (15:38 +0200)]
tests/functional: Add a function for extracting files from an archive
Some Avocado-based tests use the "archive" module from avocado.utils
to extract files from an archive. To be able to use these tests
without Avocado, we have to provide our own function for extracting
files. Fortunately, there is already the tarfile module that will
provide us with this functionality, so let's just add a nice wrapper
function around that.
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org> Tested-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20240830133841.142644-19-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Thomas Huth [Fri, 30 Aug 2024 13:38:10 +0000 (15:38 +0200)]
tests/functional: Allow asset downloading with concurrent threads
When running "make -j$(nproc) check-functional", tests that use the
same asset might be running in parallel. Improve the downloading to
detect this situation and wait for the other thread to finish the
download.
Message-ID: <20240830133841.142644-17-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Daniel P. Berrangé [Fri, 30 Aug 2024 13:38:09 +0000 (15:38 +0200)]
tests/functional: enable pre-emptive caching of assets
Many tests need to access assets stored on remote sites. We don't want
to download these during test execution when run by meson, since this
risks hitting test timeouts when data transfers are slow.
Add support for pre-emptive caching of assets by setting the env var
QEMU_TEST_PRECACHE to point to a timestamp file. When this is set,
instead of running the test, the assets will be downloaded and saved
to the cache, then the timestamp file created.
A meson custom target is created as a dependency of each test suite
to trigger the pre-emptive caching logic before the test runs.
When run in caching mode, it will locate assets by looking for class
level variables with a name prefix "ASSET_", and type "Asset".
At the ninja level
ninja test --suite functional
will speculatively download any assets that are not already cached,
so it is advisable to set a timeout multiplier.
QEMU_TEST_NO_DOWNLOAD=1 ninja test --suite functional
will fail the test if a required asset is not already cached
ninja precache-functional
will download and cache all assets required by the functional
tests
At the make level, precaching is always done by
make check-functional
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Tested-by: Richard Henderson <richard.henderson@linaro.org>
[thuth: Remove the duplicated "path = os.path.basename(...)" line]
Message-ID: <20240830133841.142644-16-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Daniel P. Berrangé [Fri, 30 Aug 2024 13:38:08 +0000 (15:38 +0200)]
tests/functional: add a module for handling asset download & caching
The 'Asset' class is a simple module that declares a downloadable
asset that can be cached locally. Downloads are stored in the user's
home dir at ~/.cache/qemu/download, using a sha256 sum of the URL.
[thuth: Drop sha1 support, use hash on file content for naming instead of URL,
add the possibility to specify the cache dir via environment variable]
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20240830133841.142644-15-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Thomas Huth [Fri, 30 Aug 2024 13:38:07 +0000 (15:38 +0200)]
tests/functional: Convert avocado tests that just need a small adjustment
These simple tests can be converted to stand-alone tests quite easily,
e.g. by just setting the machine to 'none' now manually or by adding
"-cpu" command line parameters, since we don't support the corresponding
avocado tags in the new python test framework.
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20240830133841.142644-14-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Thomas Huth [Fri, 30 Aug 2024 13:38:06 +0000 (15:38 +0200)]
tests/functional: Convert simple avocado tests into standalone python tests
These test are rather simple and don't need any modifications apart
from adjusting the "from avocado_qemu" line. To ease debugging, make
the files executable and add a shebang line and Python '__main__'
handling, too, so that these tests can now be run by executing them
directly.
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20240830133841.142644-13-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Thomas Huth [Fri, 30 Aug 2024 13:38:05 +0000 (15:38 +0200)]
tests/functional: Prepare the meson build system for the functional tests
Provide a meson.build file for the upcoming python-based functional
tests, and add some wrapper glue targets to the tests/Makefile.include
file. We are going to use two "speed" modes for the functional tests:
The "quick" tests can be run at any time (i.e. also during "make check"),
while the "thorough" tests should only be run when running a
"make check-functional" test run (since these tests might download
additional assets from the internet).
The changes to the meson.build files are partly based on an earlier
patch by Ani Sinha.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20240830133841.142644-12-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Thomas Huth [Fri, 30 Aug 2024 13:38:04 +0000 (15:38 +0200)]
tests/Makefile.include: Increase the level of indentation in the help text
The next patch is going to add some entries that need more space between
the command and the help text, so let's increase the indentation here
first.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-ID: <20240830133841.142644-11-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Thomas Huth [Fri, 30 Aug 2024 13:38:03 +0000 (15:38 +0200)]
tests/functional: Set up logging
Create log files for each test separately, one file that contains
the basic logging and one that contains the console output.
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20240830133841.142644-10-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Thomas Huth [Fri, 30 Aug 2024 13:38:02 +0000 (15:38 +0200)]
tests/functional: Add base classes for the upcoming pytest-based tests
The files are mostly a copy of the tests/avocado/avocado_qemu/__init__.py
file with some adjustments to get rid of the Avocado dependencies (i.e.
we also have to drop the LinuxSSHMixIn and LinuxTest for now).
The emulator binary and build directory are now passed via
environment variables that will be set via meson.build later.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20240830133841.142644-9-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Thomas Huth [Fri, 30 Aug 2024 13:38:01 +0000 (15:38 +0200)]
python: Install pycotap in our venv if necessary
The upcoming functional tests will require pycotap for providing
TAP output from the python-based tests. Since we want to be able
to run some of the tests offline by default, too, let's install
it along with meson in our venv if necessary (it's size is only
5 kB, so adding the wheel here should not really be a problem).
Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20240830133841.142644-8-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Thomas Huth [Fri, 30 Aug 2024 13:38:00 +0000 (15:38 +0200)]
tests/avocado/boot_linux_console: Remove the s390x subtest
We've got a much more sophisticated, Fedora-based test for s390x
("test_s390x_fedora" in another file) already, so the test in
boot_linux_console.py seems to be rather a waste of precious test
cycles. Thus move the command line check and delete the s390x
test in boot_linux_console.py.
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20240830133841.142644-7-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Thomas Huth [Fri, 30 Aug 2024 13:37:59 +0000 (15:37 +0200)]
tests/avocado/avocado_qemu: Fix the "from" statements in linuxtest.py
Without this change, the new Avocado v103 fails to find the tests
that are based on the LinuxTest class.
Suggested-by: Cleber Rosa <crosa@redhat.com> Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20240830133841.142644-6-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Cleber Rosa [Fri, 30 Aug 2024 13:37:58 +0000 (15:37 +0200)]
Bump avocado to 103.0
This bumps Avocado to latest the LTS release.
An LTS release is one that can receive bugfixes and guarantees
stability for a much longer period and has incremental minor releases
made.
Even though the 103.0 LTS release is pretty a rewrite of Avocado when
compared to 88.1, the behavior of all existing tests under
tests/avocado has been extensively tested no regression in behavior
was found.
To keep behavior of jobs as close as possible with previous version,
this version bump keeps the execution serial (maximum of one task at a
time being run).
Reference: https://avocado-framework.readthedocs.io/en/103.0/releases/lts/103_0.html Signed-off-by: Cleber Rosa <crosa@redhat.com> Tested-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
Message-ID: <20240806173119.582857-2-crosa@redhat.com> Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20240830133841.142644-5-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Cleber Rosa [Fri, 30 Aug 2024 13:37:57 +0000 (15:37 +0200)]
tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image
When the OpenBSD based tests are run in parallel, the previously
single instance of the image would become corrupt. Let's give each
test its own snapshot.
Signed-off-by: Cleber Rosa <crosa@redhat.com> Suggested-by: Alex Bennée <alex.bennee@linaro.org>
Message-ID: <20240806173119.582857-9-crosa@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Tested-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
Message-ID: <20240830133841.142644-4-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Cleber Rosa [Fri, 30 Aug 2024 13:37:56 +0000 (15:37 +0200)]
tests/avocado/boot_xen.py: fetch kernel during test setUp()
The kernel is a common blob used in all tests. By moving it to the
setUp() method, the "fetch asset" plugin will recognize the kernel and
attempt to fetch it and cache it before the tests are started.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-ID: <20240806173119.582857-7-crosa@redhat.com> Reviewed-by: Akihiko Odaki <akihiko.odaki@daynix.com> Tested-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
Message-ID: <20240830133841.142644-3-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Cleber Rosa [Fri, 30 Aug 2024 13:37:55 +0000 (15:37 +0200)]
tests/avocado: machine aarch64: standardize location and RO access
The tests under machine_aarch64_virt.py and machine_aarch64_sbsaref.py
should not be writing to the ISO files. By adding "media=cdrom" the
"ro" is automatically set.
While at it, let's use a single code style and hash for the ISO url.
Signed-off-by: Cleber Rosa <crosa@redhat.com> Tested-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
Message-ID: <20240806173119.582857-5-crosa@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20240830133841.142644-2-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
The linux-user hppa target crashes randomly for me since commit 081a0ed188d8 ("target/hppa: Do not mask in copy_iaoq_entry").
That commit dropped the masking of the IAOQ addresses while copying them
from other registers and instead keeps them with all 64 bits up until
the full gva is formed with the help of hppa_form_gva_psw().
So, when running in linux-user mode on an emulated 64-bit CPU, we need
to mask to a 32-bit address space at the very end in hppa_form_gva_psw()
if the PSW-W flag isn't set (which is the case for linux-user on hppa).
Fixes: 081a0ed188d8 ("target/hppa: Do not mask in copy_iaoq_entry") Cc: qemu-stable@nongnu.org # v9.1+ Signed-off-by: Helge Deller <deller@gmx.de> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
target/hppa: Fix PSW V-bit packaging in cpu_hppa_get for hppa64
While adding hppa64 support, the psw_v variable got extended from 32 to 64
bits. So, when packaging the PSW-V bit from the psw_v variable for interrupt
processing, check bit 31 instead the 63th (sign) bit.
This fixes a hard to find Linux kernel boot issue where the loss of the PSW-V
bit due to an ITLB interruption in the middle of a series of ds/addc
instructions (from the divU milicode library) generated the wrong division
result and thus triggered a Linux kernel crash.
Thomas Huth [Tue, 3 Sep 2024 11:47:26 +0000 (13:47 +0200)]
tests/qtest/migration: Add a check for the availability of the "pc" machine
The test_vcpu_dirty_limit is the only test that does not check for the
availability of the machine before starting the test, so it fails when
QEMU has been configured with --without-default-devices. Add a check for
the "pc" machine type to fix it.
Arman Nabiev [Thu, 22 Aug 2024 16:56:53 +0000 (19:56 +0300)]
target/ppc: Fix migration of CPUs with TLB_EMB TLB type
In vmstate_tlbemb a cut-and-paste error meant we gave
this vmstate subsection the same "cpu/tlb6xx" name as
the vmstate_tlb6xx subsection. This breaks migration load
for any CPU using the TLB_EMB CPU type, because when we
see the "tlb6xx" name in the incoming data we try to
interpret it as a vmstate_tlb6xx subsection, which it
isn't the right format for:
$ qemu-system-ppc -drive
if=none,format=qcow2,file=/home/petmay01/test-images/virt/dummy.qcow2
-monitor stdio -M bamboo
QEMU 9.0.92 monitor - type 'help' for more information
(qemu) savevm foo
(qemu) loadvm foo
Missing section footer for cpu
Error: Error -22 while loading VM state
Correct the incorrect vmstate section name. Since migration
for these CPU types was completely broken before, we don't
need to care that this is a migration compatibility break.
This affects the PPC 405, 440, 460 and e200 CPU families.
Fabiano Rosas [Wed, 28 Aug 2024 14:56:50 +0000 (11:56 -0300)]
migration/multifd: Add documentation for multifd methods
Add documentation clarifying the usage of the multifd methods. The
general idea is that the client code calls into multifd to trigger
send/recv of data and multifd then calls these hooks back from the
worker threads at opportune moments so the client can process a
portion of the data.
Suggested-by: Peter Xu <peterx@redhat.com> Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>
Fabiano Rosas [Wed, 28 Aug 2024 14:56:48 +0000 (11:56 -0300)]
migration/multifd: Fix p->iov leak in multifd-uadk.c
The send_cleanup() hook should free the p->iov that was allocated at
send_setup(). This was missed because the UADK code is conditional on
the presence of the accelerator, so it's not tested by default.
Fixes: 819dd20636 ("migration/multifd: Add UADK initialization") Reported-by: Peter Xu <peterx@redhat.com> Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>
Fabiano Rosas [Wed, 28 Aug 2024 14:56:47 +0000 (11:56 -0300)]
migration/multifd: Stop changing the packet on recv side
As observed by Philippe, the multifd_ram_unfill_packet() function
currently leaves the MultiFDPacket structure with mixed
endianness. This is harmless, but ultimately not very clean.
Stop touching the received packet and do the necessary work using
stack variables instead.
While here tweak the error strings and fix the space before
semicolons. Also remove the "100 times bigger" comment because it's
just one possible explanation for a size mismatch and it doesn't even
match the code.
CC: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>
Fabiano Rosas [Tue, 27 Aug 2024 17:46:03 +0000 (14:46 -0300)]
migration/multifd: Move nocomp code into multifd-nocomp.c
In preparation for adding new payload types to multifd, move most of
the no-compression code into multifd-nocomp.c. Let's try to keep a
semblance of layering by not mixing general multifd control flow with
the details of transmitting pages of ram.
There are still some pieces leftover, namely the p->normal, p->zero,
etc variables that we use for zero page tracking and the packet
allocation which is heavily dependent on the ram code.
Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>
Prior to moving the ram code into multifd-nocomp.c, change the code to
register the nocomp ops dynamically so we don't need to have the ops
structure defined in multifd.c.
While here, move the ops struct initialization to the end of the file
to make the next diff cleaner.
Fabiano Rosas [Tue, 27 Aug 2024 17:46:00 +0000 (14:46 -0300)]
migration/multifd: Allow multifd sync without flush
Separate the multifd sync from flushing the client data to the
channels. These two operations are closely related but not strictly
necessary to be executed together.
The multifd sync is intrinsic to how multifd works. The multiple
channels operate independently and may finish IO out of order in
relation to each other. This applies also between the source and
destination QEMU.
Flushing the data that is left in the client-owned data structures
(e.g. MultiFDPages_t) prior to sync is usually the right thing to do,
but that is particular to how the ram migration is implemented with
several passes over dirty data.
Make these two routines separate, allowing future code to call the
sync by itself if needed. This also allows the usage of
multifd_ram_send to be isolated to ram code.
Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>
Fabiano Rosas [Tue, 27 Aug 2024 17:45:59 +0000 (14:45 -0300)]
migration/multifd: Replace multifd_send_state->pages with client data
Multifd currently has a simple scheduling mechanism that distributes
work to the various channels by keeping storage space within each
channel and an extra space that is given to the client. Each time the
client fills the space with data and calls into multifd, that space is
given to the next idle channel and a free storage space is taken from
the channel and given to client for the next iteration.
This means we always need (#multifd_channels + 1) memory slots to
operate multifd.
This is fine, except that the presence of this one extra memory slot
doesn't allow different types of payloads to be processed at the same
time in different channels, i.e. the data type of
multifd_send_state->pages needs to be the same as p->pages.
For each new data type different from MultiFDPage_t that is to be
handled, this logic would need to be duplicated by adding new fields
to multifd_send_state, to the channels and to multifd_send_pages().
Fix this situation by moving the extra slot into the client and using
only the generic type MultiFDSendData in the multifd core.
Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>
Fabiano Rosas [Tue, 27 Aug 2024 17:45:58 +0000 (14:45 -0300)]
migration/multifd: Don't send ram data during SYNC
Skip saving and loading any ram data in the packet in the case of a
SYNC. This fixes a shortcoming of the current code which requires a
reset of the MultiFDPages_t fields right after the previous
pending_job finishes, otherwise the very next job might be a SYNC and
multifd_send_fill_packet() will put the stale values in the packet.
By not calling multifd_ram_fill_packet(), we can stop resetting
MultiFDPages_t in the multifd core and leave that to the client code.
Actually moving the reset function is not yet done because
pages->num==0 is used by the client code to determine whether the
MultiFDPages_t needs to be flushed. The subsequent patches will
replace that with a generic flag that is not dependent on
MultiFDPages_t.
Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>
Fabiano Rosas [Tue, 27 Aug 2024 17:45:57 +0000 (14:45 -0300)]
migration/multifd: Isolate ram pages packet data
While we cannot yet disentangle the multifd packet from page data, we
can make the code a bit cleaner by setting the page-related fields in
a separate function.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>
Fabiano Rosas [Tue, 27 Aug 2024 17:45:56 +0000 (14:45 -0300)]
migration/multifd: Remove total pages tracing
The total_normal_pages and total_zero_pages elements are used only for
the end tracepoints of the multifd threads. These are not super useful
since they record per-channel numbers and are just the sum of all the
pages that are transmitted per-packet, for which we already have
tracepoints. Remove the totals from the tracing.
Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>
Fabiano Rosas [Tue, 27 Aug 2024 17:45:55 +0000 (14:45 -0300)]
migration/multifd: Move pages accounting into multifd_send_zero_page_detect()
All references to pages are being removed from the multifd worker
threads in order to allow multifd to deal with different payload
types.
multifd_send_zero_page_detect() is called by all multifd migration
paths that deal with pages and is the last spot where zero pages and
normal page amounts are adjusted. Move the pages accounting into that
function.
Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>
Fabiano Rosas [Tue, 27 Aug 2024 17:45:54 +0000 (14:45 -0300)]
migration/multifd: Replace p->pages with an union pointer
We want multifd to be able to handle more types of data than just ram
pages. To start decoupling multifd from pages, replace p->pages
(MultiFDPages_t) with the new type MultiFDSendData that hides the
client payload inside an union.
The general idea here is to isolate functions that *need* to handle
MultiFDPages_t and move them in the future to multifd-ram.c, while
multifd.c will stay with only the core functions that handle
MultiFDSendData/MultiFDRecvData.
Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>
Fabiano Rosas [Tue, 27 Aug 2024 17:45:53 +0000 (14:45 -0300)]
migration/multifd: Make MultiFDPages_t:offset a flexible array member
We're about to use MultiFDPages_t from inside the MultiFDSendData
payload union, which means we cannot have pointers to allocated data
inside the pages structure, otherwise we'd lose the reference to that
memory once another payload type touches the union. Move the offset
array into the end of the structure and turn it into a flexible array
member, so it is allocated along with the rest of MultiFDSendData in
the next patches.
Note that other pointers, such as the ramblock pointer are still fine
as long as the storage for them is not owned by the migration code and
can be correctly released at some point.
Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>
Fabiano Rosas [Tue, 27 Aug 2024 17:45:52 +0000 (14:45 -0300)]
migration/multifd: Introduce MultiFDSendData
Add a new data structure to replace p->pages in the multifd
channel. This new structure will hide the multifd payload type behind
an union, so we don't need to add a new field to the channel each time
we want to handle a different data type.
This also allow us to keep multifd_send_pages() as is, without needing
to complicate the pointer switching.
Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>
Fabiano Rosas [Tue, 27 Aug 2024 17:45:49 +0000 (14:45 -0300)]
migration/multifd: Inline page_size and page_count
The MultiFD*Params structures are for per-channel data. Constant
values should not be there because that needlessly wastes cycles and
storage. The page_size and page_count fall into this category so move
them inline in multifd.h.
Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>
Fabiano Rosas [Tue, 27 Aug 2024 17:45:48 +0000 (14:45 -0300)]
migration/multifd: Reduce access to p->pages
I'm about to replace the p->pages pointer with an opaque pointer, so
do a cleanup now to reduce direct accesses to p->page, which makes the
next diffs cleaner.
Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>
Peter Maydell [Tue, 20 Aug 2024 14:49:12 +0000 (15:49 +0100)]
tests/qtest/migration-test: Don't leak QTestState in test_multifd_tcp_cancel()
In test_multifd_tcp_cancel() we create three QEMU processes: 'from',
'to' and 'to2'. We clean up (via qtest_quit()) 'from' and 'to2' when
we call test_migrate_end(), but never clean up 'to', which results in
this leak:
Direct leak of 336 byte(s) in 1 object(s) allocated from:
#0 0x55e984fcd328 in __interceptor_calloc (/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/asan/tests/qtest/migration-test+0x22f328) (BuildId: 710d409b68bb04427009e9ca6e1b63ff8af785d3)
#1 0x7f0878b39c50 in g_malloc0 debian/build/deb/../../../glib/gmem.c:161:13
#2 0x55e98503a172 in qtest_spawn_qemu tests/qtest/libqtest.c:397:21
#3 0x55e98502bc4a in qtest_init_internal tests/qtest/libqtest.c:471:9
#4 0x55e98502c5b7 in qtest_init_with_env tests/qtest/libqtest.c:533:21
#5 0x55e9850eef0f in test_migrate_start tests/qtest/migration-test.c:857:11
#6 0x55e9850eb01d in test_multifd_tcp_cancel tests/qtest/migration-test.c:3297:9
#7 0x55e985103407 in migration_test_wrapper tests/qtest/migration-helpers.c:456:5
Call qtest_quit() on 'to' to clean it up once it has exited.
Peter Maydell [Tue, 20 Aug 2024 14:49:11 +0000 (15:49 +0100)]
tests/qtest/migration-test: Don't strdup in get_dirty_rate()
We g_strdup() the "status" string we get out of the qdict in
get_dirty_rate(), but we never free it. Since we only use this
string while the dictionary is still valid, we don't need to strdup
at all; drop the unnecessary call to avoid this leak:
Direct leak of 18 byte(s) in 2 object(s) allocated from:
#0 0x564b3e01913e in malloc (/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/asan/tests/qtest/migration-test+0x22f13e) (BuildId: d6403a811332fcc846f93c45e23abfd06d1e67c4)
#1 0x7f2f278ff738 in g_malloc debian/build/deb/../../../glib/gmem.c:128:13
#2 0x7f2f27914583 in g_strdup debian/build/deb/../../../glib/gstrfuncs.c:361:17
#3 0x564b3e14bb5b in get_dirty_rate tests/qtest/migration-test.c:3447:14
#4 0x564b3e138e00 in test_vcpu_dirty_limit tests/qtest/migration-test.c:3565:16
#5 0x564b3e14f417 in migration_test_wrapper tests/qtest/migration-helpers.c:456:5
Peter Maydell [Tue, 20 Aug 2024 14:49:10 +0000 (15:49 +0100)]
tests/qtest/migration-helpers: Don't dup argument to qdict_put_str()
In migrate_set_ports() we call qdict_put_str() with a value string
which we g_strdup(). However qdict_put_str() takes a copy of the
value string, it doesn't take ownership of it, so the g_strdup()
only results in a leak:
Direct leak of 6 byte(s) in 1 object(s) allocated from:
#0 0x56298023713e in malloc (/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/asan/tests/qtest/migration-test+0x22f13e) (BuildId: b2b9174a5a54707a7f76bca51cdc95d2aa08bac1)
#1 0x7fba0ad39738 in g_malloc debian/build/deb/../../../glib/gmem.c:128:13
#2 0x7fba0ad4e583 in g_strdup debian/build/deb/../../../glib/gstrfuncs.c:361:17
#3 0x56298036b16e in migrate_set_ports tests/qtest/migration-helpers.c:145:49
#4 0x56298036ad1c in migrate_qmp tests/qtest/migration-helpers.c:228:9
#5 0x56298035b3dd in test_precopy_common tests/qtest/migration-test.c:1820:5
#6 0x5629803549dc in test_multifd_tcp_channels_none tests/qtest/migration-test.c:3077:5
#7 0x56298036d427 in migration_test_wrapper tests/qtest/migration-helpers.c:456:5
Peter Maydell [Tue, 20 Aug 2024 14:49:09 +0000 (15:49 +0100)]
tests/unit/crypto-tls-x509-helpers: deinit privkey in test_tls_cleanup
We create a gnutls_x509_privkey_t in test_tls_init(), but forget
to deinit it in test_tls_cleanup(), resulting in leaks
reported in hte migration test such as:
Indirect leak of 8 byte(s) in 1 object(s) allocated from:
#0 0x55fa6d11c12e in malloc (/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/asan/tests/qtest/migration-test+0x22f12e) (BuildId: 852a267993587f557f50e5715f352f43720077ba)
#1 0x7f073982685d in __gmp_default_allocate (/lib/x86_64-linux-gnu/libgmp.so.10+0xa85d) (BuildId: f110719303ddbea25a5e89ff730fec520eed67b0)
#2 0x7f0739836193 in __gmpz_realloc (/lib/x86_64-linux-gnu/libgmp.so.10+0x1a193) (BuildId: f110719303ddbea25a5e89ff730fec520eed67b0)
#3 0x7f0739836594 in __gmpz_import (/lib/x86_64-linux-gnu/libgmp.so.10+0x1a594) (BuildId: f110719303ddbea25a5e89ff730fec520eed67b0)
#4 0x7f07398a91ed in nettle_mpz_set_str_256_u (/lib/x86_64-linux-gnu/libhogweed.so.6+0xb1ed) (BuildId: 3cc4a3474de72db89e9dcc93bfb95fe377f48c37)
#5 0x7f073a146a5a (/lib/x86_64-linux-gnu/libgnutls.so.30+0x131a5a) (BuildId: 97b8f99f392f1fd37b969a7164bcea884e23649b)
#6 0x7f073a07192c (/lib/x86_64-linux-gnu/libgnutls.so.30+0x5c92c) (BuildId: 97b8f99f392f1fd37b969a7164bcea884e23649b)
#7 0x7f073a078333 (/lib/x86_64-linux-gnu/libgnutls.so.30+0x63333) (BuildId: 97b8f99f392f1fd37b969a7164bcea884e23649b)
#8 0x7f073a0e8353 (/lib/x86_64-linux-gnu/libgnutls.so.30+0xd3353) (BuildId: 97b8f99f392f1fd37b969a7164bcea884e23649b)
#9 0x7f073a0ef0ac in gnutls_x509_privkey_import (/lib/x86_64-linux-gnu/libgnutls.so.30+0xda0ac) (BuildId: 97b8f99f392f1fd37b969a7164bcea884e23649b)
#10 0x55fa6d2547e3 in test_tls_load_key tests/unit/crypto-tls-x509-helpers.c:99:11
#11 0x55fa6d25460c in test_tls_init tests/unit/crypto-tls-x509-helpers.c:128:15
#12 0x55fa6d2495c4 in test_migrate_tls_x509_start_common tests/qtest/migration-test.c:1044:5
#13 0x55fa6d24c23a in test_migrate_tls_x509_start_reject_anon_client tests/qtest/migration-test.c:1216:12
#14 0x55fa6d23fb40 in test_precopy_common tests/qtest/migration-test.c:1789:21
#15 0x55fa6d236b7c in test_precopy_tcp_tls_x509_reject_anon_client tests/qtest/migration-test.c:2614:5
(Oddly, there is no reported leak in the x509 unit tests, even though
those also use test_tls_init() and test_tls_cleanup().)
In the migration test we create several TLS certificates with
the TLS_* macros from crypto-tls-x509-helpers.h. These macros
create both a QCryptoTLSCertReq object which must be deinitialized
and also an on-disk certificate file. The migration test currently
removes the on-disk file in test_migrate_tls_x509_finish() but
never deinitializes the QCryptoTLSCertReq, which means that memory
allocated as part of it is leaked:
Indirect leak of 2 byte(s) in 1 object(s) allocated from:
#0 0x5558ba33712e in malloc (/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/asan/tests/qtest/migration-test+0x22f12e) (BuildId: 4c8618f663e538538cad19d35233124cea161491)
#1 0x7f64afc131f4 (/lib/x86_64-linux-gnu/libtasn1.so.6+0x81f4) (BuildId: 2fde6ecb43c586fe4077118f771077aa1298e7ea)
#2 0x7f64afc18d58 in asn1_write_value (/lib/x86_64-linux-gnu/libtasn1.so.6+0xdd58) (BuildId: 2fde6ecb43c586fe4077118f771077aa1298e7ea)
#3 0x7f64af8fc678 in gnutls_x509_crt_set_version (/lib/x86_64-linux-gnu/libgnutls.so.30+0xe7678) (BuildId: 97b8f99f392f1fd37b969a7164bcea884e23649b)
#4 0x5558ba470035 in test_tls_generate_cert tests/unit/crypto-tls-x509-helpers.c:234:5
#5 0x5558ba464e4a in test_migrate_tls_x509_start_common tests/qtest/migration-test.c:1058:5
#6 0x5558ba462c8a in test_migrate_tls_x509_start_default_host tests/qtest/migration-test.c:1123:12
#7 0x5558ba45ab40 in test_precopy_common tests/qtest/migration-test.c:1786:21
#8 0x5558ba450015 in test_precopy_unix_tls_x509_default_host tests/qtest/migration-test.c:2077:5
#9 0x5558ba46d3c7 in migration_test_wrapper tests/qtest/migration-helpers.c:456:5
(and similar reports).
The only function currently provided to deinit a QCryptoTLSCertReq is
test_tls_discard_cert(), which also removes the on-disk certificate
file. For the migration tests we need to retain the on-disk files
until we've finished running the test, so the simplest fix is to
provide a new function test_tls_deinit_cert() which does only the
cleanup of the QCryptoTLSCertReq, and call it in the right places.
In migrate_get_socket_address() we leak the SocketAddressList:
(cd build/asan && \
ASAN_OPTIONS="fast_unwind_on_malloc=0:strip_path_prefix=/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/asan/../../"
QTEST_QEMU_BINARY=./qemu-system-x86_64 \
./tests/qtest/migration-test --tap -k -p /x86_64/migration/multifd/tcp/tls/psk/match )
[...]
Direct leak of 16 byte(s) in 1 object(s) allocated from:
#0 0x563d7f22f318 in __interceptor_calloc (/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/asan/tests/qtest/migration-test+0x22f318) (BuildId: 2ad6282fb5d076c863ab87f41a345d46dc965ded)
#1 0x7f9de3b39c50 in g_malloc0 debian/build/deb/../../../glib/gmem.c:161:13
#2 0x563d7f3a119c in qobject_input_start_list qapi/qobject-input-visitor.c:336:17
#3 0x563d7f390fbf in visit_start_list qapi/qapi-visit-core.c:80:10
#4 0x563d7f3882ef in visit_type_SocketAddressList /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/asan/qapi/qapi-visit-sockets.c:519:10
#5 0x563d7f3658c9 in migrate_get_socket_address tests/qtest/migration-helpers.c:97:5
#6 0x563d7f362e24 in migrate_get_connect_uri tests/qtest/migration-helpers.c:111:13
#7 0x563d7f362bb2 in migrate_qmp tests/qtest/migration-helpers.c:222:23
#8 0x563d7f3533cd in test_precopy_common tests/qtest/migration-test.c:1817:5
#9 0x563d7f34dc1c in test_multifd_tcp_tls_psk_match tests/qtest/migration-test.c:3185:5
#10 0x563d7f365337 in migration_test_wrapper tests/qtest/migration-helpers.c:458:5
The code fishes out the SocketAddress from the list to return it, and the
callers are freeing that, but nothing frees the list.
Since this function is called in only two places, the simple fix is to
make it return the SocketAddressList rather than just a SocketAddress,
and then the callers can easily access the SocketAddress, and free
the whole SocketAddressList when they're done.
Peter Maydell [Tue, 20 Aug 2024 14:49:06 +0000 (15:49 +0100)]
tests/qtest/migration-test: Fix leaks in calc_dirtyrate_ready()
In calc_dirtyrate_ready() we g_strdup() a string but then never free it:
Direct leak of 19 byte(s) in 2 object(s) allocated from:
#0 0x55ead613413e in malloc (/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/asan/tests/qtest/migration-test+0x22f13e) (BuildId: e7cd5c37b2987a1af682b43ee5240b98bb316737)
#1 0x7f7a13d39738 in g_malloc debian/build/deb/../../../glib/gmem.c:128:13
#2 0x7f7a13d4e583 in g_strdup debian/build/deb/../../../glib/gstrfuncs.c:361:17
#3 0x55ead6266f48 in calc_dirtyrate_ready tests/qtest/migration-test.c:3409:14
#4 0x55ead62669fe in wait_for_calc_dirtyrate_complete tests/qtest/migration-test.c:3422:13
#5 0x55ead6253df7 in test_vcpu_dirty_limit tests/qtest/migration-test.c:3562:9
#6 0x55ead626a407 in migration_test_wrapper tests/qtest/migration-helpers.c:456:5
We also fail to unref the QMP rsp_return, so we leak that also.
Rather than duplicating the string, use the in-place value from
the qdict, and then unref the qdict.
Peter Maydell [Tue, 20 Aug 2024 14:49:05 +0000 (15:49 +0100)]
tests/qtest/migration-test: Don't leak resp in multifd_mapped_ram_fdset_end()
In multifd_mapped_ram_fdset_end() we call qtest_qmp() but forgot
to unref the response QDict we get back, which means it is leaked:
Indirect leak of 4120 byte(s) in 1 object(s) allocated from:
#0 0x55c0c095d318 in __interceptor_calloc (/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/asan/tests/qtest/migration-test+0x22f318) (BuildI
d: 07f667506452d6c467dbc06fd95191966d3e91b4)
#1 0x7f186f939c50 in g_malloc0 debian/build/deb/../../../glib/gmem.c:161:13
#2 0x55c0c0ae9b01 in qdict_new qobject/qdict.c:30:13
#3 0x55c0c0afc16c in parse_object qobject/json-parser.c:317:12
#4 0x55c0c0afb90f in parse_value qobject/json-parser.c:545:16
#5 0x55c0c0afb579 in json_parser_parse qobject/json-parser.c:579:14
#6 0x55c0c0afa21d in json_message_process_token qobject/json-streamer.c:92:12
#7 0x55c0c0bca2e5 in json_lexer_feed_char qobject/json-lexer.c:313:13
#8 0x55c0c0bc97ce in json_lexer_feed qobject/json-lexer.c:350:9
#9 0x55c0c0afabbc in json_message_parser_feed qobject/json-streamer.c:121:5
#10 0x55c0c09cbd52 in qmp_fd_receive tests/qtest/libqmp.c:86:9
#11 0x55c0c09be69b in qtest_qmp_receive_dict tests/qtest/libqtest.c:760:12
#12 0x55c0c09bca77 in qtest_qmp_receive tests/qtest/libqtest.c:741:27
#13 0x55c0c09bee9d in qtest_vqmp tests/qtest/libqtest.c:812:12
#14 0x55c0c09bd257 in qtest_qmp tests/qtest/libqtest.c:835:16
#15 0x55c0c0a87747 in multifd_mapped_ram_fdset_end tests/qtest/migration-test.c:2393:12
#16 0x55c0c0a85eb3 in test_file_common tests/qtest/migration-test.c:1978:9
#17 0x55c0c0a746a3 in test_multifd_file_mapped_ram_fdset tests/qtest/migration-test.c:2437:5
#18 0x55c0c0a93237 in migration_test_wrapper tests/qtest/migration-helpers.c:458:5
#19 0x7f186f958aed in test_case_run debian/build/deb/../../../glib/gtestutils.c:2930:15
#20 0x7f186f958aed in g_test_run_suite_internal debian/build/deb/../../../glib/gtestutils.c:3018:16
#21 0x7f186f95880a in g_test_run_suite_internal debian/build/deb/../../../glib/gtestutils.c:3035:18
#22 0x7f186f95880a in g_test_run_suite_internal debian/build/deb/../../../glib/gtestutils.c:3035:18
#23 0x7f186f95880a in g_test_run_suite_internal debian/build/deb/../../../glib/gtestutils.c:3035:18
#24 0x7f186f95880a in g_test_run_suite_internal debian/build/deb/../../../glib/gtestutils.c:3035:18
#25 0x7f186f95880a in g_test_run_suite_internal debian/build/deb/../../../glib/gtestutils.c:3035:18
#26 0x7f186f958faa in g_test_run_suite debian/build/deb/../../../glib/gtestutils.c:3109:18
#27 0x7f186f959055 in g_test_run debian/build/deb/../../../glib/gtestutils.c:2231:7
#28 0x7f186f959055 in g_test_run debian/build/deb/../../../glib/gtestutils.c:2218:1
#29 0x55c0c0a6e427 in main tests/qtest/migration-test.c:4033:11
Unref the object after we've confirmed that it is what we expect.
If you invoke the migration-test binary in such a way that it doesn't run
any tests, then we never call bootfile_create(), and at the end of
main() bootfile_delete() will try to unlink(NULL), which is not valid.
This can happen if for instance you tell the test binary to run a
subset of tests that turns out to be empty, like this:
(cd build/asan && QTEST_QEMU_BINARY=./qemu-system-x86_64 ./tests/qtest/migration-test --tap -k -p bang)
# random seed: R02S6501b289ff8ced4231ba452c3a87bc6f
# Skipping test: userfaultfd not available
1..0
../../tests/qtest/migration-test.c:182:12: runtime error: null pointer passed as argument 1, which is declared to never be null
/usr/include/unistd.h:858:48: note: nonnull attribute specified here
Handle this by making bootfile_delete() not needing to do anything
because bootfile_create() was never called.
Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Fabiano Rosas <farosas@suse.de>
[fixed conflict with aee07f2563] Signed-off-by: Fabiano Rosas <farosas@suse.de>
Fabiano Rosas [Fri, 2 Aug 2024 14:53:01 +0000 (11:53 -0300)]
tests/qtest/migration: Remove vmstate-static-checker test
I fumbled one of my last pull requests when fixing in-tree an issue
with commit 87d67fadb9 ("monitor: Stop removing non-duplicated
fds"). Basically mixed-up my `git add -p` and `git checkout -p` and
committed a piece of test infra that has not been reviewed yet.
This has not caused any bad symptoms because the test is not enabled
by default anywhere: make check doesn't use two qemu binaries and the
CI doesn't have PYTHON set for the compat tests. Besides, the test
works fine anyway, it would not break anything.
Remove this because it was never intended to be merged.
Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>
John Snow [Thu, 22 Aug 2024 20:48:03 +0000 (16:48 -0400)]
docs/sphinx: fix extra stuff in TOC after freeform QMP sections
Freeform sections with titles are currently generating a TOC entry for
the first paragraph in the section after the header, which is not what
we want.
(Easiest to observe directly in the QMP reference manual's
"Introduction" section.)
When freeform sections are parsed, we create both a section header *and*
an empty, title-less section. This causes some problems with sphinx's
post-parse tree transforms, see also 2664f317 - this is a similar issue:
Sphinx doesn't like section-less titles and it also doesn't like
title-less sections.
Modify qapidoc.py to parse text directly into the preceding section
title as child nodes, eliminating the section duplication. This removes
the extra text from the TOC.
Only very, very lightly tested: "it looks right at a glance" :tm:. I am
still in the process of rewriting qapidoc, so I didn't give it much
deeper thought.
Reported-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20240822204803.1649762-1-jsnow@redhat.com>
projects / users / dwmw2 / qemu.git / log