From: Linus Torvalds <torvalds@linux-foundation.org>
Date: Wed, 17 Jan 2024 21:03:37 +0000 (-0800)
Subject: Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm
X-Git-Url: https://www.infradead.org/git/?a=commitdiff_plain;h=09d1c6a80f2cf94c6e70be919203473d4ab8e26c;p=users%2Fjedix%2Flinux-maple.git

Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm

Pull kvm updates from Paolo Bonzini:
 "Generic:

   - Use memdup_array_user() to harden against overflow.

   - Unconditionally advertise KVM_CAP_DEVICE_CTRL for all
     architectures.

   - Clean up Kconfigs that all KVM architectures were selecting

   - New functionality around "guest_memfd", a new userspace API that
     creates an anonymous file and returns a file descriptor that refers
     to it. guest_memfd files are bound to their owning virtual machine,
     cannot be mapped, read, or written by userspace, and cannot be
     resized. guest_memfd files do however support PUNCH_HOLE, which can
     be used to switch a memory area between guest_memfd and regular
     anonymous memory.

   - New ioctl KVM_SET_MEMORY_ATTRIBUTES allowing userspace to specify
     per-page attributes for a given page of guest memory; right now the
     only attribute is whether the guest expects to access memory via
     guest_memfd or not, which in Confidential SVMs backed by SEV-SNP,
     TDX or ARM64 pKVM is checked by firmware or hypervisor that
     guarantees confidentiality (AMD PSP, Intel TDX module, or EL2 in
     the case of pKVM).

  x86:

   - Support for "software-protected VMs" that can use the new
     guest_memfd and page attributes infrastructure. This is mostly
     useful for testing, since there is no pKVM-like infrastructure to
     provide a meaningfully reduced TCB.

   - Fix a relatively benign off-by-one error when splitting huge pages
     during CLEAR_DIRTY_LOG.

   - Fix a bug where KVM could incorrectly test-and-clear dirty bits in
     non-leaf TDP MMU SPTEs if a racing thread replaces a huge SPTE with
     a non-huge SPTE.

   - Use more generic lockdep assertions in paths that don't actually
     care about whether the caller is a reader or a writer.

   - let Xen guests opt out of having PV clock reported as "based on a
     stable TSC", because some of them don't expect the "TSC stable" bit
     (added to the pvclock ABI by KVM, but never set by Xen) to be set.

   - Revert a bogus, made-up nested SVM consistency check for
     TLB_CONTROL.

   - Advertise flush-by-ASID support for nSVM unconditionally, as KVM
     always flushes on nested transitions, i.e. always satisfies flush
     requests. This allows running bleeding edge versions of VMware
     Workstation on top of KVM.

   - Sanity check that the CPU supports flush-by-ASID when enabling SEV
     support.

   - On AMD machines with vNMI, always rely on hardware instead of
     intercepting IRET in some cases to detect unmasking of NMIs

   - Support for virtualizing Linear Address Masking (LAM)

   - Fix a variety of vPMU bugs where KVM fail to stop/reset counters
     and other state prior to refreshing the vPMU model.

   - Fix a double-overflow PMU bug by tracking emulated counter events
     using a dedicated field instead of snapshotting the "previous"
     counter. If the hardware PMC count triggers overflow that is
     recognized in the same VM-Exit that KVM manually bumps an event
     count, KVM would pend PMIs for both the hardware-triggered overflow
     and for KVM-triggered overflow.

   - Turn off KVM_WERROR by default for all configs so that it's not
     inadvertantly enabled by non-KVM developers, which can be
     problematic for subsystems that require no regressions for W=1
     builds.

   - Advertise all of the host-supported CPUID bits that enumerate
     IA32_SPEC_CTRL "features".

   - Don't force a masterclock update when a vCPU synchronizes to the
     current TSC generation, as updating the masterclock can cause
     kvmclock's time to "jump" unexpectedly, e.g. when userspace
     hotplugs a pre-created vCPU.

   - Use RIP-relative address to read kvm_rebooting in the VM-Enter
     fault paths, partly as a super minor optimization, but mostly to
     make KVM play nice with position independent executable builds.

   - Guard KVM-on-HyperV's range-based TLB flush hooks with an #ifdef on
     CONFIG_HYPERV as a minor optimization, and to self-document the
     code.

   - Add CONFIG_KVM_HYPERV to allow disabling KVM support for HyperV
     "emulation" at build time.

  ARM64:

   - LPA2 support, adding 52bit IPA/PA capability for 4kB and 16kB base
     granule sizes. Branch shared with the arm64 tree.

   - Large Fine-Grained Trap rework, bringing some sanity to the
     feature, although there is more to come. This comes with a prefix
     branch shared with the arm64 tree.

   - Some additional Nested Virtualization groundwork, mostly
     introducing the NV2 VNCR support and retargetting the NV support to
     that version of the architecture.

   - A small set of vgic fixes and associated cleanups.

  Loongarch:

   - Optimization for memslot hugepage checking

   - Cleanup and fix some HW/SW timer issues

   - Add LSX/LASX (128bit/256bit SIMD) support

  RISC-V:

   - KVM_GET_REG_LIST improvement for vector registers

   - Generate ISA extension reg_list using macros in get-reg-list
     selftest

   - Support for reporting steal time along with selftest

  s390:

   - Bugfixes

  Selftests:

   - Fix an annoying goof where the NX hugepage test prints out garbage
     instead of the magic token needed to run the test.

   - Fix build errors when a header is delete/moved due to a missing
     flag in the Makefile.

   - Detect if KVM bugged/killed a selftest's VM and print out a helpful
     message instead of complaining that a random ioctl() failed.

   - Annotate the guest printf/assert helpers with __printf(), and fix
     the various bugs that were lurking due to lack of said annotation"

* tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (185 commits)
  x86/kvm: Do not try to disable kvmclock if it was not enabled
  KVM: x86: add missing "depends on KVM"
  KVM: fix direction of dependency on MMU notifiers
  KVM: introduce CONFIG_KVM_COMMON
  KVM: arm64: Add missing memory barriers when switching to pKVM's hyp pgd
  KVM: arm64: vgic-its: Avoid potential UAF in LPI translation cache
  RISC-V: KVM: selftests: Add get-reg-list test for STA registers
  RISC-V: KVM: selftests: Add steal_time test support
  RISC-V: KVM: selftests: Add guest_sbi_probe_extension
  RISC-V: KVM: selftests: Move sbi_ecall to processor.c
  RISC-V: KVM: Implement SBI STA extension
  RISC-V: KVM: Add support for SBI STA registers
  RISC-V: KVM: Add support for SBI extension registers
  RISC-V: KVM: Add SBI STA info to vcpu_arch
  RISC-V: KVM: Add steal-update vcpu request
  RISC-V: KVM: Add SBI STA extension skeleton
  RISC-V: paravirt: Implement steal-time support
  RISC-V: Add SBI STA extension definitions
  RISC-V: paravirt: Add skeleton for pv-time support
  RISC-V: KVM: Fix indentation in kvm_riscv_vcpu_set_reg_csr()
  ...
---

09d1c6a80f2cf94c6e70be919203473d4ab8e26c
diff --cc io_uring/io_uring.c
index 09b6d860deba,0efd775cb3ad..86761ec623f9
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@@ -3787,8 -3871,28 +3787,9 @@@ static int io_uring_install_fd(struct f
   */
  static struct file *io_uring_get_file(struct io_ring_ctx *ctx)
  {
- 	return anon_inode_getfile_secure("[io_uring]", &io_uring_fops, ctx,
 -	struct file *file;
 -#if defined(CONFIG_UNIX)
 -	int ret;
 -
 -	ret = sock_create_kern(&init_net, PF_UNIX, SOCK_RAW, IPPROTO_IP,
 -				&ctx->ring_sock);
 -	if (ret)
 -		return ERR_PTR(ret);
 -#endif
 -
+ 	/* Create a new inode so that the LSM can block the creation.  */
 -	file = anon_inode_create_getfile("[io_uring]", &io_uring_fops, ctx,
++	return anon_inode_create_getfile("[io_uring]", &io_uring_fops, ctx,
  					 O_RDWR | O_CLOEXEC, NULL);
 -#if defined(CONFIG_UNIX)
 -	if (IS_ERR(file)) {
 -		sock_release(ctx->ring_sock);
 -		ctx->ring_sock = NULL;
 -	} else {
 -		ctx->ring_sock->file = file;
 -	}
 -#endif
 -	return file;
  }
  
  static __cold int io_uring_create(unsigned entries, struct io_uring_params *p,
diff --cc virt/kvm/guest_memfd.c
index 000000000000,c2e2371720a9..0f4e0cf4f158
mode 000000,100644..100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@@ -1,0 -1,532 +1,532 @@@
+ // SPDX-License-Identifier: GPL-2.0
+ #include <linux/backing-dev.h>
+ #include <linux/falloc.h>
+ #include <linux/kvm_host.h>
+ #include <linux/pagemap.h>
+ #include <linux/anon_inodes.h>
+ 
+ #include "kvm_mm.h"
+ 
+ struct kvm_gmem {
+ 	struct kvm *kvm;
+ 	struct xarray bindings;
+ 	struct list_head entry;
+ };
+ 
+ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)
+ {
+ 	struct folio *folio;
+ 
+ 	/* TODO: Support huge pages. */
+ 	folio = filemap_grab_folio(inode->i_mapping, index);
+ 	if (IS_ERR_OR_NULL(folio))
+ 		return NULL;
+ 
+ 	/*
+ 	 * Use the up-to-date flag to track whether or not the memory has been
+ 	 * zeroed before being handed off to the guest.  There is no backing
+ 	 * storage for the memory, so the folio will remain up-to-date until
+ 	 * it's removed.
+ 	 *
+ 	 * TODO: Skip clearing pages when trusted firmware will do it when
+ 	 * assigning memory to the guest.
+ 	 */
+ 	if (!folio_test_uptodate(folio)) {
+ 		unsigned long nr_pages = folio_nr_pages(folio);
+ 		unsigned long i;
+ 
+ 		for (i = 0; i < nr_pages; i++)
+ 			clear_highpage(folio_page(folio, i));
+ 
+ 		folio_mark_uptodate(folio);
+ 	}
+ 
+ 	/*
+ 	 * Ignore accessed, referenced, and dirty flags.  The memory is
+ 	 * unevictable and there is no storage to write back to.
+ 	 */
+ 	return folio;
+ }
+ 
+ static void kvm_gmem_invalidate_begin(struct kvm_gmem *gmem, pgoff_t start,
+ 				      pgoff_t end)
+ {
+ 	bool flush = false, found_memslot = false;
+ 	struct kvm_memory_slot *slot;
+ 	struct kvm *kvm = gmem->kvm;
+ 	unsigned long index;
+ 
+ 	xa_for_each_range(&gmem->bindings, index, slot, start, end - 1) {
+ 		pgoff_t pgoff = slot->gmem.pgoff;
+ 
+ 		struct kvm_gfn_range gfn_range = {
+ 			.start = slot->base_gfn + max(pgoff, start) - pgoff,
+ 			.end = slot->base_gfn + min(pgoff + slot->npages, end) - pgoff,
+ 			.slot = slot,
+ 			.may_block = true,
+ 		};
+ 
+ 		if (!found_memslot) {
+ 			found_memslot = true;
+ 
+ 			KVM_MMU_LOCK(kvm);
+ 			kvm_mmu_invalidate_begin(kvm);
+ 		}
+ 
+ 		flush |= kvm_mmu_unmap_gfn_range(kvm, &gfn_range);
+ 	}
+ 
+ 	if (flush)
+ 		kvm_flush_remote_tlbs(kvm);
+ 
+ 	if (found_memslot)
+ 		KVM_MMU_UNLOCK(kvm);
+ }
+ 
+ static void kvm_gmem_invalidate_end(struct kvm_gmem *gmem, pgoff_t start,
+ 				    pgoff_t end)
+ {
+ 	struct kvm *kvm = gmem->kvm;
+ 
+ 	if (xa_find(&gmem->bindings, &start, end - 1, XA_PRESENT)) {
+ 		KVM_MMU_LOCK(kvm);
+ 		kvm_mmu_invalidate_end(kvm);
+ 		KVM_MMU_UNLOCK(kvm);
+ 	}
+ }
+ 
+ static long kvm_gmem_punch_hole(struct inode *inode, loff_t offset, loff_t len)
+ {
 -	struct list_head *gmem_list = &inode->i_mapping->private_list;
++	struct list_head *gmem_list = &inode->i_mapping->i_private_list;
+ 	pgoff_t start = offset >> PAGE_SHIFT;
+ 	pgoff_t end = (offset + len) >> PAGE_SHIFT;
+ 	struct kvm_gmem *gmem;
+ 
+ 	/*
+ 	 * Bindings must be stable across invalidation to ensure the start+end
+ 	 * are balanced.
+ 	 */
+ 	filemap_invalidate_lock(inode->i_mapping);
+ 
+ 	list_for_each_entry(gmem, gmem_list, entry)
+ 		kvm_gmem_invalidate_begin(gmem, start, end);
+ 
+ 	truncate_inode_pages_range(inode->i_mapping, offset, offset + len - 1);
+ 
+ 	list_for_each_entry(gmem, gmem_list, entry)
+ 		kvm_gmem_invalidate_end(gmem, start, end);
+ 
+ 	filemap_invalidate_unlock(inode->i_mapping);
+ 
+ 	return 0;
+ }
+ 
+ static long kvm_gmem_allocate(struct inode *inode, loff_t offset, loff_t len)
+ {
+ 	struct address_space *mapping = inode->i_mapping;
+ 	pgoff_t start, index, end;
+ 	int r;
+ 
+ 	/* Dedicated guest is immutable by default. */
+ 	if (offset + len > i_size_read(inode))
+ 		return -EINVAL;
+ 
+ 	filemap_invalidate_lock_shared(mapping);
+ 
+ 	start = offset >> PAGE_SHIFT;
+ 	end = (offset + len) >> PAGE_SHIFT;
+ 
+ 	r = 0;
+ 	for (index = start; index < end; ) {
+ 		struct folio *folio;
+ 
+ 		if (signal_pending(current)) {
+ 			r = -EINTR;
+ 			break;
+ 		}
+ 
+ 		folio = kvm_gmem_get_folio(inode, index);
+ 		if (!folio) {
+ 			r = -ENOMEM;
+ 			break;
+ 		}
+ 
+ 		index = folio_next_index(folio);
+ 
+ 		folio_unlock(folio);
+ 		folio_put(folio);
+ 
+ 		/* 64-bit only, wrapping the index should be impossible. */
+ 		if (WARN_ON_ONCE(!index))
+ 			break;
+ 
+ 		cond_resched();
+ 	}
+ 
+ 	filemap_invalidate_unlock_shared(mapping);
+ 
+ 	return r;
+ }
+ 
+ static long kvm_gmem_fallocate(struct file *file, int mode, loff_t offset,
+ 			       loff_t len)
+ {
+ 	int ret;
+ 
+ 	if (!(mode & FALLOC_FL_KEEP_SIZE))
+ 		return -EOPNOTSUPP;
+ 
+ 	if (mode & ~(FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE))
+ 		return -EOPNOTSUPP;
+ 
+ 	if (!PAGE_ALIGNED(offset) || !PAGE_ALIGNED(len))
+ 		return -EINVAL;
+ 
+ 	if (mode & FALLOC_FL_PUNCH_HOLE)
+ 		ret = kvm_gmem_punch_hole(file_inode(file), offset, len);
+ 	else
+ 		ret = kvm_gmem_allocate(file_inode(file), offset, len);
+ 
+ 	if (!ret)
+ 		file_modified(file);
+ 	return ret;
+ }
+ 
+ static int kvm_gmem_release(struct inode *inode, struct file *file)
+ {
+ 	struct kvm_gmem *gmem = file->private_data;
+ 	struct kvm_memory_slot *slot;
+ 	struct kvm *kvm = gmem->kvm;
+ 	unsigned long index;
+ 
+ 	/*
+ 	 * Prevent concurrent attempts to *unbind* a memslot.  This is the last
+ 	 * reference to the file and thus no new bindings can be created, but
+ 	 * dereferencing the slot for existing bindings needs to be protected
+ 	 * against memslot updates, specifically so that unbind doesn't race
+ 	 * and free the memslot (kvm_gmem_get_file() will return NULL).
+ 	 */
+ 	mutex_lock(&kvm->slots_lock);
+ 
+ 	filemap_invalidate_lock(inode->i_mapping);
+ 
+ 	xa_for_each(&gmem->bindings, index, slot)
+ 		rcu_assign_pointer(slot->gmem.file, NULL);
+ 
+ 	synchronize_rcu();
+ 
+ 	/*
+ 	 * All in-flight operations are gone and new bindings can be created.
+ 	 * Zap all SPTEs pointed at by this file.  Do not free the backing
+ 	 * memory, as its lifetime is associated with the inode, not the file.
+ 	 */
+ 	kvm_gmem_invalidate_begin(gmem, 0, -1ul);
+ 	kvm_gmem_invalidate_end(gmem, 0, -1ul);
+ 
+ 	list_del(&gmem->entry);
+ 
+ 	filemap_invalidate_unlock(inode->i_mapping);
+ 
+ 	mutex_unlock(&kvm->slots_lock);
+ 
+ 	xa_destroy(&gmem->bindings);
+ 	kfree(gmem);
+ 
+ 	kvm_put_kvm(kvm);
+ 
+ 	return 0;
+ }
+ 
+ static inline struct file *kvm_gmem_get_file(struct kvm_memory_slot *slot)
+ {
+ 	/*
+ 	 * Do not return slot->gmem.file if it has already been closed;
+ 	 * there might be some time between the last fput() and when
+ 	 * kvm_gmem_release() clears slot->gmem.file, and you do not
+ 	 * want to spin in the meanwhile.
+ 	 */
+ 	return get_file_active(&slot->gmem.file);
+ }
+ 
+ static struct file_operations kvm_gmem_fops = {
+ 	.open		= generic_file_open,
+ 	.release	= kvm_gmem_release,
+ 	.fallocate	= kvm_gmem_fallocate,
+ };
+ 
+ void kvm_gmem_init(struct module *module)
+ {
+ 	kvm_gmem_fops.owner = module;
+ }
+ 
+ static int kvm_gmem_migrate_folio(struct address_space *mapping,
+ 				  struct folio *dst, struct folio *src,
+ 				  enum migrate_mode mode)
+ {
+ 	WARN_ON_ONCE(1);
+ 	return -EINVAL;
+ }
+ 
 -static int kvm_gmem_error_page(struct address_space *mapping, struct page *page)
++static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *folio)
+ {
 -	struct list_head *gmem_list = &mapping->private_list;
++	struct list_head *gmem_list = &mapping->i_private_list;
+ 	struct kvm_gmem *gmem;
+ 	pgoff_t start, end;
+ 
+ 	filemap_invalidate_lock_shared(mapping);
+ 
 -	start = page->index;
 -	end = start + thp_nr_pages(page);
++	start = folio->index;
++	end = start + folio_nr_pages(folio);
+ 
+ 	list_for_each_entry(gmem, gmem_list, entry)
+ 		kvm_gmem_invalidate_begin(gmem, start, end);
+ 
+ 	/*
+ 	 * Do not truncate the range, what action is taken in response to the
+ 	 * error is userspace's decision (assuming the architecture supports
+ 	 * gracefully handling memory errors).  If/when the guest attempts to
+ 	 * access a poisoned page, kvm_gmem_get_pfn() will return -EHWPOISON,
+ 	 * at which point KVM can either terminate the VM or propagate the
+ 	 * error to userspace.
+ 	 */
+ 
+ 	list_for_each_entry(gmem, gmem_list, entry)
+ 		kvm_gmem_invalidate_end(gmem, start, end);
+ 
+ 	filemap_invalidate_unlock_shared(mapping);
+ 
+ 	return MF_DELAYED;
+ }
+ 
+ static const struct address_space_operations kvm_gmem_aops = {
+ 	.dirty_folio = noop_dirty_folio,
+ 	.migrate_folio	= kvm_gmem_migrate_folio,
 -	.error_remove_page = kvm_gmem_error_page,
++	.error_remove_folio = kvm_gmem_error_folio,
+ };
+ 
+ static int kvm_gmem_getattr(struct mnt_idmap *idmap, const struct path *path,
+ 			    struct kstat *stat, u32 request_mask,
+ 			    unsigned int query_flags)
+ {
+ 	struct inode *inode = path->dentry->d_inode;
+ 
+ 	generic_fillattr(idmap, request_mask, inode, stat);
+ 	return 0;
+ }
+ 
+ static int kvm_gmem_setattr(struct mnt_idmap *idmap, struct dentry *dentry,
+ 			    struct iattr *attr)
+ {
+ 	return -EINVAL;
+ }
+ static const struct inode_operations kvm_gmem_iops = {
+ 	.getattr	= kvm_gmem_getattr,
+ 	.setattr	= kvm_gmem_setattr,
+ };
+ 
+ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)
+ {
+ 	const char *anon_name = "[kvm-gmem]";
+ 	struct kvm_gmem *gmem;
+ 	struct inode *inode;
+ 	struct file *file;
+ 	int fd, err;
+ 
+ 	fd = get_unused_fd_flags(0);
+ 	if (fd < 0)
+ 		return fd;
+ 
+ 	gmem = kzalloc(sizeof(*gmem), GFP_KERNEL);
+ 	if (!gmem) {
+ 		err = -ENOMEM;
+ 		goto err_fd;
+ 	}
+ 
+ 	file = anon_inode_create_getfile(anon_name, &kvm_gmem_fops, gmem,
+ 					 O_RDWR, NULL);
+ 	if (IS_ERR(file)) {
+ 		err = PTR_ERR(file);
+ 		goto err_gmem;
+ 	}
+ 
+ 	file->f_flags |= O_LARGEFILE;
+ 
+ 	inode = file->f_inode;
+ 	WARN_ON(file->f_mapping != inode->i_mapping);
+ 
+ 	inode->i_private = (void *)(unsigned long)flags;
+ 	inode->i_op = &kvm_gmem_iops;
+ 	inode->i_mapping->a_ops = &kvm_gmem_aops;
+ 	inode->i_mode |= S_IFREG;
+ 	inode->i_size = size;
+ 	mapping_set_gfp_mask(inode->i_mapping, GFP_HIGHUSER);
+ 	mapping_set_unmovable(inode->i_mapping);
+ 	/* Unmovable mappings are supposed to be marked unevictable as well. */
+ 	WARN_ON_ONCE(!mapping_unevictable(inode->i_mapping));
+ 
+ 	kvm_get_kvm(kvm);
+ 	gmem->kvm = kvm;
+ 	xa_init(&gmem->bindings);
 -	list_add(&gmem->entry, &inode->i_mapping->private_list);
++	list_add(&gmem->entry, &inode->i_mapping->i_private_list);
+ 
+ 	fd_install(fd, file);
+ 	return fd;
+ 
+ err_gmem:
+ 	kfree(gmem);
+ err_fd:
+ 	put_unused_fd(fd);
+ 	return err;
+ }
+ 
+ int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args)
+ {
+ 	loff_t size = args->size;
+ 	u64 flags = args->flags;
+ 	u64 valid_flags = 0;
+ 
+ 	if (flags & ~valid_flags)
+ 		return -EINVAL;
+ 
+ 	if (size <= 0 || !PAGE_ALIGNED(size))
+ 		return -EINVAL;
+ 
+ 	return __kvm_gmem_create(kvm, size, flags);
+ }
+ 
+ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,
+ 		  unsigned int fd, loff_t offset)
+ {
+ 	loff_t size = slot->npages << PAGE_SHIFT;
+ 	unsigned long start, end;
+ 	struct kvm_gmem *gmem;
+ 	struct inode *inode;
+ 	struct file *file;
+ 	int r = -EINVAL;
+ 
+ 	BUILD_BUG_ON(sizeof(gfn_t) != sizeof(slot->gmem.pgoff));
+ 
+ 	file = fget(fd);
+ 	if (!file)
+ 		return -EBADF;
+ 
+ 	if (file->f_op != &kvm_gmem_fops)
+ 		goto err;
+ 
+ 	gmem = file->private_data;
+ 	if (gmem->kvm != kvm)
+ 		goto err;
+ 
+ 	inode = file_inode(file);
+ 
+ 	if (offset < 0 || !PAGE_ALIGNED(offset) ||
+ 	    offset + size > i_size_read(inode))
+ 		goto err;
+ 
+ 	filemap_invalidate_lock(inode->i_mapping);
+ 
+ 	start = offset >> PAGE_SHIFT;
+ 	end = start + slot->npages;
+ 
+ 	if (!xa_empty(&gmem->bindings) &&
+ 	    xa_find(&gmem->bindings, &start, end - 1, XA_PRESENT)) {
+ 		filemap_invalidate_unlock(inode->i_mapping);
+ 		goto err;
+ 	}
+ 
+ 	/*
+ 	 * No synchronize_rcu() needed, any in-flight readers are guaranteed to
+ 	 * be see either a NULL file or this new file, no need for them to go
+ 	 * away.
+ 	 */
+ 	rcu_assign_pointer(slot->gmem.file, file);
+ 	slot->gmem.pgoff = start;
+ 
+ 	xa_store_range(&gmem->bindings, start, end - 1, slot, GFP_KERNEL);
+ 	filemap_invalidate_unlock(inode->i_mapping);
+ 
+ 	/*
+ 	 * Drop the reference to the file, even on success.  The file pins KVM,
+ 	 * not the other way 'round.  Active bindings are invalidated if the
+ 	 * file is closed before memslots are destroyed.
+ 	 */
+ 	r = 0;
+ err:
+ 	fput(file);
+ 	return r;
+ }
+ 
+ void kvm_gmem_unbind(struct kvm_memory_slot *slot)
+ {
+ 	unsigned long start = slot->gmem.pgoff;
+ 	unsigned long end = start + slot->npages;
+ 	struct kvm_gmem *gmem;
+ 	struct file *file;
+ 
+ 	/*
+ 	 * Nothing to do if the underlying file was already closed (or is being
+ 	 * closed right now), kvm_gmem_release() invalidates all bindings.
+ 	 */
+ 	file = kvm_gmem_get_file(slot);
+ 	if (!file)
+ 		return;
+ 
+ 	gmem = file->private_data;
+ 
+ 	filemap_invalidate_lock(file->f_mapping);
+ 	xa_store_range(&gmem->bindings, start, end - 1, NULL, GFP_KERNEL);
+ 	rcu_assign_pointer(slot->gmem.file, NULL);
+ 	synchronize_rcu();
+ 	filemap_invalidate_unlock(file->f_mapping);
+ 
+ 	fput(file);
+ }
+ 
+ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
+ 		     gfn_t gfn, kvm_pfn_t *pfn, int *max_order)
+ {
+ 	pgoff_t index = gfn - slot->base_gfn + slot->gmem.pgoff;
+ 	struct kvm_gmem *gmem;
+ 	struct folio *folio;
+ 	struct page *page;
+ 	struct file *file;
+ 	int r;
+ 
+ 	file = kvm_gmem_get_file(slot);
+ 	if (!file)
+ 		return -EFAULT;
+ 
+ 	gmem = file->private_data;
+ 
+ 	if (WARN_ON_ONCE(xa_load(&gmem->bindings, index) != slot)) {
+ 		r = -EIO;
+ 		goto out_fput;
+ 	}
+ 
+ 	folio = kvm_gmem_get_folio(file_inode(file), index);
+ 	if (!folio) {
+ 		r = -ENOMEM;
+ 		goto out_fput;
+ 	}
+ 
+ 	if (folio_test_hwpoison(folio)) {
+ 		r = -EHWPOISON;
+ 		goto out_unlock;
+ 	}
+ 
+ 	page = folio_file_page(folio, index);
+ 
+ 	*pfn = page_to_pfn(page);
+ 	if (max_order)
+ 		*max_order = 0;
+ 
+ 	r = 0;
+ 
+ out_unlock:
+ 	folio_unlock(folio);
+ out_fput:
+ 	fput(file);
+ 
+ 	return r;
+ }
+ EXPORT_SYMBOL_GPL(kvm_gmem_get_pfn);