drivers/nvme: provide a module parameter for setting number of I/O queues

Orabug: 24914952

The current NVME driver allocates I/O queue per-cpu and alloctes IRQ
per-queue for the devices, a large number of IRQs will be allotted to
the I/O queues on a large NUMA/SMP system with multiple NVME devices
installed because of this design.

It would cause failure of CPU hotplug operations on the above mentioned
system, the problem is that the CPU cores could not be hotplugged after
certain number of them are offlined because the remaining online CPU cores
have not enough IRQ vectors to accept the large number of migrated IRQs.

This patch fixes it by providing a way to reduce the NVME queue IRQs to
an acceptable number.

Signed-off-by: Shan Hai <shan.hai@oracle.com>
Reviewed-by: Martin K. Petersen <martin.petersen@oracle.com>

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 2b9c2979da9f2f0e67c909985444b158e62e3114..fd0879823856f82d856c7709ea17ed245fa04cab 100644 (file)
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -78,6 +78,10 @@ static bool use_cmb_sqes = true;
 module_param(use_cmb_sqes, bool, 0644);
 MODULE_PARM_DESC(use_cmb_sqes, "use controller's memory buffer for I/O SQes");
 
+static unsigned int nvme_io_queues = UINT_MAX;
+module_param(nvme_io_queues, uint, 0);
+MODULE_PARM_DESC(nvme_io_queues, "set the number of nvme io queues");
+
 static DEFINE_SPINLOCK(dev_list_lock);
 static LIST_HEAD(dev_list);
 static struct task_struct *nvme_thread;
@@ -2302,7 +2306,7 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
        struct pci_dev *pdev = to_pci_dev(dev->dev);
        int result, i, vecs, nr_io_queues, size;
 
-       nr_io_queues = num_possible_cpus();
+       nr_io_queues = min(nvme_io_queues, num_possible_cpus());
        result = set_queue_count(dev, nr_io_queues);
        if (result <= 0)
                return result;