nvme-pci: fix stuck reset on concurrent DPC and HP

The PCIe error handling has the nvme driver quiesce the device, attempt
to restart it, then wait for that restart to complete.

A PCIe DPC event also toggles the PCIe link. If the slot doesn't have
out-of-band presence detection, this will trigger a pciehp
re-enumeration.

The error handling that calls nvme_error_resume is holding the device
lock while this happens. This lock blocks pciehp's request to disconnect
the driver from proceeding.

Meanwhile the nvme's reset can't make forward progress because its
device isn't there anymore with outstanding IO, and the timeout handler
won't do anything to fix it because the device is undergoing error
handling.

End result: deadlocked.

Fix this by having the timeout handler short cut the disabling for a
disconnected PCIe device. The downside is that we're relying on an IO
timeout to clean up this mess, which could be a minute by default.

Tested-by: Nilay Shroff <nilay@linux.ibm.com>
Reviewed-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 640590b2172828cf70c05bd35234b44f286d0d16..e59aad269abf8334178a362e91e0c036679e5dc8 100644 (file)
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -1411,9 +1411,20 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req)
        struct nvme_dev *dev = nvmeq->dev;
        struct request *abort_req;
        struct nvme_command cmd = { };
+       struct pci_dev *pdev = to_pci_dev(dev->dev);
        u32 csts = readl(dev->bar + NVME_REG_CSTS);
        u8 opcode;
 
+       /*
+        * Shutdown the device immediately if we see it is disconnected. This
+        * unblocks PCIe error handling if the nvme driver is waiting in
+        * error_resume for a device that has been removed. We can't unbind the
+        * driver while the driver's error callback is waiting to complete, so
+        * we're relying on a timeout to break that deadlock if a removal
+        * occurs while reset work is running.
+        */
+       if (pci_dev_is_disconnected(pdev))
+               nvme_change_ctrl_state(&dev->ctrl, NVME_CTRL_DELETING);
        if (nvme_state_terminal(&dev->ctrl))
                goto disable;
 
@@ -1421,7 +1432,7 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req)
         * the recovery mechanism will surely fail.
         */
        mb();
-       if (pci_channel_offline(to_pci_dev(dev->dev)))
+       if (pci_channel_offline(pdev))
                return BLK_EH_RESET_TIMER;
 
        /*