dm: fix bio polling to handle possibile BLK_STS_AGAIN

Expanded testing of DM's bio polling support (using more fio threads
to dm-linear ontop of null_blk) exposed the possibility for polled
bios to hang (repeatedly polling in io_uring) when null_blk responds
with BLK_STS_AGAIN (due to lack of resources):

1) io_complete_rw_iopoll() is called from blkdev_bio_end_io_async() to
   notify kiocb is done, that is the completion interface between block
   layer and io_uring

2) io_complete_rw_iopoll() is called from io_do_iopoll()

3) dm returns BLK_STS_AGAIN for one bio (on behalf of underlying
   driver), then io_complete_rw_iopoll is called, but io_do_iopoll()
   doesn't handle -EAGAIN at all (due to logic in io_rw_should_reissue)

4) reason for dm's BLK_STS_AGAIN is underlying null_blk driver ran out
   of requests (easier to reproduce by setting low hw_queue_depth).

5) dm should handle BLK_STS_AGAIN for POLLED underlying IO, and may
   retry in dm layer.

This fix adds REQ_POLLED specific BLK_STS_AGAIN handling to
dm_io_complete() that clears REQ_POLLED and requeues the bio to DM
using queue_io().

Fixes: b99fdcdc3636 ("dm: support bio polling")
Signed-off-by: Ming Lei <ming.lei@redhat.com>
[snitzer: revised header, reused dm_io_complete's REQ_POLLED case]
Signed-off-by: Mike Snitzer <snitzer@kernel.org>

diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index ad2e0bbeb559a584434a1b90da504ab0b7708c71..3c5fad7c4ee68c18d9951bdd2706eb35f4c96578 100644 (file)
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -892,13 +892,19 @@ static void dm_io_complete(struct dm_io *io)
        if (unlikely(wq_has_sleeper(&md->wait)))
                wake_up(&md->wait);
 
-       if (io_error == BLK_STS_DM_REQUEUE) {
-               /*
-                * Upper layer won't help us poll split bio, io->orig_bio
-                * may only reflect a subset of the pre-split original,
-                * so clear REQ_POLLED in case of requeue
-                */
-               bio->bi_opf &= ~REQ_POLLED;
+       if (io_error == BLK_STS_DM_REQUEUE || io_error == BLK_STS_AGAIN) {
+               if (bio->bi_opf & REQ_POLLED) {
+                       /*
+                        * Upper layer won't help us poll split bio (io->orig_bio
+                        * may only reflect a subset of the pre-split original)
+                        * so clear REQ_POLLED in case of requeue.
+                        */
+                       bio->bi_opf &= ~REQ_POLLED;
+                       if (io_error == BLK_STS_AGAIN) {
+                               /* io_uring doesn't handle BLK_STS_AGAIN (yet) */
+                               queue_io(md, bio);
+                       }
+               }
                return;
        }