net/mlx4_core: panic the system on unrecoverable errors

Mellanox catastrophic error recovery after device reset doesn't work and
in fact leads to unusable node for IB network since the HCA's ports
go down. At times hard reset is needed to get the system rebooted
which is a real problem in production environment. Once the
network outage detected, unreachable node gets evicted and rebooted
on engineered system using reboot. So hanged reboot command is
problematic. So the idea is let the kernel panic which can recover
system on its own with necessary logs captured. There was a debate
on whether to use panic or machine restart, but it was agreed to use
panic instead of silent reboot since thats the preferred option.

There is Mellanox case open to investigate this issue. As such this
is a rare case scenario and even if the issue is fixed, it is expected
to avoid leading to catas error case. This panic is limited to
only error case.

Reviewed-by: Yuval Shaia <yuval.shaia@oracle.com>
Reviewed-by: Mukesh Kacker <mukesh.kacker@oracle.com>
Signed-off-by: Santosh Shilimkar <santosh.shilimkar@oracle.com>
Orabug: 25873690
This is a change taken from QU2, it is not upstream.
(cherry picked from commit 271d694b34bd22e5632eaad41ea1d9a47f1bde3a)
Signed-off-by: Jack Vogel <jack.vogel@oracle.com>

diff --git a/drivers/net/ethernet/mellanox/mlx4/catas.c b/drivers/net/ethernet/mellanox/mlx4/catas.c
index 715de8affcc950e0ea18fd706bc8f04542d34a6f..91d8c925b7786452fc1a3728b71d1bc91c9c9084 100644 (file)
--- a/drivers/net/ethernet/mellanox/mlx4/catas.c
+++ b/drivers/net/ethernet/mellanox/mlx4/catas.c
@@ -188,6 +188,14 @@ void mlx4_enter_error_state(struct mlx4_dev_persistent *persist)
        mlx4_err(dev, "device was reset successfully\n");
        mutex_unlock(&persist->device_state_mutex);
 
+       /* Mellanox device reset and recovery has never worked and
+        * in fact ends up hanging the system which needs a hard reboot
+        * of the system. Instead of waiting for recovery which never
+        * going to happen, just panic the system so that it can capture
+        * all the necessary logs/vmcore and let the node graceful shutdown.
+        */
+       panic("MLX4 device reset due to unrecoverable catastrophic failure\n");
+
        /* At that step HW was already reset, now notify clients */
        mlx4_dispatch_event(dev, MLX4_DEV_EVENT_CATASTROPHIC_ERROR, 0);
        mlx4_cmd_wake_completions(dev);