sif: sqflush: Handle the race condition between sqflush and modify_qp

sif: sqflush: Handle the race condition between sqflush and modify_qp

Orabug: 23759723

Due to a SIF HW bug where SIF might generate duplicate completions,
the QP state must be transitioned into shadowed ERR state (HW state is
in RESET).  In this case, modify_qp(ERR) will cause the QP state
transitions(HW/SW): from ERR (ERR) to RESET (ERR).

As a result, this means that SIF driver needs to generate
FLUSHED-IN-ERR completions when IB user performs post_send while the
QP is in shadowed ERR state. HW will not generate them as the HW QP
state is already in RESET state. The SIF driver generates
FLUSHED-IN-ERR if the last_set_state is in ERR state.  last_set_state
is a "best effort" tracked state because QP mutex cannot be held in a
non-sleep context (post_send).

The issue happens in a multi-threaded scenario where one thread is
constantly performing post_send whereas another thread is performing
modify_qp (ERR). During the QP state transition from ERR (ERR) to
RESET (ERR), both HW and SIF driver generate the FLUSHED-IN-ERR and
eventually causing duplicate completion. This patch adds a
test in post_wa4074 to mask out this condition.

Signed-off-by: Wei Lin Guay <wei.lin.guay@oracle.com>
Reviewed-by: Knut Omang <knut.omang@oracle.com>