sif: qp: added persistent_state in sif_qp struct

sif: qp: added persistent_state in sif_qp struct

Orabug: 23491094

QP state needs to be referred in certain context which may not sleep.
Nevertheless, the state is not guarantee as mutex cannot be used. Thus,
this commit added new atomic persistent_state to determine the QP
state in non-sleep context.

This commit removes non-used flush_sq_done_wa4074 variable and added
mutex for sif_query_qp due to WA #3714 and WA #662. In SIF, there is
intermediate QP state from RTS->ERR and RTS->RESET. Thus, without
mutex, the sif_query_qp might gets the intermediate QP state.

Signed-off-by: Wei Lin Guay <wei.lin.guay@oracle.com>
Reviewed-by: Håkon Bugge <haakon.bugge@oracle.com>
Reviewed-by: Knut Omang <knut.omang@oracle.com>