sif: cq: Implement error tracking

sif: cq: Implement error tracking

This commit introduces a new state variable 'in_error'
in the CQ state. When a CQ error event is detected
cq->in_error is set.

This state is then checked before
* posting a req_notify_cq
* invoking any of the completion generating
  workarounds.

Also set qp->last_set_state to ERR if a fatal QP event
is detected to reduce further posting on that QP.

This commit reduces the risk of getting into a privileged
QP error scenario for some common cases of misbehaved
applications, and also enables the driver to terminate more
quickly if a priv.QP error has occurred.

Orabug: 24715634

Signed-off-by: Knut Omang <knut.omang@oracle.com>
Reviewed-by: Håkon Bugge <haakon.bugge@oracle.com>