arch/sparc: Measure receiver forward progress to avoid send mondo timeout

arch/sparc: Measure receiver forward progress to avoid send mondo timeout

A large sun4v SPARC system may have moments of intensive xcall activities,
usually caused by unmapping many pages on many CPUs concurrently. This can
flood receivers with CPU mondo interrupts for an extended period, causing
some unlucky senders to hit send-mondo timeout. This problem gets worse
as cpu count increases because sometimes mappings must be invalidated on
all CPUs, and sometimes all CPUs may gang up on a single CPU.

But a busy system is not a broken system. In the above scenario, as long
as the receiver is making forward progress processing mondo interrupts,
the sender should continue to retry.

This patch implements the receiver's forward progress meter by introducing
a per cpu counter 'cpu_mondo_counter[cpu]' where 'cpu' is in the range
of 0..NR_CPUS. The receiver increments its counter as soon as it receives
a mondo and the sender tracks the receiver's counter. Every 10000 retries,
if the receiver has stopped making forward progress, the sender declares
send-mondo-timeout and panic; otherwise, the receiver is allowed to keep
making forward progress.

Orabug: 25476541
Signed-off-by: Jane Chu <jane.chu@oracle.com>
Reviewed-By: Steve Sistare <steven.sistare@oracle.com>
Reviewed-By: Anthony Yznaga <anthony.yznaga@oracle.com>
Reviewed-by: Rob Gardner <rob.gardner@oracle.com>
Signed-off-by: Allen Pais <allen.pais@oracle.com>