net/rds: Implement ARP flushing correctly

net/rds: Implement ARP flushing correctly

If a remote peer has moved its IP address from one port to the other,
the local node may have an incorrect ARP entry in its cache. During
connection management, we will then get back a route-error-event from
the CM.

Current code attempts to flush the ARP entry from the cache. However,
1) it does not check for return values, 2) it does not supply the
device name, 3) it does not iterate over all possible device names,
and 4) its doesn't supply the correct flags.

Due to 2-4 above, the flushing doesn't work.

This commit fixes this.

On a system with a single CX-3 and 16 VFs, fail-over just after a
fail-back is reduced from ~60 seconds down to ~10 seconds with the fix
(1156 RDS connections).

Orabug: 28219857

Signed-off-by: Håkon Bugge <haakon.bugge@oracle.com>
Reviewed-by: Santosh Shilimkar <santosh.shilimkar@oracle.com>
---

V1 -> V2:
- Move NOTE section down as suggested by Santosh
- Remove unnecessary initialization of ret
- Fix a nit in the commit message (2-4 instead of 1-4)
- Use pr_err() instead of printk() as suggested by Yuval
- Use sizeof(*r) as suggested by Yuval

NOTE:
	* The uek-4-QU7 version of this bug has an approved exception from
	  Greg Marsden, that it can be merged first in uek-4-QU7, before
	  uek5/master-next.

Signed-off-by: Brian Maly <brian.maly@oracle.com>