RDS: IB: Run rds_fmr_flush WQ closer to ib_device

RDS: IB: Run rds_fmr_flush WQ closer to ib_device

Orabug 22269408

rds_fmr_flush workqueue is calling ib_unmap_fmr
to invalidate a list of FMRs. Today, this workqueue
can be scheduled at any CPUs. In a NUMA-aware system,
schedule this workqueue to run on a CPU core closer to
ib_device can improve performance. As for now, we use
"sequential-low" policy. This policy selects two lower
cpu cores closer to HCA. In a non-NUMA aware system,
schedule rds_fmr_flush workqueue in a fixed cpu core
improves performance.

The mapping of cpu to the rds_fmr_flush workqueue
can be enabled/disabled via  sysctl and it is enable
by default. To disable the feature, use below sysctl.

rds_ib_sysctl_disable_unmap_fmr_cpu = 1

Putting down some of the rds-stress performance number
comparing default and sequential-low policy in a NUMA
system with Oracle M4 QDR and Mellanox CX3.

rds-stress 4 conns, 32 threads, 16 depths, RDMA write
and unidirectional (higher is better).

(Oracle M4 QDR)
default		: 645591 IOPS
sequential-low	: 806196 IOPS

(Mellanox CX3)
default		: 473836 IOPS
sequential-low	: 544187 IOPS

Reviewed-by: Håkon Bugge <haakon.bugge@oracle.com>
Reviewed-by: Knut Omang <knut.omang@oracle.com>
Reviewed-by: Santosh Shilimkar <santosh.shilimkar@oracle.com>
Signed-off-by: Wei Lin Guay <wei.lin.guay@oracle.com>