Revert x86/apic/x2apic: set affinity of a single interrupt to one cpu

The commit 092aa78c11f0
("x86/apic/x2apic: set affinity of a single interrupt to one cpu")
was causing performance regression on block storage server on X5.

On OCI X5 server, they were not binding irqs to CPUs 1:1,
irq to cpu affinity was set to multiple cpus
(/proc/$irq/smp_affinity: 00,003ffff0,0003ffff, cpu0-17 and 36-53).
This is not the default behavior of bnxt_en. From bnxt_en,
driver, when NIC link is up, it sets irq affinity, OCI assumed that
most of the bnxt_en interrupts will go to cpu3.

After the patch "x86/apic/x2apic:
set affinity of a single interrupt to one cpu",
if we set irq to cpu 1:1, it works fine, but if we set irq affinity
to multiple cpus, it only sets irq_cfg->domain/cpumask to the first
online cpu which is on the cpu affinity list. With the current setting
which caused the perf issue, although /proc/$irq/smp_affinity is set
to multiple cpus, irq_cfg->domain cpumask only has cpu 0, this lead all
ens4f0-TxRx interrupts to route to cpu0, also iscsi target application
was being run on CPU0 during the testing which led to the performace issue.
The issue is no longer seen after the patch was reverted.

Orabug: 29449976
Signed-off-by: Mridula Shastry <mridula.c.shastry@oracle.com>
Reviewed-by: Joe Jin <joe.jin@oracle.com>
Signed-off-by: Somasundaram Krishnasamy <somasundaram.krishnasamy@oracle.com>

diff --git a/arch/x86/kernel/apic/x2apic_cluster.c b/arch/x86/kernel/apic/x2apic_cluster.c
index fb2edecc470a5d427218a968a2381d75fbff03b4..ab3219b3fbda3794eeeded5c9002dd85d3a5eb29 100644 (file)
--- a/arch/x86/kernel/apic/x2apic_cluster.c
+++ b/arch/x86/kernel/apic/x2apic_cluster.c
@@ -219,7 +219,19 @@ static const struct cpumask *x2apic_cluster_target_cpus(void)
 static void cluster_vector_allocation_domain(int cpu, struct cpumask *retmask,
                                             const struct cpumask *mask)
 {
-       cpumask_copy(retmask, cpumask_of(cpu));
+       /*
+        * To minimize vector pressure, default case of boot, device bringup
+        * etc will use a single cpu for the interrupt destination.
+        *
+        * On explicit migration requests coming from irqbalance etc,
+        * interrupts will be routed to the x2apic cluster (cluster-id
+        * derived from the first cpu in the mask) members specified
+        * in the mask.
+        */
+       if (mask == x2apic_cluster_target_cpus())
+               cpumask_copy(retmask, cpumask_of(cpu));
+       else
+               cpumask_and(retmask, mask, per_cpu(cpus_in_cluster, cpu));
 }
 
 static struct apic apic_x2apic_cluster = {