perf/x86/intel/ds: Simplify the PEBS records processing for adaptive PEBS

The current code may iterate all the PEBS records in the DS area several
times. The first loop is to find all active events and calculate the
available records for each event. Then iterate the whole buffer again
and again to process available records until all active events are
processed.

The algorithm is inherited from the old generations. The old PEBS
hardware does not deal well with the situation when events happen near
each other. SW has to drop the error records. Multiple iterations are
required.

The hardware limit has been addressed on newer platforms with adaptive
PEBS. A simple one-iteration algorithm is introduced.

The samples are output by record order with the patch, rather than the
event order. It doesn't impact the post-processing. The perf tool always
sorts the records by time before presenting them to the end user.

In an NMI, the last record has to be specially handled. Add a last[]
variable to track the last unprocessed record of each event.

Test:

11 PEBS events are used in the perf test. Only the basic information is
collected.
perf record -e instructions:up,...,instructions:up -c 2000003 benchmark

The ftrace is used to record the duration of the
intel_pmu_drain_pebs_icl().

The average duration reduced from 62.04us to 57.94us.

A small improvement can be observed with the new algorithm.
Also, the implementation becomes simpler and more straightforward.

Suggested-by: Stephane Eranian <eranian@google.com>
Signed-off-by: Kan Liang <kan.liang@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Dapeng Mi <dapeng1.mi@linux.intel.com>
Link: https://lore.kernel.org/r/20241119135504.1463839-5-kan.liang@linux.intel.com

diff --git a/arch/x86/events/intel/ds.c b/arch/x86/events/intel/ds.c
index 79a3467c747bebc303bd869289dd46fd2a8a8d07..8dcf90f6fb59069a85b1b3ed76b6b21c4f3724a9 100644 (file)
--- a/arch/x86/events/intel/ds.c
+++ b/arch/x86/events/intel/ds.c
@@ -2425,8 +2425,12 @@ static void intel_pmu_drain_pebs_nhm(struct pt_regs *iregs, struct perf_sample_d
 static void intel_pmu_drain_pebs_icl(struct pt_regs *iregs, struct perf_sample_data *data)
 {
        short counts[INTEL_PMC_IDX_FIXED + MAX_FIXED_PEBS_EVENTS] = {};
+       void *last[INTEL_PMC_IDX_FIXED + MAX_FIXED_PEBS_EVENTS];
        struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
        struct debug_store *ds = cpuc->ds;
+       struct x86_perf_regs perf_regs;
+       struct pt_regs *regs = &perf_regs.regs;
+       struct pebs_basic *basic;
        struct perf_event *event;
        void *base, *at, *top;
        int bit;
@@ -2448,30 +2452,41 @@ static void intel_pmu_drain_pebs_icl(struct pt_regs *iregs, struct perf_sample_d
                return;
        }
 
-       for (at = base; at < top; at += cpuc->pebs_record_size) {
+       if (!iregs)
+               iregs = &dummy_iregs;
+
+       /* Process all but the last event for each counter. */
+       for (at = base; at < top; at += basic->format_size) {
                u64 pebs_status;
 
-               pebs_status = get_pebs_status(at) & cpuc->pebs_enabled;
-               pebs_status &= mask;
+               basic = at;
+               if (basic->format_size != cpuc->pebs_record_size)
+                       continue;
+
+               pebs_status = basic->applicable_counters & cpuc->pebs_enabled & mask;
+               for_each_set_bit(bit, (unsigned long *)&pebs_status, X86_PMC_IDX_MAX) {
+                       event = cpuc->events[bit];
 
-               for_each_set_bit(bit, (unsigned long *)&pebs_status, X86_PMC_IDX_MAX)
-                       counts[bit]++;
+                       if (WARN_ON_ONCE(!event) ||
+                           WARN_ON_ONCE(!event->attr.precise_ip))
+                               continue;
+
+                       if (counts[bit]++) {
+                               __intel_pmu_pebs_event(event, iregs, regs, data, last[bit],
+                                                      setup_pebs_adaptive_sample_data);
+                       }
+                       last[bit] = at;
+               }
        }
 
        for_each_set_bit(bit, (unsigned long *)&mask, X86_PMC_IDX_MAX) {
-               if (counts[bit] == 0)
+               if (!counts[bit])
                        continue;
 
                event = cpuc->events[bit];
-               if (WARN_ON_ONCE(!event))
-                       continue;
-
-               if (WARN_ON_ONCE(!event->attr.precise_ip))
-                       continue;
 
-               __intel_pmu_pebs_events(event, iregs, data, base,
-                                       top, bit, counts[bit],
-                                       setup_pebs_adaptive_sample_data);
+               __intel_pmu_pebs_last_event(event, iregs, regs, data, last[bit],
+                                           counts[bit], setup_pebs_adaptive_sample_data);
        }
 }