ras-arm-handler: cope with latest upstream changes
Unfortunately, rasdaemon support for the firmware first
CPER ARM processor extended trace was added years before
having it merged upstream. That's bad, specially since
upstream revision requested a change on some fields.
Fix support for it by aligning with latest upstream version:
https://lore.kernel.org/linux-edac/3853853f820a666253ca8ed6c7c724dc3d50044a.1720679234.git.mchehab+huawei@kernel.org/T/#m17003e47912b228e91e57ac6e4f90ea30061aa3b
A backward-compatible logic was added to avoid breaking with
existing OOT support.
scripts/checkpatch.pl: some improvements to reduce false positives
- camelcase is OK for printk inttypes.h;
- strncpy is OK;
- accept up to 120 chars on lines without warnings;
- stop complaining about "BACKTRACE=" strings split on multiple lines;
- remove PREFER_DEFINED_ATTRIBUTE_MACRO, as this is kernel-specific;
- remove MACRO_ARG_REUSE, as this applies mostly to multithreading;
- don't warn on using do{} while(0) with single line statements;
There were lots of changes on this version. The summary at
ChangeLog contains a sanitized version of it.
It should be noticed that the next version will likely bring
an uAPI incompatible change. Unfortunately, UEFI CPER record
trace for ARM processor is currently incomplete upstream.
Rasdaemon gained support for an extended arm trace event that
supports all fields of the CPER record, but it depends on a
patch that it is not upstreamed yet.
While looked on such patches, there are some changes needed
to get it merged, meaning that future versions of rasdaemon
may not be compatible with the downstream patch anymore.
scripts/checkpatch.pl: add a script to check coding style
We sort of follow Kernel coding style. Import a version of it,
making it compatible with rasdaemon coding style by removing
stuff that doesn't fix here.
Ruidong Tian [Thu, 23 Nov 2023 09:47:25 +0000 (17:47 +0800)]
rasdaemon: add mc_event trigger
Allow users to run a trigger when RAS mc_event occurs, The mc_event
trigger is separated into CE trigger and UE trigger, this is because
CE is more frequent than UE, and the CE trigger will lead to more
performance hits. Users can choose different triggers for CE/UE to
reduce this effect.
Users can config trigger in /etc/sysconfig/rasdaemon:
TRIGGER_DIR: The trigger diretory
MC_CE_TRIGGER: The script executed when corrected error occurs.
MC_UE_TRIGGER: The script executed when uncorrected error occurs.
No script will be executed if MC_CE_TRIGGER/MC_UE_TRIGGER is null.
util/arm_einj.py: add an utility for ARM error injection via QEMU
Testing rasdaemon is not easy, as it depends on either having
real hardware producing events or a test BIOS. This is usually
not available and/or not too reliable.
So, take a different approach by adding a QEMU QAPI designed for
doing hardware error injection. The QEMU patches are at:
ras-arm-handler: be compatible with upstream Kernel
Changeset e37eb2f11a82 ("Add code to decode Ampere specific error")
broke ARM event record with upstream Kernel, as it requires a different
trace event than the one that it is on upstream Kernel, and it is
part of a pending pull request:
Restore its behavior by making parsing the UEFI 2.6+ N.17 and N.16
table extra fields to be optional. That should make it compatible
with current upstream Kernels again.
Fixes: e37eb2f11a82 ("Add code to decode Ampere specific error") Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Avadhut Naik [Fri, 10 May 2024 18:20:19 +0000 (13:20 -0500)]
rasdaemon: Update SMCA bank error descriptions
Update error descriptions of SMCA bank types to support AMD's new Family
1Ah-based processors.
Also, modify some existing error descriptions to better reflect the error
received.
Shiju Jose [Wed, 20 Mar 2024 12:16:05 +0000 (12:16 +0000)]
rasdaemon: Fix for vendor errors are not recorded in the SQLite database if some cpus are offline
Fix for vendor errors are not recorded in the SQLite database if some cpus
are offline at the system start.
Issue:
This issue is reproducible by offline some cpus, run
./rasdaemon -f --record & and
inject vendor specific error supported in the rasdaemon.
Reason:
When the system starts with some of the cpus offline and then run
the rasdaemon, read_ras_event_all_cpus() exit with error and switch to
the multi thread way. However read() in read_ras_event() return error in
threads for each of the offline CPUs and does clean up including calling
ras_ns_finalize_vendor_tables(), which invokes sqlite3_finalize() on vendor
tables created. Thus the vendor error data does not stored in the SQLite
database when such error is reported next time.
Solution:
In ras_ns_add_vendor_tables() and ras_ns_finalize_vendor_tables() use
reference count and close vendor tables which created in ras_ns_add_vendor_tables()
based on the reference count.
Reported-by: Junhao He <hejunhao3@huawei.com> Signed-off-by: Shiju Jose <shiju.jose@huawei.com> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
mce-amd-smca: update smca_hwid to use smca_bank_types
bank_type is used as smca_bank_types everywhere, there's no point in
declaring it as unsigned int. It also upsets covscan:
3. rasdaemon-0.6.7/mce-amd-smca.c:914: assignment: Assigning: "bank_type" = "s_hwid->bank_type".
7. rasdaemon-0.6.7/mce-amd-smca.c:926: cond_at_most: Checking "bank_type >= 64U" implies that "bank_type" and "s_hwid->bank_type" may be up to 63 on the false branch.
14. rasdaemon-0.6.7/mce-amd-smca.c:942: overrun-local: Overrunning array "smca_mce_descs" of 38 16-byte elements at element index 63 (byte offset 1023) using index "bank_type" (which evaluates to 63).
# 940| /* Only print the descriptor of valid extended error code */
# 941| if (xec < smca_mce_descs[bank_type].num_descs)
# 942|-> mce_snprintf(e->mcastatus_msg,
# 943| "%s. Ext Err Code: %d",
# 944| smca_mce_descs[bank_type].descs[xec],
rasdaemon: Add support to parse microcode field of mce tracepoint
Support for exporting the Microcode Revision is being added to the
mce_record tracepoint.
Add the required, corresponding support in the rasdaemon for the field
to be parsed and logged or added to the database and viewed later through
ras-mc-ctl utility.
rasdaemon: Add support to parse the PPIN field of mce tracepoint
Support for exporting the PPIN (Protected Processor Inventory Number)
is being added to the mce_record tracepoint.
Add the required, corresponding support in the rasdaemon for the field
to be parsed and logged or added to the database and viewed later through
ras-mc-ctl utility.
Avadhut Naik [Tue, 26 Mar 2024 04:06:08 +0000 (23:06 -0500)]
rasdaemon: ras-mc-ctl: Add support to display mcastatus_msg string
Currently, the mcastatus_msg string of struct mce_event is added to the
SQLite database by the rasdaemon when it is recording errors. The same
however, is not outputted by the ras-mc-ctl utility.
The string provides important error information relating to the received
MCE. For example, on AMD SMCA systems, the string outputs extended error
code and description. As such, the string should be present in the
output of ras-mc-ctl utility.
Add support to output the string through the ras-mc-ctl utility.
rasdaemon: Add error decoding for MCA_CTL_SMU extended bits
Enable error decoding support for the newly added extended
error bit descriptions from MCA_CTL_SMU.
b'0:11 can be decoded from existing array smca_smu2_mce_desc.
Define a function to append the newly defined b'58:62 to the
smca_smu2_mce_desc. This reduces the maintaining Reserved bits
from b'12:57 in the code.
Walter Sonius [Sun, 11 Feb 2024 22:30:25 +0000 (23:30 +0100)]
rasdaemon: labels/apple add MacPro 1,1 and 2,1 models
For the Apple MacPro 1,1 (Mac-F4208DC8) and MacPro 2,1 (Mac-F4208DA9)
these are the correct labels for the DIMM numbers 1-4 on each DIMM Riser
A&B for a total of 8 DIMMS. The MacPro 1,1 vendor is actually called
"Apple Computer, Inc." vs "Apple Inc." for the MacPro 2,1 and 3,1.
Another note is that the MacPro 1,1 and 2,1 require the kernel parameter
noefi for their efi32 firmware to boot a 64bit kernel using the
debian-12.4.0-amd64-netinst.iso.
The upper Riser is called A the lower Riser is called B. However
compared to MacPro 3,1 the riser labels A & B are branch swapped on the
memory controller on MacPro1,1 and 2,1 not its physical location in the
case (double checked it)! The so called slot 2 and slot 3 found by
ras-mc-ctl --layout are not available as slots or risers on the
motherboard. The ras-mc-ctl --guess-labels showed right labels but the
DIMM numbers are indistinguishable, however this commit is needed to
link them to the right memory location.
Signed-off-by: Walter Sonius <walterav1984@gmail.com> Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
We don't use Fedorahosted for a long time; the URL was updated,
but right now it is a way more common to receive patches via github
than from other repositories, so change the repository order.
Walter Sonius [Mon, 29 Jan 2024 17:21:37 +0000 (18:21 +0100)]
apple macpro 2008 3,1 dimm1-4 labels riser A&B
For the Apple Mac Pro 3,1( 2008) Mac-F42C88C8 these are the correct labels for the DIMM numbers 1-4 on each DIMM Riser A&B for a total of 8 DIMMS.
The upper Riser is called A the lower Riser is called B. The so called `slot 2` and `slot 3` found by `ras-mc-ctl --layout` are not available as slots or risers on the motherboard. The `ras-mc-ctl --guess-labels` showed right labels but the DIMM numbers are indistinguishable, however this commit is needed to link them to the right memory location.
$ ras-mc-ctl --guess-labels
memory stick 'DIMM 1' is located at 'DIMM Riser B'
memory stick 'DIMM 2' is located at 'DIMM Riser B'
memory stick 'DIMM 1' is located at 'DIMM Riser A'
memory stick 'DIMM 2' is located at 'DIMM Riser A'
memory stick 'DIMM 3' is located at 'DIMM Riser B'
memory stick 'DIMM 4' is located at 'DIMM Riser B'
memory stick 'DIMM 3' is located at 'DIMM Riser A'
memory stick 'DIMM 4' is located at 'DIMM Riser A'
```
Signed-off-by: Walter Sonius <walterav1984@gmail.com>
zhuofeng [Thu, 7 Dec 2023 02:26:56 +0000 (10:26 +0800)]
Fix potential overflow with some arrays at page-isolation logic
Overflows may happen in the `threshold_string` and `cycle_string` arrays.
If the PAGE_CE_THRESHOLD value in page isolation is set to 50 bits,
there is a risk of array overflow. Because sprintf is an insecure
function, use snprintf instead.
An error is reported when the AddressSanitizer is used.
rasdaemon: Improper PAGE_CE_ACTION, set to default soft
rasdaemon: Page offline choice on Corrected Errors is soft
=================================================================
==221920==ERROR: AddressSanitizer: stack-buffer-overflow on address 0xffffdd91d932 at pc 0xffffa24071c4 bp 0xffffdd91d720 sp 0xffffdd91ced8
WRITE of size 55 at 0xffffdd91d932 thread T0
#0 0xffffa24071c0 in vsprintf (/usr/lib64/libasan.so.6+0x5c1c0)
#1 0xffffa24073cc in sprintf (/usr/lib64/libasan.so.6+0x5c3cc)
#2 0x459558 in parse_env_string /home/rasdaemon/ras-page-isolation.c:185
#3 0x4596f4 in page_isolation_init /home/rasdaemon/ras-page-isolation.c:202
#4 0x459934 in ras_page_account_init /home/rasdaemon/ras-page-isolation.c:211
#5 0x40f700 in handle_ras_events /home/rasdaemon/ras-events.c:902
#6 0x405b8c in main /home/rasdaemon/rasdaemon.c:211
#7 0xffffa20b6f38 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
#8 0xffffa20b7004 in __libc_start_main_impl ../csu/libc-start.c:409
#9 0x4038ec in _start (/home/rasdaemon/rasdaemon+0x4038ec)
Address 0xffffdd91d932 is located in stack of thread T0 at offset 82 in frame
#0 0x459574 in page_isolation_init /home/rasdaemon/ras-page-isolation.c:190
This frame has 2 object(s):
[32, 82) 'threshold_string' (line 191)
[128, 178) 'cycle_string' (line 192) <== Memory access at offset 82 partially underflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
(longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-overflow (/usr/lib64/libasan.so.6+0x5c1c0) in vsprintf
Shadow bytes around the buggy address:
0x200ffbb23ad0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x200ffbb23ae0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x200ffbb23af0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x200ffbb23b00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x200ffbb23b10: 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 f1 f1
=>0x200ffbb23b20: 00 00 00 00 00 00[02]f2 f2 f2 f2 f2 00 00 00 00
0x200ffbb23b30: 00 00 02 f3 f3 f3 f3 f3 00 00 00 00 00 00 00 00
0x200ffbb23b40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x200ffbb23b50: f1 f1 f1 f1 f1 f1 04 f2 00 00 f2 f2 00 00 00 00
0x200ffbb23b60: 00 00 00 f2 f2 f2 f2 f2 00 00 00 00 00 00 00 f2
0x200ffbb23b70: f2 f2 f2 f2 00 00 00 00 00 00 00 00 f2 f2 f2 f2
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Shadow gap: cc
==221920==ABORTING
Hunter He [Wed, 6 Dec 2023 06:52:03 +0000 (14:52 +0800)]
rasdaemon:Add support for creating vendor tables at startup.
When rasdaemon is running without non-standard error, those
tables are not created in the database file. Then ras-mc-ctl
script breaks trying to query data from non-existent tables.
Add support for creating vendor tables at startup.
Signed-off-by: Hunter He <hunter.he@jaguarmicro.com>
caixiaomeng 00662745 [Wed, 29 Nov 2023 06:31:46 +0000 (14:31 +0800)]
Add dynamic switch of ras events support.
Rasdaemon does not support a way to disable some events by config.
If user want to disable specified event(eg:block_rq_complete), he
should recompile rasdaemon, which is not so convenient.
This patch add dynamic switch of ras event support.You can add
events you want to disabled in /etc/sysconfig/rasdaemon.For example,
`DISABLE="ras:mc_event,block:block_rq_complete"`.Then restart
rasdaemon, these two events will be disabled without recompilation.
[mchehab: make is_disabled_event() static] Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
Avadhut Naik [Tue, 21 Nov 2023 20:04:19 +0000 (14:04 -0600)]
rasdaemon: Add support for vendor-specific machine check error information
Some CPU vendors may provide additional vendor-specific machine check
error information. AMD, for example, provides FRU Text through SYND 1/2
registers if BIT 9 of SMCA_CONFIG register is set.
Add support to display the additional vendor-specific error information,
if any.
rasdaemon: log non_standard_event at just one line
It is more reasonable log non_standard_event in one line exclude errors
dump. So you can easily to get decoded non_standard_event log in one
line if you implement a decoder like other event.
Avadhut Naik [Thu, 31 Aug 2023 07:23:48 +0000 (02:23 -0500)]
rasdaemon: Fix SMCA bank type decoding
On AMD systems with Scalable MCA (SMCA), the (HWID, MCATYPE) tuple from
the MCA_IPID MSR, bits 43:32 and 63:48 respectively, are used for SMCA
bank type decoding. On occurrence of an SMCA error, the cached tuples are
compared against the tuple read from the MCA_IPID MSR to determine the
SMCA bank type.
Currently however, all high 32 bits of the MCA_IPID register are cached in
the rasdaemon for all SMCA bank types. Bits 47:44 which do not play a part
in bank type decoding are zeroed out. Likewise, when an SMCA error occurs,
all high 32 bits of the MCA_IPID register are read and compared against
the cached values in smca_hwid_mcatypes array.
This can lead to erroneous bank type decoding since the bits 47:44 are
not guaranteed to be zero. They are either reserved or, on some modern
AMD systems viz. Genoa, denote the InstanceIdHi value. The bits therefore,
should not be associated with SMCA bank type decoding.
Import the HWID_MCATYPE macro from the kernel to ensure that only the
relevant fields i.e. (HWID, MCATYPE) tuples are used for SMCA bank type
decoding on occurrence of an SMCA error.
Muralidhara M K [Thu, 27 Jul 2023 10:18:12 +0000 (10:18 +0000)]
rasdaemon: Identify the DIe Number in multidie system
Some AMD systems have 4 dies in each socket and Die ID represents
whether the error occured on cpu die or gpu die.
Also, respective Die used for FRU identification.
Signed-off-by: Muralidhara M K <muralidhara.mk@amd.com> Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
Muralidhara M K [Fri, 30 Jun 2023 11:19:42 +0000 (11:19 +0000)]
rasdaemon: Handle reassigned bit definitions for UMC bank
On some AMD systems some of the existing bit definitions in the
CTL register of SMCA bank type are reassigned without defining
new HWID and McaType. Consequently, the errors whose bit
definitions have been reassigned in the CTL register are being
erroneously decoded.
Add new error description structure to compensate for the
reassigned bit definitions, by new software defined SMCA bank
type by utilizing the hardware-reserved values for HWID.
The new SMCA bank type will only be employed for UMC error
decoding on affected models and the existing error description
structure for UMC bank type is still valid.
Signed-off-by: Muralidhara M K <muralidhara.mk@amd.com> Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
Avadhut Naik [Mon, 22 May 2023 22:13:17 +0000 (22:13 +0000)]
rasdaemon: Add support for post-processing MCA errors
Currently, the rasdaemon performs detailed error decoding of received
MCA errors on the system only whence it is running, either as a daemon
or in the foreground.
As such, error decoding cannot be undertaken for any MCA errors received
whence the rasdaemon wasn't running. Additionally, if the error decoding
modules like edac_mce_amd too have not been loaded, error records in the
demsg buffer might correspond to raw values in associated MSRs, compelling
users to undertake decoding manually. The scenario seems more plausible on
AMD systems with Scalabale MCA (SMCA) with plans in place to remove SMCA
Extended Error Descriptions from the edac_mce_amd module in an effort to
offload SMCA Error Decoding to the rasdaemon.
As such, add support to post-process and decode MCA Errors received on AMD
SMCA systems from raw MSR values. Support for post-processing and decoding
of MCA Errors received on CPUs of other vendors can be added in the future,
as needed.
rasdaemon: Handle reassigned bit definitions for CS SMCA
Currently, on AMD systems with Scalable MCA (SMCA), each machine check
error of a SMCA bank type has an associated bit position in the bank's
control (CTL) register used for enabling / disabling reporting of the
very error. An error's bit position in the CTL register is also used
during error decoding for offsetting into the corresponding bank's error
description structure. As new errors are being added in newer AMD systems
for existing SMCA bank types, the underlying SMCA architecture guarantees
that the bit positions of existing errors are not altered.
However, on some AMD systems viz. Genoa, some of the existing bit
definitions in the CTL register of the Coherent Slave (CS) SMCA bank type
are reassigned without defining new HWID and McaType. Consequently, the
very errors whose bit definitions have been reassigned in the CTL register
are being erroneously decoded.
As a solution, create a new software defined SMCA bank type by utilizing
one of the hardware-reserved values for HWID. The new SMCA bank type will
only be employed for CS error decoding on affected CPU models.
Additionally, since the existing error description structure for the CS
SMCA bank type is still valid, add new error description structure to
compensate for the reassigned bit definitions.
Update, reword some existing SMCA bank type error descriptions to extend
SMCA error decoding functionality for modern AMD processors. Additionally,
also add new error descriptions for missing SMCA bank types.
When the number of CPUs detected is greater than the number of CPUs in
the system, rasdaemon will crash when it receives some events.
Looking deeper, we also fail to use the poll method for similar reasons
in this case.
All of this can be prevented by checking to see how many CPUs are
currently online (sysconf(_SC_NPROCESSORS_ONLN)) instead of how many
CPUs the current kernel was configured to support
(sysconf(_SC_NPROCESSORS_CONF)).
For the kernel side of the discussion, see https://lore.kernel.org/lkml/CAM6Wdxft33zLeeXHhmNX5jyJtfGTLiwkQSApc=10fqf+rQh9DA@mail.gmail.com/T/ Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
Xiaofei Tan [Thu, 11 May 2023 02:54:26 +0000 (10:54 +0800)]
rasdaemon: fix return value type issue of read/write function from unistd.h
The return value type of read/write function from unistd.h is ssize_t.
It's signed normally, and return -1 on error. Fix incorrect use in the
function read_ras_event_all_cpus().
BTW, make setting buffer_percent as a separate function.
Fixes: 94750bcf9309 ("rasdaemon: Fix poll() on per_cpu trace_pipe_raw blocks indefinitely") Signed-off-by: Xiaofei Tan <tanxiaofei@huawei.com> Signed-off-by: Shiju Jose <shiju.jose@huawei.com> Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
Ayush Jain [Tue, 23 May 2023 06:55:36 +0000 (12:25 +0530)]
Rasdaemon: Fix autoreconf build error
When building rasdaemon with autoreconf, on certain distros
we see the following error message.
Makefile.am: error: required file './README' not found
Autoreconf looks for README file instead of README.md
Fix this by passing 'foreign' to AM_INIT_AUTOMAKE.
hubin [Thu, 18 May 2023 08:14:41 +0000 (16:14 +0800)]
ras-events: quit loop in read_ras_event when kbuf data is broken
when kbuf data is broken, kbuffer_next_event() may move kbuf->index back to
the current kbuf->index position, causing dead loop.
In this situation, rasdaemon will repeatedly parse an invalid event, and
print warning like "ug! negative record size -8!", pushing cpu utilization
rate to 100%.
when kbuf data is broken, discard current page and continue reading next page
kbuf.
projects / users / mchehab / rasdaemon.git / log