drm/amdkfd: Ensure consistent barrier state saved in gfx12 trap handler

It is possible for some waves in a workgroup to finish their save
sequence before the group leader has had time to capture the workgroup
barrier state.  When this happens, having those waves exit do impact the
barrier state.  As a consequence, the state captured by the group leader
is invalid, and is eventually incorrectly restored.

This patch proposes to have all waves in a workgroup wait for each other
at the end of their save sequence (just before calling s_endpgm_saved).

Signed-off-by: Lancelot SIX <lancelot.six@amd.com>
Reviewed-by: Jay Cornwall <jay.cornwall@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org # 6.12.x

diff --git a/drivers/gpu/drm/amd/amdkfd/cwsr_trap_handler.h b/drivers/gpu/drm/amd/amdkfd/cwsr_trap_handler.h
index 984f0e7050788785f8ddffc5350f0836cb942ba8..651660958e5b15aeeee4333761175c3fed23a57c 100644 (file)
--- a/drivers/gpu/drm/amd/amdkfd/cwsr_trap_handler.h
+++ b/drivers/gpu/drm/amd/amdkfd/cwsr_trap_handler.h
@@ -4121,7 +4121,8 @@ static const uint32_t cwsr_trap_gfx12_hex[] = {
        0x0000ffff, 0x8bfe7e7e,
        0x8bea6a6a, 0xb97af804,
        0xbe804ec2, 0xbf94fffe,
-       0xbe804a6c, 0xbfb10000,
+       0xbe804a6c, 0xbe804ec2,
+       0xbf94fffe, 0xbfb10000,
        0xbf9f0000, 0xbf9f0000,
        0xbf9f0000, 0xbf9f0000,
        0xbf9f0000, 0x00000000,

diff --git a/drivers/gpu/drm/amd/amdkfd/cwsr_trap_handler_gfx12.asm b/drivers/gpu/drm/amd/amdkfd/cwsr_trap_handler_gfx12.asm
index 1740e98c6719d22b29b9b561119b58700c13ebd2..7b9d36e5fa437222f161cf1d7da5746e11abf1da 100644 (file)
--- a/drivers/gpu/drm/amd/amdkfd/cwsr_trap_handler_gfx12.asm
+++ b/drivers/gpu/drm/amd/amdkfd/cwsr_trap_handler_gfx12.asm
@@ -1049,6 +1049,10 @@ L_SKIP_BARRIER_RESTORE:
        s_rfe_b64       s_restore_pc_lo                                         //Return to the main shader program and resume execution
 
 L_END_PGM:
+       // Make sure that no wave of the workgroup can exit the trap handler
+       // before the workgroup barrier state is saved.
+       s_barrier_signal        -2
+       s_barrier_wait  -2
        s_endpgm_saved
 end