x86: Fix spectre/kpti integration

The issue is that DISABLE_IBRS (and pretty much all of the _IBRS) first
operation is touching an kernel variable. The restore_c_regs_and_iret is
already in user-space cr3 so we page fault.

The fix is simple - do not run any of the IBRS macros from within
restore_c_regs_and_iret. Which means that the three functions that
used to call it now have to call IBRS_DISABLE by themselves:
retint_swapgs, opportunistic_sysret_failed, and nmi.

Adding in the IBRS_DISABLE in opportunistic_sysret_failed also
fixes another bug - which is more clearly explained in
"x86/enter: Use IBRS on syscall and interrupts  - fix ia32 path"

Orabug: 27333760
CVE: CVE-2017-5754
Signed-off-by: Khalid Aziz <khalid.aziz@oracle.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Signed-off-by: Kirtikar Kashyap <kirtikar.kashyap@oracle.com>

diff --git a/arch/x86/kernel/entry_64.S b/arch/x86/kernel/entry_64.S
index 14959bc7349b389137d120e5d4aa127b1d19d7a0..40970c5403b8e0818310b9b4b4615894d52f3f33 100644 (file)
--- a/arch/x86/kernel/entry_64.S
+++ b/arch/x86/kernel/entry_64.S
@@ -514,6 +514,7 @@ syscall_return_via_sysret:
        CFI_RESTORE_STATE
 
 opportunistic_sysret_failed:
+       DISABLE_IBRS
        /*
         * This opens a window where we have a user CR3, but are
         * running in the kernel.  This makes using the CS
@@ -813,6 +814,7 @@ retint_swapgs:              /* return to user-space */
        DISABLE_INTERRUPTS(CLBR_ANY)
        TRACE_IRQS_IRETQ
 
+       DISABLE_IBRS
        SWITCH_USER_CR3
        SWAPGS
        jmp     restore_c_regs_and_iret
@@ -840,7 +842,6 @@ retint_kernel:
  * which come from interrupts/exception and from syscalls, merge.
  */
 restore_c_regs_and_iret:
-       DISABLE_IBRS
        RESTORE_C_REGS
        REMOVE_PT_GPREGS_FROM_STACK 8
        INTERRUPT_RETURN
@@ -1759,6 +1760,7 @@ ENTRY(nmi)
 2:
 #endif
        call    do_nmi
+       DISABLE_IBRS
 #ifdef CONFIG_PAGE_TABLE_ISOLATION
        /*
         * Unconditionally restore CR3.  I know we return to