Intel x86-64 OS Internals

Abstract

This is it! This is the class that actually teaches you how ring 0 vs. ring 3 privilege separation works at a deep deep level on Intel hardware!!! But so much more too! This class gives you an incredibly strong foundation for Windows, Linux, *BSD, or macOS OS Internals classes. Because it covers the hardware-specific requirements that every OS (or VMM or BIOS) must deal with if they want to run on Intel hardware.

After this class you will feel exploring wherever you want in the Intel manuals, because this class covers some of the most difficult to understand underlying technologies that OSes depend on.

Topics include, but are not limited to:

• Physical and virtual memory and how a limited amount of physical memory is represented as much more virtual memory through a multilevel paging system. We will also talk about memory segmentation.
• The hardware basis for kernel versus userspace separation and how software transitions between the two. This portion answers the question of why does x86 have 4 “rings”, with ring 0 being the most privileged, and ring 3 being the least.
• Hardware and software interrupts, and how they are the basis for debugging.
• Input/Output instructions and how these allow the CPU to talk to peripherals.

Example applications of this knowledge include showing how hardware and memory mechanisms are used for software exploits, anti-debug techniques, and rootkit hiding.

If your engineers need to know how fundamental information that every x86-based OS and hypervisor and firmware depends on, at a low level, reach out to us with info about how many students you’d like us to teach, and where.

---

Full class outline

• Introduction
• CPUID
  • CPU Feature Identification instruction
• Processor Execution Modes
  • Real mode, protected mode, IA-32e mode, system management mode
• Model Specific Registers (MSRs)
  • Reading and writing MSRs (rdmsr, wrmsr)
• Privilege Rings & Segmentation
  • Privilege rings start
  • Segment selectors & segment registers
  • Global Descriptor Table (GDT) & Local Descriptor Table (LDT)
  • Segment descriptors
  • Privilege rings finish
  • Call gates
  • Return to RTFM
  • Implicit and explicit use of segmentation
• Interrupts
  • Interrupts vs. Exceptions
  • Tasks and the Task State Segment (TSS)
  • Interrupt Descriptor Table (IDT)
  • Interrupt Descriptors
  • Interrupt masking
  • Red Pill and virtualization detection
• System Calls
  • System Call instructions (syscall/sysret, sysenter/sysexit)
  • Syscall-adjacent techniques & instructions (swapgs, {rd,wr}{fs,gs}base)
• Read the Time Stamp Counter (RDTSC)
• Paging and Virtual Memory
  • Introduction
  • Paging and the Control Registers
  • Page Tables
    - MAXPHYADDR
    - 32 bit linear to 32 bit physical, 4KB pages
    - 32 bit linear to 32 bit physical, 4KB pages
    - 32 bit linear to 32 bit physical, 4MB pages
    - 32 bit linear to 40 bit physical, Physical Address Extensions (PAE)
    - 48 bit linear to 52 bit physical, 4-level paging, 4KB, 2MB, 1GB pages
    - (Optional) 57 bit linear to 52 bit physical, 5-level paging
  • Page Table Entries
    - CR3
    - PML4E (Page Map Level 4 Entry)
    - Exploit Mitigation Aside - XD, SMEP, SMAP
    - PDPTE (Page Directory Pointer Table Entry)
    - PDE (Page Directory Entry)
    - PTE (Page Table Entry)
  • Canonical addresses
  • Page faults
  • Translation Lookaside Buffer & Shadow Walker rootkit
  • Non-executable Memory (NX/XD bit)
• Interrupts & Debugging
  • Software breakpoints
  • Hardware breakpoints
  • Normal break on execute, write, read/write, port IO
  • Break on mov to debug registers
  • Trap Flag (TF)
  • Resume Flag (RF) and single step exceptions
• Port IO
  • In/out instruction and accessing VMWare “backdoor” IO port
• Conclusion

Request Training Quote