Hypervisor in 1,000 Lines
repository·main·Indexed 43 days ago
https://github.com/nuta/hypervisor-in-1000-linesAn educational reference implementation of a RISC-V hypervisor written in Rust. The project demonstrates hypervisor construction through a concise implementation, featuring a custom BumpAllocator, guest page table management (Sv48x4), a Linux kernel loader, and VCpu state management for a no_std environment.
What's inside Hypervisor in 1,000 Lines
- This repository provides the source code for the website Hypervisor in 1,000 Lines and includes a reference implementation of a hypervisor. It is designed as an educational resource for understanding hypervisor construction through a concise implementation.
SBI Call Conventions for Hypercalls
mainWhen implementing hypercall handlers, follow the SBI specification conventions:
- Control Transfer: Use
ECALLto transfer control from the supervisor to the SEE (Supervisor Execution Environment/Hypervisor). - Extension ID (EID): Encoded in register
a7. - Function ID (FID): Encoded in register
a6(for SBI v0.2+). - Register Preservation: The callee must preserve all registers except
a0anda1. - Return Values: Functions must return a pair of values in
a0anda1, wherea0contains the error code.
- Control Transfer: Use
Understand RISC-V Guest Modes (VU-mode and VS-mode)
mainThe RISC-V hypervisor extension introduces virtualization modes that allow the guest operating system to run with its own user and kernel modes, separate from the host's modes.
- VU-mode (Virtual User mode): The guest's user mode.
- VS-mode (Virtual Supervisor mode): The guest's kernel mode.
- U-mode (User mode): The host's user mode.
- HS-mode (Hypervisor-extended Supervisor mode): The host's kernel mode.
In a hardware-assisted hypervisor, the CPU handles transitions between VU-mode and VS-mode (like guest system calls) automatically. The hypervisor acts as an event handler, primarily intervening during VM exits (e.g., when the guest performs memory-mapped I/O) by jumping to a trap handler.
How hypercalls work in the hypervisor
mainHypercalls are the mechanism used by a guest OS to request services from the hypervisor, similar to how system calls work for an OS. The flow is as follows:
- The guest program executes the
ecallinstruction with parameters passed in registers. - The CPU switches to HS-mode (Hypervisor Supervisor mode) and jumps to the trap handler defined in the
stvecCSR. - The trap handler saves the guest state and determines the cause of the trap by reading the
scauseCSR (e.g., anecallfrom the guest). - After the hypervisor handles the request, the trap handler restores the guest state and returns control to the guest.
- The guest program executes the
Understand Guest-physical vs. Host-physical addresses
mainIn hardware-assisted virtualization (like RISC-V), the CPU uses four address spaces to provide isolation between VMs. When the CPU is in guest mode, address translation occurs in two stages:
- Stage 1: Guest-virtual $\rightarrow$ guest-physical (controlled by the page table in
satp). - Stage 2: Guest-physical $\rightarrow$ host-physical (controlled by the guest page table in
hgatp).
This two-stage translation allows the hypervisor to map each guest's perceived physical memory to distinct regions of the actual host physical memory, enabling multiple VMs to run simultaneously without direct access to host memory.
- Stage 1: Guest-virtual $\rightarrow$ guest-physical (controlled by the page table in
Implement a trap handler to debug CPU events
mainA trap handler is triggered when the CPU encounters events like invalid memory access or illegal instructions. To implement one:
- Read CSRs: Use a macro to read Control and Status Registers (CSRs) like
scause(cause),sepc(program counter), andstval(trap-specific value). - Map Causes: Match the
scausevalue against known RISC-V trap causes (e.g.,2forillegal instruction). - Register the Handler: Write the handler's address to the
stvecCSR. - Linker Configuration: Ensure the handler is placed in a specific section (e.g.,
.text.stvec) using the#[unsafe(link_section = ".text.stvec")]attribute so the linker can align it correctly.
Once implemented, you can use
llvm-addr2linewith the address fromsepcto find the exact line of code that caused the trap.#[unsafe(link_section = ".text.stvec")] pub fn trap_handler() -> ! { let scause = read_csr!("scause"); let sepc = read_csr!("sepc"); let stval = read_csr!("stval"); // ... match scause to string ... panic!("trap handler: {} at {:#x} (stval={:#x})", scause_str, sepc, stval); }- Read CSRs: Use a macro to read Control and Status Registers (CSRs) like
Add the `spin` crate for thread-safe allocation
mainTo make your allocator thread-safe in a
no_stdenvironment, add thespincrate, which provides aMuteximplementation using spinlocks.$ cargo add spinEnable the Hypervisor Extension in QEMU
mainTo support virtualization modes, you must enable the hypervisor extension in QEMU by adding the
h=trueflag to the-cpuoption.-cpu rv64,h=trueInstall QEMU for emulation
mainQEMU is required to emulate the environment where your hypervisor will run. Installation commands vary by operating system:
- macOS: Use Homebrew.
- Ubuntu: Install the RISC-V 64-bit system emulator.
- Windows: Install Windows Subsystem for Linux (WSL2) and follow the Ubuntu instructions.
### macOS ```sh brew install qemuUbuntu
sudo apt install qemu-system-riscv64Build and run the hypervisor with QEMU
mainTo build the hypervisor, use
cargo buildwith specificRUSTFLAGSto pass the linker script and specify therust-lldlinker. The resulting ELF file is then executed viaqemu-system-riscv64.Build Command Requirements:
RUSTFLAGS="-C link-arg=-Thypervisor.ld -C linker=rust-lld"- Target:
riscv64gc-unknown-none-elf
QEMU Execution Flags:
-machine virtand-cpu rv64-bios default-m 128M(Memory size)-d cpu_reset,unimp,guest_errors,int -D qemu.log(Logging/Debugging)-kernel hypervisor.elf
#!/bin/sh set -ev RUSTFLAGS="-C link-arg=-Thypervisor.ld -C linker=rust-lld" \ cargo build --bin hypervisor --target riscv64gc-unknown-none-elf cp target/riscv64gc-unknown-none-elf/debug/hypervisor hypervisor.elf qemu-system-riscv64 \ -machine virt \ -cpu rv64 \ -bios default \ -smp 1 \ -m 128M \ -nographic \ -d cpu_reset,unimp,guest_errors,int -D qemu.log \ -serial mon:stdio \ --no-reboot \ -kernel hypervisor.elfUse the `alloc` crate for dynamic collections
mainOnce the global allocator is initialized, you can use dynamic memory features by adding
extern crate alloc;to your entry point. Note that you do not need to addallocto yourCargo.tomlas it is bundled with the standard library.You must explicitly
useobjects from thealloccrate (e.g.,alloc::vec::Vec).#![no_std] #![no_main] extern crate alloc; fn main() -> ! { // Ensure allocator is initialized first allocator::GLOBAL_ALLOCATOR.init(&raw mut __heap, &raw mut __heap_end); let mut v = alloc::vec::Vec::new(); v.push('a'); v.push('b'); v.push('c'); loop {} }Enter Guest Mode (HS-mode) using RISC-V CSRs
mainTo transition the CPU from the host kernel into the guest kernel (HS-mode), you must configure the Hypervisor Status register (
hstatus) and the Supervisor Exception Program Counter (sepc), then execute thesretinstruction.Key components:
hstatus: A register containing flags that describe the CPU state for the transition.sepc: The program counter address where the CPU will begin execution in guest mode.sret: The trap-return instruction that performs the transition based on the values inhstatusandsepc.
fn main() -> ! { /* ... */ let mut hstatus: u64 = 0; hstatus |= 2 << 32; // VSXL: XLEN for VS-mode (64-bit) hstatus |= 1 << 7; // SPV: Supervisor Previous Virtualization mode let sepc: u64 = 0x1234abcd; unsafe { asm!( "csrw hstatus, {hstatus}", "csrw sepc, {sepc}", "sret", hstatus = in(reg) hstatus, sepc = in(reg) sepc, ); } unreachable!(); }