syscall-detect

repository·master·Indexed 42 days ago

https://github.com/jackullrich/syscall-detect

A Proof of Concept tool designed to detect manual system call execution from usermode to identify evasion techniques. It utilizes the Windows ProcessInstrumentationCallback mechanism via NtSetInformationProcess to verify that syscall return addresses reside within legitimate modules like ntdll.dll or win32u.dll.

Tokens: 893
Snippets: 3
Records: 5
Agent score: 3%

What's inside syscall-detect

Overview of syscall-detect
master
syscall-detect is a Proof of Concept (PoC) designed to detect manual system calls (syscalls) originating from usermode. It is intended for security research and detection engineering to identify attempts to bypass standard API monitoring by executing syscalls directly.
https://github.com/jackullrich/syscall-detect/blob/master/README.md
How syscall-detect works via Instrumentation Callbacks
master
The tool uses the Windows ProcessInstrumentationCallback mechanism to intercept transitions from kernel mode to user mode.
When a syscall completes, the kernel executes a callback function. syscall-detect registers a thunk (InstrumentationCallbackThunk) that points to InstrumentationCallback. This callback inspects the CPU context (PCONTEXT) to verify that the return address (Rip) resides within a known, legitimate module (specifically ntdll.dll or win32u.dll).
If the return address is outside these verified modules, the tool identifies it as an unverified module return (potentially indicating direct syscalls or shellcode execution) and triggers a DebugBreak() to halt execution.
https://github.com/jackullrich/syscall-detect/blob/master/main.cpp
Registering the instrumentation callback via NtSetInformationProcess
master
The tool enables detection by calling NtSetInformationProcess with the ProcessInstrumentationCallback information class (integer value 40). This must be done during DLL_PROCESS_ATTACH.
It requires a PROCESS_INSTRUMENTATION_CALLBACK_INFORMATION structure containing a pointer to the callback thunk.
```
PROCESS_INSTRUMENTATION_CALLBACK_INFORMATION nirvana;
nirvana.Callback = (PVOID)(ULONG_PTR)InstrumentationCallbackThunk;
nirvana.Reserved = 0;
nirvana.Version = 0;

NtSetInformationProcess(
    GetCurrentProcess(),
    (PROCESS_INFORMATION_CLASS)40,
    &nirvana,
    sizeof(nirvana));
```
https://github.com/jackullrich/syscall-detect/blob/master/main.cpp

Reference: PROCESS_INSTRUMENTATION_CALLBACK_INFORMATION

master

The structure used to pass the callback information to NtSetInformationProcess.

typedef struct _PROCESS_INSTRUMENTATION_CALLBACK_INFORMATION
{
	ULONG Version;
	ULONG Reserved;
	PVOID Callback;
} PROCESS_INSTRUMENTATION_CALLBACK_INFORMATION, * PPROCESS_INSTRUMENTATION_CALLBACK_INFORMATION;

InstrumentationCallback
master
The core logic function that processes the CPU context during a syscall return. It performs the following steps:
1. Context Restoration: Retrieves the previous instruction pointer (Rip) and stack pointer (Rsp) from the Thread Environment Block (TEB) to ensure the execution flow can be correctly analyzed.
2. Recursion Prevention: Temporarily disables the instrumentation callback via the TEB (TEB->InstrumentationCallbackDisabled) to prevent an infinite loop during the detection process.
3. Module Verification: Uses RIP_SANITY_CHECK to verify if the current Rip is within the memory bounds of ntdll.dll or win32u.dll.
4. Symbol Resolution (Debug Mode): If compiled in _DEBUG, it uses SymFromAddr to resolve the function name and attempts a reverse lookup via GetProcAddress to confirm the function's legitimacy.
5. Security Enforcement: If the Rip is not within the verified modules, it prints a warning and calls DebugBreak().
6. Cleanup: Re-enables the instrumentation callback and restores the context.
```
VOID InstrumentationCallback(PCONTEXT ctx)
```
https://github.com/jackullrich/syscall-detect/blob/master/main.cpp