This might be a weird question, and it may very well be impossible, but is there a way to intercept all syscalls of a specific program? I have full control over the file, which mean I can scan and modify it, which led me to think I could maybe replace all 0F 05 instructions (syscall) by a jmp instruction, but I don't think this will work, because I also want to intercept syscall made by the standard library. I am on windows x86-64, but there is no problem for me to install a Linux virtual machine - I already know there's a syscall logger that works on Linux, strace. ChatGPT suggested a VEH hooking, but I'm not quite sure it works the way he describes it.

For instance, for this code, parsed to executable code:

#include <cstdio>

int main() {
    FILE *file = fopen("example.txt", "w");
    fclose(file);

    return 0;
}

I would like to intercept the NtCreateFile syscall. Is it possible? If i cannot intercept, is there a way of at least logging it?

EDIT: Clarifications

I want to intercept only the syscalls of a specific program. Not of all of them, so a kernel driver won't do the trick. Administrator mode is no problem, I can run the intercepter in Admin mode. A solution I thought of is to scan the .exe file to locate standard library calls. I'm not sure whether that's doable or not...? but it would suffice me, given I can get the functions' name.

This might be a weird question, and it may very well be impossible, but is there a way to intercept all syscalls of a specific program? I have full control over the file, which mean I can scan and modify it, which led me to think I could maybe replace all 0F 05 instructions (syscall) by a jmp instruction, but I don't think this will work, because I also want to intercept syscall made by the standard library. I am on windows x86-64, but there is no problem for me to install a Linux virtual machine - I already know there's a syscall logger that works on Linux, strace. ChatGPT suggested a VEH hooking, but I'm not quite sure it works the way he describes it.

For instance, for this code, parsed to executable code:

#include <cstdio>

int main() {
    FILE *file = fopen("example.txt", "w");
    fclose(file);

    return 0;
}

I would like to intercept the NtCreateFile syscall. Is it possible? If i cannot intercept, is there a way of at least logging it?

EDIT: Clarifications

I want to intercept only the syscalls of a specific program. Not of all of them, so a kernel driver won't do the trick. Administrator mode is no problem, I can run the intercepter in Admin mode. A solution I thought of is to scan the .exe file to locate standard library calls. I'm not sure whether that's doable or not...? but it would suffice me, given I can get the functions' name.

• windows
• debugging
• x86-64
• exe
• system-calls

• It's definitely possible -- I've seen it done -- but in a context where knowledge of any non-obvious details all falls into the category of trade secrets owned by a former employer; so this question needs to be left for someone else to answer. – Charles Duffy Commented Mar 10 at 21:42
• On Linux, the ptrace(PTRACE_SYSCALL) system call is what a debugger (or other intrusive process) can use to run the tracee until its next system call. Effectively like setting a breakpoint on every system-call by another process. ptrace(GETREGS) / ptrace(PTRACE_SETREGS) allow modifying it. Of course this doesn't help you at all for doing things to a native Windows process; those won't be visible outside the VM. Even for WSL1 which is just a Windows subsystem, unlike WSL2 where a hardware VM runs a paravirtualized Linux kernel. Linux processes can only trace other Linux processes. – Peter Cordes Commented Mar 10 at 23:55
• 1 There's an handful of ways to do that. As you pointed out, syscall instructions are in the loaded DLLs on Windows, so scanning the original file won't work. The easiest way is to write a program that single step the original binary and check each instruction. But it's slow. Another possibility is to launch the program suspended, scan it and its modules, eventually hooking the low level API that map a PE image to scan dynamically loaded modules. You can also write a driver that hook the syscall table for your process only. Mostly depends on how general you want to be. – Margaret Bloom Commented Mar 11 at 14:02
• I don't really care about speed. What do you mean by running the program step by step? Is there a tool to do that? For your other suggestion, I don't want to impact any other program other than this specific one, so making a kernell driver won't work. – anom907 zat Commented Mar 11 at 16:18
• @anom907zat I mean making a very very simple debugger. For example, with the debug api or more unconventionally by starting the process suspended, loading a DLL of your in the program, setting the TF flag in the process context and an exception handler handling the single step exception. This is easily detectable by the program itself, FWIW. – Margaret Bloom Commented Mar 14 at 12:43

1 Answer 1

I would like to intercept the NtCreateFile syscall.

My answer is for the Windows platform and is based on my experience writing a minifilter file system driver, but conceptually, it is similar for other platforms.

The problem with writing an application like this - OS will separate each application into its own execution contexts: memory address space, resources, handles and so on. Tapping into another application requires administrator permissions, and some actions cannot be done in the user space. For example, in the user space you may be able to read the memory of another process. However, you cannot put execution hooks when another application calls NtCreateFile. Windows OS does not allow this for security reasons.

To intercept / hook an API call of another process you have to write a kernel driver. For this specific task, the driver you are looking for is called a filesystem minifilter driver. A great example of this is a passthrough minifilter driver available on github.

Minifilter driver will intercept each and every Windows API call you hook (for example IRP_MJ_CREATE), and route it to your handling function. You must be careful to correctly handle the request, otherwise, Windows will crash with BSOD. There are certain parts of functionality available at certain execution time, and there are very strict behaviour limitation as to what the code can do (such as memory allocations, delayed IO operations, anything that may take time to execute).

Because this runs in a kernel, and if you want to release the code, you have to go through a driver signing. For debug and testing purposes, you could allow test-signed or unsigned (maybe?) drivers to run.

Once you have a skeleton of a driver, you can then identify specific application, either by name, path or some other way, and only run handling logic when the filter matches the application.

I would also like to add, that you could see many file system related calls in an application called procmon, where you can configure filters to the specific file / application, which may be helpful in some cases.