Guest kernel uses ring-0 interrupts

[chc4]

In https://info.varnish-software.com/blog/tinykvm-the-fastest-sandbox it says:

We also enter in usermode and try to avoid kernel mode except where impossible. Did you know that you can handle CPU exceptions in usermode on x86?

However, tinykvm doesn't seem to actually do this.

tinykvm/lib/tinykvm/amd64/idt.cpp

Line 63 in b517667

set_entry(idt.entry[vec], handler, 0x8, IDT_PRESENT | IDT_CPL0 | IDT_GATE_INTR);

only sets up a single IDT gate which runs at ring-0 (CPL0), and with ring-0 segment selectors.

I am using tinykvm for a side project which wants specifically ring-3 -> ring-3 interrupts for speed reasons, and so this behavior was surprising to me. I was able to add a second CPL3 gate, along with another usermode accessible mapping of the kernel interrupt code and usermode accessible interrupt stack, fairly easily so this isn't a blocker, but I wasn't sure if this was unintentional behavior or a design decision which has changed since the block post was written.

[fwsGonzo]

Yep, I don't remember why I did that. Perhaps because I didn't have the usermode assembly at the time, or perhaps because the PIT timer I was programming required kernel mode and I wanted to have an array of (sequential) interrupts? I just don't remember, but... I found my old question, and apparently I was doing it initially: https://stackoverflow.com/questions/66233771/handle-cpu-exceptions-outside-of-kvm/66656234#66656234

It's nice to hear that it still works! Perhaps this should be revisited again. There might be interrupts (except page fault) that really don't need to execute in kernel mode. Page fault has to be CPL0 because of invlpg and invpcid. For the case where you are running a kernel without copy-on-write, you can of course run everything in usermode using an alternate interrupts assembly.

[chc4]

Ah, that makes sense. I'm only using a ring-3 -> ring-3 interrupt for a specific interrupts, so the CoW fault would still run at the correct privilege level. I don't know how useful it will be to you, but chc4/tinycov@64371cd is the hacky commit I made for adding a usermode interrupt for reference.

[fwsGonzo]

That's really cool shit. Btw. I have never measured if it's faster to just set exception handler to unmapped memory, which might trigger a KVM_EXIT_MMIO (or something like that) directly. It could be faster?

In any case, the exception handling API is (as you saw) very rudimentary, and it deserves a little polish. If nothing else, to add support for usermode exceptions. It's interesting that it's faster than privilige switching. Also would be interesting to see if SYSCALL+SYSRET combo is faster than a privilege switch, since it's supposed to be only MSR-dependent, and not use memory. The latest design in TinyKVM avoids most CPU exceptions, though, as the reset mechanism uses non-temporal streaming to roll back to a previous good state after a request. But, for debugging, and for your use-case, there are probably many interesting ideas still to be tried.

[chc4]

I did a small benchmark, and using a ring 3 -> ring 0 interrupt in an int3-heavy workload is 57% (!) slower than the ring 3 -> ring 3 one. In practice I expect this to be much less impactful for most of tinykvm's other usecases, which aren't spending a large percentage of their time servicing interrupts, but still potentially a useful touchpoint.

[fwsGonzo]

If you use forking, will it be able to run faster than current state-of-the-art in sandboxed fuzzing?

[chc4]

Oh definitely not 😅 The current state of the art would be Nyx, or afl++'s nyx_mode, which also use KVM (plus some kernel patches) for hypervisor based fuzzing with a VMM and a fast snapshot reset mechanism instead of a naive Linux forkserver. Nyx however uses Intel PT to collect coverage, which is ~10% performance overhead. TinyCOV's int3 instrumentation is something like 30x instead. My project is mostly for fun and because my laptop doesn't have Intel PT.

The overhead is probably even multiple of afl++ unicorn_mode, which uses either 1) a normal forkserver or 2) persistent mode which does userspace snapshot resetting of the QEMU guest. The overhead of the actual TCG JIT is low enough that even against the "simplest" forkserver mode, doing KVM based instrumentation would probably be slower for most binaries. unicorn_mode is the qemu_mode variant that does system instead of user emulation and provides a security boundary, but I think it would be about equivalent as well.

[fwsGonzo]

That's a very thorough response, thank you. It's probably true that the overhead of the KVM context switching will kill any potential when TCG JIT doesn't, which could be as low as a stack and register switch, or even an opaque function call depending on how it actually works. I have no idea.

... but I also never thought I would end up competing against native Deno simply because my (now improved) reset mechanism ends up being nearly equal (with lower p99), like a flat arena being dropped after rendering a frame in a game engine, compared to having to invoke GC for fairly complex page rendering.

I remember you said that perhaps you could do more in the tiny guest kernel without having to drop out. Did you ever get a chance to go that route and see if amortizing the KVM exit has any chance here?

My project is mostly for fun and because my laptop doesn't have Intel PT.

The best reason of all. And same here, I don't have any Intel PCs anymore.

[chc4]

Yup, I am handling the coverage hook in the guest kernel now: it only takes a VMExit in cases where it hit a new block or is taking a dynamic dispatch jump. In particular, https://github.com/chc4/tinycov/blob/c997f2a6e9a7321f0cfadadb83eac45a3826e5aa/lib/tinykvm/amd64/builtin/guest.cpp#L70 with OPTIMIZE_VMEXIT making the interrupts.asm entrypoint skip over the out instruction in cases where the guest has fully handled the coverage hook.

The numbers quoted and in the README are including that optimization, and without it it's several times slower. Dynamic uninstalling the coverage hooks for branches which I've seen both true and false branch would be the next big improvement, since it would make running the program under KVM in the steadystate be faster than the equivalent code under QEMU which must do SoftMMU translations, but also would give a fairly weird performance profile (where always-taken dead code branches that normally have perfect branch prediction can never be uninstalled and thus higher performance overhead than badly predicted branches). It would also mean you'd no longer be able to use N-gram branch history when computing the bitmap entry instead of only looking at if the single branch was taken or not, which helps in actual fuzzer exploration by exposing more program state as feedback. And it also wouldn't be possible to record a precise coverage trace like that, and you can only record a coverage bitmap (which is less important, since you usually only look at a full trace when triaging crashes or otherwise investigating a single run in a non-performance sensitive way).