Hello everyone!

I'm thinking of writing a hypervisor based kernel mode debugger and I was wondering if anyone has any books/articles/whatsoever on writing kernel mode debuggers? Im inexperienced in writing debuggers, however I do know how to code in x86 assembly, write drivers for windows, etc. I've already used google in hope to get some information but with no success. I'm thinking of using fasm in combination with C/C++ for the project. So if anyone has any information to direct me in right way, I'd be very grateful.

PS: I didnt really know where to put this thread since it's not retated to any particular OS. So if I've putten it to the wrong section, please move it to the right one. Thanks in advance!

Regards,
smoke

Hi, this team did something you would like. At the time I read their work (source code) it lacked a lot of things I would like to have there (more than year ago, maybe they already implemented that).
http://code.google.com/p/hyperdbg/
In principle, your hypervisor intercepts keyboard IO to handle keystrokes so it has some input, you scan PCI bus to find graphic adapter framebuffer, map it into hypervisor and write there (direct pixels to screen) so it has some output.
Another choice is to have 2 PCs and send data via cable (protocol for serial port is simple, for USB or firewire more complicated).
Personaly I developed something else, I have hypervisor as replacement for ms win x64 debug / memory / threads / processes system calls. It is capable to debug ring3 applications, not ring0. Calls hypervisor for freezing threads / manipulate memory / manipulate registers / handle debug events. Does not call ms win APIs for that. Something like stealth debugger.
http://fdbg.x86asm.net/hdbg/hdbg.html
here some video where it steps through HideFromDebugger (you must switch to full screen and switch to HD quality to read the small text there, else it is hardly readable !!!)
http://www.metacafe.com/watch/6104225/hdbg/

I may give you advices / correct mistakes. Cannot release source code / binaries as big part of the work is property of one company I developed some technologies for. Later I extended the work and made that debugger (hdbg) as private activity.

Cheers, Feryno

I did "full" hypervisor debugger few years ago, eg. one where also user interface was running in hypervisor host mode, outside of debugged OS, similar to good-old Soft-Ice, but then I had to discontinue for various reasons. I still dig the idea, and hope someone will sometimes produce something like that.

Thanks for the info Feryno I'll look into that, and thanks for pointing me in the right direction, i honestly didnt know where and what to look for Any ideas what to start working on first? Disasm engine, hypervisor?

vid: Yeah, thats exactly what I want to do, I'm inspired by SoftICE and I hate it that we dont have a decent kernel debugger for x86-64 yet (not counting Windbg for Windows ofcourse ).

Last time I checked Syser is 32bit

smoke: It's trickier than you may think. Virtualization wasn't designed for this. Keep us posted if you start any actual work on this.

certainly hypervisor

which platform do you plane (AMD/Intel) ?
If you have only Intel CPU you may develop AMD hypervisor using SimNow simulator (I did it in such way)
http://developer.amd.com/tools/simnow/pages/default.aspx
for Intel hypervisor the best way is to have CPU (although some free emulators are capable to emulate intel virtualization technology)

vid: yeah, I can imagine I gotta ask though.. are the developer manuals enough to get a complete picture of how to implement the hypervisor the right way? Or will I need to list through the sources of open source virtualization solutions?

Feryno: I have an AMD cpu, so I'll probably start working on the AMD's version of the hypervisor and then upgrade it to Intel. Which one's better anyways?

PS: I'll keep you posted on my work

Manuals are 95% enough, but you need to get a very good grasp of them. There was one exceptionally badly explained thing (something like "VMMON region" I think), but otherwise they were okay. I went through them dozens of times while looking for where the F-ing error could be.

Hopefully AMD's documents are better. Well.. talk to you when I have something!

Regards,
smoke

no worry, I'll help you
for AMD it is a bit easier to setup VMCB but the vm exit handler is much more difficult than on intel
will you start you hypervisor as driver?

Feryno: Thanks, I'd be very glad Yeah I'm thinking of writing a driver based hypervisor for Win 7 64bit as a start

modify your boot.ini or use bcdedit depending your OS (bcdedit for win7) to run on 1 CPU instead of SMP
bcdedit /set onecpu on
then
MmAllocateContiguousMemorySpecifyCache with MmCached
http://msdn.microsoft.com/en-us/library/windows/hardware/ff554464%28v=vs.85%29.aspx
I suggest you to allocate 2 MB (you'll later support more CPUs, e.g. 64 CPUs will eat about 1 MB)
then
MmGetPhysicalAddress
http://msdn.microsoft.com/en-us/library/windows/hardware/ff554547%28v=VS.85%29.aspx
then setup VMCB according AMD manual and prepare vm exit handler also

after you succeed, I'll tell you how to run that on all CPUs in SMP (about 5 OS specific system calls)

certainly install something like windbg so you can analyze crash dump file (I got certainly more than 1000 reboots/BSODs/hangs during few years)

1 trick helped me a lot to know where I got (how far) - I put few instructions which turned on all LEDs on my keyboard (e.g. I had hang but know that something passed OK so the hang was later than the procedure turning on LEDs)

be patient enough (few months of studying manuals, programming, trying, rebooting), never resign

Thanks, I'll do that. Can I do all the initialization from DriverEntry? Or should I initialize the hypervisor from a dispatch routine?

better from dispatch (because later you'll add turning off hypervisor on the running OS without necessity of OS restarting)
but no problem to do it from init
let the first 1MB of mem for shared things of hypervisor (paging tables, code, ...) and split the second 1 MB into 64 parts for 64 CPUs, every CPU will have 4 kB for HSAVE, 4kb for VMCB, 8 kB for stack (at first you setup only CPU00, later you implement SMP)
setup also GS base for hypervisor so you can access per CPU private data (analogy of ms win KPRCB) using GS prefix from hypervisor, I suggest to hold at least GDT, TSS, GS base (self pointer) there
the easiest vm exit loop is something like:
http://board.flatassembler.net/topic.php?p=110033#110033

Hey Feryno, I'm sorry for the late reply but i was a bit busy with school so i couldnt work on the project as much as i'd hoped.

I have a question (maybe a stupid one). I'm kinda half way done with setting up the VMCB. When setting up the Save State area.. I can just use the values of segment registers, stack pointer, control registers, etc right?

Greets,
smoke

Hi, I was busy the whole last week also.

here sample how to setup something:

; rsi virtual memory
; rdi physical memory
; rbp CPU number * 4kB (every CPU uses 4 pages of 4 kB, 1 page for HSAVE, 1 page for VMCB, 2 pages for stack)
; N.B. big offsets like the 00301000 because hypervisor occupies 4 MB of memory, first 2 MB are used for translation tables for memory virtualization, next 1 MB for paging tables of hypervisor, its code, ..., the last 1 MB is used for 64 CPUs, every CPU occupies 16 kB (4 pages)


0000000000010A75: lea rax,[rdi+rbp*4+00301000] ; pointer to physical memory of the VMCB of the current CPU
0000000000010A7D: vmsave ; read manual for more, the rest will be saved manually as follows:
0000000000010A80: sgdt [rsp+06]
0000000000010A85: sidt [rsp+16]
0000000000010A8A: movzx r8d,word [rsp+06]
0000000000010A90: mov rdx,[rsp+08]
0000000000010A95: movzx eax,word [rsp+16]
0000000000010A9A: mov rcx,[rsp+18]
0000000000010A9F: mov [rsi+rbp*4+00301464],r8d
0000000000010AA7: mov [rsi+rbp*4+00301468],rdx
0000000000010AAF: mov [rsi+rbp*4+00301484],eax
0000000000010AB6: mov [rsi+rbp*4+00301488],rcx
0000000000010ABE: mov ecx,es
0000000000010AC0: mov [rsi+rbp*4+00301400],es
0000000000010AC7: lar eax,cx
0000000000010ACA: jnz 0000000000010AD9
0000000000010ACC: shr eax,08
0000000000010ACF: shr ah,04
0000000000010AD2: mov [rsi+rbp*4+00301402],eax
0000000000010AD9: lsl eax,cx
0000000000010ADC: jnz 0000000000010AF3
0000000000010ADE: mov [rsi+rbp*4+00301404],eax
0000000000010AE5: call 0000000000010F00
0000000000010AEA: jc 0000000000010AF3
0000000000010AEC: mov [rsi+rbp*4+00301408],eax

....

0000000000010F00: push rcx
0000000000010F01: push rdx
0000000000010F02: push rbx
0000000000010F03: lea ebx,[r8+01]
0000000000010F07: test cl,04
0000000000010F0A: jz 0000000000010F30
0000000000010F0C: sldt word eax
0000000000010F0F: lsl ebx,ax
0000000000010F12: stc
0000000000010F13: jnz 0000000000010F4C
0000000000010F15: inc ebx
0000000000010F17: mov eax,[rdx+rcx*1+02]
0000000000010F1B: mov rdx,[rdx+rcx*1+04]
0000000000010F20: and eax,00FFFFFF
0000000000010F25: and rdx,FFFFFFFFFF000000
0000000000010F2C: or rdx,rax
0000000000010F2F: nop
0000000000010F30: and cl,F8
0000000000010F33: cmp ebx,ecx
0000000000010F35: jc 0000000000010F4C
0000000000010F37: mov eax,[rdx+rcx*1+02]
0000000000010F3B: mov ecx,[rdx+rcx*1+04]
0000000000010F3F: and eax,00FFFFFF
0000000000010F44: and ecx,FF000000
0000000000010F4A: or eax,ecx
0000000000010F4C: pop rbx
0000000000010F4D: pop rdx
0000000000010F4E: pop rcx
0000000000010F4F: ret

Gavel Republik...