Message board for the users of flat assembler.
> Projects and Ideas > Writing a Hypervisor - ring -1 debugger
Goto page 1, 2 Next
smoke 10 Oct 2011, 16:54
I'm thinking of writing a hypervisor based kernel mode debugger and I was wondering if anyone has any books/articles/whatsoever on writing kernel mode debuggers? Im inexperienced in writing debuggers, however I do know how to code in x86 assembly, write drivers for windows, etc. I've already used google in hope to get some information but with no success. I'm thinking of using fasm in combination with C/C++ for the project. So if anyone has any information to direct me in right way, I'd be very grateful.
PS: I didnt really know where to put this thread since it's not retated to any particular OS. So if I've putten it to the wrong section, please move it to the right one. Thanks in advance!
|10 Oct 2011, 16:54||
Feryno 11 Oct 2011, 07:19
Hi, this team did something you would like. At the time I read their work (source code) it lacked a lot of things I would like to have there (more than year ago, maybe they already implemented that).
In principle, your hypervisor intercepts keyboard IO to handle keystrokes so it has some input, you scan PCI bus to find graphic adapter framebuffer, map it into hypervisor and write there (direct pixels to screen) so it has some output.
Another choice is to have 2 PCs and send data via cable (protocol for serial port is simple, for USB or firewire more complicated).
Personaly I developed something else, I have hypervisor as replacement for ms win x64 debug / memory / threads / processes system calls. It is capable to debug ring3 applications, not ring0. Calls hypervisor for freezing threads / manipulate memory / manipulate registers / handle debug events. Does not call ms win APIs for that. Something like stealth debugger.
here some video where it steps through HideFromDebugger (you must switch to full screen and switch to HD quality to read the small text there, else it is hardly readable !!!)
I may give you advices / correct mistakes. Cannot release source code / binaries as big part of the work is property of one company I developed some technologies for. Later I extended the work and made that debugger (hdbg) as private activity.
|11 Oct 2011, 07:19||
vid 11 Oct 2011, 08:09
I did "full" hypervisor debugger few years ago, eg. one where also user interface was running in hypervisor host mode, outside of debugged OS, similar to good-old Soft-Ice, but then I had to discontinue for various reasons. I still dig the idea, and hope someone will sometimes produce something like that.
|11 Oct 2011, 08:09||
smoke 11 Oct 2011, 14:04
Thanks for the info Feryno I'll look into that, and thanks for pointing me in the right direction, i honestly didnt know where and what to look for Any ideas what to start working on first? Disasm engine, hypervisor?
vid: Yeah, thats exactly what I want to do, I'm inspired by SoftICE and I hate it that we dont have a decent kernel debugger for x86-64 yet (not counting Windbg for Windows ofcourse ).
|11 Oct 2011, 14:04||
ouadji 11 Oct 2011, 14:16
we dont have a decent kernel debugger for x86-64 yet
i use Syser (ring0 degugger) every day 18h/24h
(sometimes even at night)
|11 Oct 2011, 14:16||
smoke 11 Oct 2011, 14:35
Last time I checked Syser is 32bit
|11 Oct 2011, 14:35||
ouadji 11 Oct 2011, 15:30
Indeed, you're right. Syser is a 32bits degugger, not 64bit.
For 64bit, there are only WinDbg.
WinDbg is a big shit, really !
|11 Oct 2011, 15:30||
vid 12 Oct 2011, 09:25
smoke: It's trickier than you may think. Virtualization wasn't designed for this. Keep us posted if you start any actual work on this.
|12 Oct 2011, 09:25||
Feryno 12 Oct 2011, 13:34
Any ideas what to start working on first? Disasm engine, hypervisor?
which platform do you plane (AMD/Intel) ?
If you have only Intel CPU you may develop AMD hypervisor using SimNow simulator (I did it in such way)
for Intel hypervisor the best way is to have CPU (although some free emulators are capable to emulate intel virtualization technology)
|12 Oct 2011, 13:34||
smoke 12 Oct 2011, 16:51
vid: yeah, I can imagine I gotta ask though.. are the developer manuals enough to get a complete picture of how to implement the hypervisor the right way? Or will I need to list through the sources of open source virtualization solutions?
Feryno: I have an AMD cpu, so I'll probably start working on the AMD's version of the hypervisor and then upgrade it to Intel. Which one's better anyways?
PS: I'll keep you posted on my work
|12 Oct 2011, 16:51||
vid 12 Oct 2011, 17:56
Manuals are 95% enough, but you need to get a very good grasp of them. There was one exceptionally badly explained thing (something like "VMMON region" I think), but otherwise they were okay. I went through them dozens of times while looking for where the F-ing error could be.
|12 Oct 2011, 17:56||
smoke 15 Oct 2011, 18:41
Hopefully AMD's documents are better. Well.. talk to you when I have something!
|15 Oct 2011, 18:41||
Feryno 17 Oct 2011, 07:09
no worry, I'll help you
for AMD it is a bit easier to setup VMCB but the vm exit handler is much more difficult than on intel
will you start you hypervisor as driver?
|17 Oct 2011, 07:09||
smoke 17 Oct 2011, 08:37
Feryno: Thanks, I'd be very glad Yeah I'm thinking of writing a driver based hypervisor for Win 7 64bit as a start
|17 Oct 2011, 08:37||
Feryno 17 Oct 2011, 12:14
modify your boot.ini or use bcdedit depending your OS (bcdedit for win7) to run on 1 CPU instead of SMP
bcdedit /set onecpu on
MmAllocateContiguousMemorySpecifyCache with MmCached
I suggest you to allocate 2 MB (you'll later support more CPUs, e.g. 64 CPUs will eat about 1 MB)
then setup VMCB according AMD manual and prepare vm exit handler also
after you succeed, I'll tell you how to run that on all CPUs in SMP (about 5 OS specific system calls)
certainly install something like windbg so you can analyze crash dump file (I got certainly more than 1000 reboots/BSODs/hangs during few years)
1 trick helped me a lot to know where I got (how far) - I put few instructions which turned on all LEDs on my keyboard (e.g. I had hang but know that something passed OK so the hang was later than the procedure turning on LEDs)
be patient enough (few months of studying manuals, programming, trying, rebooting), never resign
|17 Oct 2011, 12:14||
smoke 17 Oct 2011, 12:33
Thanks, I'll do that. Can I do all the initialization from DriverEntry? Or should I initialize the hypervisor from a dispatch routine?
|17 Oct 2011, 12:33||
Feryno 18 Oct 2011, 08:08
better from dispatch (because later you'll add turning off hypervisor on the running OS without necessity of OS restarting)
but no problem to do it from init
let the first 1MB of mem for shared things of hypervisor (paging tables, code, ...) and split the second 1 MB into 64 parts for 64 CPUs, every CPU will have 4 kB for HSAVE, 4kb for VMCB, 8 kB for stack (at first you setup only CPU00, later you implement SMP)
setup also GS base for hypervisor so you can access per CPU private data (analogy of ms win KPRCB) using GS prefix from hypervisor, I suggest to hold at least GDT, TSS, GS base (self pointer) there
the easiest vm exit loop is something like:
|18 Oct 2011, 08:08||
smoke 03 Nov 2011, 15:11
Hey Feryno, I'm sorry for the late reply but i was a bit busy with school so i couldnt work on the project as much as i'd hoped.
I have a question (maybe a stupid one). I'm kinda half way done with setting up the VMCB. When setting up the Save State area.. I can just use the values of segment registers, stack pointer, control registers, etc right?
|03 Nov 2011, 15:11||
Feryno 07 Nov 2011, 10:55
Hi, I was busy the whole last week also.
here sample how to setup something:
; rsi virtual memory
; rdi physical memory
; rbp CPU number * 4kB (every CPU uses 4 pages of 4 kB, 1 page for HSAVE, 1 page for VMCB, 2 pages for stack)
; N.B. big offsets like the 00301000 because hypervisor occupies 4 MB of memory, first 2 MB are used for translation tables for memory virtualization, next 1 MB for paging tables of hypervisor, its code, ..., the last 1 MB is used for 64 CPUs, every CPU occupies 16 kB (4 pages)
0000000000010A75: lea rax,[rdi+rbp*4+00301000] ; pointer to physical memory of the VMCB of the current CPU
0000000000010A7D: vmsave ; read manual for more, the rest will be saved manually as follows:
0000000000010A80: sgdt [rsp+06]
0000000000010A85: sidt [rsp+16]
0000000000010A8A: movzx r8d,word [rsp+06]
0000000000010A90: mov rdx,[rsp+08]
0000000000010A95: movzx eax,word [rsp+16]
0000000000010A9A: mov rcx,[rsp+18]
0000000000010A9F: mov [rsi+rbp*4+00301464],r8d
0000000000010AA7: mov [rsi+rbp*4+00301468],rdx
0000000000010AAF: mov [rsi+rbp*4+00301484],eax
0000000000010AB6: mov [rsi+rbp*4+00301488],rcx
0000000000010ABE: mov ecx,es
0000000000010AC0: mov [rsi+rbp*4+00301400],es
0000000000010AC7: lar eax,cx
0000000000010ACA: jnz 0000000000010AD9
0000000000010ACC: shr eax,08
0000000000010ACF: shr ah,04
0000000000010AD2: mov [rsi+rbp*4+00301402],eax
0000000000010AD9: lsl eax,cx
0000000000010ADC: jnz 0000000000010AF3
0000000000010ADE: mov [rsi+rbp*4+00301404],eax
0000000000010AE5: call 0000000000010F00
0000000000010AEA: jc 0000000000010AF3
0000000000010AEC: mov [rsi+rbp*4+00301408],eax
0000000000010F00: push rcx
0000000000010F01: push rdx
0000000000010F02: push rbx
0000000000010F03: lea ebx,[r8+01]
0000000000010F07: test cl,04
0000000000010F0A: jz 0000000000010F30
0000000000010F0C: sldt word eax
0000000000010F0F: lsl ebx,ax
0000000000010F13: jnz 0000000000010F4C
0000000000010F15: inc ebx
0000000000010F17: mov eax,[rdx+rcx*1+02]
0000000000010F1B: mov rdx,[rdx+rcx*1+04]
0000000000010F20: and eax,00FFFFFF
0000000000010F25: and rdx,FFFFFFFFFF000000
0000000000010F2C: or rdx,rax
0000000000010F30: and cl,F8
0000000000010F33: cmp ebx,ecx
0000000000010F35: jc 0000000000010F4C
0000000000010F37: mov eax,[rdx+rcx*1+02]
0000000000010F3B: mov ecx,[rdx+rcx*1+04]
0000000000010F3F: and eax,00FFFFFF
0000000000010F44: and ecx,FF000000
0000000000010F4A: or eax,ecx
0000000000010F4C: pop rbx
0000000000010F4D: pop rdx
0000000000010F4E: pop rcx
|07 Nov 2011, 10:55||
guignol 05 Jan 2012, 15:59
, but then I had to discontinue for various reasons.
|05 Jan 2012, 15:59||
|Goto page 1, 2 Next
< Last Thread | Next Thread >
Copyright © 1999-2020, Tomasz Grysztar. Also on GitHub, YouTube, Twitter.
Website powered by rwasa.