Hello, is anyone here wanting to develop hypervisors for AMD or Intel?
I do not know in which subsection of the forum start this thread. I found the most suitable subsection in OS development. If administrators decide to move it to better subforum please do it.
The development is closest to OS development. If you are able to write you own tiny x64 OS (look for Tomasz example 'Entering long mode' from FASM examples) you are very likely able to do hypervisor development as well.
Hypervisor or your skills could be used for these purposes: secretly watch OS, run few virtual machines on one CPU (only as a part of a team and using C programming language), research vulnerabilities of already used hypervisors.
I was doing hypervisors development for the past 15 years since AMD FX and Intel Core 2 Duo launched (core 2 duo was limited to start hypervisor only from ms win driver due to lack of real mode virtualization, core i had already features for real mode virtualization to launch hypervisor easily from BIOS and UEFI).
Hypervisor could be started from UEFI (UEFI runtime driver / UEFI boot executable / UEFI application from UEFI shell), BIOS (similarly as bootkit) or from running OS (e.g. ms win x64 driver) - I implemented all of these 3 ways.
If anyone interested, I will post at first some easy to understand examples for paging, how to write system structures (VMCB for AMD, VMCS for Intel), how a VM exit handler could look like, some examples to debug/trace in emulators (SimNow 4.6.2, BOCHS). You will need to study CPU manuals, expect something like 1000 pages for Intel and 500 pages for AMD, AMD manuals are more compact than Intel, but the amount of information is comparable).
Before even starting I would like to warn you that debugging could be a nightmare especially on hardware and best if you have a machines with serial ports connected with null modem serial cable. But the development and tests are usually done in emulators and only then you finally retest on real hardware (baremetal). If you are not very very very patient or you are unable to withstand weeks of debugging without seeing progress (trying to find a bug in your work) you should rather do something else to prevent your hair get gray or even losing your hair from head entirely.
There are some published C source code for hypervisors but most of them just copy/paste someone else' work without deep understanding. We will of course write hypervisors in FASM. To have some idea and estimation about its size, the simplest hypervisor written in FASM could be around 8 kB binary, maybe it could be pushed into 5 KB, but rather expect 8 kB.
If you would like to secretly watch/monitor ms windows OS, they added detections for hypervisors (KiErrata704Present, KiErrataSkx55Present, KiErrata671Present, KiErrata361Present) but the detections are harmless for properly written hypervisor. Also today ms windows OS enables hyper-v by default, then your hypervisor may pretend virtualization not present or disabled so hyper-v does not start or it is possible to add nesting (here you can expect your hypervisor size to grow to around 32 kB binary).
To start a hypervisor at all CPUs/cores/threads, I can explain APIC INIT-SIPI interprocessor interrupts (when started from BIOS) or multiprocessor UEFI calls (when started from UEFI), or ms win x64 interprocessor kernel calls (when started from ms win driver).
Suspend and resume from S3 also disables virtualization, but it can be hooked so hypervisor survives.
Simple hypervisor with static paging tables covering terra/petabytes of RAM running on 64 CPUs/cores/threads can consume upto 16 MB of RAM even its binary only 8 kB, every CPU core may consume upto 16-32 kB of RAM (stack + system structures), paging tables may consume slightly more than 4 MB for physical memory virtualization and slightly more than 4 MB for hypervisor virtual memory with identity map of the whole physical memory.
I won't help any newly registered member unless proven not to be malware creator.
I will help and cooperate with active forum members with long enough history of posts where it is clear that they are peaceful, helpful to other FASM forum members.

Unfortunately I don't have time for that. But I appreciate that you are willing to share your knowledge on this advanced topic.

Hello,Feryno
I appreciate your ideal. but is this possible? This should be an act of the state, not the act of certain individuals.

1, You should maintain a development team of at least 5000 people per year, including an office and salary. this will run for 20 years.
2, M$ is there, preemptively occupied this track. Many CPU manufacturers have always supported him. could you done better than him?

I would just like to share knowledge so more people will be educated and skilled.
Most of hypervisors are developed by a big team (hyper-v, KVM, VMware, virtualbox, xen, ...).
I'm doing hypervisor development completely alone in the past 15 years although there are more people in the team, they are developing other parts (GUI, AI, ...). My part is single-person task since beginning.
There are few guys on internet who share code written in C on github and similar platforms and these projects also seem to be one-man work, usually copy/paste an older work of someone else and then further improving (they always mention who was their ancestor).
I do not expect to create a team or to develop here some commercial tool based on hypervisor.
If few people start to develop their own hypervisor it would be great.
It's hard to start by only reading CPU manuals or by analyzing something huge like hyper-v (closed source, only binary available) or KVM (open source). Better if someone become skilled by debugging/tracing something tiny to fully understand it so later can develop something advanced or be able to analyze something bigger and eventually find there mistakes/bugs.
I do not expect this to become any commercial project. Better if everyone involved become skilled to play with hypervisors and be able to develop their own without necessity of copy someone' source code.
I also do not like to make new posts if there will be no one interested.
If there will be interest I will post some theory to understand paging, how to setup hypervisor structures, some small binaries to debug in emulators to understand what vm entry / vm exit does, how to start at every CPU/core/thread etc.
Only x64 platform (AMD and Intel), not ARM. Only FASM, not C.

The first question that comes to my mind: What does the hypervisor need? Memory manager, I/O support, etc ... We can't leave UEFI and create a hypervisor directly, or can we? Or you're suggesting starting a hypervisor in UEFI? What is the expected support, if any?

The other question would be simplification on modern hardware. We have UEFI, ECAM, etc. ... creating a modern OS is very simple code if we abandon all the legacy cruft. Is it the same with hypervisors?

Traffic is low and specialized here. Gauging interest is difficult.

(Most my hypervisor experience is through WHP in Windows - no concern for the low-level details is required, but it's really just a different abstraction. My current understanding is that a hypervisor cannot be created in user mode on Windows 11, but there are a number of Windows features that rely on hypervisors. This is why all the emulators in Windows are built on WHP.)

SVM support for fasmg/2: Edit: corrected. Still under testing.

Hi bitRAKE.
Mem manager - for simplest hypervisor it is unnecessary. During startup it is enough to create static paging tables and do not change them anymore. The pages could be constructed for identity mapping so hypervisor can access physical memory at the same linear/virtual memory address. For more advanced hypervisor (e.g. running more operating systems on the CPU) the hypervisor will require memory manager and adjusting paging tables on demand.
IO support is not essential or only minimal, hypervisor can let it be almost fully handled by operating system. Hypervisor may reserve e.g. serial port IO to send diagnostic data and then prevent OS to access these io ports.
I can post methods how to start hypervisor from UEFI, from BIOS, from running ms win x64 (as a driver). After the hypervisor starts the code execution returns back to UEFI/BIOS/running OS so the execution continues further (UEFI loads OS, BIOS loads MBR, running OS continues to run). From running OS the hypervisor can be started by a ring0 driver, never from only usermode.
We can discuss here how to create and start hypervisor for AMD and also for Intel. I can create some small hypervisor samples to debug in emulators (Bochs, SimNow 4.6.2) to better understand how they operate. I can fix your mistakes / errors if someone will develop their own hypervisor.
We can create here something like teaching course with lessons, where some basics will be explained with some trivial homework at the end to check whether you understood the topic and correct answers will be in the next post, you may ask if something not clear.
The AMD support you posted is correct, the simplest AMD hypervisor may use 4 instructions clgi, vmsave, vmload, vmrun during startup and then instruction vmmcall to trigger vm exit to check whether everything is working well (the vm exit handler will execute vmsave, vmload, vmrun).
On Intel the minimal instructions required are vmxon, vmclear, vmptrld, vmwrite, vmlaunch, then vmread, vmwrite, vmresume in exit handler and vmcall to trigger vm exit (but also cpuid is capable to cause vm exit).

First step to start - you need CPU manuals.

AMD64 Architecture
Programmer’s Manual
Volume 2:
System Programming
it is here:
https://docs.amd.com/v/u/en-US/24593_3.45_APM_Vol2_PUB

Intel® 64 and IA-32 Architectures
Software Developer’s Manual
Volume 3 (3A, 3B, 3C, & 3D):
System Programming Guide
it is here:
https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html
you can download the complete manual (Combined Volumes: 1, 2A, 2B, 2C, 2D, 3A, 3B, 3C, 3D, and 4) or only volume 3 (Combined Volumes 3A, 3B, 3C, and 3D)

I think the main problem here isn't how to launch the hypervisor (the structures are well known), but rather how to implement its core functionality. Let's say the hypervisor code has gained control—what should it do next? For instance, consider the virtualization of a network adapter or some other device.

Core i7 - virtualization of a network adapter or some other device is possible. Hypervisor can catch OS reads/writes of certain IO ports, intercept and create an IRQ assigned to the virtualized device.
Around 10 years ago I tried to emulate SMC device of Apple machine (IO ports 300h-31Fh) on Intel non-apple machines (hackintosh), not fully functional, just to pretend to be present to cheat installer to install Apple OS to a non-apple machine without SMC.
MS added support for last branch record and single step on branching instructions into kernel for older CPUs and later did not update the kernel for recent CPUs, I used hypervisor to pretend a recent CPU was an older model (suppored by ms kernel) so ms kernel enabled these debugging features for the new unsupported CPU (this was done be cheating some CPUID instructions by modifying their return values).
You can do various cool things to the OS.
We will start lessons from basics so even beginners understand.
First lessons will be about paging, how to create paging structures. Then lessons how to setup hypervisor structures (which are well known but beginners do not know). Then lessons how to construct vm exit handler. Then few lessons with some binaries available which you can single step in emulators to understand how everything works together, what causes vm exit and how does it behave, how to do vm entry (intel) / vmrun (amd). Then some lessons how to do something advances which you asked in discussion.

Paging.
This topic is split into 2 lessons.
In the first lesson only pointers of paging tables will be explained and in the second lesson the remaining bits of paging structures.

Abbreviation PA is used for Physical Address which is address somewhere in RAM. MBR is loaded by BIOS to PA 7C00h. Reset vector of BIOS is at PA FFFFFFF0h which is truncated in real mode to FFFF:0000 so the instruction 'jmp FFFF:0000' resets computer running in real mode. BIOS creates real mode IDT at PA in range 0000-03FFh.
When paging is enabled CR0.PG=1 (bit 31.) which also requires CR0.PE=1 (bit 0.) the memory reference instructions, e.g. 'cmp gs:[rcx],esi' , 'mov [rdx],al' do not access directly physical memory PA (RAM) but the memory is translated through paging tables. These paging tables translate linear / virtual memory address (gs_base + rcx in the first previous sample, rdx in the second previous sample) into physical memory address.
Abbreviation VA is used for Virtual Address which is equal to linear memory address in x64 as this architecture uses flat memory model. Correctly Linear Address should be used but everyone uses VA as they are equal.

The x64 paging uses pages of size 4096 bytes (4 KiB) aligned at 4 KiB boundary. Every page contains 512 entries of qwords size (8 bytes = 64 bits).
512 * 8 = 4096 (decimal) = 1000h (hexadecimal). Not all 512 entries have to be set, but at least one entry properly set in every level of paging tables is necessary. If only one entry in every level is properly set then the memory translation is capable to translate only 1 page, the more entries are set the more virtual memory VA could be translated into physical memory PA.
Default x64 paging uses 4 levels of paging tables.
All pointers in paging structures contain physical memory addresses PA (RAM addresses), these pointer are in bits 12 and higher bits of each qword. Bits 11-0 are used as properties of pages translation (e.g. readonly, writeable, uncacheable), sometimes highest bits are used as attributes too (e.g. bit 63. to mark page as not executable). To make things simpler and easier to understand (for beginners) we will ignore bits 11-0 in the first lesson and here we focus only on pointer part (bits 63-12). The low bits 11-0 will be explained in the next lesson.

CR3 register contains a pointer to first level of paging tables and the table has a name PML4. It is 4096 bytes table containing 512 entries, each entry of qword size = 8 bytes = 64 bits. The entries of PML4 are indexed from 0 to 511 so the first entry is PML4E[0], the second PML4E[1], the third PML4E[2] and the last PML4E[511].
Let at least one entry of PML4 is properly set so the PML4 entry with index 'a' PML4E[a] contains a pointer to the next level of paging tables. The next level of paging tables has a name PDP.

The PDP has size 4096 bytes, is aligned at 4096 bytes boundary and contains 512 entries, each entry of qword size = 8 bytes = 64 bits. The entries are indexed from 0 to 511, so the first entry is PDPE[0], the second PDPE[1]... the last PDPE[511].
Let at least one entry of PDP is properly set, PDP entry with index 'b' PDPE[ b] contains a pointer to the next level of paging tables. The next level of paging tables has a name PD.

The PD has size 4096 bytes, is aligned at 4096 bytes boundary, contains 512 entries, each entry of qword size = 8 bytes = 64 bits. Entries are indexed from 0 to 511, first entry PDE[0], the last entry PDE[511].
Let at least one entry of PD is properly set, PD entry with index 'c' PDE[c] contains a pointer to the next level of paging tables. The next level of paging tables has a name PT.

The PT has size 4096 bytes, is aligned at 4096 bytes boundary, contains 512 entries, each entry of qword size = 8 bytes = 64 bits. Entries are indexed from 0 to 511, first entry PTE[0], the last entry PTE[511].
Let at least one entry of PT is properly set, PT entry with index 'd' PTE[d] contains the final physical memory address PA (RAM address) where the virtual memory is translated so the PT is the last level of paging structures.

The indices a, b, c, d originate from VA and the formula is:

'shr x' means the same as FASM instruction shift to right by x bits = divide by 2^x
511 (decimal) = 111111111b (binary)
a = (VA shr 39) and 511
b = (VA shr 30) and 511
c = (VA shr 21) and 511
d = (VA shr 12) and 511

bits 47-39 of VA are index 'a'
bits 38-30 of VA are index 'b'
bits 29-21 of VA are index 'c'
bits 20-12 of VA are index 'd'
bits 63-48 of VA are sign extension of bit 47, so if bit 47 is zero then also bits 63-48 of VA are all zeros, if bit 47 is 1 then bits 63-48 of VA are all 1 (canonical address form)
bits 11-0 of VA are offset into PA (e.g. access of VA xyz62Ch is translated into accessing PA klm62Ch)

now some examples

you want to construct paging tables such that VA 0000h - 0FFFh (1 page of 4 KiB) is translated into physical memory address 0000h-0FFFh, this is an example of identity mapping where you constructed such paging tables where VA = PA so you can access a physical memory at the same virtual memory address
then:
a = (0 shr 39) and 511 = 0 and you need to properly set first entry of PML4 with index 0 which is PML4E[0] which is the first qword of the PML4
b = (0 shr 30) and 511 = 0 and you need to properly set first entry of PDP with index 0 which is PDPE[0] which is the first qword of the PDP
c = (0 shr 21) and 511 = 0 and you need to properly set first entry of PD with index 0 which is PDE[0] which is the first qword of the PD
d = (0 shr 12) and 511 = 0 and you need to properly set first entry of PT with index 0 which is PTE[0] which is the first qword of the PT and the entry will reference the physical memory (RAM) where is the memory access translated so the entry in bits 63-12 will contain the PA which is 0

you want to construct paging tables such that VA 1000h - 1FFFh (1 page of 4 KiB) is translated into physical memory address 1000h-1FFFh, this is an example of identity mapping where you constructed such paging tables where VA = PA so you can access physical memory at the same virtual memory address
then:
a = (1000h shr 39) and 511 = 0 and you need to properly set first entry of PML4 with index 0 which is PML4E[0] which is the first qword of the PML4
b = (1000h shr 30) and 511 = 0 and you need to properly set first entry of PDP with index 0 which is PDPE[0] which is the first qword of the PDP
c = (1000h shr 21) and 511 = 0 and you need to properly set first entry of PD with index 0 which is PDE[0] which is the first qword of the PD
d = (1000h shr 12) and 511 = 1 and you need to properly set second entry of PT with index 1 which is PTE[1] which is the second qword of the PT and the entry will reference the physical memory (RAM) = 1000h where is the memory access translated

you want to construct paging tables such that VA 200000h - 200FFFh (1 page of 4 KiB) is translated into physical memory address 200000h-200FFFh, this is an example of identity mapping where you constructed such paging tables where VA = PA so you can access physical memory at the same virtual memory address
then:
a = (200000h shr 39) and 511 = 0 and you need to properly set first entry of PML4 with index 0 which is PML4E[0] which is the first qword of the PML4
b = (200000h shr 30) and 511 = 0 and you need to properly set first entry of PDP with index 0 which is PDPE[0] which is the first qword of the PDP
c = (200000h shr 21) and 511 = 1 and you need to properly set second entry of PD with index 1 which is PDE[1] which is the second qword of the PD
d = (200000h shr 12) and 511 = 0 and you need to properly set first entry of PT with index 0 which is PTE[0] which is the first qword of the PT and the entry will reference the physical memory (RAM) = 200000h where is the memory access translated

you want to construct paging tables such that VA 201000h - 201FFFh (1 page of 4 KiB) is translated into physical memory address 201000h-201FFFh, this is an example of identity mapping where you constructed such paging tables where VA = PA so you can access physical memory at the same virtual memory address
then:
a = (201000h shr 39) and 511 = 0 and you need to properly set first entry of PML4 with index 0 which is PML4E[0] which is the first qword of the PML4
b = (201000h shr 30) and 511 = 0 and you need to properly set first entry of PDP with index 0 which is PDPE[0] which is the first qword of the PDP
c = (201000h shr 21) and 511 = 1 and you need to properly set second entry of PD with index 1 which is PDE[1] which is the second qword of the PD
d = (201000h shr 12) and 511 = 1 and you need to properly set second entry of PT with index 1 which is PTE[1] which is the second qword of the PT and the entry will reference the physical memory (RAM) = 201000h where is the memory access translated

you want to construct paging tables such that VA 40000000h - 40000FFFh (1 page of 4 KiB) is translated into physical memory address 40000000h-40000FFFh, this is an example of identity mapping where you constructed such paging tables where VA = PA so you can access physical memory at the same virtual memory address
then:
a = (40000000h shr 39) and 511 = 0 and you need to properly set first entry of PML4 with index 0 which is PML4E[0] which is the first qword of the PML4
b = (40000000h shr 30) and 511 = 1 and you need to properly set second entry of PDP with index 1 which is PDPE[1] which is the second qword of the PDP
c = (40000000h shr 21) and 511 = 0 and you need to properly set first entry of PD with index 0 which is PDE[0] which is the first qword of the PD
d = (40000000h shr 12) and 511 = 0 and you need to properly set first entry of PT with index 0 which is PTE[0] which is the first qword of the PT and the entry will reference the physical memory (RAM) = 40000000h where is the memory access translated

you want to construct paging tables such that VA 40001000h - 40001FFFh (1 page of 4 KiB) is translated into physical memory address 40001000h-40001FFFh, this is an example of identity mapping where you constructed such paging tables where VA = PA so you can access physical memory at the same virtual memory address
then:
a = (40001000h shr 39) and 511 = 0 and you need to properly set first entry of PML4 with index 0 which is PML4E[0] which is the first qword of the PML4
b = (40001000h shr 30) and 511 = 1 and you need to properly set second entry of PDP with index 1 which is PDPE[1] which is the second qword of the PDP
c = (40001000h shr 21) and 511 = 0 and you need to properly set first entry of PD with index 0 which is PDE[0] which is the first qword of the PD
d = (40001000h shr 12) and 511 = 1 and you need to properly set second entry of PT with index 1 which is PTE[1] which is the second qword of the PT and the entry will reference the physical memory (RAM) = 40001000h where is the memory access translated

you want to construct paging tables such that VA 40200000h - 40200FFFh (1 page of 4 KiB) is translated into physical memory address 40200000h-40200FFFh, this is an example of identity mapping where you constructed such paging tables where VA = PA so you can access physical memory at the same virtual memory address
then:
a = (40200000h shr 39) and 511 = 0 and you need to properly set first entry of PML4 with index 0 which is PML4E[0] which is the first qword of the PML4
b = (40200000h shr 30) and 511 = 1 and you need to properly set second entry of PDP with index 0 which is PDPE[1] which is the second qword of the PDP
c = (40200000h shr 21) and 511 = 1 and you need to properly set second entry of PD with index 1 which is PDE[1] which is the second qword of the PD
d = (40200000h shr 12) and 511 = 0 and you need to properly set first entry of PT with index 0 which is PTE[0] which is the first qword of the PT and the entry will reference the physical memory (RAM) = 40200000h where is the memory access translated

you want to construct paging tables such that VA 40201000h - 40201FFFh (1 page of 4 KiB) is translated into physical memory address 40201000h-40201FFFh, this is an example of identity mapping where you constructed such paging tables where VA = PA so you can access physical memory at the same virtual memory address
then:
a = (40201000h shr 39) and 511 = 0 and you need to properly set first entry of PML4 with index 0 which is PML4E[0] which is the first qword of the PML4
b = (40201000h shr 30) and 511 = 1 and you need to properly set second entry of PDP with index 0 which is PDPE[1] which is the second qword of the PDP
c = (40201000h shr 21) and 511 = 1 and you need to properly set second entry of PD with index 1 which is PDE[1] which is the second qword of the PD
d = (40201000h shr 12) and 511 = 1 and you need to properly set second of PT with index 1 which is PTE[1] which is the second qword of the PT and the entry will reference the physical memory (RAM) = 40201000h where is the memory access translated

you want to construct paging tables such that VA 40202000h - 40202FFFh (1 page of 4 KiB) is translated into physical memory address 40202000h-40202FFFh, this is an example of identity mapping where you constructed such paging tables where VA = PA so you can access physical memory at the same virtual memory address
then:
a = (40200000h shr 39) and 511 = 0 and you need to properly set first entry of PML4 with index 0 which is PML4E[0] which is the first qword of the PML4
b = (40200000h shr 30) and 511 = 1 and you need to properly set second entry of PDP with index 0 which is PDPE[1] which is the second qword of the PDP
c = (40200000h shr 21) and 511 = 1 and you need to properly set second entry of PD with index 1 which is PDE[1] which is the second qword of the PD
d = (40200000h shr 12) and 511 = 2 and you need to properly set third entry of PT with index 2 which is PTE[2] which is the third qword of the PT and the entry will reference the physical memory (RAM) = 40202000h where is the memory access translated

your tiny kernel written in FASM has size not more than 4 KiB (so it fits into 1 memory page), it is loaded in physical memory (RAM) address 9000h-9FFFh and you want to construct paging tables such that it is accessed from VA FFFF800000000000h - FFFF800000000FFFh (1 page of 4 KiB)
then:
a = (FFFF800000000000h shr 39) and 511 = 256 (decimal) = 100h (hexadecimal) and you need to properly set 257th entry of PML4 with index 256 which is PML4E[256] which is the 257th qword of the PML4
b = (FFFF800000000000h shr 30) and 511 = 0 and you need to properly set first entry of PDP with index 0 which is PDPE[0] which is the first qword of the PDP
c = (FFFF800000000000h shr 21) and 511 = 0 and you need to properly set first entry of PD with index 0 which is PDE[0] which is the first qword of the PD
d = (FFFF800000000000h shr 12) and 511 = 0 and you need to properly set first entry of PT with index 0 which is PTE[0] which is the first qword of the PT and the entry will reference the physical memory (RAM) = 9000h where is the memory access translated
for example CR3 contains value 2000h which is PA of PML4
the PML4 is thus located at PA=2000h and it looks like this:
PML4E[0] qword at PA 2000h = 0 (not used)
PML4E[1] qword at PA 2008h = 0 (not used)
...
PML4E[255] qword at PA 27F8h = 0 (not used)
PML4E[256] qword at PA 2800h = 3000h which is PA of next level of paging tables = address of PDP
PDPE[0] qword at PA 3000h = 4000h which is PA of next level of paging tables = address of PD
PDE[0] qword at PA 4000h = 5000h which is PA of next level of paging tables = address of PT
PTE[0] qword at PA = 9000h which is PA of the kernel loaded in physical memory (RAM) at 9000h

your kernel written in FASM has size not more than 8 KiB (so it fits into 2 memory pages), it is already loaded in physical memory (RAM) address 5000h-6FFFh, you want your kernel be located at VA FFFF800000000000h - FFFF800000001FFFh (2 pages of 4 KiB), you want to construct paging tables such that CR3 contains value 1000h so PML4E is located at PA=1000h-1FFFh, you want to create PDP at PA=2000h-2FFFh, PD at 3000-3FFFh, PT at 4000h-4FFFh
then:
a = (FFFF800000000000h shr 39) and 511 = 256 (decimal) = 100h (hexadecimal) and you need to properly set 257th entry of PML4 with index 256 which is PML4E[256] which is the 257th qword of the PML4
b = (FFFF800000000000h shr 30) and 511 = 0 and you need to properly set first entry of PDP with index 0 which is PDPE[0] which is the first qword of the PDP
c = (FFFF800000000000h shr 21) and 511 = 0 and you need to properly set first entry of PD with index 0 which is PDE[0] which is the first qword of the PD
d = (FFFF800000000000h shr 12) and 511 = 0 and you need to properly set first and second entry of PT with indices 0, 1 which are PTE[0], PTE[1] which are the first and second qwords of the PT and the entries will reference the physical memory (RAM) = 5000h (PTE[0]) and 6000h (PTE[1]) into which is the memory access translated
CR3 = 1000h which is PA of PML4
the PML4 is located at PA=1000h and it looks like this:
PML4E[0] qword at PA 1000h = 0
PML4E[1] qword at PA 1008h = 0
...
PML4E[255] qword at PA 17F8h = 0
PML4E[256] qword at PA 1800h = 2000h which is PA of next level of paging tables = address of PDP
PDPE[0] qword at PA 2000h = 3000h which is PA of next level of paging tables = address of PD
PDE[0] qword at PA 3000h = 4000h which is PA of next level of paging tables = address of PT
PTE[0] qword at PA 4000h = 5000h which is PA of the first page of the kernel loaded in physical memory (RAM) at 5000h
PTE[1] qword at PA 4008h = 6000h which is PA of the second page of the kernel loaded in physical memory (RAM) 2 pages of kernel are loaded in PA range 5000h-6FFFh so the second page starts from PA 6000h

your kernel fits into 12 KiB (3 memory pages), it is already loaded in physical memory (RAM) address 5000h-7FFFh, you want your kernel be located at VA FFFFFFFFFFFFD000h - FFFFFFFFFFFFFFFFh (3 pages of 4 KiB), you want to construct paging tables such that CR3 contains value 1000h so PML4E is located at PA=1000h-1FFFh, you want to create PDP at PA=2000h-2FFFh, PD at 3000-3FFFh, PT at 4000h-4FFFh
then:
a = (FFFFFFFFFFFFD000h shr 39) and 511 = 511 (decimal) = 1FFh (hexadecimal) and you need to properly set 512th entry of PML4 with index 511 which is PML4E[511] which is the 512th qword of the PML4E
b = (FFFFFFFFFFFFD000h shr 30) and 511 = 0 and you need to properly set first entry of PDP with index 0 which is PDPE[0] which is the second qword of the PDP
c = (FFFFFFFFFFFFD000h shr 21) and 511 = 0 and you need to properly set first entry of PD with index 0 which is PDE[0] which is the second qword of the PD
d = (FFFFFFFFFFFFD000h shr 12) and 511 = 509 and you need to properly set entries of PT with indices 509, 510, 511 which are PTE[509], PTE[510], PTE[511] which are the last 3 qwords of the PT and the entries will reference the physical memory (RAM) 5000h (PTE[509]), 6000h (PTE[510]), 7000h (PTE[511]) into which is the memory access translated
CR3 = 1000h which is PA of PML4
the PML4 is located at PA=1000h and it looks like this:
PML4E[0] qword at PA 1000h = 0 (not used)
PML4E[1] qword at PA 1008h = 0 (not used)
...
PML4E[510] qword at PA 1FF0h = 0 (not used)
PML4E[511] qword at PA 1FF8h = 2000h which is PA of next level of paging tables = address of PDP
PDPE[0] qword at PA 2000h = 0 (not used)
...
PDPE[510] qword at PA 2FF0h = 0 (not used)
PDPE[511] qword at PA 2FF8h = 3000h which is PA of next level of paging tables = address of PD
PDE[0] qword at PA 3000h = 0 (not used)
...
PDE[510] qword at 3FF0h = 0 (not used)
PDE[511] qword at 3FF8h = 4000h which is PA of next level of paging tables = address of PT
PTE[0] qword at 4000h = 0 (not used)
...
PTE[508] qword at 4FE0h = 0 (not used)
PTE[509] qword at PA 4FE8h = 5000h which is PA of the first page of the kernel loaded in physical memory (RAM) range 5000h-7FFFh
PTE[510] qword at PA 4FF0h = 6000h which is PA of the second page of the kernel loaded in physical memory (RAM) range 5000h-7FFFh
PTE[511] qword at PA 4FF8h = 7000h which is PA of the third page of the kernel loaded in physical memory (RAM) range 5000h-7FFFh


now we will play with live OS and explore its paging tables using different sight, we will perform paging tables walk from CR3 to the last level and find at which VA is APIC mapped


command 'r cr3' prints the value of CR3 register which contains pointer to PML4 which is located at PA 146000h
command 'dq /p' prints the value of qword located at some PA, 'L1' tells to print only 1 qword while 'L40' tells to print 40h = 64d qwords
command 'dq /p 0146000+FF8 L1' prints the last qword of PML4E located at PA=146000 + offset FF8 (FF8 hexadecimal / 8 = 511 decimal so it is PML4E[511]), it contains value 124063h, we will neglect for now lowest bits containing value 63h (they will be explained in the next lesson) and are insterested only in bits 63-12 which contain physical address of next level of paging tables, the value is 124000h, so the PDP is located at PA=124000h
command 'dq /p 124000+FF8 L1' prints the last qword of PDP located at PA=124000 + offset FF8 (FF8 hexadecimal / 8 = 511 decimal so it is PDPE[511]), it contains value 86063h, we will neglect for now lowest bits containing value 63h (they will be explained in the next lesson) and are insterested only in bits 63-12 which contain physical address of next level of paging tables, the value is 86000h, so the PD is located at PA=86000h
command 'dq /p 86000+FF8 L1' prints the last qword of PD located at PA=86000 + offset FF8 (FF8 hexadecimal / 8 = 511 decimal so it is PDE[511]), it contains value 1e163h, we will neglect for now lowest bits containing value 163h (they will be explained in the next lesson) and are insterested only in bits 63-12 which contain physical address of next level of paging tables, the value is 1e000h, so the PT is located at PA=1e000h
command 'dq /p 1e000+f00 L1' prints the 481th qword of PT located at PA=1e000 + offset F00 (F00 hexadecimal / 8 = 480 decimal so it is PTE[480]), it contains value 00000000`fee0017b, we will neglect lowest bits 11-0 containing value 17Bh (they will be explained in the next lesson) and are interested only in bits 63-12 which contain physical address with value 00000000fee00000 which is typical PA of APIC
command 'dq /p 00000000fee00000 L40' prints first 40h qwords of the APIC page
from the above it is possible to reconstruct VA where the APIC page is mapped:
a = 0FF8h / 8 = 1FFh (bits 47-39 of VA) = 111111111b (binary)
b = 0FF8h / 8 = 1FFh (bits 38-30 of VA) = 111111111b (binary)
c = 0FF8h / 8 = 1FFh (bits 29-21 of VA) = 111111111b (binary)
d = 0F00h / 8 = 1E0h (bits 20-12 of VA) = 111100000b (binary)
VA = (a shl 39) + (b shl 30) + (c shl 21) + (d shl 12)
VA = (111111111b shl 39) + (111111111b shl 30) + (111111111b shl 21) + (111100000b shl 12)
VA = 111111111111111111111111111111100000000000000000b
VA = FFFFFFFE0000h
but because highest bit 47. is set to 1, we must sign extend the bit 47. into bits 63-48 so all these bits are also 1 (canonical address form)
so the cannonical address of VA is:
VA = 1111111111111111111111111111111111111111111111100000000000000000b
VA = FFFFFFFFFFFE0000h
we decoded the VA where is APIC mapped so we can access it by the VA, the command is also 'dq' just without the '/p' switch, the '/p' switch means physical address and without the switch the dump is by virtual address
command 'dq FFFFFFFFFFFE0000 L40' prints first 40h qwords of the APIC page using its VA
as you can see, the output of dq commands using PA and VA is the same (well some of the values may meanwhile change as OS writes values into the APIC)

homework:
you already obtained physical memory address of APIC by this code:



your paging tables are constructed as such:
CR3 contains value 3000h so PML4 is located at PA 3000h
PDP is located at PA 4000h
PD is located at PA 5000h
PT is located at PA 6000h

you want to access APIC page (1 page of 4 kB which PA is now in RAX) at virtual address FFFF800000000000h - FFFF800000000FFFh

calculate the indices a, b, c, d
what value of pointer will be at qword PML4E[a] which is qword at PA [3000h + a*8]
what value of pointer will be at qword PDPE[ b] which is qword at PA [4000h + b*8]
what value of pointer will be at qword PDE[c] which is qword at PA [5000h + c*8]
what value of pointer will be at qword PTE[d] which is qword at PA [6000h + d*8]

answer the same above questions when you want to create paging tables such that APIC is accessed from VA FFFFFFFFFFE00000h - FFFFFFFFFFE00FFFh

answer the same above questions when you want to create paging tables such that APIC is accessed at VA FFFFFFFFFFFFF000h - FFFFFFFFFFFFFFFFh (highest possible last 4KiB page of virtual address space)

why not take existing hypervisor and modify it?
e.g. Cheat Engine hypervisor (windows-only) starts in ring0 and virtualizes the whole OS.

Feryno, allow me to express my opinion on this matter.
Hypervisors are certainly not a topic intended for beginners. Those who don't know what a byte, word, or qword is won't understand a thing you've said here, no matter how much they might want to. Therefore, I think you should address users who already have the necessary background knowledge and avoid getting sidetracked by basics. Otherwise, you end up with a lot of text but very little substance.

Now, regarding the homework assignment...

Incidentally, the Windows kernel employs a "Self-Reference Entry" technique within x64 page tables. Thanks to this trick, the entire page table hierarchy becomes accessible to the kernel as a contiguous array of virtual addresses.

This is achieved by having one of the entries in the PML4 (or PXE) point back to the PML4 table itself. Consequently, when accessing a virtual address that uses the same index across all four levels (PML4, PDPT, PD, PT), the processor traverses the hierarchy and lands back in the PML4. This allows the directory to reference itself, enabling access to the tables just like a standard array in memory.

In Win7_x64, this self-referential entry is located at index 0x1ED, corresponding to the virtual address 0xFFFFF6FB7DBED000. In newer versions like Win10/11, this index is selected randomly during system boot, in conjunction with KASLR randomization. This specific entry was the root cause of the CVE-2018-1038 vulnerability ("Total Meltdown"), where a patch erroneously permitted user-mode access to it.
https://www.bleepingcomputer.com/news/microsoft/meltdown-patch-opened-bigger-security-hole-on-windows-7/#1

I happen to be running Win7, and by using the `!cmkd` extension from the Codemachine group, I can generate a map like this: https://codemachine.com/downloads/cmkd.html

Core i7 - well done, correct results!

Ali.Z - we won't copy here someone else' work. We would like to be able to understand what we are doing and develop from complete beginning by ourselves. If you want to copy you can download source code from github or similar. Written in HLL. Nothing in FASM.

Now paging part 2 - things not included in the part 1 of paging (which was focused only on pointer part of paging table entries).

Another fact specific for hypervisors. Similarly as operating system uses paging tables to translate VA to PA, hypervisor usually uses another extra paging structures to virtualize physical memory to run more operating systems at one CPU (e.g. under VMware, QEMU, KVM, Xen you can run Linux + MS-DOS + win95 win NT + win XP + win 7 and so on). That way hypervisor creates an illusion of a physical memory exposed to every running OS. Every running OS will have its own physical memory (guest PA) starting from PA=0 and gradually increasing to top of guest physical memory (how much physical memory will be allocated to every OS running in the system). This mechanism is named NP (Nested Paging AMD) / EPT (Extended Page Table mechanism Intel). Hypervisors use NP / EPT to translate guest PA (guest thinks it is RAM and for guest it looks like real RAM) to system physical address (SPA = real RAM).
So finally there will be 2 paging structures used to do these 2 translations while guest is running:
guest VA to guest PA (using guest CR3 which points to guest PML4)
guest PA to system PA (using nested paging NP / EPT which points to different paging tables)

Hypervisor do not need to do anything with the first translation (by guest CR3), these paging tables are managed by guest = by running OS. An advanced hypervisor may want to lookup these translation tables e.g. to read/write some guest VA but that is an extended technique and it is not necessary for the simplest hypervisor, this advanced technique could be incorporated later after having some simple hypervisor running well and stable.
Hypervisor needs to setup the tables for translation of guest PA to system PA (or it is possible to not enable this translation and expose the whole system PA to the guest so guest PA = system PA but such hypervisor would be very limited and unprotected).
For NP (AMD) the paging structures and all bits for translation of guest PA to system PA are identical with paging structures for translation of guest VA to guest PA (guest CR3 points to the translation tables)
For EPT (Intel) the tables differ in control bits (pointer part = bits 12. and higher are luckily kept the similar principle without any difference).

Hypervisor also lives in its own virtual address space it needs to setup its own paging tables, when hypervisor is running (host mode) its VA is directly translated to system PA (using host CR3).
When guest is running (guest mode, OS is running), there are 2 translations active, first one translates guest VA to guest PA (using guest CR3) and the second translates guest PA to system PA (NP on AMD / EPT on Intel).

Transitions from guest (running OS) into host (hypervisor) is named VM exit. Transition from host (hypervisor) to guest (running OS) is named VM entry.
VM entry is done by executing the VMRUN instruction on AMD or by VMLAUNCH or VMRESUME on Intel (the first time it needs to be VMLAUNCH and subsequently could be done by VMRESUME, the VMRESUME is a little faster but for the first time it needs to be VMLAUNCH).
Examples which cause vm exit which is transition from guest (running OS) into host (hypervisor) could be instructions like CPUID, RDMSR/WRMSR, VMCALL (Intel) / VMMCALL (AMD) and much more. Some of them could be disabled/enabled. Every vm exit + vm entry consume some CPU time so hypervisor usually enables only as little of vm exit instructions as it requires. Our first simplest hypervisor will typically use only like 3-4 instructions for vm exits (e.g. instructions RDMSR, WRMSR, VMMCALL on AMD, instructions CPUID, RDMSR, WRMSR, VMCALL on Intel).

So now we will discus control bits identical in paging translation on AMD (guest VA to guest PA translation using guest CR3, guest PA to system PA translation using NP) and Intel (guest VA to guest PA translation using guest CR3) and then what is different (guest PA to guest VA on Intel using EPT). NP (AMD) and EPT (Intel) translation is active when enabled (a hypervisor typically needs it to be enabled although it could be also disabled) and guest is running = in guest mode = after VM entry.
The translation using hypervisor CR3 behaves as standard translation (as if you are writing your own OS). This translation is active when hypervisor is running = in host mode (guest execution is temporarily suspended) = after VM exit. Hypervisor VA is then directly translated to system PA (RAM).

Bit 0. of paging table entry is Present bit. If this bit is set to 1 the entry is valid and its pointer part is used for the translation. If this bit is 0 the entry is not valid and is not used for pages translation. You need this bit to be set to 1 for pages translation to work.

Bit 1. pf paging table entry is Writeable bit. If it is set to 1 the translation allows writing into the VA. If it is 0 the translation is only for read only.

Bit 2. is User/Supervisor. If 0 only supervisor can access the page. If 1 everything is allowed to access the page. For AMD NP you need to enable this bit. For Intel the EPT uses different schema which will be discussed later.

Bits 3., 4., bit 7. or 12. select PAT (in the last level of paging tables: in 4-level 4 KiB it is bit 7. of PTE, in 3-level 2 MiB it is bit 12. of PDE, in 2-level 1 GiB it is bit 12. of PDPE) e.g. in my previous post and the sample from running windows server 2003 the last level entry was PTE and contained value 00000000`fee0017b where bit 7. was 0, bit 4. was 1, bit 3. was 1 so in binary form it was 011b = 3h which selects uncacheable memory type by PA3 (bits 26-24 of MSR PAT). Writing into APIC must be performed instantly and directly into hardware and not temporarily into memory caches like L1, L2, L3 CPU caches.
Guest PA to system PA translation where is allocated RAM for guest is on AMD done using PA0 (lowest bits of MSR PAT) which is writeback (the best performance for memory) so then all bits 7, 4, 3 are all 0.
Translation where are gaps in RAM (usually holes in RAM like 3 GiB to 4 GiB) are mapped uncacheable (there is e.g. APIC, BIOS shadowed to RAM and so on).
The memory where is graphics linear framebuffer (LFB) should be mapped WC (write combining)
After reset the default value of MSR_PAT_0277h = 0007040600070406h (PA7=00h, PA6=07h, PA5=04h, PA4=06h, PA3=00h, PA2=07h, PA1=04h, PA0=06h)
memory types:
Uncacheable (UC) 00H
Write Combining (WC) 01H
Write-through (WT) 04H
Write-protected (WP) 05H
Writeback (WB) 06H
Uncached (UC-) 07h
the PA0-PA7 from the PAT is selected using PAT, PCD, PWT bits of page translation entry from the last level of page translation:


Bit 5. (A, Accessed) and 6. (D, Dirty) are set when there was already memory access (bit 5) or memory write (bit 6). When only read the bit 5. is set to 1, when write both bits 5. and 6. are set to 1). If a hypervisor initially sets all paging entries bits A, D to 0, it can trace which memory pages were written or read. This slows down execution a little because CPU has to change these bits from 0 to 1 after access/write. If you want to make a little speedup favor for a hypervisor and do not need to watch which pages were read/written, setting these both bits A, D to 1 makes some tiny speedup advantage as CPU does not need to modify these bits from 0 to 1. CPU may also execute instructions speculatively so some of the bits may be flipped from 0 to 1 by a CPU even there was no memory access e.g.

in the above sample the CPU may touch the memory referenced by RDX even in case eax=0, it may speculatively execute the instruction and be prepared for the case eax is not 0, the CPU may even touch the memory referenced by RDX few instruction prior executing the test eax,eax instruction
Inspecting A, D bits is never 100% proof that the memory was really accessed or written.

Bit 8. (G, Global) you do not need to enable this bit, it is used by OS, because when OS executes mov to CR3 instruction that instruction execution erases from TLB (Translation Lookaside Buffer) all translations which are not global and keeps in TLB global translations.

Bit 63. no execute bit - you typically do not need this bit and let i 0, OS uses it for making pages not executable to prevent some malware from execution

Bit 7. PS (page size) in paging tables is used to select to bigger pages - typical translations uses 4 levels (PML4, PDP, PD, PT) which maps 4 KiB page, enabling this bit changes paging from 4-level 4 KiB to 3-level 2 MiB (bit 7. of PDE is enabled) or even 2-level 1 GiB translation (bit 7. of PDPE is enabled). Using bigger pages you can save memory space and map more memory using less paging tables which also a little speeds up whole system performance.
4-level 4 KiB translation is present at every x64 CPU, 3-level 2 MiB paging is also present since first x64 CPU generation but you should consult CPUID instruction whether 3-level 2 MiB paging is present. 2-level 1 GiB paging is not present in the first generation of x64 AMD CPUs (Athlon), you need to consult CPUID instruction whether it is present. IIRC 2-level 1 GiB paging is already present since Bulldozer CPUs like FX-4100. Anyway in any case you need to consult CPUID instruction whether the feature is present.
When 3-level 2 MiB paging is present you can enable bit 7. (PS bit) in PD entry, so then the PD is the last level of paging and points into final physical memory (it does not point to PT).
When 2-level 1 GiB paging is present you can enable bit 7, (PS bit) in PDP entry, so then the PDP is the last level of paging and points into final physical memory (it does not point to PD). Enabling this bit when CPUID does not report that this feature is present will lead into CPU exception (or vm exit by violation of NP/EPT)

here the CPUID detection:


Now about Intel EPT.
Here the bits in paging structures differ from standard paging. Currently everything is described in Intel CPU manual in these chapters (the number will change in future as CPU manuals will grow up)
30.3 The Extended Page Table Mechanism (EPT).
26.6.11 Extended-Page-Table Pointer (EPTP)

For EPTP pointer (which is an analogy of CR3 used by operating system) set bits 2-0 to 6 = 110b allow writeback memory (highest performance) - if you set them to 0 the guest will run but extremely slowly (like 100 times slower) in uncacheable memory. Bits 5-3 must be set to the EPT page table walk length minus one (that's value 3 for 4-level paging or value 4 for 5-level paging)
e.g. RAX holds the value of EPTP (pointer part = bits 12. and higher) and you want to construct EPT using 4-level 4-KiB page size (4-1) and writeback memory type (6)

note you can still use 3-level 2 MiB and 2-level 1 GiB translation by setting corresponding bit 7. (PS bit) in PD and PDP

bit 0. read
bit 1. write
bit 2. execute (this bit differs from standard paging and differs from NP used by AMD)
you typically need to enable all these bits for the simplest hypervisor

bits 5-3 memory type, e.g. Writeback=6 for most of guest physical memory, then some guest physical memory ranges uncacheable, write combining etc
0 = UC; 1 = WC; 4 = WT; 5 = WP; and 6 = WB
Intel is very sensitive for this field and it should be not messed with MTRR settings, read all MTRR registers (MSR) and set bits 5-3 appropriately. I got Machine check exceptions (#MC) when did not care and set all NP entries writeback, the #MC occurred when guest used realmode int 10h to write text to screen, #MC was triggered from inside of int 10h handler from VGA BIOS somewhere behind C000:0000)

There are much more bits in EPT translation but they are not required in simplest hypervisor. You can add them later after having some basic simple hypervisor running and stable.

Most recent CPUs even support 5-level paging where highest level table is PML5, an entry from PML5 (which has name PML5E and there are 512 entries from PML5E[0] to PML5E[1FFh]) points into PML4. You need to consult CPUID whether it is present and then enable corresponding bit in CR4 (CR4.LA57, bit 12.)

The simplest hypervisor runs only 1 guest where operating system is running (e.g. Linux, ms win etc.), creates NP / EPT with identity map (so it exposes whole RAM to guest and is constructed in a way that guest PA = system PA), it just hides its own SPA where hypervisor body is present). The mapping and translation of guest PA to system PA is by this paging size and principle:
mem 0-2 MB using 4-level 4 KiB pages (and consulting Fixed-Range MTRR to setup caching type correctly)
mem 2 MB and above using 3-level 2 MiB paging
mem above 512 GB using 2-level 1 GiB paging
the pages where is real RAM should be writeback
where hardware memory type should be uncacheable = usually memory in range like 3-4 GiB, LFB video buffer WC (write combining) - these types need to be obtained by reading corresponding variable range MTRRs, on AMD you can also consult these 2 MSRs:
MSR_AMD64_TOP_MEM = 0C001001Ah
MSR_AMD64_TOP_MEM2 = 0C001001Dh
the memory on AMD between 0-TOP_MEM is read RAM and should be writeback
range from TOP_MEM to 4 GiB is not RAM (should not be writeback and should be obtained from variable range MTRRs
the memory on AMD above TOP_MEM2 is not RAM and should not be writeback (consult MTRRs or if not found in MTRRs set it as uncacheable, some hardware devices may still use this memory range like graphic cards)

In the next lesson we will discuss how to prepare, read, write hypervisor structures: VMCB on AMD and VMCS on Intel.

Homework
Near the end of my previous post there is a code to obtain physical address of APIC, the address is in RAX register, this is a PA of the APIC.
You want to map the APIC using NP (AMD) / EPT (Intel) and thus expose the APIC (system PA) to guest (guest PA) so guest is able read/write the APIC. APIC is a hardware feature, it is not in RAM so you need to map it uncacheable (memory existing in RAM needs to be mapped with fastest performance which is writeback type = 6 but that's not the case of APIC).
The PA of APIC is already in the RAX register and you are going to write the value from RAX into PT table into the corresponding entry (as Core i7 calculated where to write it). Before writing the PTE (PT entry) you need to setup some bits in the RAX:
- enable bit 0. so the entry translation will be present thus also readable (NP on AMD Present bit, EPT on Intel Readable bit)
- enable bit 1. (W = writeable) so the memory translation will allow writing into the memory
- for AMD enable bit 2. so the entry will be accessible from not only supervisor (hypervisor, ring -1) but also from less privileged level (guest OS, ring 0), for Intel the bit 2. is executable and you do not need to enable it, on Intel set it to zero because the memory of APIC is never executed, it is only read or written
- the value of MSR PAT is now 0007040600070406h so uncacheable memory type (0) is at position of PA3 (bits 26-24 of PAT, or as well bits 56-59 = PA7 could be used as they are also 0) so you need to set the PT entry to use PA3, so 3 in binary form is 011b so the PTE bit 7. has to be 0, bit 4. has to be 1, bit 3. has to be 1
(erase RAX bit 7., set bit 4. of RAX to one, set bit 3. of RAX to one)

Write the FASM instructions with the RAX register to do the necessary operations with these bits 7., 4., 3., 2., 1., 0. for AMD using NP Nested Paging and for Intel using EPT Extended Page Translation.

Write the FASM instructions with the RAX register for AMD platform using NP Nested paging when you want to do the same as above plus you want to enable also the bits 6. (D, dirty = already written) and 5. (A, accessed = already read or written).