flat assembler
Message board for the users of flat assembler.

Index > OS Construction > Educational "CodebyOS" for the collection

Author
Thread Post new topic Reply to topic
Core i7



Joined: 14 Nov 2024
Posts: 133
Location: Socket on motherboard
Core i7 27 Jun 2025, 08:48
Hello everyone!
I write an OS in my free time, and decided to share the result here.
The capabilities of BIOS are used (not uefi), direct output to video memory 0xB800 in 80x25/16 mode, menu selection with keys 1-7. In principle, nothing new - the bootloader reads the kernel at address 0x0600, leaving the stack in the same place 0x7C00.

In addition to the legacy ATA interface, there is support for modern SATA in AHCI mode. Later, I plan to add initialization of the NVM controller on the PCI bus, as well as NVMe on the PCI-Express bus. The software model of AHCI, NVM, NVMe is different, so I will have to implement three different modules. Since their ports are mapped to MMIO memory, the real mode kernel code has a switch to "unreal" mode, with access to 4GB of memory through the FS register.

At the moment I am working on the x64 entry with the "Long" memory model - it is not finished yet, so the menu item (6) is empty for now. The archive contains the entire source code, as well as a 1.44 MB floppy image of "CodebyOS". To conduct tests on real hardware, you can manually create a bootable USB-Flash, and in the BOOT folder there is the source code of the USB-loader. Here is the result.


Description:
Filesize: 16.4 KB
Viewed: 1076 Time(s)

Example.png


Description:
Download
Filename: CodebyOS_v3.zip
Filesize: 48.23 KB
Downloaded: 47 Time(s)

Post 27 Jun 2025, 08:48
View user's profile Send private message Reply with quote
Core i7



Joined: 14 Nov 2024
Posts: 133
Location: Socket on motherboard
Core i7 02 Jul 2025, 12:58
I have a question about the "PageTable" structure in x64?
I don't understand how many PML4.PDPT.PDT.PT directories need to be created in paging? It is known that in a 48-bit linear address, 9 bits are allocated for each of them, which means a maximum of 512 entries in each directory.

If PML4 exists in a single copy, then there can be 512 PDPT directories, then 512 * 512 = 262,144 PDT directories, and the last PT directories are 262,144 * 512 = 134,217,728, and each also has 512 entries, which in total gives 68,719,476,736 pointers to physical PFN frames with a size of 4 KB. In total, we get 256 TeraBytes of available virtual memory in x64 mode.

If we take into account that each "PageTableEntry" entry is 8 bytes in size, then a fully filled table requires 68719476736*8=512 GB of memory - this fact is confirmed by the WinDBG debugger extension !cmkd:

Code:
0: kd> !cmkd.kvas    ;<------ Kernel VA Space

###  Start             End                                Length   Type
000  ffff080000000000  fffff67fffffffff    ee8000000000 ( 238 TB)  SystemSpace
001  fffff68000000000  fffff6ffffffffff      8000000000 ( 512 GB)  PageTables     ;<---------//
002  fffff70000000000  fffff77fffffffff      8000000000 ( 512 GB)  HyperSpace
003  fffff78000000000  fffff78000000fff            1000 (   4 KB)  SharedSystemPage
004  fffff78000001000  fffff7ffffffffff      7ffffff000 ( 511 GB)  CacheWorkingSet
005  fffff80000000000  fffff87fffffffff      8000000000 ( 512 GB)  LoaderMappings
006  fffff88000000000  fffff89fffffffff      2000000000 ( 128 GB)  SystemPTEs
007  fffff8a000000000  fffff8bfffffffff      2000000000 ( 128 GB)  PagedPool
008  fffff90000000000  fffff97fffffffff      8000000000 ( 512 GB)  SessionSpace
009  fffff98000000000  fffffa7fffffffff     10000000000 (   1 TB)  DynamicKernelVa
010  fffffa8000000000  fffffa80038fffff         3900000 (  57 MB)  PfnDatabase
011  fffffa8003800000  fffffa80c01fffff        bca00000 (   2 GB)  NonPagedPool
012  ffffffffffc00000  ffffffffffffffff          400000 (   4 MB)  HalReserved

0: kd>    


The calculations described above are valid when the physical DRAM memory is 256 TB, but in fact we have much less in the range of 4-128 GB. Here I am confused.

Now, in the QEMU settings I have 1 GB, and to get the number of directories at each level, I first find the total number of frames PFN = 1073741824 / 4096 = 262.144, after which I do a PFN/512 cycle until the remainder is zero. At the final stage, to get the number of directories from the number of "Entry" records, it is enough to shift all the values down, i.e. for 128 MB we get: PML4=0,PDPT=0,PDT=0,PT=64, and for 1024 MB: PML4=0,PDPT=0,PDT=1,PT=512.

There are no problems in such a scheme, as long as the virtual address matches the physical one. And then the CPU will generate a "PageFault" exception, write the virtual address it accessed to the CR2 register, and call the int-0Eh interrupt (the vector can be changed via APIC).

But what should the OS memory manager do at this point? Do I need to dynamically create an additional directory PML4-PDPT/PDT for the missing address, or does the directory structure remain the same, and only the free entry in PFN that I have already registered is modified? I read "Intel-SDM" and other pages on the Internet, but I still have questions.


Description:
Filesize: 2.09 KB
Viewed: 930 Time(s)

PFNs.png


Post 02 Jul 2025, 12:58
View user's profile Send private message Reply with quote
macomics



Joined: 26 Jan 2021
Posts: 1180
Location: Russia
macomics 02 Jul 2025, 23:59
Virtual addressing does not have to project all the physical memory. Moreover, you can repeatedly project one physical memory page to different virtual addresses. Moreover, there are page absence flags at each of the projection levels, which suggests a branch break in the virtual memory map tree.

bot-0 = Present

If you need it, then the minimum virtual address space will be absolutely necessary: 32 kb = 4kb (PLM4) + 4kb (PDPTE) + 4kb (PDE) + 4 kb (PTB) + 4kb (GDT) + 4kb (IDT) + 4kb (code) + 4kb (stack+data). That is, the approach is as follows. First, you need to split the memory ranges into pages, and then you can use these pages as needed and place them in the virtual memory map of the process.
Post 02 Jul 2025, 23:59
View user's profile Send private message Reply with quote
Core i7



Joined: 14 Nov 2024
Posts: 133
Location: Socket on motherboard
Core i7 03 Jul 2025, 03:29
macomics wrote:
Virtual addressing does not have to project all the physical memory.

This means that the directories are created dynamically, as the number of requested virtual addresses increases, and their order is arbitrary, not linear. Thank you, I understand.
Code:
    PML4  PDPT  PDT  PT
     0     0     0    0  -->  PFN #0
           5    210   1  -->  PFN #293
           1     2   18  -->  PFN #5
 etc..
    

And apparently this is only the beginning, because further there is a separate memory of the state of physical frames "PfnDatabase" (free, busy, prototype, etc.), and for each process there is also a list of pages allocated to it VAD - Virt Addr Descriptor. In general, the mechanism of supporting virtual memory is very complex, and good logic is needed for fast search in large arrays of data for the required values.
Post 03 Jul 2025, 03:29
View user's profile Send private message Reply with quote
Core i7



Joined: 14 Nov 2024
Posts: 133
Location: Socket on motherboard
Core i7 04 Jul 2025, 04:05
No, I still don't understand something.
Let's say we have 1000 frames of physical memory PFN, and each is 4 KB.
In order to display all the virtual memory x64 = 256 TB in them, I have to save 1 PFN to disk (for example PFN #50), and rewrite it with new data. This can be repeated infinitely. But if later the CPU requests a virtual address that pointed to the original PFN=50, I will have to restore it from disk. This is impossible to do without remembering all the virtual addresses that the CPU accesses. This is exactly why the PML4..PT directories were invented. But then the size of these directories will grow exponentially up to 512 GB. I wonder how this problem is solved?

I have already looked at all the virtual memory support structures in Windows - mainly _MMPFN in "PfnDatabase", and _MMPTE in directory entries. But these structures also do not have a field with a virtual address. Apparently there is some little thing here that I cannot understand.

The global PFN database stores the state of each frame, which is described by such a _MMPFN structure of 48 bytes (30h). That is, for every 4K of physical memory we have 48 bytes of service information:
Code:
0: kd> dt _MmPfn -r1
nt!_MMPFN
   +0x000 u1                 : <unnamed-tag>
      +0x000 Flink              : Uint8B
      +0x000 WsIndex            : Uint4B
      +0x000 Event              : Ptr64 _KEVENT
      +0x000 Next               : Ptr64 Void
      +0x000 VolatileNext       : Ptr64 Void
      +0x000 KernelStackOwner   : Ptr64 _KTHREAD
      +0x000 NextStackPfn       : _SINGLE_LIST_ENTRY

   +0x008 u2                 : <unnamed-tag>
      +0x000 Blink              : Uint8B
      +0x000 ImageProtoPte      : Ptr64 _MMPTE
      +0x000 ShareCount         : Uint8B

   +0x010 PteAddress         : Ptr64 _MMPTE
   +0x010 VolatilePteAddress : Ptr64 Void
   +0x010 Lock               : Int4B
   +0x010 PteLong            : Uint8B

   +0x018 u3                 : <unnamed-tag>
      +0x000 ReferenceCount     : Uint2B
      +0x002 e1                 : _MMPFNENTRY
      +0x000 e2                 : <unnamed-tag>

   +0x01c UsedPageTableEntry : Uint2B
   +0x01e VaType             : UChar
   +0x01f ViewCount          : UChar
   +0x020 OriginalPte        : _MMPTE

   +0x028 u4                 : <unnamed-tag>
      +0x000 PteFrame           : Pos 0, 52 Bits
      +0x000 Unused             : Pos 52, 3 Bits
      +0x000 PfnImageVerified   : Pos 55, 1 Bit
      +0x000 AweAllocation      : Pos 56, 1 Bit
      +0x000 PrototypePte       : Pos 57, 1 Bit
      +0x000 PageColor          : Pos 58, 6 Bits
    

And this is the _MMPTE structure, which describes one entry in the PML4..PT directories. There are 7 types of frames, and the ones of interest now are "Hardware" with the zero bit (V) set, and "Software" for those unloaded to the swap file on the disk. As we can see, there are no virtual addresses for matching with PFN frames, which means the addresses are stored in the "PageTable" directories.
Code:
0: kd> dt _MmPte -r1
nt!_MMPTE
   +0x000 u                : <unnamed-tag>
      +0x000 VolatileLong     : Uint8B
      +0x000 Hard             : _MMPTE_HARDWARE      ;<--- valid frame in memory
      +0x000 Proto            : _MMPTE_PROTOTYPE     ;<--- shared frame for 2+ processes
      +0x000 Soft             : _MMPTE_SOFTWARE      ;<--- saved to disk in Pagefile.sys
      +0x000 TimeStamp        : _MMPTE_TIMESTAMP 
      +0x000 Trans            : _MMPTE_TRANSITION
      +0x000 Subsect          : _MMPTE_SUBSECTION
      +0x000 List             : _MMPTE_LIST

0: kd> dt _MMPTE_HARDWARE
nt!_MMPTE_HARDWARE
   +0x000 Valid            : Pos 0,  1  Bit
   +0x000 Dirty1           : Pos 1,  1  Bit
   +0x000 Owner            : Pos 2,  1  Bit
   +0x000 WriteThrough     : Pos 3,  1  Bit
   +0x000 CacheDisable     : Pos 4,  1  Bit
   +0x000 Accessed         : Pos 5,  1  Bit
   +0x000 Dirty            : Pos 6,  1  Bit
   +0x000 LargePage        : Pos 7,  1  Bit
   +0x000 Global           : Pos 8,  1  Bit
   +0x000 CopyOnWrite      : Pos 9,  1  Bit
   +0x000 Unused           : Pos 10, 1  Bit
   +0x000 Write            : Pos 11, 1  Bit
   +0x000 PageFrameNumber  : Pos 12, 36 Bits
   +0x000 reserved1        : Pos 48, 4  Bits
   +0x000 SoftwareWsIndex  : Pos 52, 11 Bits
   +0x000 NoExecute        : Pos 63, 1  Bit

0: kd> dt _MMPTE_SOFTWARE
nt!_MMPTE_SOFTWARE
   +0x000 Valid            : Pos 0,  1  Bit
   +0x000 Unused           : Pos 1,  2  Bits
   +0x000 InStore          : Pos 3,  1  Bit
   +0x000 SwizzleBit       : Pos 4,  1  Bit
   +0x000 Protection       : Pos 5,  5  Bits
   +0x000 Prototype        : Pos 10, 1  Bit
   +0x000 Transition       : Pos 11, 1  Bit
   +0x000 PageFileLow      : Pos 12, 4  Bits
   +0x000 UsedPTableEntry  : Pos 16, 10 Bits
   +0x000 Reserved         : Pos 26, 6  Bits
   +0x000 PageFileHigh     : Pos 32, 32 Bits

0: kd>
    
Post 04 Jul 2025, 04:05
View user's profile Send private message Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 4260
Location: vpcmpistri
bitRAKE 04 Jul 2025, 04:58
A fully populated set of page tables for a 256 TB address space would indeed be enormous. The solution is that page tables are created on demand.
  • When a process is created, it does not get 512 GB of page tables. It gets a single PML4 table.
  • Most entries in that PML4 are empty. Only when the program tries to access a virtual address in a region that doesn't have a page table will the OS allocate one.
  • For example, if you access an address that requires the 5th entry in the PML4, the OS will see that this entry is empty, allocate a new page for a PDPT, and update the 5th PML4 entry to point to it. This continues down the hierarchy.
  • Large, unused gaps in the 256 TB virtual address space will have no corresponding page tables allocated at all, consuming zero memory. This makes the system efficient.


You have pinpointed the exact source of your confusion, and you are very close to the answer. The critical piece you're missing is that the mapping is not searched for; it is calculated. The virtual address itself is the key that directly leads to the location of its data, whether that data is in RAM or on disk.

The system does not need to store the virtual address within the _MMPFN or on disk because the entire page table structure is designed to use the virtual address as an index.


You are correct that the _MMPFN does not contain a virtual address. Its purpose is the reverse mapping. For every physical frame (PFN), the _MMPFN entry tells the OS about the state of that physical frame. A crucial piece of information it contains is a pointer back to the Page Table Entry (_MMPTE) that maps this frame.

This is essential when the OS decides to page out the data in a particular frame (e.g., PFN #50). The OS needs to do two things:
  1. Write the data from PFN #50 to the disk.
  2. Find the _MMPTE that currently points to PFN #50 and invalidate it.
The OS uses the PFN database to solve step 2. It looks up entry #50 in the PfnDatabase, finds the pointer to the corresponding _MMPTE, and modifies that _MMPTE to mark it as invalid and store the new swap file location.

(Perhaps the supporting code in the Windows Research Kernel would be helpful.)
Post 04 Jul 2025, 04:58
View user's profile Send private message Visit poster's website Reply with quote
macomics



Joined: 26 Jan 2021
Posts: 1180
Location: Russia
macomics 04 Jul 2025, 07:57
Core i7 wrote:
And this is the _MMPTE structure, which describes one entry in the PML4..PT directories. There are 7 types of frames, and the ones of interest now are "Hardware" with the zero bit (V) set, and "Software" for those unloaded to the swap file on the disk. As we can see, there are no virtual addresses for matching with PFN frames, which means the addresses are stored in the "PageTable" directories.

The virtual address itself does not need to be stored. All you need is information about the use of a page that has already been used in the virtual space (bit-5 Access) and information about its position (_MMPTE). When switching virtual address spaces, the OS resets the access bit in all entries at all levels of the map (in the map that will be used now, but in the old one they remain unchanged). This way, she learns about the pages that the tasks really needed during their execution. This means that the rest can be painlessly download from memory because they will be used less often than the first ones. If a memory page is uploaded to disk, then you need to record information about the map to which it belonged (CR3) and its position in this map. You don't need the rest of the information.

ADD: I do not exclude that the OS can calculate the frequency of page usage, and not just rely on the availability of an access bit. That is, enter your own counter for each page and increase it when she get a=1.


Description:
Filesize: 151.91 KB
Viewed: 781 Time(s)

Снимок экрана_20250704_114838.png


Description:
Filesize: 56.78 KB
Viewed: 781 Time(s)

Снимок экрана_20250704_114645.png


Post 04 Jul 2025, 07:57
View user's profile Send private message Reply with quote
macomics



Joined: 26 Jan 2021
Posts: 1180
Location: Russia
macomics 04 Jul 2025, 08:45
A more mundane example. Your task is to load your own library into RAM. To locate the entire image of this library, physical memory pages were found and used, and then arranged in a linear block of virtual addresses through the location of pointers to nothing in the corresponding map. If possible, the block of virtual addresses for the library is selected according to its PE.ImageBase, so that relocation does not have to be performed (but I digress). After ensuring that the entire memory range is available in the virtual space for loading the image, it is read from disk into this memory. I made a reservation about his own library for a reason, because it will not be divided between different memory maps (for simplicity) it will be present in the memory map only of the current process. Then, when this process is running, a=1 bits will occur in this memory range only in those places that are actually used by the code. In addition, let say the library occupies 10 MB of memory (5*PDE=3*PDE header and code + 2*PDE data and resources). That is, the OS can create 5*PDE and fill in 5*512 PTEs for them, or use 5*PDE with 2Mb pages (I got distracted again). There are few accesses to the file header while using the DLL, and it will often be a=0. But not all of the DLL code will work either too. Only those pages that actually running. The situation with data is slightly different, but with resources it is exactly the same as with the header and code. All these pages use read-only data from an image (a file on disk). That is, it is pointless to download it to a pagefile. All this data can be re-read from the library image from disk. But here is the data (data and bss section) just what the library recorded, and it's not in the image. Only 2*PDEs with data are subject to paging.

That is, the header, the part of the code that is not executed, and the resources are simply unloaded from the physical memory pages and it is released. All this information is simply re-read from the corresponding DLL file from the disk as needed. But the data sections, stack and heap will already be saved in the pagefile.
Post 04 Jul 2025, 08:45
View user's profile Send private message Reply with quote
Core i7



Joined: 14 Nov 2024
Posts: 133
Location: Socket on motherboard
Core i7 04 Jul 2025, 17:46
bitRAKE, macomics, Thank you very much for the detailed explanation!
As I understand it, now my algorithm should be approximately as described below.
For simplicity, there are no user processes, and virtual memory with page swapping is used only for the OS kernel:

1. Calculate the available size of physical memory, and divide by 4096 to get the number of PFN frames.
2. Find free regions in the memory space, for example, with the 0xE820 int-15h function.
3. Create a "PfnDatabase" in memory, which will contain as many 48-byte structures as the PFN found in stage (1).
4. Create a global "PageTable" of the kernel with a physical address in the CR3 register - at each PML4..PT level, only 1 base directory.
5. Create PTE records at all levels only in the "PageFault" exception handler.
6. Each PTE entry is an index into a _MMPFN structure in the "PfnDatabase", which should be filled as follows:
Code:
0: kd> dt _mmpfn -r1
nt!_MMPFN
   +0x000 Flink                : Uint8B 
   +0x008 Blink                : Uint8B
   +0x010 PteAddress           : Ptr64 _MMPTE
   +0x018 ReferenceCount       : Uint4B
   +0x01c UsedPageTableEntries : Uint2B
   +0x01e VaType               : UChar
   +0x01f ViewCount            : UChar
   +0x020 OriginalPte          : _MMPTE
   +0x028 PteFrame             : Uint8B
0: kd>    

The first 2 fields "Flink/Blink" store the index of the next/previous record of the same type (free, busy, etc.), then the virtual address of the PTE in the directory, then the process counter (if the frame is common for 2+ processes), flags, and in the "OriginalPte" field an exact copy of the PTE record from the directory. I am haunted by the connection of virtual addresses with PFN frames, and apparently the "PteAddress" field somehow solves this problem, .. but how, if this is the "address of the record in the directory", and not the "linear address of the cpu access".

Let's take the address of the PEB structure of the user process "calc.exe" for example, and try to find a mention of it by searching in directories and other system structures. As a result, this virtual address does not appear anywhere, which I cannot understand.
Code:
0: kd> !process 0 0 calc.exe     ;<--- DirBase = CR3 = phy.address of PML4 process
PROCESS fffffa8005e3eb00
    SessionId: 1         Cid: 0a60    Peb: 7fffffdf000  ParentCid : 06b8
    DirBase  : 50e95000  ObjectTable: fffff8a00cbfd1a0  HandleCount:  97
    Image    : calc.exe

0: kd> .process /P fffffa8005e3eb00           ;<--- Switch debugger context to CALC.EXE
Implicit process is now fffffa80`05e3eb00

0: kd> !vtop 50e95000 7fffffdf000             ;<-------- PEB address relative to CR3
Amd64VtoP: Virt 000007ff`fffdf000, pagedir 50e95000

Amd64VtoP: PML4E  50e95078
Amd64VtoP: PDPE   1`18cc9ff8
Amd64VtoP: PDE    bfecaff8
Amd64VtoP: PTE    af04bef8

Virtual address 7fffffdf000 translates to physical address 9454c000.
    

Thus, the virtual address 7fffffdf000 is mapped to PFN=9454c.
The !cmkd extension shows the same result in more detail, including the indexes of records in all PML4..PT tables.
Apparently, each of the tables is completely filled, since with a max value of 200h, all indexes are almost at the end:
Code:
0: kd> !cmkd.ptelist -v 7fffffdf000
VA=000007FFFFFDF000
  PXE Idx=00F  Va=FFFFF6FB7DBED078  Contents=6380000118CC9867  Hard Pfn=00118CC9  Attr=---DA--UWEV
  PPE Idx=1FF  Va=FFFFF6FB7DA0FFF8  Contents=26100000BFECA867  Hard Pfn=000BFECA  Attr=---DA--UWEV
  PDE Idx=1FF  Va=FFFFF6FB41FFFFF8  Contents=00A00000AF04B867  Hard Pfn=000AF04B  Attr=---DA--UWEV
  PTE Idx=1DF  Va=FFFFF683FFFFFEF8  Contents=831000009454C867  Hard Pfn=0009454C  Attr=---DA--UW-V
    

Now, having received the PTE address, we can read the attributes in the first 12 bits of PFN=9554c, although they are indicated above:
Code:
0: kd> dt _mmpte_hardware FFFFF683FFFFFEF8
nt!_MMPTE_HARDWARE
   +0x000 Valid            : Pos 0,   1
   +0x000 Owner            : Pos 2,   1
   +0x000 WriteThrough     : Pos 3,   0
   +0x000 CacheDisable     : Pos 4,   0
   +0x000 Accessed         : Pos 5,   1
   +0x000 Dirty            : Pos 6,   1
   +0x000 LargePage        : Pos 7,   0
   +0x000 Global           : Pos 8,   0
   +0x000 CopyOnWrite      : Pos 9,   0
   +0x000 Write            : Pos 11,  1
   +0x000 PageFrameNumber  : Pos 12,  0x00009454c  ;<------
   +0x000 Reserved         : Pos 48,  0
   +0x000 SoftwareWsIndex  : Pos 52,  0x31
   +0x000 NoExecute        : Pos 63,  1
0: kd>    

The last place where you can find the virtual address PEB=7fffffdf000 of the calc.exe process is the entries in "PfnDatabase", but as we can see, this is also empty.
Code:
0: kd> !pfn 9454C
    PFN 0009454C at address FFFFFA8001BCFE40
    flink       00000031    blink / shared  00000001     pteaddress FFFFF683FFFFFEF8
    reference count 0001    used entry count    0000     color 0    priority 5
    restore pte 00000080    containing page   0AF04B     Active     Modified

0: kd> dt _mmpfn FFFFFA8001BCFE40
nt!_MMPFN
   +0x000 Flink                : 0x31
   +0x008 Blink                : 1
   +0x010 PteAddress           : 0xfffff683`fffffef8 _MMPTE
   +0x018 ReferenceCount       : 1
   +0x01c UsedPageTableEntries : 0
   +0x01e VaType               : 0 ''
   +0x01f ViewCount            : 0 ''
   +0x020 OriginalPte          : _MMPTE
   +0x028 PteFrame             : 0xaf04b
0: kd>    

So the OS really doesn't store the virtual address anywhere, and it's just an index of a record in "PageTable". But then how can I find the PFN=9454c that was unloaded to disk if the system gives my PFN to another virtual address, and even worse, if it starts distributing it to two or three addresses in turn (i.e. I'll be the third one to use it)? Each new owner will set the (V) bit in the PFN, which will be reflected in the records in "PfnDatabase". In general, I'm completely confused.. Simple truths just don't reach me.
Post 04 Jul 2025, 17:46
View user's profile Send private message Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 4260
Location: vpcmpistri
bitRAKE 04 Jul 2025, 19:16
You are 100% correct in your conclusion: the OS does not explicitly store the virtual address anywhere in the _MMPFN or other structures. You have proven this to yourself. The connection is entirely implicit, and you have all the pieces. Let's put them together.

The key to your question—"how can I find the PFN=9454C that was unloaded to disk"—is to stop thinking from the perspective of the PFN. Instead, think from the perspective of the virtual address. The VA is the permanent key; the PFN is just a temporary location.

Here is the sequence of events that resolves your confusion:

----

The "PteAddress" Field is the Key

You are correct that the PteAddress field in the _MMPFN is the crucial link. It solves the reverse mapping problem. When the OS decides it needs to reclaim physical frame 9454C, it performs these steps:
  1. Looks up PFN 9454c in the PfnDatabase.
  2. Finds the _MMPFN structure at that location (FFFFFA8001BCFE40 in your example).
  3. Reads the PteAddress field from that structure, which is FFFFF683FFFFFEF8.
Now the OS knows the exact address of the single _MMPTE that is using this physical frame. It doesn't need to know the user-mode virtual address (7FFFFFDF000), it only needs to know how to break the mapping.

----

The Page-Out and Page-In Lifecycle

Let's walk through what happens when your PFN 9454C (containing the PEB) is paged out and then paged back in.

State 1: Initial Mapping (What you observed)
  • Virtual Address 7FFFFFDF000 is requested by calc.exe.
  • The MMU walks the page tables and finds the PTE at FFFFF683FFFFFEF8.
  • This PTE contains the hardware address of PFN 9454C and has its Valid bit set to 1.
  • The _MMPFN entry for 9454c has its PteAddress field pointing back to this PTE.

State 2: The Page Out

Now, the system comes under memory pressure and the Memory Manager decides to reclaim PFN 9454C.
  1. Write to Disk: The OS copies the 4KB of data from physical frame 9454C to a free slot in the page file (e.g., pagefile.sys). Let's say it's written to slot #5821.
  2. Modify the PTE: The OS uses the PteAddress from the _MMPFN entry (FFFFF683FFFFFEF8) to find the PTE. It then completely changes that PTE's contents:
    • - The Valid bit is set to 0.
    • - The bits that used to hold the PageFrameNumber are overwritten with information encoding the location on disk (e.g., "Page File #1, Slot #5821"). This is now a "Software PTE".
  3. Free the PFN: The _MMPFN entry for 9454C is now put on the "free" or "standby" list. Its PteAddress is cleared. Physical frame 9454C no longer has any association with your calc.exe process. It can now be given to another process for a completely different virtual address.
This is the most important part: the information about where your data went is stored in the PTE, whose location is permanently tied to the virtual address 7FFFFFDF000.

State 3: The Page In (Page Fault)

Sometime later, calc.exe runs again and its code tries to read from the PEB at virtual address 7FFFFFDF000.
  1. Hardware Walk: The CPU's MMU does exactly what it did before. It uses the bits from 7FFFFFDF000 to walk the PML4 -> PDPT -> PD -> PT, arriving at the exact same PTE address: FFFFF683FFFFFEF8.
  2. Fault! The MMU reads the PTE and sees the Valid bit is 0. It cannot continue. It triggers a #PF (Page Fault) exception and hands control to the OS kernel. The CPU also reports the faulting virtual address (7FFFFFDF000).
  3. OS Handler: The page fault handler looks at the PTE that caused the fault. It sees it's a "software" PTE and decodes the contents: "Aha, the data for this virtual address is in Page File #1, Slot #5821".
  4. Restore from Disk: The OS finds a new free physical frame (let's say PFN ABCDE), reads the data from the page file into that new frame, and updates the PTE at FFFFF683FFFFFEF8 once more.
  5. Update the PTE (again): The PTE is now changed to be a "Hardware PTE" again. The Valid bit is set to 1, and the PageFrameNumber is set to ABCDE.
  6. Update the PFN: The _MMPFN for the new frame ABCDE is updated. Its PteAddress is set to FFFFF683FFFFFEF8.
  7. Resume Execution: The OS returns from the exception. The CPU re-runs the instruction that faulted. This time, the MMU walks the tables, finds the PTE, sees Valid=1, gets PFN ABCDE, and the memory access succeeds.
The process is completely unaware that any of this happened.

----

In short, the OS isn't "looking for a PFN" in a general search. It's performing highly organized list management. It decides which list to take a PFN from and then uses that PFN as a key to update the rest of the system. Paging in requires only the faulting virtual address 7FFFFFDF000, and the page tables tell you where its data is right now.

_________________
¯\(°_o)/¯ AI may [not] have aided with the above reply.


Last edited by bitRAKE on 04 Jul 2025, 20:48; edited 1 time in total
Post 04 Jul 2025, 19:16
View user's profile Send private message Visit poster's website Reply with quote
Core i7



Joined: 14 Nov 2024
Posts: 133
Location: Socket on motherboard
Core i7 04 Jul 2025, 19:57
bitRAKE, firstly, I take my hat off to your technical knowledge of the OS, and secondly, that you can express your thoughts so clearly! Thank you very much - this is truly valuable information that opened my eyes.

PS: My mistake was that I completely forgot about the existence of lists in the "Working set".
Post 04 Jul 2025, 19:57
View user's profile Send private message Reply with quote
Core i7



Joined: 14 Nov 2024
Posts: 133
Location: Socket on motherboard
Core i7 05 Jul 2025, 06:34
Just for fun..
The "PageFault" exception counter at calc.exe startup = 4249.
Virtual memory consumption 98 MB, private pages 1630.
Why does a regular calculator use such huge resources? Sad
Code:
0: kd> !process 0 1 calc.exe

PROCESS fffffa80045abb00
    SessionId: 1          Cid: 0c8c    Peb: 7fffffd6000  ParentCid : 06b0
    DirBase  : 11bf92000  ObjectTable: fffff8a004438660  HandleCount:  96
    Image    : calc.exe

    VadRoot    fffffa8003b450e0  Vads 214. Clone 0. Private 1630. Modified 0. Locked 0.
    DeviceMap  fffff8a00149f980
    Token                             fffff8a00aa5ea30
    ElapsedTime                       00:00:02.429
    UserTime                          00:00:00.000
    KernelTime                        00:00:00.000
    QuotaPoolUsage[PagedPool]         0
    QuotaPoolUsage[NonPagedPool]      0
    Working Set Sizes (now,min,max)  (3784, 50, 345) (15136KB, 200KB, 1380KB)
    PeakWorkingSetSize                3784
    VirtualSize                       93 Mb
    PeakVirtualSize                   98 Mb
    PageFaultCount                    4249         ;<-------------
    MemoryPriority                    BACKGROUND
    BasePriority                      8
    CommitCharge                      1855

0: kd>    
Post 05 Jul 2025, 06:34
View user's profile Send private message Reply with quote
macomics



Joined: 26 Jan 2021
Posts: 1180
Location: Russia
macomics 05 Jul 2025, 10:29
It is Windows...

Core i7 wrote:
Code:
Working Set Sizes (now,min,max)  (3784, 50, 345) (15136KB, 200KB, 1380KB)    

On the other hand. The really working program code affected only 200 kb, and the rest are gifts from the Danites (exhaust and utility from .NET).
Post 05 Jul 2025, 10:29
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 20700
Location: In your JS exploiting you and your system
revolution 05 Jul 2025, 10:34
All the AI? All the "phone home"? All the "rate this app in the store"? All the embedded spyware? All the bloated generic libraries?

All of the above?
Post 05 Jul 2025, 10:34
View user's profile Send private message Visit poster's website Reply with quote
macomics



Joined: 26 Jan 2021
Posts: 1180
Location: Russia
macomics 05 Jul 2025, 10:39
The decorations on the buttons in the interface are still forgotten.
Post 05 Jul 2025, 10:39
View user's profile Send private message Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 4260
Location: vpcmpistri
bitRAKE 05 Jul 2025, 12:39
Core i7 wrote:
Why does a regular calculator use such huge resources? Sad
The working set is only 15MB - which means most the faults were for DLLs already in memory (soft faults). My explanation is not an endorsement. Some would say the "regular calculator" is Windows Calculator app -- which is an order of magnitude larger!

The next "regular calculator" will probably be an AI agent providing dynamic lectures on potential calculations and their applications. The UI will require the Edge browser to communicate with a cloud super-cluster to generate the experience. Twisted Evil
Post 05 Jul 2025, 12:39
View user's profile Send private message Visit poster's website Reply with quote
Core i7



Joined: 14 Nov 2024
Posts: 133
Location: Socket on motherboard
Core i7 05 Jul 2025, 15:33
That calculator was on Win7, and here is its brother on Win10.
Okay, the "PageFault" counter (there is only 4Gb of physical memory here), but "Working Set" was 15 and became 88 MB, and the virtual memory is 5GB! Microsoft is openly laughing at users, selling tons of useless data like candy.
Code:
0: kd> !process 0 1 CalculatorApp.exe
PROCESS ffffe004ca727080
    SessionId: 1          Cid: 1d70    Peb: 42fef46000   ParentCid:  0c64
    DirBase  : 16d0c6000  ObjectTable: ffffa103e0353840  HandleCount: 601
    Image    : CalculatorApp.exe

    VadRoot   ffffe004caa5a200  Vads 231. Clone 0. Private 6828. Modified 684. Locked 1916.
    DeviceMap ffffa103d5e16280
    Token                             ffffa103d60e9870
    ElapsedTime                       00:00:10.070
    UserTime                          00:00:00.046
    KernelTime                        00:00:00.031
    QuotaPoolUsage[PagedPool]         1021816
    QuotaPoolUsage[NonPagedPool]      45104
    Working Set Sizes (now,min,max)  (21970, 50, 345) (87880KB, 200KB, 1380KB)
    PeakWorkingSetSize                22032
    VirtualSize                       4940 Mb
    PeakVirtualSize                   4951 Mb
    PageFaultCount                    23995
    MemoryPriority                    BACKGROUND
    BasePriority                      8
    CommitCharge                      8549
    Job                               ffffe004c438e060

0: kd>    
Post 05 Jul 2025, 15:33
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 20700
Location: In your JS exploiting you and your system
revolution 06 Jul 2025, 07:32
bitRAKE wrote:
The UI will require the Edge browser to communicate with a cloud super-cluster to generate the experience. Twisted Evil
And even with offloading the work to the remote server it will still require 10GB of memory just to show the one sentence response. And it will cost you $0.179 per response because cloud super-clusters aren't cheap to run.
Post 06 Jul 2025, 07:32
View user's profile Send private message Visit poster's website Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 4260
Location: vpcmpistri
bitRAKE 06 Jul 2025, 19:08
{Trying to stay on topic.}

However "good" the present AI has gotten the problem of out-of-distribution trajectories remains. For example, if I tell the AI that the PFN database memory is not in the PFN database in my OS implementation; the AI flat out refuses to work in such a domain with any reliability. From the AI's perspective such an implementation introduces too many problems - this is clearly not the case.

Otherwise, the AI's are far better than any searching: both Gemini 2.5 and ChatGPT o3 give links to source information, and in-distribution hallucinations are almost non-existent.

The paradox is that if one is learning they don't know when they're out-of-distribution, and the AI is not going to tell them. If one is engineering something new then using the AI is like bashing one's head against a brick wall.

_________________
¯\(°_o)/¯ AI may [not] have aided with the above reply.
Post 06 Jul 2025, 19:08
View user's profile Send private message Visit poster's website Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  


< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2025, Tomasz Grysztar. Also on GitHub, YouTube.

Website powered by rwasa.