flat assembler
Message board for the users of flat assembler.
![]() |
Author |
|
Core i7 27 Jun 2025, 08:48
Hello everyone!
I write an OS in my free time, and decided to share the result here. The capabilities of BIOS are used (not uefi), direct output to video memory 0xB800 in 80x25/16 mode, menu selection with keys 1-7. In principle, nothing new - the bootloader reads the kernel at address 0x0600, leaving the stack in the same place 0x7C00. In addition to the legacy ATA interface, there is support for modern SATA in AHCI mode. Later, I plan to add initialization of the NVM controller on the PCI bus, as well as NVMe on the PCI-Express bus. The software model of AHCI, NVM, NVMe is different, so I will have to implement three different modules. Since their ports are mapped to MMIO memory, the real mode kernel code has a switch to "unreal" mode, with access to 4GB of memory through the FS register. At the moment I am working on the x64 entry with the "Long" memory model - it is not finished yet, so the menu item (6) is empty for now. The archive contains the entire source code, as well as a 1.44 MB floppy image of "CodebyOS". To conduct tests on real hardware, you can manually create a bootable USB-Flash, and in the BOOT folder there is the source code of the USB-loader. Here is the result.
|
||||||||||||||||||||
![]() |
|
macomics 02 Jul 2025, 23:59
Virtual addressing does not have to project all the physical memory. Moreover, you can repeatedly project one physical memory page to different virtual addresses. Moreover, there are page absence flags at each of the projection levels, which suggests a branch break in the virtual memory map tree.
bot-0 = Present If you need it, then the minimum virtual address space will be absolutely necessary: 32 kb = 4kb (PLM4) + 4kb (PDPTE) + 4kb (PDE) + 4 kb (PTB) + 4kb (GDT) + 4kb (IDT) + 4kb (code) + 4kb (stack+data). That is, the approach is as follows. First, you need to split the memory ranges into pages, and then you can use these pages as needed and place them in the virtual memory map of the process. |
|||
![]() |
|
Core i7 03 Jul 2025, 03:29
macomics wrote: Virtual addressing does not have to project all the physical memory. This means that the directories are created dynamically, as the number of requested virtual addresses increases, and their order is arbitrary, not linear. Thank you, I understand. Code: PML4 PDPT PDT PT 0 0 0 0 --> PFN #0 5 210 1 --> PFN #293 1 2 18 --> PFN #5 etc.. And apparently this is only the beginning, because further there is a separate memory of the state of physical frames "PfnDatabase" (free, busy, prototype, etc.), and for each process there is also a list of pages allocated to it VAD - Virt Addr Descriptor. In general, the mechanism of supporting virtual memory is very complex, and good logic is needed for fast search in large arrays of data for the required values. |
|||
![]() |
|
Core i7 04 Jul 2025, 04:05
No, I still don't understand something.
Let's say we have 1000 frames of physical memory PFN, and each is 4 KB. In order to display all the virtual memory x64 = 256 TB in them, I have to save 1 PFN to disk (for example PFN #50), and rewrite it with new data. This can be repeated infinitely. But if later the CPU requests a virtual address that pointed to the original PFN=50, I will have to restore it from disk. This is impossible to do without remembering all the virtual addresses that the CPU accesses. This is exactly why the PML4..PT directories were invented. But then the size of these directories will grow exponentially up to 512 GB. I wonder how this problem is solved? I have already looked at all the virtual memory support structures in Windows - mainly _MMPFN in "PfnDatabase", and _MMPTE in directory entries. But these structures also do not have a field with a virtual address. Apparently there is some little thing here that I cannot understand. The global PFN database stores the state of each frame, which is described by such a _MMPFN structure of 48 bytes (30h). That is, for every 4K of physical memory we have 48 bytes of service information: Code: 0: kd> dt _MmPfn -r1 nt!_MMPFN +0x000 u1 : <unnamed-tag> +0x000 Flink : Uint8B +0x000 WsIndex : Uint4B +0x000 Event : Ptr64 _KEVENT +0x000 Next : Ptr64 Void +0x000 VolatileNext : Ptr64 Void +0x000 KernelStackOwner : Ptr64 _KTHREAD +0x000 NextStackPfn : _SINGLE_LIST_ENTRY +0x008 u2 : <unnamed-tag> +0x000 Blink : Uint8B +0x000 ImageProtoPte : Ptr64 _MMPTE +0x000 ShareCount : Uint8B +0x010 PteAddress : Ptr64 _MMPTE +0x010 VolatilePteAddress : Ptr64 Void +0x010 Lock : Int4B +0x010 PteLong : Uint8B +0x018 u3 : <unnamed-tag> +0x000 ReferenceCount : Uint2B +0x002 e1 : _MMPFNENTRY +0x000 e2 : <unnamed-tag> +0x01c UsedPageTableEntry : Uint2B +0x01e VaType : UChar +0x01f ViewCount : UChar +0x020 OriginalPte : _MMPTE +0x028 u4 : <unnamed-tag> +0x000 PteFrame : Pos 0, 52 Bits +0x000 Unused : Pos 52, 3 Bits +0x000 PfnImageVerified : Pos 55, 1 Bit +0x000 AweAllocation : Pos 56, 1 Bit +0x000 PrototypePte : Pos 57, 1 Bit +0x000 PageColor : Pos 58, 6 Bits And this is the _MMPTE structure, which describes one entry in the PML4..PT directories. There are 7 types of frames, and the ones of interest now are "Hardware" with the zero bit (V) set, and "Software" for those unloaded to the swap file on the disk. As we can see, there are no virtual addresses for matching with PFN frames, which means the addresses are stored in the "PageTable" directories. Code: 0: kd> dt _MmPte -r1 nt!_MMPTE +0x000 u : <unnamed-tag> +0x000 VolatileLong : Uint8B +0x000 Hard : _MMPTE_HARDWARE ;<--- valid frame in memory +0x000 Proto : _MMPTE_PROTOTYPE ;<--- shared frame for 2+ processes +0x000 Soft : _MMPTE_SOFTWARE ;<--- saved to disk in Pagefile.sys +0x000 TimeStamp : _MMPTE_TIMESTAMP +0x000 Trans : _MMPTE_TRANSITION +0x000 Subsect : _MMPTE_SUBSECTION +0x000 List : _MMPTE_LIST 0: kd> dt _MMPTE_HARDWARE nt!_MMPTE_HARDWARE +0x000 Valid : Pos 0, 1 Bit +0x000 Dirty1 : Pos 1, 1 Bit +0x000 Owner : Pos 2, 1 Bit +0x000 WriteThrough : Pos 3, 1 Bit +0x000 CacheDisable : Pos 4, 1 Bit +0x000 Accessed : Pos 5, 1 Bit +0x000 Dirty : Pos 6, 1 Bit +0x000 LargePage : Pos 7, 1 Bit +0x000 Global : Pos 8, 1 Bit +0x000 CopyOnWrite : Pos 9, 1 Bit +0x000 Unused : Pos 10, 1 Bit +0x000 Write : Pos 11, 1 Bit +0x000 PageFrameNumber : Pos 12, 36 Bits +0x000 reserved1 : Pos 48, 4 Bits +0x000 SoftwareWsIndex : Pos 52, 11 Bits +0x000 NoExecute : Pos 63, 1 Bit 0: kd> dt _MMPTE_SOFTWARE nt!_MMPTE_SOFTWARE +0x000 Valid : Pos 0, 1 Bit +0x000 Unused : Pos 1, 2 Bits +0x000 InStore : Pos 3, 1 Bit +0x000 SwizzleBit : Pos 4, 1 Bit +0x000 Protection : Pos 5, 5 Bits +0x000 Prototype : Pos 10, 1 Bit +0x000 Transition : Pos 11, 1 Bit +0x000 PageFileLow : Pos 12, 4 Bits +0x000 UsedPTableEntry : Pos 16, 10 Bits +0x000 Reserved : Pos 26, 6 Bits +0x000 PageFileHigh : Pos 32, 32 Bits 0: kd> |
|||
![]() |
|
bitRAKE 04 Jul 2025, 04:58
A fully populated set of page tables for a 256 TB address space would indeed be enormous. The solution is that page tables are created on demand.
You have pinpointed the exact source of your confusion, and you are very close to the answer. The critical piece you're missing is that the mapping is not searched for; it is calculated. The virtual address itself is the key that directly leads to the location of its data, whether that data is in RAM or on disk. The system does not need to store the virtual address within the _MMPFN or on disk because the entire page table structure is designed to use the virtual address as an index. You are correct that the _MMPFN does not contain a virtual address. Its purpose is the reverse mapping. For every physical frame (PFN), the _MMPFN entry tells the OS about the state of that physical frame. A crucial piece of information it contains is a pointer back to the Page Table Entry (_MMPTE) that maps this frame. This is essential when the OS decides to page out the data in a particular frame (e.g., PFN #50). The OS needs to do two things:
(Perhaps the supporting code in the Windows Research Kernel would be helpful.) |
|||
![]() |
|
macomics 04 Jul 2025, 07:57
Core i7 wrote: And this is the _MMPTE structure, which describes one entry in the PML4..PT directories. There are 7 types of frames, and the ones of interest now are "Hardware" with the zero bit (V) set, and "Software" for those unloaded to the swap file on the disk. As we can see, there are no virtual addresses for matching with PFN frames, which means the addresses are stored in the "PageTable" directories. The virtual address itself does not need to be stored. All you need is information about the use of a page that has already been used in the virtual space (bit-5 Access) and information about its position (_MMPTE). When switching virtual address spaces, the OS resets the access bit in all entries at all levels of the map (in the map that will be used now, but in the old one they remain unchanged). This way, she learns about the pages that the tasks really needed during their execution. This means that the rest can be painlessly download from memory because they will be used less often than the first ones. If a memory page is uploaded to disk, then you need to record information about the map to which it belonged (CR3) and its position in this map. You don't need the rest of the information. ADD: I do not exclude that the OS can calculate the frequency of page usage, and not just rely on the availability of an access bit. That is, enter your own counter for each page and increase it when she get a=1.
|
|||||||||||||||||||
![]() |
|
macomics 04 Jul 2025, 08:45
A more mundane example. Your task is to load your own library into RAM. To locate the entire image of this library, physical memory pages were found and used, and then arranged in a linear block of virtual addresses through the location of pointers to nothing in the corresponding map. If possible, the block of virtual addresses for the library is selected according to its PE.ImageBase, so that relocation does not have to be performed (but I digress). After ensuring that the entire memory range is available in the virtual space for loading the image, it is read from disk into this memory. I made a reservation about his own library for a reason, because it will not be divided between different memory maps (for simplicity) it will be present in the memory map only of the current process. Then, when this process is running, a=1 bits will occur in this memory range only in those places that are actually used by the code. In addition, let say the library occupies 10 MB of memory (5*PDE=3*PDE header and code + 2*PDE data and resources). That is, the OS can create 5*PDE and fill in 5*512 PTEs for them, or use 5*PDE with 2Mb pages (I got distracted again). There are few accesses to the file header while using the DLL, and it will often be a=0. But not all of the DLL code will work either too. Only those pages that actually running. The situation with data is slightly different, but with resources it is exactly the same as with the header and code. All these pages use read-only data from an image (a file on disk). That is, it is pointless to download it to a pagefile. All this data can be re-read from the library image from disk. But here is the data (data and bss section) just what the library recorded, and it's not in the image. Only 2*PDEs with data are subject to paging.
That is, the header, the part of the code that is not executed, and the resources are simply unloaded from the physical memory pages and it is released. All this information is simply re-read from the corresponding DLL file from the disk as needed. But the data sections, stack and heap will already be saved in the pagefile. |
|||
![]() |
|
Core i7 04 Jul 2025, 17:46
bitRAKE, macomics, Thank you very much for the detailed explanation!
As I understand it, now my algorithm should be approximately as described below. For simplicity, there are no user processes, and virtual memory with page swapping is used only for the OS kernel: 1. Calculate the available size of physical memory, and divide by 4096 to get the number of PFN frames. 2. Find free regions in the memory space, for example, with the 0xE820 int-15h function. 3. Create a "PfnDatabase" in memory, which will contain as many 48-byte structures as the PFN found in stage (1). 4. Create a global "PageTable" of the kernel with a physical address in the CR3 register - at each PML4..PT level, only 1 base directory. 5. Create PTE records at all levels only in the "PageFault" exception handler. 6. Each PTE entry is an index into a _MMPFN structure in the "PfnDatabase", which should be filled as follows: Code: 0: kd> dt _mmpfn -r1 nt!_MMPFN +0x000 Flink : Uint8B +0x008 Blink : Uint8B +0x010 PteAddress : Ptr64 _MMPTE +0x018 ReferenceCount : Uint4B +0x01c UsedPageTableEntries : Uint2B +0x01e VaType : UChar +0x01f ViewCount : UChar +0x020 OriginalPte : _MMPTE +0x028 PteFrame : Uint8B 0: kd> The first 2 fields "Flink/Blink" store the index of the next/previous record of the same type (free, busy, etc.), then the virtual address of the PTE in the directory, then the process counter (if the frame is common for 2+ processes), flags, and in the "OriginalPte" field an exact copy of the PTE record from the directory. I am haunted by the connection of virtual addresses with PFN frames, and apparently the "PteAddress" field somehow solves this problem, .. but how, if this is the "address of the record in the directory", and not the "linear address of the cpu access". Let's take the address of the PEB structure of the user process "calc.exe" for example, and try to find a mention of it by searching in directories and other system structures. As a result, this virtual address does not appear anywhere, which I cannot understand. Code: 0: kd> !process 0 0 calc.exe ;<--- DirBase = CR3 = phy.address of PML4 process PROCESS fffffa8005e3eb00 SessionId: 1 Cid: 0a60 Peb: 7fffffdf000 ParentCid : 06b8 DirBase : 50e95000 ObjectTable: fffff8a00cbfd1a0 HandleCount: 97 Image : calc.exe 0: kd> .process /P fffffa8005e3eb00 ;<--- Switch debugger context to CALC.EXE Implicit process is now fffffa80`05e3eb00 0: kd> !vtop 50e95000 7fffffdf000 ;<-------- PEB address relative to CR3 Amd64VtoP: Virt 000007ff`fffdf000, pagedir 50e95000 Amd64VtoP: PML4E 50e95078 Amd64VtoP: PDPE 1`18cc9ff8 Amd64VtoP: PDE bfecaff8 Amd64VtoP: PTE af04bef8 Virtual address 7fffffdf000 translates to physical address 9454c000. Thus, the virtual address 7fffffdf000 is mapped to PFN=9454c. The !cmkd extension shows the same result in more detail, including the indexes of records in all PML4..PT tables. Apparently, each of the tables is completely filled, since with a max value of 200h, all indexes are almost at the end: Code: 0: kd> !cmkd.ptelist -v 7fffffdf000 VA=000007FFFFFDF000 PXE Idx=00F Va=FFFFF6FB7DBED078 Contents=6380000118CC9867 Hard Pfn=00118CC9 Attr=---DA--UWEV PPE Idx=1FF Va=FFFFF6FB7DA0FFF8 Contents=26100000BFECA867 Hard Pfn=000BFECA Attr=---DA--UWEV PDE Idx=1FF Va=FFFFF6FB41FFFFF8 Contents=00A00000AF04B867 Hard Pfn=000AF04B Attr=---DA--UWEV PTE Idx=1DF Va=FFFFF683FFFFFEF8 Contents=831000009454C867 Hard Pfn=0009454C Attr=---DA--UW-V Now, having received the PTE address, we can read the attributes in the first 12 bits of PFN=9554c, although they are indicated above: Code: 0: kd> dt _mmpte_hardware FFFFF683FFFFFEF8 nt!_MMPTE_HARDWARE +0x000 Valid : Pos 0, 1 +0x000 Owner : Pos 2, 1 +0x000 WriteThrough : Pos 3, 0 +0x000 CacheDisable : Pos 4, 0 +0x000 Accessed : Pos 5, 1 +0x000 Dirty : Pos 6, 1 +0x000 LargePage : Pos 7, 0 +0x000 Global : Pos 8, 0 +0x000 CopyOnWrite : Pos 9, 0 +0x000 Write : Pos 11, 1 +0x000 PageFrameNumber : Pos 12, 0x00009454c ;<------ +0x000 Reserved : Pos 48, 0 +0x000 SoftwareWsIndex : Pos 52, 0x31 +0x000 NoExecute : Pos 63, 1 0: kd> The last place where you can find the virtual address PEB=7fffffdf000 of the calc.exe process is the entries in "PfnDatabase", but as we can see, this is also empty. Code: 0: kd> !pfn 9454C PFN 0009454C at address FFFFFA8001BCFE40 flink 00000031 blink / shared 00000001 pteaddress FFFFF683FFFFFEF8 reference count 0001 used entry count 0000 color 0 priority 5 restore pte 00000080 containing page 0AF04B Active Modified 0: kd> dt _mmpfn FFFFFA8001BCFE40 nt!_MMPFN +0x000 Flink : 0x31 +0x008 Blink : 1 +0x010 PteAddress : 0xfffff683`fffffef8 _MMPTE +0x018 ReferenceCount : 1 +0x01c UsedPageTableEntries : 0 +0x01e VaType : 0 '' +0x01f ViewCount : 0 '' +0x020 OriginalPte : _MMPTE +0x028 PteFrame : 0xaf04b 0: kd> So the OS really doesn't store the virtual address anywhere, and it's just an index of a record in "PageTable". But then how can I find the PFN=9454c that was unloaded to disk if the system gives my PFN to another virtual address, and even worse, if it starts distributing it to two or three addresses in turn (i.e. I'll be the third one to use it)? Each new owner will set the (V) bit in the PFN, which will be reflected in the records in "PfnDatabase". In general, I'm completely confused.. Simple truths just don't reach me. |
|||
![]() |
|
bitRAKE 04 Jul 2025, 19:16
You are 100% correct in your conclusion: the OS does not explicitly store the virtual address anywhere in the _MMPFN or other structures. You have proven this to yourself. The connection is entirely implicit, and you have all the pieces. Let's put them together.
The key to your question—"how can I find the PFN=9454C that was unloaded to disk"—is to stop thinking from the perspective of the PFN. Instead, think from the perspective of the virtual address. The VA is the permanent key; the PFN is just a temporary location. Here is the sequence of events that resolves your confusion: ---- The "PteAddress" Field is the Key You are correct that the PteAddress field in the _MMPFN is the crucial link. It solves the reverse mapping problem. When the OS decides it needs to reclaim physical frame 9454C, it performs these steps:
---- The Page-Out and Page-In Lifecycle Let's walk through what happens when your PFN 9454C (containing the PEB) is paged out and then paged back in. State 1: Initial Mapping (What you observed)
State 2: The Page Out Now, the system comes under memory pressure and the Memory Manager decides to reclaim PFN 9454C.
State 3: The Page In (Page Fault) Sometime later, calc.exe runs again and its code tries to read from the PEB at virtual address 7FFFFFDF000.
---- In short, the OS isn't "looking for a PFN" in a general search. It's performing highly organized list management. It decides which list to take a PFN from and then uses that PFN as a key to update the rest of the system. Paging in requires only the faulting virtual address 7FFFFFDF000, and the page tables tell you where its data is right now. _________________ ¯\(°_o)/¯ AI may [not] have aided with the above reply. Last edited by bitRAKE on 04 Jul 2025, 20:48; edited 1 time in total |
|||
![]() |
|
Core i7 04 Jul 2025, 19:57
bitRAKE, firstly, I take my hat off to your technical knowledge of the OS, and secondly, that you can express your thoughts so clearly! Thank you very much - this is truly valuable information that opened my eyes.
PS: My mistake was that I completely forgot about the existence of lists in the "Working set". |
|||
![]() |
|
Core i7 05 Jul 2025, 06:34
Just for fun..
The "PageFault" exception counter at calc.exe startup = 4249. Virtual memory consumption 98 MB, private pages 1630. Why does a regular calculator use such huge resources? ![]() Code: 0: kd> !process 0 1 calc.exe PROCESS fffffa80045abb00 SessionId: 1 Cid: 0c8c Peb: 7fffffd6000 ParentCid : 06b0 DirBase : 11bf92000 ObjectTable: fffff8a004438660 HandleCount: 96 Image : calc.exe VadRoot fffffa8003b450e0 Vads 214. Clone 0. Private 1630. Modified 0. Locked 0. DeviceMap fffff8a00149f980 Token fffff8a00aa5ea30 ElapsedTime 00:00:02.429 UserTime 00:00:00.000 KernelTime 00:00:00.000 QuotaPoolUsage[PagedPool] 0 QuotaPoolUsage[NonPagedPool] 0 Working Set Sizes (now,min,max) (3784, 50, 345) (15136KB, 200KB, 1380KB) PeakWorkingSetSize 3784 VirtualSize 93 Mb PeakVirtualSize 98 Mb PageFaultCount 4249 ;<------------- MemoryPriority BACKGROUND BasePriority 8 CommitCharge 1855 0: kd> |
|||
![]() |
|
macomics 05 Jul 2025, 10:29
It is Windows...
Core i7 wrote:
On the other hand. The really working program code affected only 200 kb, and the rest are gifts from the Danites (exhaust and utility from .NET). |
|||
![]() |
|
revolution 05 Jul 2025, 10:34
All the AI? All the "phone home"? All the "rate this app in the store"? All the embedded spyware? All the bloated generic libraries?
All of the above? |
|||
![]() |
|
macomics 05 Jul 2025, 10:39
The decorations on the buttons in the interface are still forgotten.
|
|||
![]() |
|
bitRAKE 05 Jul 2025, 12:39
Core i7 wrote: Why does a regular calculator use such huge resources? The next "regular calculator" will probably be an AI agent providing dynamic lectures on potential calculations and their applications. The UI will require the Edge browser to communicate with a cloud super-cluster to generate the experience. ![]() |
|||
![]() |
|
Core i7 05 Jul 2025, 15:33
That calculator was on Win7, and here is its brother on Win10.
Okay, the "PageFault" counter (there is only 4Gb of physical memory here), but "Working Set" was 15 and became 88 MB, and the virtual memory is 5GB! Microsoft is openly laughing at users, selling tons of useless data like candy. Code: 0: kd> !process 0 1 CalculatorApp.exe PROCESS ffffe004ca727080 SessionId: 1 Cid: 1d70 Peb: 42fef46000 ParentCid: 0c64 DirBase : 16d0c6000 ObjectTable: ffffa103e0353840 HandleCount: 601 Image : CalculatorApp.exe VadRoot ffffe004caa5a200 Vads 231. Clone 0. Private 6828. Modified 684. Locked 1916. DeviceMap ffffa103d5e16280 Token ffffa103d60e9870 ElapsedTime 00:00:10.070 UserTime 00:00:00.046 KernelTime 00:00:00.031 QuotaPoolUsage[PagedPool] 1021816 QuotaPoolUsage[NonPagedPool] 45104 Working Set Sizes (now,min,max) (21970, 50, 345) (87880KB, 200KB, 1380KB) PeakWorkingSetSize 22032 VirtualSize 4940 Mb PeakVirtualSize 4951 Mb PageFaultCount 23995 MemoryPriority BACKGROUND BasePriority 8 CommitCharge 8549 Job ffffe004c438e060 0: kd> |
|||
![]() |
|
revolution 06 Jul 2025, 07:32
bitRAKE wrote: The UI will require the Edge browser to communicate with a cloud super-cluster to generate the experience. |
|||
![]() |
|
bitRAKE 06 Jul 2025, 19:08
{Trying to stay on topic.}
However "good" the present AI has gotten the problem of out-of-distribution trajectories remains. For example, if I tell the AI that the PFN database memory is not in the PFN database in my OS implementation; the AI flat out refuses to work in such a domain with any reliability. From the AI's perspective such an implementation introduces too many problems - this is clearly not the case. Otherwise, the AI's are far better than any searching: both Gemini 2.5 and ChatGPT o3 give links to source information, and in-distribution hallucinations are almost non-existent. The paradox is that if one is learning they don't know when they're out-of-distribution, and the AI is not going to tell them. If one is engineering something new then using the AI is like bashing one's head against a brick wall. _________________ ¯\(°_o)/¯ AI may [not] have aided with the above reply. |
|||
![]() |
|
< Last Thread | Next Thread > |
Forum Rules:
|
Copyright © 1999-2025, Tomasz Grysztar. Also on GitHub, YouTube.
Website powered by rwasa.