flat assembler
Message board for the users of flat assembler.

Index > OS Construction > Accessing real mode data in long mode

Goto page Previous  1, 2
Author
Thread Post new topic Reply to topic
smiddy



Joined: 31 Oct 2004
Posts: 559
smiddy
revolution wrote:
smiddy wrote:
BTW, is a paging file "required" to work in this virtual environment?
You would need to define "virtual environment" first. Generally speaking paging files are used to store parts of memory in disc but it is also possible to not store anything and simply make all virtual memory be "reloaded" as zeros (or some other fixed set of constant values) each time it is used. I don't know of a use case for such a thing but the OS could be programmed for that if wanted. Although in a way memory mapped files are a type of virtual memory that is not using a specific paging file.
That is clear, thanks! I was fearing I had to get a controller and file system going before I had paging going, if I was forced into using a page file.
Post 09 Dec 2014, 03:18
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17624
Location: In your JS exploiting you and your system
revolution
Yes, you could simply define a fenced-off section of the HDD as virtual memory without needing to give it a filename and sector map etc. It would give less flexibility than a file defined within a file system but it would be easier to implement.
Post 09 Dec 2014, 03:28
View user's profile Send private message Visit poster's website Reply with quote
smiddy



Joined: 31 Oct 2004
Posts: 559
smiddy
l_inc wrote:
smiddy
Quote:
it is a process towards learning

It is a process towards confusion.

Maybe that's why it is confusing?! LOL! I keep going until I understand, I am a learner, constantly trying to understand. The more I know, the less I know I know, you know?
Quote:
Asked another way

l_inc wrote:
Another way is good, cause understanding the former way requires profound skills in cryptanalysis. Smile

Who says I've never listened to beeps and squeaks, or dabbled in mathematical algorythm work?! Wink
Quote:
can linear address space point to the physical address space such that they are aligned the same, address 0 linear address space equals 0 physical memory address?

l_inc wrote:
That's what's called "identity mapping". Sure it can.

Intel Chapter 4 wrote:
Chapter 3 explains how segmentation converts logical addresses to linear addresses. Paging (or linear-address
translation) is the process of translating linear addresses so that they can be used to access memory or I/O
devices. Paging translates each linear address to a physical address and determines, for each translation, what
accesses to the linear address are allowed (the address’s access rights) and the type of caching used for such
accesses (the address’s memory type).

I can see how that is interpreted as Identity Mapping. A database of pointers, keeping track of both physical and linear address translation.
Quote:
And still see the original contents prior to being placed in that virtual environment?

l_inc wrote:
Are you suggesting to access physical memory using the same physical addresses, but obtain different data? As I said before, there's the memory controller hub between the physical address space and physical memory. If you'd be able to program the MCH (which you are normally not, cause BIOS does that and locks the corresponding settings) it would allow you to do that for some memory ranges and at specific circumstances, but not in general. In general, you should assume that if you write something to the physical address X that is mapped to RAM, then you overwrite the data previously saved at physical address X.

No, to be clear, I want to setup everything needing to be done in real mode prior to transitioning into either of the protected modes, leaving those nuggets of information in a locale I am aware of, mapped as I specify, then be able to retrieve them, with (a little as possible) translation to linear address (as opposed to 16 bit segmented addresses) with paging. I've done this with a flat 32 bit memory model, which as the Intel document states, is a page mode, but I never change in and out or multitask switch anything, I'd actually would prefer to make it flat. The problem is you cannot make a 2^x (where x is the number of bits of physical memory, as I have read thus far) wide page, you have to use many pages, I'm gathering, to cover the width of the physical addresses, and it isn't clear to me if you use more than one page, if the second one can be contiguous addresses off of the first either. I then wish to be able to change modes, 64 bit, 32 bit, and 16 bit, and back and forth, so there is a lot of house keeping to keep track of everything. I don't want to get into 64 bit and not transition back to 16 bit if I want to. All of that house keeping has to be laid out pretty well, in specified locales in order to keep it all straight.
Quote:
BTW, is a paging file "required" to work in this virtual environment?

l_inc wrote:
A page file is just an implementation detail for virtual memory. Neither a page file is required for implementing virtual memory, nor virtual memory is required to be implemented by an OS.

Kewl! That's a detail I wish to avoid up front, for a while.
revolution wrote:
simply make all virtual memory be "reloaded" as zeros (or some other fixed set of constant values) each time it is used

l_inc wrote:
This would break the concept of virtualization, which prescribes to create an illusion of something real.

I've been called a magician a time or two...but that has nothing to do with this. Wink

I guess, I'm trying to place a boundary on the possibilities in asking what I'm asking. I will be experimenting with it at some point. Like I said, I'm working to get my storage controllers working, IDE is working, I'm working out the kinks on AHCI at the moment. I can ID ports, but the command list is giving me fits. Slow process of learning.

I appreciate your indulgence in my elementary questions. The curve is steep, but conquerable.

Smiddy
Post 09 Dec 2014, 03:59
View user's profile Send private message Reply with quote
l_inc



Joined: 23 Oct 2009
Posts: 881
l_inc
Quote:
with (a little as possible) translation to linear address (as opposed to 16 bit segmented addresses) with paging

To make it clear, paging (page tables) translates linear addresses to physical addresses.
Quote:
I've done this with a flat 32 bit memory model, which as the Intel document states, is a page mode

I don't think they state that. Flat model is a model that minimizes the effect of segmentation mechanisms by setting all segment bases to zero and all segment limits to the maximum value. It's not directly related to paging.
Quote:
it isn't clear to me if you use more than one page, if the second one can be contiguous addresses off of the first either

If we assume the simplest case of 32-bit paging and use large pages, then the page directory could look like this:
Code:
a)Page dir:                pages:           b)Page dir:                pages:           c)Page dir:                pages:
 __________              __________          __________              __________          __________              __________
|_0x400000_|----------->|physical: |        |_0x400000_|----------->|physical: |        |_0x400000_|----------->|physical: |
|_0x800000_|------\     |  0x400000|        |_0xc00000_|------\     |  0x400000|        |__________|            |  0x400000|
|__________|      |     |linear:   |        |__________|      |     |linear:   |        |_0x800000_|------\     |linear:   |
|__________|      |     |  0x000000|        |__________|      |     |  0x000000|        |__________|      |     |  0x000000|
|..........|      |     |__________|        |..........|      |     |__________|        |..........|      |     |__________|
                  |      __________                           |      __________                           |      __________
                  \---->|phys:     |                          \---->|physical: |                          \---->|physical: |
                        |  0x800000|                                |  0xc00000|                                |  0x800000|
                        |linear:   |                                |linear:   |                                |linear:   |
                        |  0x400000|                                |  0x400000|                                |  0xc00000|
                        |__________|                                |__________|                                |__________|    

In the case a both physical and linear address ranges are contiguous: physical from 0x400000 to 0xbfffff and linear from 0 to 0x7fffff.
In the case b the linear address ranges are contiguous, but physical are not: you may continuously read the data starting from address 0 up to the address 0x7fffff, but you'll obtain data that was located in the address ranges 0x400000 to 0x7fffff and 0xc00000 to 0xffffff before paging was enabled, as if the range 0x800000 to 0xbfffff just magically disappeared.
In the case c the physical ranges are contiguous, but the linear are not. In order to read the data that was contiguously located in the range 0x400000 to 0xbfffff before paging was enabled, you'll need to skip the range 0x400000 to 0x7fffff, while reading data from address 0 to 0xbfffff.

Note that the values I used for the sake of simplicity are just addresses rather than valid PDEs with all the proper flags.
Quote:
I then wish to be able to change modes, 64 bit, 32 bit, and 16 bit, and back and forth

I suppose, you're confusing paging modes and code execution modes. E.g., you can execute 32 bit code no matter what paging mode is currently engaged. Switching paging modes does not make sense: every adequate OS enables its native paging mode once and does not ever change it (except may be for the very first boot-up stages).

Having said that I should mention that I had exotic situations, in which I needed to switch to PAE mode in a Windows driver (in case Windows has been set up to work with 32-bit paging) on-the-fly for a very short period of time by replacing the Windows native page tables with my custom hybrid page tables.

_________________
Faith is a superposition of knowledge and fallacy
Post 09 Dec 2014, 22:46
View user's profile Send private message Reply with quote
smiddy



Joined: 31 Oct 2004
Posts: 559
smiddy
Whoa, the layout makes it clear, thanks for this!

I went a reread the Intel manual, you're on it, it is segmentation comparisons where they mention the similarities, not in paging, or using pages.

I was talking about code execution, and going between those modes, yes. I've done this with 32 bit, and using the flat model (one segment to cover the entire 4 GB space) without separating code and data, mixing them. I traverse between 16-bit and 32-bit in order to get to the BIOS interrupts. This also means keeping track of the PICs, IVT/IDTs, etcetera. It works quite well, however I don't use paging, a simple GDT, no page tables. I'd like to understand the overhead better before trying it in 64-bit, so I will try 32-bit first with some form paging, then transition to 64-bit thereafter.

But, as I can "visually see" from the diagram you provided, the physical memory from 0 to x, can be used by a page and accessing the data therein should be ok (so long as you adhere to the RAM/ROM possibilities, that come from the E820 memory map, PCI devices, and CMOS, I expect. Are there any other gotchas?

So, working in Windows, I have wondered why Windows (Windows 7 64-bit) says it can use physical memory up to 2 TB, but never seems to get beyond 4 GB on my system at home. Are you saying that Windows doesn't use 64-bit paging, or are you saying that of the 32-bit variety?

Again, thanks man, that was a huge push in the right direction in understanding paging.
Post 10 Dec 2014, 15:14
View user's profile Send private message Reply with quote
l_inc



Joined: 23 Oct 2009
Posts: 881
l_inc
smiddy
Quote:
I traverse between 16-bit and 32-bit in order to get to the BIOS interrupts

Virtual 8086 is normally used for that. But it's unavailable in 64-bit mode. Thus I think, the only way to use BIOS interrupts would be to run them in non-root operation as unrestricted guest. Thus in a general case, if you wanna have a 64-bit OS, you should forget about BIOS interrupts (maybe other than at early bootup). It is possible however to run custom 16-bit code in compatibility mode in a standard 16-bit segment (code segment descriptor bits L=0 and D=0).
Quote:
as I can "visually see" from the diagram you provided, the physical memory from 0 to x, can be used by a page and accessing the data therein should be ok

Sure, but to make it clear: a page does not use memory, it provides a way to access the memory. Thus if you wanna access physical memory at address 0, you must create an entry in the page tables for that.
Quote:
so long as you adhere to the RAM/ROM possibilities, that come from the E820 memory map, PCI devices, and CMOS

It doesn't matter what underlies the physical address regions. If you want to access PCI MMIO, you need to create the corresponding page table entries the same way you'd do that for RAM.
Quote:
I have wondered why Windows (Windows 7 64-bit) says it can use physical memory up to 2 TB, but never seems to get beyond 4 GB on my system at home

The RAM usage restrictions in Windows may be imposed by the Windows edition. However there's no 64-bit Windows edition that is limited to 4GB (even Home Basic allow for up to 8 GB).
Quote:
Are you saying that Windows doesn't use 64-bit paging

There's no other choice for a 64-bit OS. It has to use 64-bit paging (which is called IA-32e paging in the Intel manuals).

Quote:
are you saying that of the 32-bit variety?

I was talking about 32-bit Windows with PAE explicitly disabled. Then it's not possible to access physical addresses above 4GB other than to mess with page tables directly (Windows doesn't seem to make use of PSE-36).

_________________
Faith is a superposition of knowledge and fallacy
Post 12 Dec 2014, 00:26
View user's profile Send private message Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  
Goto page Previous  1, 2

< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2020, Tomasz Grysztar. Also on GitHub, YouTube, Twitter.

Website powered by rwasa.