flat assembler
Message board for the users of flat assembler.

Index > OS Construction > Accessing real mode data in long mode

Goto page 1, 2  Next
Author
Thread Post new topic Reply to topic
Aulendil



Joined: 13 Jul 2014
Posts: 14
Aulendil 24 Nov 2014, 21:20
So in my OS I have 'saved' some data while in real mode at memory locations such as:

some_data dw 0
more_data dd 0

When I switch to protected then long mode, how would one access data in those memory locations? I may be missing something really obvious but I don't think this is possible due to having setup paging.

My kernel is loaded at 0000:1800 into memory,then I proceed with setting up a gdt, paging and an idt. I am using 1mb pages (one for kernel, one for user).
Post 24 Nov 2014, 21:20
View user's profile Send private message Reply with quote
l_inc



Joined: 23 Oct 2009
Posts: 881
l_inc 25 Nov 2014, 00:06
Aulendil
x86/x64 do not have 1MB pages. Large pages are 4MB for 32-bit paging and 2MB for PAE and 64 bit.

In order to access your "real mode variables", you should set up gdt/ldt and page tables so that the translation results in the same physical address you were using to access those variables in real mode. If you were able to enter the long mode, this shouldn't make any problems for you.

_________________
Faith is a superposition of knowledge and fallacy
Post 25 Nov 2014, 00:06
View user's profile Send private message Reply with quote
smiddy



Joined: 31 Oct 2004
Posts: 557
smiddy 05 Dec 2014, 16:43
I_inc is there a tutorial on paging that explains this somewhere (I willl be searching soon, just thought you may know of a good one before I search)? Up until recently I never considering using paging, but now I want to use 64 bit, which kind of forces my hand into doing paging, based on the few things I've read on the subject. Like Aulendil's question, here are a few others:

How do pages get mapped, specifically to the real memory locales?
In running code in paging, do you jump into a page, and run your OS (or ring 0 code) from there? Is it paged memory, or does the OS continue to run in non-paged environment?
For variables in real mode, or segmented memory or 16-bit memory realm, there is a potential problem within the first 2 MB of memory, where there are ROMs, Video RAM buffer, and BIOS, do you still map that 2 MB as continuous memory area, in a specific page?
How do you get out of paging, and back to running in a specific physical memory area?

I have several more, and will post those questions once I re-post in one of the other posts where I owe some code.

Smiddy
Post 05 Dec 2014, 16:43
View user's profile Send private message Reply with quote
l_inc



Joined: 23 Oct 2009
Posts: 881
l_inc 06 Dec 2014, 00:34
smiddy
Quote:
I willl be searching soon, just thought you may know of a good one before I search

I'm not gonna try to be ingenious here and would suggest the official documentation. If you take the Intel's one, then you need to read the first five chapters of the Volume 3. The Chapter 4 is specifically dedicated to paging, obviously.

Quote:
How do pages get mapped, specifically to the real memory locales?

You'll find that quite comprehensively explained in the manual. But remember, that page tables translate to physical addressing space, not to physical memory. There is one more level of indirection between physical addressing space and physical memory called memory controller hub, which combines the device MMIO and possibly different parts of the physical memory and maps all of it to the physical addressing space.
Quote:
In running code in paging, do you jump into a page, and run your OS (or ring 0 code) from there?

If you jump somewhere, you normally don't bother with the details of paging, but just jump. Unless you write highly optimized code, taking TLBs and caches into account. The time to think of page tables normally comes only when you need to create or modify them.

Quote:
Is it paged memory, or does the OS continue to run in non-paged environment?

Be careful with the wording, because "paged memory" often refers to memory that is a subject to offloading to a hard drive.

When paging is enabled, all the code works with logical addresses that are a subject to a logical-to-linear address translation defined by GDT/LDT and then linear-to-physical address translation defined by the page tables. The only values in the whole system that remain physical addresses are the values from the page tables themselves and the pointer to the page tables located in the CR3 register. Well, you'd also need to deal with physical addresses when programming DMA.

Quote:
there is a potential problem within the first 2 MB of memory, where there are ROMs, Video RAM buffer, and BIOS, do you still map that 2 MB as continuous memory area

It's not necessarily 2 MBs and not only the first MBs. The standard procedure of booting an OS includes requesting the memory map from the BIOS at the very beginning. The memory map describes free and occupied memory regions. Your systems memory manager is free to use the free memory regions by creating entries for them in the page tables.

Quote:
How do you get out of paging, and back to running in a specific physical memory area?

A well-written OS does not do that. But the procedure is just the reverse of enabling the paging. You'll need an idempotent mapping at least for the pages with the code that performs the switching procedure.

_________________
Faith is a superposition of knowledge and fallacy
Post 06 Dec 2014, 00:34
View user's profile Send private message Reply with quote
smiddy



Joined: 31 Oct 2004
Posts: 557
smiddy 06 Dec 2014, 02:51
Thanks, you've given me a lot to think about with loads of homework ahead. With 64-bit memory addressing just became very complicated (for me). Your last sentence:
Quote:
You'll need an idempotent mapping at least for the pages with the code that performs the switching procedure.
This is certainly the intention, but more specific mapping, which is why I asked the questions. My intention is the be able to move between the different modes of operation, when needed, which will require very specific mapping in order to keep it all straight.

Off to read those Intel chapters. I order the CDs probably four years ago, never had the opportunity crack them open until recently. Again, thanks!
Post 06 Dec 2014, 02:51
View user's profile Send private message Reply with quote
smiddy



Joined: 31 Oct 2004
Posts: 557
smiddy 07 Dec 2014, 04:52
I got a chance to read most of chapter 4. I find the semantics a little different from your vernacular, but I get it. Using strictly 64 (or 48 to 52) bits is a one way street, so to speak. The paging mechanism is pretty complex so I will be re-reading it several times. Along with the surrounding information.
Post 07 Dec 2014, 04:52
View user's profile Send private message Reply with quote
l_inc



Joined: 23 Oct 2009
Posts: 881
l_inc 07 Dec 2014, 16:37
smiddy
Quote:
I find the semantics a little different from your vernacular

Could you, please, specify, what exactly you mean by that? I normally try to stick to the Intel terminology and to be as precise as possible.

_________________
Faith is a superposition of knowledge and fallacy
Post 07 Dec 2014, 16:37
View user's profile Send private message Reply with quote
Aulendil



Joined: 13 Jul 2014
Posts: 14
Aulendil 07 Dec 2014, 21:51
Thanks for all the info you have posted l_inc - very useful, and you have given me a lot to think about as well.
Post 07 Dec 2014, 21:51
View user's profile Send private message Reply with quote
BAiC



Joined: 22 Mar 2011
Posts: 272
Location: California
BAiC 07 Dec 2014, 21:54
l_inc: Intel uses Identity mapped where you use idempotent mapped.
Post 07 Dec 2014, 21:54
View user's profile Send private message Visit poster's website Reply with quote
l_inc



Joined: 23 Oct 2009
Posts: 881
l_inc 07 Dec 2014, 22:56
BAiC
Oh, thank you. Even though I don't think Intel is the authority for creating own definitions of general mathematical concepts, I must admit that "identity-mapping" is here much more appropriate to say.

_________________
Faith is a superposition of knowledge and fallacy
Post 07 Dec 2014, 22:56
View user's profile Send private message Reply with quote
smiddy



Joined: 31 Oct 2004
Posts: 557
smiddy 08 Dec 2014, 05:36
The schema on memory, in general. BAiC caught the one that stands out most. I think Intel uses Linear Addressing as a coin for a virtual memory space, to a physical address (in my original put, real memory, not to be confused with the mode). Even within Chapter 3 Intel confuses by interchanging terminology, especially going down the path of segmentation, and using linear address space, then switching to strictly paging for 64-bit or long mode (not compatibility). The pictorials are not very intuitive though, so keeping four items straight for this one mode will be cumbersome, but has to be done, in order to switch between the modes.

In between reading this (not applying it, yet), I working on IDE Controller and AHCI Controller, oh and my regularly scheduled life and work. Smile
Post 08 Dec 2014, 05:36
View user's profile Send private message Reply with quote
l_inc



Joined: 23 Oct 2009
Posts: 881
l_inc 08 Dec 2014, 13:02
smiddy
Quote:
I think Intel uses Linear Addressing as a coin for a virtual memory space, to a physical address

This statement doesn't make any sense to me. The term "virtual memory space" doesn't exist. There are "virtual address" (which is out of scope of the Intel's documentation, because it should be defined by the OS), "virtual memory" (which is not a part of a CPU architecture and is only mentioned by Intel as something that can be implemented by an OS using paging) and "address space" (which is just a set of all possible numeric values of an address of whatever kind).
Quote:
Even within Chapter 3 Intel confuses by interchanging terminology

I don't think, they confuse anything. Please provide a couple of statements from the documentation that, as you think, are inconsistent.

_________________
Faith is a superposition of knowledge and fallacy
Post 08 Dec 2014, 13:02
View user's profile Send private message Reply with quote
smiddy



Joined: 31 Oct 2004
Posts: 557
smiddy 08 Dec 2014, 14:39
I should probably not write this, because I get the impression this is not going to go well.

l_inc wrote:
smiddy
Quote:
I think Intel uses Linear Addressing as a coin for a virtual memory space, to a physical address

This statement doesn't make any sense to me. The term "virtual memory space" doesn't exist. There are "virtual address" (which is out of scope of the Intel's documentation, because it should be defined by the OS), "virtual memory" (which is not a part of a CPU architecture and is only mentioned by Intel as something that can be implemented by an OS using paging) and "address space" (which is just a set of all possible numeric values of an address of whatever kind).
Quote:
Even within Chapter 3 Intel confuses by interchanging terminology

I don't think, they confuse anything. Please provide a couple of statements from the documentation that, as you think, are inconsistent.


Virtual memory space very much does exist: https://software.intel.com/en-us/articles/recap-virtual-memory-and-cache#Virtual_Memory

It is simply a definition of something not physical, in this case memory. This is intertwines in using the term linear address space.

It is my way of making a comparison to "other" terminology too. Does that make sense now?

BTW, I said they confuse me, not you (that is implied in my sentence). Is Intel inconsistent? I don't know, which is why it confuses, comprende? Why did you take it personal? I wasn't attacking you, based on your approach I'm assuming. Perhaps you are not, don't know. I am not saying how you interpret or understand their document. Your initial terminology was different, and I said so, and I said I understood. It is ok to be fuzzy...
Post 08 Dec 2014, 14:39
View user's profile Send private message Reply with quote
l_inc



Joined: 23 Oct 2009
Posts: 881
l_inc 08 Dec 2014, 16:34
smiddy
Quote:
Your initial terminology was different

That's the point that it was not. You thought differently and it was a sign for me that you needed some help in understanding. I didn't try to offend you and by no means felt offended. You can safely assume, that I'm just trying to be helpful.

No, it does not. The link points to the description of "virtual memory", not "virtual memory space". It seems that you didn't read the first sentence of my previous post completely.
Quote:
BTW, I said they confuse me, not you (that is implied in my sentence)

I just wanna make sure you don't have to decide who you trust more. If I say something incompatible to the Intel manuals, chances are that I'm wrong. If you think that I say something incompatible to the Intel manuals, it's important for you to know, why you are wrong. Smile
Quote:
Why did you take it personal?

I should probably start to embed more smilies into my posts. It's strange that people interpret emotionless posts as a sign of personal offense.

_________________
Faith is a superposition of knowledge and fallacy
Post 08 Dec 2014, 16:34
View user's profile Send private message Reply with quote
smiddy



Joined: 31 Oct 2004
Posts: 557
smiddy 08 Dec 2014, 21:04
Intel Chapter 3 wrote:
3.1 MEMORY MANAGEMENT OVERVIEW
The memory management facilities of the IA-32 architecture are divided into two parts: segmentation and paging.
Segmentation provides a mechanism of isolating individual code, data, and stack modules so that multiple
programs (or tasks) can run on the same processor without interfering with one another. Paging provides a mechanism
for implementing a conventional demand-paged, virtual-memory system where sections of a program’s
execution environment are mapped into physical memory as needed. Paging can also be used to provide isolation
between multiple tasks. When operating in protected mode, some form of segmentation must be used. There is
no mode bit to disable segmentation. The use of paging, however, is optional.
These two mechanisms (segmentation and paging) can be configured to support simple single-program (or singletask)
systems, multitasking systems, or multiple-processor systems that used shared memory.
As shown in Figure 3-1, segmentation provides a mechanism for dividing the processor’s addressable memory
space (called the linear address space) into smaller protected address spaces called segments. Segments can
be used to hold the code, data, and stack for a program or to hold system data structures (such as a TSS or LDT).
If more than one program (or task) is running on a processor, each program can be assigned its own set of
segments. The processor then enforces the boundaries between these segments and insures that one program
does not interfere with the execution of another program by writing into the other program’s segments. The
segmentation mechanism also allows typing of segments so that the operations that may be performed on a particular
type of segment can be restricted.
All the segments in a system are contained in the processor’s linear address space. To locate a byte in a particular
segment, a logical address (also called a far pointer) must be provided. A logical address consists of a segment
selector and an offset. The segment selector is a unique identifier for a segment. Among other things it provides an
offset into a descriptor table (such as the global descriptor table, GDT) to a data structure called a segment
descriptor. Each segment has a segment descriptor, which specifies the size of the segment, the access rights and
privilege level for the segment, the segment type, and the location of the first byte of the segment in the linear
address space (called the base address of the segment). The offset part of the logical address is added to the base
address for the segment to locate a byte within the segment. The base address plus the offset thus forms a linear
address in the processor’s linear address space.
3-2 Vol. 3A
PROTECTED-MODE MEMORY MANAGEMENT
If paging is not used, the linear address space of the processor is mapped directly into the physical address space
of processor. The physical address space is defined as the range of addresses that the processor can generate on
its address bus.
Because multitasking computing systems commonly define a linear address space much larger than it is economically
feasible to contain all at once in physical memory, some method of “virtualizing” the linear address space is
needed. This virtualization of the linear address space is handled through the processor’s paging mechanism.
Paging supports a “virtual memory” environment where a large linear address space is simulated with a small
amount of physical memory (RAM and ROM) and some disk storage. When using paging, each segment is divided
into pages (typically 4 KBytes each in size), which are stored either in physical memory or on the disk. The operating
system or executive maintains a page directory and a set of page tables to keep track of the pages. When a
program (or task) attempts to access an address location in the linear address space, the processor uses the page
directory and page tables to translate the linear address into a physical address and then performs the requested
operation (read or write) on the memory location.
If the page being accessed is not currently in physical memory, the processor interrupts execution of the program
(by generating a page-fault exception). The operating system or executive then reads the page into physical
memory from the disk and continues executing the program.
When paging is implemented properly in the operating-system or executive, the swapping of pages between physical
memory and the disk is transparent to the correct execution of a program. Even programs written for 16-bit IA-
32 processors can be paged (transparently) when they are run in virtual-8086 mode.


Virtual Address Space, not verbatim, but inferred when this is all put together, virtual memory environment (a mechanism) where linear address space is a simulation, I combined the two, virtual address space, my vernacular...

YMMV (your mileage may vary)
Post 08 Dec 2014, 21:04
View user's profile Send private message Reply with quote
l_inc



Joined: 23 Oct 2009
Posts: 881
l_inc 08 Dec 2014, 21:37
smiddy
Virtual address space is by definition a set of all possible numeric values for a virtual address (see my previous posts). As I already said before, virtual address is defined by an OS. E.g. see here.

Inventing your own definitions won't help you understand the manuals.

_________________
Faith is a superposition of knowledge and fallacy
Post 08 Dec 2014, 21:37
View user's profile Send private message Reply with quote
smiddy



Joined: 31 Oct 2004
Posts: 557
smiddy 08 Dec 2014, 23:20
I think it does, it is a process towards learning. Linear address space is a part of the virtual memory environment, the linear addresses are virtual addresses, and they occupy space, hence the numeric values keeping track of everything to the physical addresses. The linear addresses protected for anything, are virtualized, such that everything has a 0 thru [SIZE] memory area, in which to operate, not directly addressing the memory locales in the physical memory. That's how I'm reading it...

My original question, in paging (whoops, HDD gets paged, which is not what I'm saying here, as you pointed out above, there's a difference in a page table; which a page goes to disk and memory, and I am refering to paging as the mechanism, which is not correct), does the OS begin operating in a linear address space, from 0 to 2 MB say, if you assign that physical address to the OS's operating page from 0 to 2 MB? This is still unclear to me. So, if you wanted the physical memory locales 1 MB to 1.25 MB, because you set them up with real mode data, the you could set up a linear address space, from 1 MB to 1.5 MB, and its address range seen by the code would be 0 to 0.25 MB (I know the page size is not correct, but the idea and mechanization thereof).

Asked another way, can linear address space point to the physical address space such that they are aligned the same, address 0 linear address space equals 0 physical memory address? [EDIT] And still see the original contents prior to being placed in that virtual environment?

I will be rereading chapter 4 again later this evening.
Post 08 Dec 2014, 23:20
View user's profile Send private message Reply with quote
smiddy



Joined: 31 Oct 2004
Posts: 557
smiddy 08 Dec 2014, 23:23
BTW, is a paging file "required" to work in this virtual environment?
Post 08 Dec 2014, 23:23
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 20295
Location: In your JS exploiting you and your system
revolution 08 Dec 2014, 23:44
smiddy wrote:
BTW, is a paging file "required" to work in this virtual environment?
You would need to define "virtual environment" first. Generally speaking paging files are used to store parts of memory in disc but it is also possible to not store anything and simply make all virtual memory be "reloaded" as zeros (or some other fixed set of constant values) each time it is used. I don't know of a use case for such a thing but the OS could be programmed for that if wanted. Although in a way memory mapped files are a type of virtual memory that is not using a specific paging file.
Post 08 Dec 2014, 23:44
View user's profile Send private message Visit poster's website Reply with quote
l_inc



Joined: 23 Oct 2009
Posts: 881
l_inc 09 Dec 2014, 00:26
smiddy
Quote:
it is a process towards learning

It is a process towards confusion.
Quote:
Asked another way

Another way is good, cause understanding the former way requires profound skills in cryptanalysis. Smile
Quote:
can linear address space point to the physical address space such that they are aligned the same, address 0 linear address space equals 0 physical memory address?

That's what's called "identity mapping". Sure it can.
Quote:
And still see the original contents prior to being placed in that virtual environment?

Are you suggesting to access physical memory using the same physical addresses, but obtain different data? As I said before, there's the memory controller hub between the physical address space and physical memory. If you'd be able to program the MCH (which you are normally not, cause BIOS does that and locks the corresponding settings) it would allow you to do that for some memory ranges and at specific circumstances, but not in general. In general, you should assume that if you write something to the physical address X that is mapped to RAM, then you overwrite the data previously saved at physical address X.
Quote:
BTW, is a paging file "required" to work in this virtual environment?

A page file is just an implementation detail for virtual memory. Neither a page file is required for implementing virtual memory, nor virtual memory is required to be implemented by an OS.

revolution
Quote:
simply make all virtual memory be "reloaded" as zeros (or some other fixed set of constant values) each time it is used

This would break the concept of virtualization, which prescribes to create an illusion of something real.

_________________
Faith is a superposition of knowledge and fallacy
Post 09 Dec 2014, 00:26
View user's profile Send private message Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  
Goto page 1, 2  Next

< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2024, Tomasz Grysztar. Also on GitHub, YouTube.

Website powered by rwasa.