flat assembler
Message board for the users of flat assembler.

Index > OS Construction > Best Memory Model

Goto page Previous  1, 2, 3  Next

Which memory model is the best?
Paged
40%
 40%  [ 6 ]
Paged with Segmentation (segments within pages)
0%
 0%  [ 0 ]
Flat
60%
 60%  [ 9 ]
Total Votes : 15

Author
Thread Post new topic Reply to topic
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17270
Location: In your JS exploiting you and your system
revolution
tom tobias wrote:
Paging
1. definition: Operating System requirement (not option) to copy portions of task FROM hard disk TO system memory, AND then, to return unused portions of the task BACK to the hard disk again.
2. Philosophy: this is (yet another) example of wasteful, and utterly stupid activity (not evil) which is derived from the ERA of EXPENSIVE system memory. At that time, 1960's, no one could imagine a computer with 2 GB memory. How many tasks require that much memory? Apart from Astronomers and Mathematicians, who else is using 2 GB of memory, whether on a hard disk, (paged into memory) or sitting entirely in memory, for a single task?
3. Alternative: Load the whole task into main memory, one time. No further requirement to "page", hence, no overhead at all. Yes, it will take longer to COMMENCE the task, compared with a paged model.
tom tobias: Your definition for paging is not correct. What you have described is virtual memory.

Paging is the reordering of physical addresses to virtual addresses.

[edit]Quoted tom tobias' text because it went to the next page.[/edit]
Post 15 Sep 2008, 11:16
View user's profile Send private message Visit poster's website Reply with quote
sinsi



Joined: 10 Aug 2007
Posts: 693
Location: Adelaide
sinsi
Paging used by itself has no real overhead, since every memory access is verified by the CPU anyway.
Paging used with virtual memory is the killer - HDD access is so s l o w.

Either do a 1-to-1 map of all memory and forget paging or do it the Windows way. It depends on what you want your OS to do (multitask etc.).
Post 15 Sep 2008, 11:29
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17270
Location: In your JS exploiting you and your system
revolution
I just want to make a note here. Using virtual memory will NOT automatically make your OS slower. It will only be slower when your apps are trying to use more memory than you have available in RAM. IF you are perpetually running apps that force the OS to make use of virtual memory then it is time to buy more RAM. However, IF your computer spends most it's time needing less memory than you have RAM then buying extra RAM for just the occasional task that is greedy is probably not cost effective in most cases.

The only alternative to VM is to have each and every app that needs lots of RAM to have written, as part of their own code, routines to open a file on disk and use that file as it's own private swap file. As most people will automatically spot here, it is naive to expect every programmer that writes code to also include VM code just in case the user is low on RAM. It is by far, much more efficient and to have the OS handle the shortage of memory without the app needing to worry about such minutiae.

My personal experience here is with fasm. I have one job that requires >700M to assemble. If I disabled the OS VM then it would not be possible for me to assemble this job at all. The only way around it in this case would be to have fasm include code to swap memory to a temporary disc file. This would increase to opportunity for bugs in fasm and also increase the time to write it.
Post 15 Sep 2008, 11:44
View user's profile Send private message Visit poster's website Reply with quote
neville



Joined: 13 Jul 2008
Posts: 507
Location: New Zealand
neville
The common use of the same or very similar words to mean completely different things is very unfortunate.

For example, "virtual address" is often used to mean the relocated memory address defined by the page translation process. This of course is completely different to "virtual memory" (VM) which is hard disk storage space defined by the disk swapping/caching process.

Virtual Memory is unfortunately an established term (but is not memory at all), so instead of "virtual address" it would be better to use an alternative word, such as "logical address". So "paging" translates logical addresses (or logical memory) to physical addresses or physical memory, but has nothing to do with virtual addresses or virtual memory.

We also need to consider the meanings of flat memory, segmented memory and linear memory. IMO flat memory is actually segmented memory where the segment is set to 0. This is what flat real mode is, but flat memory can still be paged i.e. flat memory can be relocated between logical addresses and actual physical memory addresses.

Linear memory is often used to mean the same thing as flat memory, but linear addresses are really the combined value of the segment and offset parts of segmented memory.


Now just to repond again to revolution:
Your suggested alternative to VM is exactly what I have proposed in FAMOS - if an application programmer really wants to use virtual memory for any reason then FAMOS won't stop him from doing it (because FAMOS imposes very few restrictions on programmers) but the system software provides no support for it.

I regard using VM a bit like "living beyond your means" - when you have to borrow from the bank when you don't have enough. But it is better to either wait until you do have enough, or change your habits to use what you've got. Actually you don't have to wait very long because technology moves very fast. This way you don't have to use incredibly slow old technology which will slow your application down.

In a preemptive multi-tasking OS if an individual application is using its own VM, it will have no effect on the performance of the OS. But if the OS also uses VM for its own system utilities then it will affect the performance of the OS. Of course if the OS is not preemptive then a slow application can also slow down the entire OS.

_________________
FAMOS - the first memory operating system
Post 16 Sep 2008, 20:57
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17270
Location: In your JS exploiting you and your system
revolution
neville wrote:
I regard using VM a bit like "living beyond your means" - when you have to borrow from the bank when you don't have enough. But it is better to either wait until you do have enough, or change your habits to use what you've got. Actually you don't have to wait very long because technology moves very fast.
But I need my job assemble NOW. I can't afford to wait 2-3 years for technology to catch up, my boss would fire me. Let the computer do the work and pretend there is extra RAM, it costs me nothing except a little bit of time (and I can do other things while waiting so the time is not really lost).
Post 16 Sep 2008, 21:05
View user's profile Send private message Visit poster's website Reply with quote
neville



Joined: 13 Jul 2008
Posts: 507
Location: New Zealand
neville
Tut tut, revolution you're just too impatient Wink

How much memory do you need N.O.W ? 4GB? 4TB??

Besides, I'm surprised your boss hasn't fired you already, you spend so much time here Laughing

_________________
FAMOS - the first memory operating system
Post 17 Sep 2008, 00:13
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17270
Location: In your JS exploiting you and your system
revolution
neville wrote:
Tut tut, revolution you're just too impatient Wink
"I want it all and I want it now"
neville wrote:
How much memory do you need N.O.W ? 4GB? 4TB??
Huh, only 4TB! I want 1023 YiB
neville wrote:
Besides, I'm surprised your boss hasn't fired you already, you spend so much time here Laughing
Shh, don't say that out loud. Keep it to yourself please. Okay?
Post 17 Sep 2008, 06:48
View user's profile Send private message Visit poster's website Reply with quote
tom tobias



Joined: 09 Sep 2003
Posts: 1320
Location: usa
tom tobias
revolution wrote:
Your definition for paging is not correct. What you have described is virtual memory....
Hmm. Appears to be time for a rebuttal.
Operating Systems Lecture Notes Lecture 10 Issues in Paging and Virtual Memory:
Page Table Structure. Where to store page tables, issues in page table design.
In a real machine, page tables stored in physical memory. Several issues arise:
1. How much memory does the page table take up?
2. How to manage the page table memory. Contiguous allocation? Blocked allocation? What about paging the page table?
On TLB misses, OS must access page tables. Issue: how the page table design affects the TLB miss penalty.
Real operating systems provide the abstraction of sparse address spaces. Issue: how well does a particular page table design support sparse address spaces?
Linear Page Table.
1. OS now faces variable-sized allocation problem for the page table storage.
2. Page table may occupy significant amount of physical memory. Page table for 32-bit address space with 4K byte pages has 232 / 212 = 220 entries. If each entry is 32 bits, need 4M bytes of memory to store page table.
3. Does not support sparse address spaces well - too many wasted entries.
4. TLB miss handler is very simple - just index the page table.
Two-Level Page Table. Page Table itself is broken up into pages. An outer page table indexes pages of page table. Assuming 4K byte pages and 32 byte page table entries, each page can hold 212 / 4 = 210 entries. There are 10 bits left in virtual address for index of outer page table. Virtual address now has 10 bit outer page table index, 10 bit inner page table offset and 12 bit page offset......
Question: Was this 1960's approach--Paging-- required because the top of the line IBM desktop computers in 1975 (Model 5100, costing only $18,000) contained a maximum of 64 kilobytes of memory? (The less expensive version with only half as much memory, cost 1/3 less, 12K dollars.) Has textbook style computer science, as spouted and touted by "revolution", (and doubted by me) evolved (very NON-revolutionary) dogma based on the supposition that a single computer, for one person's use, containing 2 gigabytes of memory, would cost millions of dollars?
Today, 2 gb of DDR II/667MHz costs less than Twenty five dollars, but we are still writing software using 1975 pricing/scarcity models.
WE ARE ENGAGED IN OBSOLETE SOFTWARE DESIGN, arguing about the crucial importance of paging, as though it were still 1975. The name for that behavioural profile is REACTIONARY, (i.e. status quo preserving) not revolutionary.
Post 17 Sep 2008, 10:40
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17270
Location: In your JS exploiting you and your system
revolution
tom tobias: Memory requirements of a lot of software packages today still outstrips available memory. How else to cope with it other than applying OS virtual memory or forcing each and every app to provide it's own private virtual memory? And I predict that this will always be the case no matter how abundant or how cheap memory becomes, we will always want to use more and have to use methods like VM to overcome the shortage.

Your quote above does not rebut my statement, sorry, but paging and virtual memory are not the same thing.
Post 17 Sep 2008, 11:06
View user's profile Send private message Visit poster's website Reply with quote
neville



Joined: 13 Jul 2008
Posts: 507
Location: New Zealand
neville
tom tobias, I think you're agreeing 100% with my earlier post, except memory paging .ne. virtual memory. You can have paging without VM, and theoretically you could have VM without paging i.e. a flat implementation of VM, but VM usually uses paged memory.

revolution, I think there is a time when memory availability = memory needs/wants, and that time is NOW! Should we really choose to keep our computers dependent on incredibly slow rotating machinery?

_________________
FAMOS - the first memory operating system
Post 18 Sep 2008, 03:56
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17270
Location: In your JS exploiting you and your system
revolution
neville wrote:
revolution, I think there is a time when memory availability = memory needs/wants, and that time is NOW! Should we really choose to keep our computers dependent on incredibly slow rotating machinery?
Hmm, I think a lot of people here seriously misunderstand how VM works. Folks, virtual memory is only used when it is needed. If all your programs running use less than the total RAM installed (less a little overhead) then you are not using any VM.
Post 18 Sep 2008, 04:25
View user's profile Send private message Visit poster's website Reply with quote
neville



Joined: 13 Jul 2008
Posts: 507
Location: New Zealand
neville
revolution wrote:
Hmm, I think a lot of people here seriously misunderstand how VM works. Folks, virtual memory is only used when it is needed. If all your programs running use less than the total RAM installed (less a little overhead) then you are not using any VM.

If only that were true of all VM OS's.

Some VM OS's (without mentioning any by name) have such inefficient VM management that they even cache their own system components and then fail to find them in the cache even when RAM is less than 20% utilised, resulting in needless disk accesses.

Also, despite what some VM OS's say, it is impossible to fully disable their VM, because they keep their system caches at all times.

Furthermore, some applications which run under those same VM OS's, REFUSE to run if the VM is turned off, even when RAM usage is also less than 20% and the application with all it's plugins and data would need no more than another 20% of available RAM.

You can prove this yourself with some simple tests - would you like the OS and application names? Wink

_________________
FAMOS - the first memory operating system
Post 18 Sep 2008, 06:01
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17270
Location: In your JS exploiting you and your system
revolution
neville: All your arguments are OS specific. I agree that some OSes are very bad at how they handle memory (all memory, not just VM), but this should not detract from the concept of VM.
Post 18 Sep 2008, 06:09
View user's profile Send private message Visit poster's website Reply with quote
neville



Joined: 13 Jul 2008
Posts: 507
Location: New Zealand
neville
revolution wrote:
neville: All your arguments are OS specific. I agree that some OSes are very bad at how they handle memory (all memory, not just VM),


Agreed Smile

VM increases the complexity of the memory management task out of sight, which is why it is almost inevitable that VM will adversely affect non-VM management, and the OS as a whole.

Saying NO to VM, by contrast, leaves the OS with a beautifully simple memory management task, which will be highly efficient and very fast.

So, its easy to ...
Quote:
detract from the concept of VM
Wink

_________________
FAMOS - the first memory operating system
Post 18 Sep 2008, 07:52
View user's profile Send private message Visit poster's website Reply with quote
tom tobias



Joined: 09 Sep 2003
Posts: 1320
Location: usa
tom tobias
revolution wrote:
...this should not detract from the concept of VM.
I thought you were the one arguing that VM is distinct from paging:
revolution wrote:
paging and virtual memory are not the same thing.
revolution wrote:
Folks, virtual memory is only used when it is needed. If all your programs running use less than the total RAM installed (less a little overhead) then you are not using any VM.
Adam asked about PAGING. My argument is that PAGING implemented into the kernel of an operating system will result in degraded performance, EVEN IF THERE IS NO REQUIREMENT for virtual memory. I sought to explain that above, but evidently, not to revolution's satisfaction. Let me try a different source:

In computer operating systems that have their main memory divided into pages, paging (sometimes called swapping) is a transfer of pages between main memory and an auxiliary store, such as hard disk drive.[1] Paging is an important part of virtual memory implementation in most contemporary general-purpose operating systems, allowing them to use disk storage for data that does not fit into physical RAM. Paging is usually implemented as architecture-specific code built into the kernel of the operating system....The main functions of paging are performed when a program tries to access pages that do not currently reside(present) in RAM. This situation is known as a page fault. The Operating System must then take control and handle the page fault, in a manner invisible to the program.
Revolution's perspective is that IF an application can fit into the existing memory, then there will be no page faults, hence no virtual memory requirement, whereas my point of view, on the contrary, is that REGARDLESS of the size of the application, there will be overhead, inconvenience, and code obfuscation creating page tables to swap page tables to swap memory, built into the kernel, and that is neither easy to implement, nor to test, nor to debug, compared with a flat model, that ignores paging, and simply REQUIRES the user to ensure sufficient memory for the applications running on the computer.

Again, in my opinion, PAGING exists SOLELY as a manifestation of a non-existant problem: lack of volatile memory. There will surely be, in Adam's nascent operating system, enough OTHER bugs to solve, without introducing the complexity of paging. Yes, revolution is surely correct, upon completion of his operating system, Adam will have a profoundly impaired operating system, one which can ONLY function correctly, provided it operates on machines with sufficient memory. Hmm. Where else do we see this dilemma? Oh, how about EVERY part of society, from banking to hospitals to military to art studios. Ever try painting a very large image using only a very small canvas? If your canvas is only 2 meters square, how will you be able to reproduce Diego Rivera? Must every artist employ 20 meter length canvas, because that's what he used? Yes, Adam's operating system, if he follows the advice of ignoring paging, will FAIL, upon commencing some application which exceeds the size of memory on that computer. To my way of thinking, that is a reasonable limitation. I consider the requirement to introduce paging in a hobby os as an unreasonable demand.
Smile
Post 18 Sep 2008, 09:56
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17270
Location: In your JS exploiting you and your system
revolution
Paging exists to facilitate the OS memory allocation in user space that are contiguous blocks and also provides a simple mechanism for process separation.

There is no requirement to place any RAM storage onto the HDD when using paging. However because historically paging has also been used to facilitate VM this causes many people to mistakenly assume that paging necessarily means transfer of data to/from the HDD.
Post 18 Sep 2008, 10:05
View user's profile Send private message Visit poster's website Reply with quote
tom tobias



Joined: 09 Sep 2003
Posts: 1320
Location: usa
tom tobias
revolution wrote:
Paging exists (1) to facilitate the OS memory allocation (2) in user space that are (3) contiguous blocks ...
Three good points which highlight the distinction between our respective ways of thinking about the creation of Adam's "hobby" operating system.
1. No need to "facilitate" OS memory allocation. Adam (or any other creator) will specify precisely where each of the different aspects of the OS are located. He has no need for paging to perform that task for him.
2. "user space" ??? Ooops. It is ALL user space. All 2-4 gbytes. The os is just a tool to manage the various tasks. "user space" is a 1960's era buzzword, when a single cpu and its tiny memory, had to be apportioned among a dozen different users simultaneously. Each user had his or her own (VERY TINY) private memory space from which to operate. The biggest computer memories in those days, mid 60's, were measured in hundreds of kilobytes. If ten users, each user received about 20 kb of space to use for his/her own task. Ten years later, in mid 70's, single desktop computer had 64 whole kilobytes of memory, none of which had to be shared with anyone else!!!!
3. "contiguous blocks", again, something important if one is copying data from memory to a hard drive, absolutely useless in the context of a new OS running autonomously on a dedicated cpu with a couple of gigabytes of memory available. Should he prefer to do so, Adam could easily separate each section of OS code with several blank few megabytes of space, to allow future modifications. There is absolutely no need for contiguity.
Paging, like its miserable stepchild, segmentation, is obsolete.
Post 18 Sep 2008, 17:55
View user's profile Send private message Reply with quote
neville



Joined: 13 Jul 2008
Posts: 507
Location: New Zealand
neville
tom tobias wrote:
Quote:
Again, in my opinion, PAGING exists SOLELY as a manifestation of a non-existant problem: lack of volatile memory.

Tom, you mean VIRTUAL MEMORY here, not PAGING. Have you seen my numerous previous posts in this thread trying to explain this Question Question

I could implement FLAT VIRTUAL MEMORY in 32-bit flat real FAMOS if I wanted to, but of course I would never do so because it would be contrary to the very ethos of FAMOS.

Apart from the overheads, inefficiencies and severe speed penalties it would prevent FAMOS from being a true MEMORY OS, because it would make FAMOS dependent on the existence and operation of particular mass storage devices i.e. disk drives. This is what sets FAMOS apart from the rest - all media access mechanisms are completely outside of the kernel. The kernel deals with MEMORY only - beautifully simple!

Tom, we are singing from the same song sheet, but please acknowledge the above Smile

Also, it must be realised (as far as I can tell at the moment) that 64-bit FAMOS will have to use PAGED memory because LONG MODE requires it! There is no flat 64-bit CPU mode AFAIK right now. But it definitely still won't use any kind virtual memory.

_________________
FAMOS - the first memory operating system
Post 18 Sep 2008, 20:03
View user's profile Send private message Visit poster's website Reply with quote
baldr



Joined: 19 Mar 2008
Posts: 1651
baldr
Correct me, please, if I'm wrong (all this regarding IA-32 without IntelĀ® 64 architecture and strictly is MHO):

1. Program code works with logical addresses (selector:offset).
2. Logical addresses are translated to linear addresses (32-bit) via segmentation. Flat memory model use several segment descriptors to map offsets in these segments 1:1 to linear addresses. These segments are zero-based, cover entire linear address space and always present (bit P==1 in segment descriptor).
3. Paging (not swapping!) further translates linear addresses to physical addresses. If it is disabled, mapping is also 1:1.

So we have: 46-bit logical address (artificially shortened to 32-bit for flat memory model) -> 32-bit linear address -> 32-bit physical address (PAE-36 and PSE-36 are out of scope for now).

Does it mean that with flat memory model and disabled paging all processes share single logical/linear/physical address space with full access rights to each other and the kernel?
I think, it does. No comments.
Post 20 Sep 2008, 18:42
View user's profile Send private message Reply with quote
neville



Joined: 13 Jul 2008
Posts: 507
Location: New Zealand
neville
Quote:
Correct me, please, if I'm wrong (all this regarding IA-32 without IntelĀ® 64 architecture and strictly is MHO):


I think you are correct, with the proviso that you have described IA-32 Protected Mode, but not IA-32 Flat Real mode. In FR mode there is only physical memory and physical addresses with 16-bit segments and 32-bit offsets. There is no logical memory in FR mode because translation (paging) is not implemented. The segments define fixed 20-bit base addresses = segment value * 16.

Quote:
Does it mean that with flat memory model and disabled paging all processes share single logical/linear/physical address space with full access rights to each other and the kernel?
I think, it does. No comments.

Yes, I think this is true too, certainly in FR Mode although (correct me if I'm wrong) in Pmode privilege levels 0-3 could still be used for segment access control even with paging disabled and using a "flat" model, but with segment limits set to less than 4GB.

Now I think I need to defend my use of FR mode in FAMOS Wink
In a hostile newtork environment of course it would be crashed very quickly.
And even a well-intentioned applications programmer could quite easily cause it to crash.
But I have only used FR mode in this first version of FAMOS to demonstrate the concept of a (32-bit) memory-based OS. FR mode has given me the freedom to concentrate on the overall design of the OS, rather than worry about issues such as security.
Of course, one day FAMOS will be given the best possible security which will probably be in the 64-bit Long Mode version.
In the meantime it is hoped and assumed that FAMOS will be used in friendly environments only. The 32-bit FAMOS Memory Editor ("FAME") has R/W access to all installed RAM so could be used to crash FAMOS very easily, or even to write an entirely new OS... For that reason only FAME "LE" (Limited Edition) is available on the FD/CD images I've posted. FAME LE is prevented from accessing "system" memory - the first 3 megabytes - but is still quite capable of crashing FAMOS Smile

_________________
FAMOS - the first memory operating system
Post 21 Sep 2008, 09:03
View user's profile Send private message Visit poster's website Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  
Goto page Previous  1, 2, 3  Next

< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2020, Tomasz Grysztar.

Powered by rwasa.