flat assembler
Message board for the users of flat assembler.

Index > Heap > Whats the limit of memory on windows?

Author
Thread Post new topic Reply to topic
xanatose



Joined: 09 Jan 2004
Posts: 57
xanatose
I need to make a program that takes huges amount of data an do calculations on them as fast as possible (thus no swapping) the processors speed of todays hardware are ok for the calculations, but I need at least 16GB of ram to put one month data, (ideally I would like to use a year of data, but im not sure on limits). I wonder, what are the limits of memory of an AMD64 in 64 bit windows, linux, or bsd. And where can I get a board that allows for huge amounts of memory.
Post 10 Jan 2008, 11:49
View user's profile Send private message Reply with quote
edfed



Joined: 20 Feb 2006
Posts: 4237
Location: 2018
edfed
16GB of ram???
and if you compress the datas? can you obtain a more convenient ram need?
Post 10 Jan 2008, 12:08
View user's profile Send private message Visit poster's website Reply with quote
coconut



Joined: 02 Apr 2004
Posts: 326
Location: usa
coconut
What is the memory limit for Windows XP Professional x64 Edition?

Windows XP Professional x64 Edition will support up to 128 GB of RAM. This is a limit imposed by the client OS. (Some x64 versions of Windows Server support larger amounts of memory.) Of course the hardware platform may support more or less RAM. For example, the xw4200 chipset only supports 4GB of RAM so you are limited by the system.
Post 10 Jan 2008, 16:21
View user's profile Send private message Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 2915
Location: [RSP+8*5]
bitRAKE
This is a very common question:
http://it.slashdot.org/article.pl?sid=08/01/01/0016211&from=rss

I'm not suggesting that you don't understand your work, but why must it be in memory all at once? Maybe, the data could be organized to improve average access time while having a very bad worst case? Is this some type of simulation?
Post 10 Jan 2008, 18:43
View user's profile Send private message Visit poster's website Reply with quote
Madis731



Joined: 25 Sep 2003
Posts: 2141
Location: Estonia
Madis731
If its 16GB random, then I get it - there's now way to cache or predict it. Some analysis are like that. I've never done myself any of these, but I heard about them in school and such. When the problem can't be solved, then read on Smile

Okay, back to the limits - the limit of 4GB came from the 32-bit mode where you couldn't address more. Btw, the 4GB isn't like when you put 4GB in and hope its true. Actually BIOS, peripherals need all be mapped in that same space (graphics take a lot sometimes w/ AGP aperture). It gets "eaten". There were some fixes like PAE (Physical Address Extension), but it only gave us this far - 64GB. Now with 64-bit OS-s there are no such limits and we reach them only in sizes like 17179869184GB or shortly 16EB (Exabytes). The memory isn't artificially shown to the user, but its the actual flat memory the OS sees itself. The limits in 2003 Server are a bit more real, though:
Wikipedia wrote:
The 64-bit version of Windows Server 2003, Enterprise Edition [64-bit] is capable of addressing up to 1 Terabyte (1024 GB) of RAM


I don't know why they even mention the values like this - they seem to namely disable bigger addressing capabilities, but that's OS-politics.
i.e. Standard (I'm behind one right now) supports "only" 32GB while two steps ahead, the datacenter says 2TB.
Post 10 Jan 2008, 22:51
View user's profile Send private message Visit poster's website Yahoo Messenger MSN Messenger Reply with quote
edfed



Joined: 20 Feb 2006
Posts: 4237
Location: 2018
edfed
i insist, if you need to have all datas on memory, you need to compress it.
and then the treatment will be done by a non windows code.
a boot code to be exact, because windows will slow down your program.

if you manage this really big amount of memory, you will need to make a bootable program able to manage cache L1,L2,L3(ram) and L4 (hdd)

speed improvement will be made depending on the needed algorythm and data structure...
Post 10 Jan 2008, 23:48
View user's profile Send private message Visit poster's website Reply with quote
Madis731



Joined: 25 Sep 2003
Posts: 2141
Location: Estonia
Madis731
@xanatose: Now that people got interested in the matter, maybe you could fill in the details. Then we know exactly how to conquer this problem.

And one more thing - you probably do need a server motherboard because usually there are only 4 slots on PC for RAM and the maximum one-chip size I could find was 2GB so in total 8GB. You need 4GB chips to work it out.
Post 11 Jan 2008, 06:42
View user's profile Send private message Visit poster's website Yahoo Messenger MSN Messenger Reply with quote
MazeGen



Joined: 06 Oct 2003
Posts: 975
Location: Czechoslovakia
MazeGen
Madis731 wrote:
Now with 64-bit OS-s there are no such limits and we reach them only in sizes like 17179869184GB or shortly 16EB (Exabytes).

For some reasons (to simplify current chip architecture etc.) the chip itself doesn't support full 64-bit address space. Both AMD64 and Intel 64 architectures today supports only 48 bits, so only 256TB can be physically reached. For some other reasons, today's OSes support even less Smile
Post 11 Jan 2008, 07:56
View user's profile Send private message Visit poster's website Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3170
Location: Denmark
f0dder
MazeGen: is the "only 48 bits" a physical ram or virtual address space limitation? Wikipedia says address space, but I don't trust WP to be 100% correct always, and I don't have time to look up the manuals right now Smile

Btw, 32bit XP used to support 4GB of physical memory - with XP SP2, this was changed to 4GB of address space instead. This means a lot if you need a "remap memory" option in your BIOS to use 4gb, since XP SP2 then only sees 2GB.
Post 11 Jan 2008, 10:15
View user's profile Send private message Visit poster's website Reply with quote
Feryno



Joined: 23 Mar 2005
Posts: 454
Location: Czech republic, Slovak republic
Feryno
Hi guys,
2^52 is architectural (AMD64) limit of physical memory (RAM), today CPUs have the limit even lower.
But one surprise - I read a news in AMD64 manual - newer AMD64 CPUs support 2-level paging with 1 GB pages (other possibilities are 3-level paging with 2 MB pages, 4-level paging with 4 kB pages)
So for mapping 16 GB of physical memory (RAM) into virtual space, you need only 2*4 kB of memory to construct paging tables (TLB). I think this is not implemented in today OS-es yet (they almost use 4-level with 4 kB pages, perhaps some of them are able to use 2 MB pages with 3-level paging mechanism). Just calculate how much memory you waste to map 16 GB of physical memory (RAM) with 4kB page_size (more than 32 MB!!!!!) and compare with 8kB required by 2-level method with 1GB page_size ! And the OS overload for handling more than 8000 tables in TLB !!!
Post 11 Jan 2008, 12:03
View user's profile Send private message Visit poster's website ICQ Number Reply with quote
edfed



Joined: 20 Feb 2006
Posts: 4237
Location: 2018
edfed
CPU without mother board is nothing, and i don't know any mother board that can contain more than 16*1Gb ram with a single prossessor.
Post 11 Jan 2008, 13:01
View user's profile Send private message Visit poster's website Reply with quote
MazeGen



Joined: 06 Oct 2003
Posts: 975
Location: Czechoslovakia
MazeGen
f0dder: Shit, I can't copypaste here the info from Intel manuals, they are locked now Sad

Intel says in Basic architecture manual that first implementation of Intel 64 architecture supports 48-bit linear address. In System programming manual is a small note that physical address size is implementation-specific and indicated by CPUID.80000008H.EAX[bits 7-0].

All what I found in AMD manuals is:

Application programming manual:
Quote:

2.2.2 64-Bit Canonical Addresses
Long mode defines 64 bits of virtual address, but implementations of the AMD64 architecture may
support fewer bits of virtual address.


System programming manual:
Quote:

5.1 Page Translation Overview
...
Currently, the AMD64 architecture defines a mechanism for translating 48-bit virtual addresses to 52-bit physical addresses. The mechanism used to translate a full 64-bit virtual address is reserved and
will be described in a future AMD64 architectural specification.
Post 11 Jan 2008, 13:34
View user's profile Send private message Visit poster's website Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3170
Location: Denmark
f0dder
Hm, so only 48bit linear addresses for now, that's good to know.

Not that I expect it to be a problem, even if there was a half-and-half kernel/user address space split like on win32, you'd still be able to map files i excess of 120 terabytes Razz

Ugh, intel manuals locked? How moronic >_<. I wonder if foxit reader respects the locking... and how hard it would be making it not respect it, if it does Smile
Post 12 Jan 2008, 13:18
View user's profile Send private message Visit poster's website Reply with quote
vid
Verbosity in development


Joined: 05 Sep 2003
Posts: 7105
Location: Slovakia
vid
Quote:
I wonder if foxit reader respects the locking

it does Sad
Post 12 Jan 2008, 14:00
View user's profile Send private message Visit poster's website AIM Address MSN Messenger ICQ Number Reply with quote
Madis731



Joined: 25 Sep 2003
Posts: 2141
Location: Estonia
Madis731
Adobe's own reader (8.1) doesn't! The only problem with all PDF-s is that the copypasted material is NEVER WYSIWYG. It always loses formatting and it never knows which way is up or right, or what does a PAGE mean Razz or tables and their orders Wink

Otherwise PDF-s are (almost) the best way to exchange data (of course the best being plain ASCII). Very Happy
Post 12 Jan 2008, 20:30
View user's profile Send private message Visit poster's website Yahoo Messenger MSN Messenger Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3170
Location: Denmark
f0dder
Madis731 wrote:
Otherwise PDF-s are (almost) the best way to exchange data (of course the best being plain ASCII). Very Happy


PDFs are good for printing, and nothing else. They suck for screen reading, such for searching, suck for... etc.

The best is HTML, because it can be rendered just about anywhere, you can do markup with it, you can index it, you can search it easily, etc.

_________________
Image - carpe noctem
Post 13 Jan 2008, 14:11
View user's profile Send private message Visit poster's website Reply with quote
rugxulo



Joined: 09 Aug 2005
Posts: 2341
Location: Usono (aka, USA)
rugxulo
Charles W. Sandmann wrote:

As the 4Gb address space gets more crowded with devices and memory and
DPMI programs we will see more problems. Windows 9x already has problems
with over 512Mb of memory (which get progressively worse up to 1Gb) - and
you can't use more than 2Gb of address space under NT/Win2K/XP without
a special boot switch (/3Gb) that only works with "server" versions - and
even then you must use special link flags, exe header flags, etc and are
limited to 3Gb.


Also, I heard that some DOS extenders (or DPMI servers) can access > 2 GB (e.g. unfinished CWSDPMI r6, X32, HDPMI32, DOS/32A), but I don't have that much RAM to test with (barely anybody does, at least with popular desktop cpus).
Post 25 Jan 2008, 04:45
View user's profile Send private message Visit poster's website Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3170
Location: Denmark
f0dder
Windows doesn't have a problem accessing >2GB of memory, but workstation versions are limited to 4GB of address space (prior to XP SP2 the limit was on physical ram rather than address space...). This is a rather superficial limit, since PAE has been available since, oh what, Pentium Pro?
Post 25 Jan 2008, 11:49
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17279
Location: In your JS exploiting you and your system
revolution
Since no one seems to have mentioned it yet.

For normal 32bit system setup the per-process limit is slightly less than 2GB.

For 32bit systems with /3gb support the per-process limit is slightly less than 3GB.
Post 25 Jan 2008, 13:54
View user's profile Send private message Visit poster's website Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3170
Location: Denmark
f0dder
revolution wrote:
Since no one seems to have mentioned it yet.

For normal 32bit system setup the per-process limit is slightly less than 2GB.

For 32bit systems with /3gb support the per-process limit is slightly less than 3GB.


And to take use of the 3:1 split, a process needs to be flagged "large memory aware" in the PE header, otherwise it will get standard 2:2 split even on boot.ini/3gb systems.

There's also the AWE (usermode process use of PAE) stuff, but you're better off porting to 64bit instead of using that.

_________________
Image - carpe noctem
Post 25 Jan 2008, 14:20
View user's profile Send private message Visit poster's website Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  


< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum


Copyright © 1999-2020, Tomasz Grysztar.

Powered by rwasa.