flat assembler
Message board for the users of flat assembler.

flat assembler > Projects and Ideas > RAMDisk for Windows

Goto page Previous  1, 2, 3  Next
Author
Thread Post new topic Reply to topic
kalambong



Joined: 08 Nov 2008
Posts: 165
Wonder if anyone has any new thought on this RAMdisk thing?
Post 23 Jan 2013, 05:55
View user's profile Send private message Reply with quote
Spool



Joined: 08 Jan 2013
Posts: 154
[]


Last edited by Spool on 17 Mar 2013, 04:31; edited 1 time in total
Post 23 Jan 2013, 07:54
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 16050
Location: 112 Ocean Avenue, Amityville
Spool wrote:
If I have a 100GB RAM I would use RAMDisk.
I wouldn't. I would use it for caching on demand. That way I would keep the flexibility of having RAM available for application usage if it is needed.
Post 23 Jan 2013, 08:07
View user's profile Send private message Visit poster's website Reply with quote
Spool



Joined: 08 Jan 2013
Posts: 154
[]


Last edited by Spool on 17 Mar 2013, 04:31; edited 1 time in total
Post 23 Jan 2013, 10:36
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 16050
Location: 112 Ocean Avenue, Amityville
Spool wrote:
revolution wrote:
Spool wrote:
If I have a 100GB RAM I would use RAMDisk.
I wouldn't. I would use it for caching on demand. That way I would keep the flexibility of having RAM available for application usage if it is needed.


How about for 1TB RAM?
Same. Indeed, for any RAM size it would be the same for me. Much better use of resources IMO.
Post 23 Jan 2013, 10:39
View user's profile Send private message Visit poster's website Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3172
Location: Denmark
revolution wrote:
Spool wrote:
If I have a 100GB RAM I would use RAMDisk.
I wouldn't. I would use it for caching on demand. That way I would keep the flexibility of having RAM available for application usage if it is needed.

Whether this is better use depends a lot on what you're doing. Read caching works fairly well (there might still be some situations where you'd like to keep data readily available in memory even though the OS would have evicted it to cache other stuff).

But for write caching it's a completely different matter. First of all, an OS must flush it's write buffers relatively often to avoid data loss. Second, journaling filesystems will make sure to keep at least the FS metadata in a good state (which means small, uncachable enforced writes with write barriers). There's a lot of (write-heavyt) things that can be sped up a fair amount by being on a ramdisk.

On my 16gig workstation, I've got a permanent (and disk-persisted) 1gig ramdisk (%temp% and my firefox profile), and it's quite nice being able to pull in large on-demand disks when, well, the demand is there.

_________________
Image - carpe noctem
Post 23 Jan 2013, 14:57
View user's profile Send private message Visit poster's website Reply with quote
zir_blazer



Joined: 05 Dec 2006
Posts: 66
The only thing I can say is that I have been happily using Gavotte Ramdisk during the last 4 months ago with no issues, having been able to put to use the entire 4 GB of RAM under classic 32 Bit WXP SP3 with the Pagefile-in-RAMDisk trick. Actually, with PAE support, the theoric limit should be 64 GB. However, due to still having 4 GB of RAM I couldn't use the RAMDisk for my original purpose, that was installing games there to load them from.
Post 26 Jan 2013, 03:51
View user's profile Send private message MSN Messenger Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3172
Location: Denmark
zir_blazer wrote:
with the Pagefile-in-RAMDisk trick.

>_<

_________________
Image - carpe noctem
Post 26 Jan 2013, 21:24
View user's profile Send private message Visit poster's website Reply with quote
zir_blazer



Joined: 05 Dec 2006
Posts: 66
f0dder wrote:
zir_blazer wrote:
with the Pagefile-in-RAMDisk trick.

>_<

I know, and I still think than the basis of Pagefile-in-RAMDisk is stupid due to adding overhead (Maintaining a file system to put the Pagefile on it and letting the OS manage the "move from RAM to Pagefile" business, that considering that is moving from and to the same place, are just wasted redundant read and writes).

...That is assuming that you have X amount of RAM that the OS can see and use, then decide to make a RAMDisk reducing your usable RAM to put a Pagefile on it, so you would still have the same X total memory capacity but with added overhead. But you are forgetting than this is allowing me to use all my 4 GB of RAM on 32 Bits WXP, via PAE. While my WXP SP3 only sees 2.87 GB, via PAE I can make use of the other 1.13 GB left to make a RAMDisk. Putting the Pagefile there, I can actually use all that RAM that before I couldn't use as WXP doesn't directly uses it. So technically I found a niche usefulness for it: Making 32 Bits WXP put to use the unseen RAM as actual RAM via Pagefile-in-RAMDisk.
I could use it for browser cache or something else instead, but if you are doing MMO powergaming (Namely, having multiple clients of a game open in simultaneous), you need tons of RAM, so going 2.87 GB to 4 made a VERY BIG difference, and I actually need much more than that. Specially now that we live on a era of Internet browsers memory leaks... http://img694.imageshack.us/img694/7324/taskmanagermr.png
Post 26 Jan 2013, 22:30
View user's profile Send private message MSN Messenger Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3172
Location: Denmark
zir_blazer wrote:
While my WXP SP3 only sees 2.87 GB, via PAE I can make use of the other 1.13 GB left to make a RAMDisk. Putting the Pagefile there, I can actually use all that RAM to keep applications open that before I couldn't use as WXP doesn't directly uses it. So technically I found a niche usefulness for it.

Wow, that's a lot of wasted RAM - I thought normally you'd only lose some 250-500 megabyte of the total? With that much wasted RAM, it actually does make sense to create a ramdisk for the pagefile.

The sane thing would still be installing a 64bit OS, though Smile

_________________
Image - carpe noctem
Post 26 Jan 2013, 22:33
View user's profile Send private message Visit poster's website Reply with quote
zir_blazer



Joined: 05 Dec 2006
Posts: 66
f0dder wrote:
Wow, that's a lot of wasted RAM - I thought normally you'd only lose some 250-500 megabyte of the total?

It depends on how many expansion cards you have installed. My Motherboard has a Chipset with IGP (AMD 785G/Radeon 4200), and it also brings a Sideport (Marketing name for a RAM chip installed on the Motherboard intended to be used as exclusive RAM for the IGP). If I had the Radeon 4200 set in BIOS to only use the Sideport and no shared memory, I was able to see 3.5 GB. Actually, that is more than what most other people that I know can see.
After that, I installed a Radeon 5770 Video Card, and if I recall correctly, WXP only saw 3.25 GB with it, that seems to be the common standard. With two Radeons 5770, as I got currently, the second one seems to drain a whole 512 MB from the addressable space for some reason, as I only see 2.87 GB. MultiGPU users are the guys than the 32 Bits addressing limit was capable of hurting the most.
Post 26 Jan 2013, 23:03
View user's profile Send private message MSN Messenger Reply with quote
baldr



Joined: 19 Mar 2008
Posts: 1651
revolution wrote:
Same. Indeed, for any RAM size it would be the same for me. Much better use of resources IMO.
I second that. It seems that your opponents are missing part of the whole picture: even RAMDisk accesses will go through cache manager (unless smartass program detects that those files are on RAMDisk and use FILE_FLAG_NO_BUFFERING or memory mapping for them, though I doubt it a lot ;-).
Post 27 Jan 2013, 10:06
View user's profile Send private message Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3172
Location: Denmark
baldr wrote:
I second that. It seems that your opponents are missing part of the whole picture: even RAMDisk accesses will go through cache manager (unless smartass program detects that those files are on RAMDisk and use FILE_FLAG_NO_BUFFERING or memory mapping for them, though I doubt it a lot Wink.

Well, windows does have a DRIVE_RAMDISK class, I wonder if it disabled the cache for those? Could be really interesting wrt. memory-mapped files... haven't researched it, though.

And at any rate, even without optimizations like that, RAMdisks can be beneficial. It depends on what you're doing, of course. But a couple of scenarios that have been nice for me:

1) ISO building. Full read speed from my HDD, targeting a ramdrive. This was especially nice while getting just the right tweaked version of Win7 for my new workstation build, I did a lot of ISO builds. Was also good for speed when doing (yet a) vmware test install, with the ISO guaranteed in memory.

2) I occasionally deal with datasets that consist of a crapload of tiny files. Extracting such a dataset to my 10k-rpm velociraptor takes ~1:30 and ~0:30 to a ramdrive - on the ramdrive, I can be 100% certain that the dataset stays in ram, and that there won't be any annoying speed hiccups while I work with it. Furthermore, once done, I don't have to delete all the files (which takes a while), I simply unmount the ramdisk.

3) %TEMP%, %TMP%, Firefox profile. Putting the FF profile on a ramdisk makes it noticably snappier - none of the temporary stutter I'd get every once in a while. Having TEMP on ramdisk doesn't give much of a speed boost for normal situations, but it does mean my main harddrive can be kept spinned down a lot of the time.

Whether you get a benefit really depends on what you're doing - speedwise, the biggest boost is when dealing with (writes to) a crapload of small files. For reads to be sped up (after the first cache-cold read, of course) you need some relatively special situations. Some games can reportedly get decent speed boosts when loading (I guess they're doing FILE_FLAG_NO_BUFFERING but not being as efficient as they thought they were?), whereas for other it makes absolutely no difference (I tested with Far Cry 2 some years ago, it was the only not-super-small game I was playing that was small enough that I could fit it in a ramdisk back then Smile).

_________________
Image - carpe noctem
Post 27 Jan 2013, 13:47
View user's profile Send private message Visit poster's website Reply with quote
sleepsleep



Joined: 05 Oct 2006
Posts: 7616
Location: ˛                              ⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣ Posts: 6699
ctl3d32 wrote:
For those who are searching for a nice ramdisk drive software, try imdisk.
http://www.ltr-data.se/opencode.html/


seems like a great tool!
thanks! Smile

never thought about using RAM disk as building iso placeholder, thanks for the idea, f0dder
Post 27 Jan 2013, 16:31
View user's profile Send private message Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3172
Location: Denmark
sleepsleep wrote:
ctl3d32 wrote:
For those who are searching for a nice ramdisk drive software, try imdisk.
http://www.ltr-data.se/opencode.html/


seems like a great tool!
thanks! Smile

You might also want to check out SoftPerfect, it's pretty well-polished (used to be a commercial product) and has great performance, and a nice featureset (differential saves, and the ability to save every X minutes means your logoff/shutdown time won't be too bad when you use persistant disks).

sleepsleep wrote:
never thought about using RAM disk as building iso placeholder, thanks for the idea, f0dder

There's a few situations like that where a ramdisk can be a nice speed boost, if you've got enough RAM available. It can also be nice for building stuff like boost - you'd probably need insane amounts to be able to build something like KDE in-ram ;-P

baldr and revolutions remarks are valid, though - there's a lot of other situations where you won't get much performance from using a ramdisk, and will actually pessimize system performance because you have less memory available for filesystem caching. You need to do a bit of thinking and a bit of performance monitoring to figure out where it's suitable Smile

_________________
Image - carpe noctem
Post 27 Jan 2013, 17:40
View user's profile Send private message Visit poster's website Reply with quote
sleepsleep



Joined: 05 Oct 2006
Posts: 7616
Location: ˛                              ⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣ Posts: 6699
i got one question regarding this ramdisk,

suppose i got a text file around 500MB that need to be processed by text utility software

this 500MB located in X:/ (ramdisk)

would this software read the .txt file into memory (again) since the 500MB already inside RAM.

is there any "method" or "programming method" to let the software know, that is loaded into RAM already, please process it directly.

of the best method to use ramdisk is for "write purpose"?
so in above case,

set X:/ as the output of the software?
Post 05 Feb 2013, 08:02
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 16050
Location: 112 Ocean Avenue, Amityville
sleepsleep wrote:
would this software read the .txt file into memory (again) since the 500MB already inside RAM.
It depends upon which text editor you use, but probably for most editors out there the answer is yes, it would load it all into RAM first before you could edit it.
sleepsleep wrote:
is there any "method" or "programming method" to let the software know, that is loaded into RAM already, please process it directly.
Sure. In windows there are file mapping APIs. But depending upon what you do with the file it might be slower than just a simple read/modify/write editor.
sleepsleep wrote:
of the best method to use ramdisk is for "write purpose"?
No such thing as best. It all depends upon what you are doing.
Post 05 Feb 2013, 08:22
View user's profile Send private message Visit poster's website Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3172
Location: Denmark
revolution wrote:

sleepsleep wrote:
is there any "method" or "programming method" to let the software know, that is loaded into RAM already, please process it directly.
Sure. In windows there are file mapping APIs. But depending upon what you do with the file it might be slower than just a simple read/modify/write editor.

[/quote]
And in some cases, it can be a decent bit faster - but it depends a lot on your access patterns. Worst case scenario with mapped files, the actual file I/O is no faster than what you can achieve manually, and you get a pagefault for each 4k you process. Best case scenario, the file is already in filesystem cache, and you can access the data directly without buffer copying (this is fast).

revolution wrote:

sleepsleep wrote:
of the best method to use ramdisk is for "write purpose"?
No such thing as best. It all depends upon what you are doing.

Indeed - I sometimes see people recommend mapped files as if they are the holy grail, which they definitely aren't... but even when not the fastest solution, they can be extremely convenient and easy to work with Smile

_________________
Image - carpe noctem
Post 05 Feb 2013, 15:52
View user's profile Send private message Visit poster's website Reply with quote
baldr



Joined: 19 Mar 2008
Posts: 1651
f0dder,

Devise a conscious access pattern for a text file (except for sequential reading at least once; not as a whole at a time, probably).
Post 05 Feb 2013, 21:06
View user's profile Send private message Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3172
Location: Denmark
baldr wrote:
Devise a conscious access pattern for a text file (except for sequential reading at least once; not as a whole at a time, probably).

For standard text file reading, you'd do a one-pass sequential scan - unless the file was already cached, doing it via mmf might be slower: one #PF per 4kb, compared to large blockreads with fewer r3<>r0 switches... OTOH, you could be endline scanning while more disk I/O is being done (not sure how much readahead you get when doing MMF, but regardless of readhead I do believe you still get a #PF per page).

Didn't we have a thread here a while ago about alternate ways of processing a text file, though, or am I remembering wrong? Something about multiple parallel line-scanning threads... there's definitely some CPU overhead in managing those, though, and you'd need to be careful (or on a fast SSD) to not kill any eventual performance boost by seek overhead. Not sure if it's worth it.

Another approach that might not buy you performance gains while loading, but could turn out interesting for an editor, would be linescanning the MMF, and keeping each line as a {index, length} pair, keeping the MMF open in SHARE_READ for the lifetime of the editor... you'd obviously need a bit of logic (span buffers thingies?) to handle edits, and in-place saving could turn out really tricky. The upside would be that in low-memory situations, the OS being could discard memory pages (for later source file pagein) rather than pageout+pagein. Not sure if the complexity is worth it in the end, but I think it's an interesting idea at least. And I get a sense of déjà vu writing this, so I've probably babbled about the idea before Smile

_________________
Image - carpe noctem
Post 05 Feb 2013, 21:32
View user's profile Send private message Visit poster's website Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  
Goto page Previous  1, 2, 3  Next

< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2018, Tomasz Grysztar.

Powered by rwasa.