flat assembler
Message board for the users of flat assembler.

Index > Windows > High Performance Windows Storage

Author
Thread Post new topic Reply to topic
donn



Joined: 05 Mar 2010
Posts: 178
donn
Hi,

Does anyone have experience with high-performance storage operations (read/write) with Windows APIs, possibly using Solid State Drives?

I've just been using createfile, SetEndOfFile, writeFile, closeHandle (or keep it open if handle used again), and a few other calls.

Storage I/O is the slowest aspect of my projects by far. I find some speedups can be achieved by making minor tweaks, think the last one I made was with flushing, but can't remember.

When I exclude the storage portion of some of these projects, performance becomes 'real-time,' I expect this can be achieved with disk I/O also, based on the file sizes I've worked with and especially when using m.2 SSDs, still researching this. If anyone has any tips/experience, would be much appreciated.
Post 27 Apr 2020, 02:19
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17339
Location: In your JS exploiting you and your system
revolution
Are you using threading? Because you can overlap the I/O with the CPU processing. All you need is a few extra buffers, some threads, and a bit of synchronising.
Post 27 Apr 2020, 03:51
View user's profile Send private message Visit poster's website Reply with quote
donn



Joined: 05 Mar 2010
Posts: 178
donn
I am, but not on this! Great idea.

So far, found you can also specify asynchronous (non-blocking) I/O using a single thread by filling out a FILE_FLAG_OVERLAPPED structure. Seems you can also specify some flags depending upon the operations you are performing. FILE_FLAG_WRITE_THROUGH works on sequential operations, FILE_FLAG_NO_BUFFERING should NOT be set if you want to write to intermediary buffers first instead of one at a time. Thankfully did not have this no-buffering flag set.

My slowest use case is just writing 8MB of entries into a log file and 2MB into a separate log file, no reading, so these sequential options may work and synchronization should not be tough.

Combining this with actual threading should yield some good results.
Post 27 Apr 2020, 16:53
View user's profile Send private message Reply with quote
Ali.Z



Joined: 08 Jan 2018
Posts: 348
Ali.Z
i have m.2 ssd too, but whats the latency you are talking about.

so far i have not found any issues, maximum disk write i did was 100 +/- MBs, i was generating some data for specific task i had to accomplish; so yeah it was pretty fast without me noticing actually.

maybe i can run some tests for you?, provide your code here or in a PM.

_________________
Asm For Wise Humans
Post 27 Apr 2020, 18:06
View user's profile Send private message Reply with quote
donn



Joined: 05 Mar 2010
Posts: 178
donn
I just clocked it, 8.5 seconds on the first 2MB of the first file. I think because it writes each of the 127,000 lines with individual writefile calls, the delay occurs.

Without logging in these files, the delay is probably under 1 second, I haven't clocked it though.

This is a very experimental personal project, involving self-modifying instructions. I'll strongly consider sending it, but it will very likely crash before reaching the first 2MB of logging and will seem like a bug! It initializes parallel compute VK devices which have 'Queues' that prepare to dispatch SPIR-V.

So, running it is very device-specific and this computer is very fast, but it runs with similar speed on a tablet!
Post 28 Apr 2020, 00:02
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17339
Location: In your JS exploiting you and your system
revolution
donn wrote:
I just clocked it, 8.5 seconds on the first 2MB of the first file. I think because it writes each of the 127,000 lines with individual writefile calls, the delay occurs.
Buffer those lines. And make a write call when the buffer is full. 2MB should take only a few milliseconds to write to a fast disk.
Post 28 Apr 2020, 00:07
View user's profile Send private message Visit poster's website Reply with quote
donn



Joined: 05 Mar 2010
Posts: 178
donn
Totally! I'm trying to upload a sample log as an attachment (getting an error, posted in Feedback section).

Buffering before calling CreateFile would just need to take informational versus real debugging messages into consideration. If there is a crash, would need a way to turn buffering off, resize the buffer, or stick to buffering only on informational messages. If there's no crash, buffering would be a win regardless.

A few milliseconds sounds like a good goal.
Post 28 Apr 2020, 04:40
View user's profile Send private message Reply with quote
donn



Joined: 05 Mar 2010
Posts: 178
donn
Log file zipped.


Description:
Download
Filename: DebugMsgs.7z
Filesize: 90.65 KB
Downloaded: 45 Time(s)

Post 28 Apr 2020, 08:40
View user's profile Send private message Reply with quote
Furs



Joined: 04 Mar 2016
Posts: 1483
Furs
If you want even more speed then make it binary (more compact, less to write) and maybe try to memory map it and see if it makes a difference.
Post 28 Apr 2020, 12:35
View user's profile Send private message Reply with quote
donn



Joined: 05 Mar 2010
Posts: 178
donn
Storing as binary is definitely important. I have a use for that with some numeric data I'm trying to store and retrieve, doesn't have to be readable in the file itself.

Memory mapping files has been a question mark since I first starting logging these debug messages. Seems like it works best with non-sequential file updates. There are a few things to consider, like the the size of the file view, how often it is persisted, and I'm trying to compare and contrast with non-blocking I/O, multi-threaded I/O, the buffering provided by this flag FILE_FLAG_NO_BUFFERING and implementing buffering before using the file apis as mentioned above. Maybe some of these strategies are equivalent, still looking into them... Combining a few of these will definitely yields some fast performance, haha!
Post 29 Apr 2020, 19:36
View user's profile Send private message Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  


< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2020, Tomasz Grysztar. Also on YouTube, Twitter.

Website powered by rwasa.