Hi,

Does anyone have experience with high-performance storage operations (read/write) with Windows APIs, possibly using Solid State Drives?

I've just been using createfile, SetEndOfFile, writeFile, closeHandle (or keep it open if handle used again), and a few other calls.

Storage I/O is the slowest aspect of my projects by far. I find some speedups can be achieved by making minor tweaks, think the last one I made was with flushing, but can't remember.

When I exclude the storage portion of some of these projects, performance becomes 'real-time,' I expect this can be achieved with disk I/O also, based on the file sizes I've worked with and especially when using m.2 SSDs, still researching this. If anyone has any tips/experience, would be much appreciated.

Are you using threading? Because you can overlap the I/O with the CPU processing. All you need is a few extra buffers, some threads, and a bit of synchronising.

I am, but not on this! Great idea.

So far, found you can also specify asynchronous (non-blocking) I/O using a single thread by filling out a FILE_FLAG_OVERLAPPED structure. Seems you can also specify some flags depending upon the operations you are performing. FILE_FLAG_WRITE_THROUGH works on sequential operations, FILE_FLAG_NO_BUFFERING should NOT be set if you want to write to intermediary buffers first instead of one at a time. Thankfully did not have this no-buffering flag set.

My slowest use case is just writing 8MB of entries into a log file and 2MB into a separate log file, no reading, so these sequential options may work and synchronization should not be tough.

Combining this with actual threading should yield some good results.

i have m.2 ssd too, but whats the latency you are talking about.

so far i have not found any issues, maximum disk write i did was 100 +/- MBs, i was generating some data for specific task i had to accomplish; so yeah it was pretty fast without me noticing actually.

maybe i can run some tests for you?, provide your code here or in a PM.

I just clocked it, 8.5 seconds on the first 2MB of the first file. I think because it writes each of the 127,000 lines with individual writefile calls, the delay occurs.

Without logging in these files, the delay is probably under 1 second, I haven't clocked it though.

This is a very experimental personal project, involving self-modifying instructions. I'll strongly consider sending it, but it will very likely crash before reaching the first 2MB of logging and will seem like a bug! It initializes parallel compute VK devices which have 'Queues' that prepare to dispatch SPIR-V.

So, running it is very device-specific and this computer is very fast, but it runs with similar speed on a tablet!

Buffer those lines. And make a write call when the buffer is full. 2MB should take only a few milliseconds to write to a fast disk.

Totally! I'm trying to upload a sample log as an attachment (getting an error, posted in Feedback section).

Buffering before calling CreateFile would just need to take informational versus real debugging messages into consideration. If there is a crash, would need a way to turn buffering off, resize the buffer, or stick to buffering only on informational messages. If there's no crash, buffering would be a win regardless.

A few milliseconds sounds like a good goal.

Log file zipped.

If you want even more speed then make it binary (more compact, less to write) and maybe try to memory map it and see if it makes a difference.

Storing as binary is definitely important. I have a use for that with some numeric data I'm trying to store and retrieve, doesn't have to be readable in the file itself.

Memory mapping files has been a question mark since I first starting logging these debug messages. Seems like it works best with non-sequential file updates. There are a few things to consider, like the the size of the file view, how often it is persisted, and I'm trying to compare and contrast with non-blocking I/O, multi-threaded I/O, the buffering provided by this flag FILE_FLAG_NO_BUFFERING and implementing buffering before using the file apis as mentioned above. Maybe some of these strategies are equivalent, still looking into them... Combining a few of these will definitely yields some fast performance, haha!