flat assembler
Message board for the users of flat assembler.
![]() Goto page 1, 2, 3, 4 Next |
Author |
|
LostCoder 01 May 2012, 16:22
Realloc? Allocate some memory, for example 4096 bytes, when it full add 4096 bytes to existing size, then realloc buffer and continue reading.
|
|||
![]() |
|
f0dder 01 May 2012, 17:52
Does the buffer need to be 100% contiguous? If not, do a linked list of decently-sized chunks.
If it does, consider mmap'ing a region that will "surely be large enough", but only reserve the region - then commit pages as you go. |
|||
![]() |
|
typedef 01 May 2012, 17:55
LostCoder wrote: Realloc? Allocate some memory, for example 4096 bytes, when it full add 4096 bytes to existing size, then realloc buffer and continue reading. Yes this is how I do it in C. Read and allocate +2 bytes until CR/LF. in C Code: int main(int argc, char* argv[]) { char * buffer = 0; char _char = 0; int _read = 0; while( (_char = getchar()) != '\n') { buffer = (char*)realloc(buffer,_read+++2); if(!buffer) // if the first call of realloc fails break; buffer[_read-1] = _char; buffer[_read] = '\000'; } puts("Read: "); puts(buffer); if(buffer) free(buffer); getchar(); return 0; } FASM Code: You do it. Last edited by typedef on 01 May 2012, 18:24; edited 2 times in total |
|||
![]() |
|
f0dder 01 May 2012, 17:56
typedef wrote:
Are you insane? Realloc in itself is bad enough, you'll potentially end up with an über-fragmented heap... and allocating just 2 bytes extra? Ugh. PS: on some OSes you might be able to use stat() or similar system calls, but it's by no means portable, and you might get the size of some internal pipe buffer rather than the actual file size. I'd recommend the mmap solution if you really need the buffer to be contiguous, or linkedlist of chunks if you can adapt whatever processing code. _________________ ![]() |
|||
![]() |
|
rugxulo 01 May 2012, 18:05
JohnFound wrote: I post the question here, because I need both solutions for Linux and Win32. I think you answered your own question. If it's a file, it's not a stream, so why limit yourself to stream activities? If it's always going to be a file, treat it as a file and ignore the stream stuff. EDIT: At the very least, you can probably externally list and then pipe the filesize into STDIN first, then start reading .... |
|||
![]() |
|
typedef 01 May 2012, 18:27
f0dder wrote:
It all depends on what you are trying to with it. For example, fscanf is more vulnerable to BOF attacks and this way you can sort of bypass that. Also having a fucked up heap is because you don't use realloc right. You'll need an extra byte so as not to corrupt it. |
|||
![]() |
|
JohnFound 01 May 2012, 18:43
Well, my current solution is just that: reading chunks of the buffer, until the read byte count is smaller that the requested.
The every next chunk request is double of the previous in order to keep reallocations count as small as possible. It works fine, but at the price of non elegant sollution and possibly allocation of double of the needed memory. And I really can't avoid STDIN stream here. The problem is that I need it for a contest for size/speed optimization, so I wanted to use the best solution... Anyway. |
|||
![]() |
|
LostCoder 01 May 2012, 18:55
typedef, you can get memory leakage at this point if realloc fail:
Code: buffer = (char*)realloc(buffer,_read+++2); if(!buffer) // if the first call of realloc fails break; Code: char *newbuffer = (char*)realloc(buffer,_read+++2); if(!newbuffer) // if the call of realloc fails break; buffer = newbuffer; 1. Do you need seeking? 2. How much data size can be? 3. What kind of data is supposed? |
|||
![]() |
|
JohnFound 01 May 2012, 19:11
LostCoder wrote: JohnFound, some counter-questions: 1. No, I need everything from the STDIN to be read in the memory buffer. Then I will make several passes through the data in order to create the result. 2. Probably 20..30 MBytes. But the limit is the memory I can allocate. 3. The data is UTF-8 characters. (it is artificially constructed test file that is piped to the STDIN The code I use now is this: Code: START_SIZE = 1024 proc ReadTheInput begin mov [SourceSize], START_SIZE stdcall GetMem, [SourceSize] mov edi, eax xor esi, esi .readloop: mov ebx, [SourceSize] lea eax, [esi+edi] sub ebx, esi stdcall FileRead, [STDIN], eax, ebx cmp eax, ebx jne .endoffile add esi, ebx shl [SourceSize], 1 stdcall ResizeMem, edi, [SourceSize] mov edi, eax jmp .readloop .endoffile: add esi, eax mov [SourceSize], esi xor eax, eax mov [pSourceBuffer], edi mov [edi+esi], eax ; zero terminated... return endp It is not polished to the every possible byte, but it is not needed actually. The test file of 78Mbytes, piped to STDIN was loaded for 1100ms (intel atom 1.6GHz) Is it fast or not? Is it possible to load it faster? I don't know. _________________ Tox ID: 48C0321ADDB2FE5F644BB5E3D58B0D58C35E5BCBC81D7CD333633FEDF1047914A534256478D9 |
|||
![]() |
|
r22 01 May 2012, 19:47
If you want to optimize for speed make your START_SIZE something egregious like ~1MB or 512KB. The optimal size would be processor (data cache) & OS (internal buffer size) specific.
The less calls you make to ReadFile and ResizeMem APIs the faster it will run. |
|||
![]() |
|
JohnFound 01 May 2012, 20:29
Even 4Mbytes start size decrease the load time by 10%.
But if I set start size enough to take the data (80Mbytes), the time decrease 3..4 times! to 250..280ms |
|||
![]() |
|
typedef 01 May 2012, 20:51
well. If speed is not a problem you can read and write to a temporary file and count the bytes written. Then rewind the file, allocate the needed memory at once and read the temp file.
WHILE READ_FROM_STDIN WRITE_TO_TEMP, BYTES, N_BYTES_READ TOTAL_BYTES += N_BYTES_READ ENDW pBuff = ALLOC, TOTAL_BYTES + 1 READ_FROM_TEMP,pBuff,TOTAL_BYTES ...... FREE, pBuff Overkill ? |
|||
![]() |
|
revolution 01 May 2012, 22:21
In Windows reading the STDIN does support GetFileSize. When reading from a file you can just get the size, allocate memory and then read. Easy. If it is not a file then GetFilesize returns an error and you can then deal with it in whatever way suits your program.
|
|||
![]() |
|
JohnFound 01 May 2012, 22:47
Hm, it really works in Windows. I though GetFileSize does not work for STDIN.
But what about Linux? |
|||
![]() |
|
f0dder 02 May 2012, 01:03
GetFileSize() vs stat()... yay, support two operating systems - or at least two specific kernel versions of two operating systems. Way to go. You're not even supporting two classes of OSes (win32 vs posix), just two very specific versioned APIs. And for very specific use cases.
Please specify your needs a bit more directly. Do you need to optimized just "getting STDIN entirely to a contiguous memory buffer" because of some extremely lame framework needs or synthetic benchmark, or do you have some more reasonable needs? And please stay away from anything involving realloc - it might work decently on whatever OS you're testing on, but it might also end up with horrible results... take a look at how something like the windows heap APIs have changed over time, for instance. |
|||
![]() |
|
typedef 02 May 2012, 01:56
AFAIK, stdin is a stream and bytes will always flow in. It's never a fixed size how would stat() work on it.
Have you actually tried it? ![]() |
|||
![]() |
|
JohnFound 02 May 2012, 08:19
revolution, thanks for the hint. It works great in Windows. Now I have to check it in Linux, but I have some troubles with STAT structure definition.
f0dder, I am not very sure what you mean. I need exactly what I described - to read the whole content redirected through STDIN. I need it in most optimal and resource friendly manner. So, I don't want to use realloc at all. typedef - it seems that STDIN is not exactly a stream, at least in Windows. |
|||
![]() |
|
f0dder 02 May 2012, 09:56
JohnFound wrote: f0dder, I am not very sure what you mean. I need exactly what I described - to read the whole content redirected through STDIN. I need it in most optimal and resource friendly manner. So, I don't want to use realloc at all. JohnFound wrote: typedef - it seems that STDIN is not exactly a stream, at least in Windows. _________________ ![]() |
|||
![]() |
|
JohnFound 02 May 2012, 10:24
f0dder, the processing of the file is better to be in one block in memory. Of course, if the attempt to determine the exact data size fails, I can always fall back to the the block reading algorithm, described above.
But allocating memory and reading at once is 4..5 times faster... |
|||
![]() |
|
Goto page 1, 2, 3, 4 Next < Last Thread | Next Thread > |
Forum Rules:
|
Copyright © 1999-2025, Tomasz Grysztar. Also on GitHub, YouTube.
Website powered by rwasa.