WriteFile() API wastes all memory.

Index > Windows > WriteFile() API wastes all memory.

Author

Thread

programmer

Joined: 05 Jun 2006
Posts: 6

programmer 20 Jun 2006, 10:38

After I write 100 MB data to the file, it wastes 100 MB of the RAM. How to delete the buffer after WriteFile function?

Code:

format PE GUI 4.0
entry start

include 'win32ax.inc'

ID_BUTTON1 = 1

section '.data' data readable writeable

        wnd_class db 'pro_wnd',0
        wc WNDCLASS 0,WndProc,0,0,0,0,0,COLOR_BTNFACE+1,0,wnd_class
        msg MSG
        hFont dd ?
        hMain dd ?
        hButton1 dd ?
        nobr dd ?
        nobw dd ?
        hFile1 dd ?
        hFile2 dd ?
        FileSize dd ?
        
        Buffer db 6000000 dup(?)
        
section '.code' code readable executable

        start:
                invoke GetModuleHandle,0
                mov [wc.hInstance],eax
                invoke LoadIcon,0,IDI_APPLICATION
                mov [wc.hIcon],eax
                invoke LoadCursor,0,IDC_ARROW
                mov [wc.hCursor],eax
                invoke RegisterClass,wc
                invoke CreateWindowEx,0,wnd_class,'Files',WS_VISIBLE + WS_DLGFRAME + WS_SYSMENU,300,300,320,240,0,0,[wc.hInstance],0
                mov [hMain],eax
                
        message_loop:
                invoke GetMessage,msg,0,0,0
                or eax,eax
                jz exit
                invoke TranslateMessage,msg
                invoke DispatchMessage,msg
                jmp message_loop
                
        exit:
                invoke ExitProcess,[msg.wParam]
        
        proc WndProc hwnd,wmsg,wparam,lparam
                cmp [wmsg],WM_COMMAND
                je wm_command
                cmp [wmsg],WM_CREATE
                je wm_create
                cmp [wmsg],WM_DESTROY
                je wm_destroy
                
                def_wndproc:
                        invoke DefWindowProc,[hwnd],[wmsg],[wparam],[lparam]
                        ret
                        
                wm_create:
                        invoke CreateFont,13,0,0,0,1,0,0,0,DEFAULT_CHARSET,OUT_DEFAULT_PRECIS,CLIP_DEFAULT_PRECIS,DEFAULT_QUALITY,FF_DONTCARE,'Tahoma'
                        mov [hFont],eax
                        ;»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»
                        invoke CreateWindowEx,WS_EX_CLIENTEDGE,'BUTTON','Process',WS_CHILD + WS_VISIBLE + BS_PUSHBUTTON,5,5,100,27,[hwnd],ID_BUTTON1,0,0
                        mov [hButton1],eax
                        invoke SendMessage,eax,WM_SETFONT,[hFont],1
                        ;»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»»
                        jmp def_wndproc
                        
                wm_destroy:
                        invoke PostQuitMessage,0
                        ret
                        
                wm_command:
                        mov eax,[wparam]
                        shr eax,16
                        cmp eax,BN_CLICKED
                        jne @f
                        mov eax,[wparam]
                        and eax,0xFFFF
                        cmp eax,ID_BUTTON1
                        je .button1_click
                        @@:
                        ret
                
                .button1_click:
                        invoke CreateFile,'C:\file_1.exe',GENERIC_READ,0,0,OPEN_EXISTING,FILE_ATTRIBUTE_NORMAL,0
                        mov [hFile1],eax
                        invoke CreateFile,'C:\file_2.exe',GENERIC_READ + GENERIC_WRITE,0,0,CREATE_ALWAYS,FILE_ATTRIBUTE_NORMAL,0
                        mov [hFile2],eax
                        
                        invoke GetFileSize,[hFile1],0
                        mov [FileSize],eax
                        
                        invoke ReadFile,[hFile1],Buffer,[FileSize],nobr,0
                        invoke WriteFile,[hFile2],Buffer,[FileSize],nobw,0
                        
                        invoke CloseHandle,[hFile1]
                        invoke CloseHandle,[hFile2]
                        
                        invoke MessageBox,[hMain],'Done!','',MB_OK
                        ret
                
                endp
                
section '.idata' import data readable writeable

        library kernel32,'kernel32.dll',\
                        user32,'user32.dll',\
                        gdi32,'gdi32.dll'
        
        include 'apia\kernel32.inc'
        include 'apia\user32.inc'
        include 'apia\gdi32.inc'

20 Jun 2006, 10:38

Torrey

Joined: 12 Oct 2003
Posts: 78

Torrey 20 Jun 2006, 10:54

Why not use VirtualAlloc and VirtualFree since you have the file size? It'd be similar to the method you're using, but would free up the RAM it used. This method wouldn't be recommended for very large files.

20 Jun 2006, 10:54

quasar

Joined: 09 Oct 2005
Posts: 16

quasar 20 Jun 2006, 11:12

You can't. To do this as it should you have to allocate your buffer through GlobalAlloc/VirtualAlloc and GlobalFree it when you don't need it anymore.

20 Jun 2006, 11:12

jbojarczuk

Joined: 21 Jun 2006
Posts: 27

jbojarczuk 09 Sep 2006, 01:58

Use VirtualAlloc to alloc the memory; this will make sure the memory is sector-aligned.

When you call CreateFile, use the FILE_FLAG_NO_BUFFERING flag.

When you write the file, pass in this buffer. Because of the flag, windows will not use an intermediate buffer on this file, and so will not hold resources.

The caveat for this is that buffers for reading/writing to disk must be sector aligned, otherwise... (It must be REALLY BAD for the manual to not say what happens!!! Surprised

) )

09 Sep 2006, 01:58

LocoDelAssembly
Your code has a bug

Joined: 06 May 2005
Posts: 4624
Location: Argentina

LocoDelAssembly 09 Sep 2006, 03:08

Try with http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/flushfilebuffers.asp

Since those buffers are stored to disk after calling that function the memory manager can use that memory when it wants without wainting for any I/O.

09 Sep 2006, 03:08

f0dder

Joined: 19 Feb 2004
Posts: 3175
Location: Denmark

f0dder 09 Sep 2006, 11:00

Process file in chunks instead of one read and one write.

09 Sep 2006, 11:00

LocoDelAssembly
Your code has a bug

Joined: 06 May 2005
Posts: 4624
Location: Argentina

LocoDelAssembly 09 Sep 2006, 14:44

f0dder, he process the file in 6000000 bytes chucks, the problen here is that he says that after processing 100 MB he looses 100 MB of RAM memory.

mmmm, looking deeper in the code I see that if the file is more that 6000000 bytes the buffer will overflow. You have to read in chucks as f0dder says and if that wastes you RAM then use FlushFileBuffers in the loop.

Sorry if I'd assume wrong that those 100 MB was after executing WriteFile many times, not only once.

09 Sep 2006, 14:44

f0dder

Joined: 19 Feb 2004
Posts: 3175
Location: Denmark

f0dder 09 Sep 2006, 14:50

Also, "losing" memory is a pretty loose description.

The application shouldn't keep having a high memory footprint if the buffers are allocated; the total memory stats shown by taskmgr will keep being high, but that's because memory is used for filesystem cache.

That is *not* a problem, the memory will be freed if it becomes necessary.

Recall the old saying "unused memory is wasted memory". Filesystem cache is a good thing.

09 Sep 2006, 14:50

LocoDelAssembly
Your code has a bug

Joined: 06 May 2005
Posts: 4624
Location: Argentina

LocoDelAssembly 09 Sep 2006, 15:16

Yes, but I think that Windows prefer to cache some file contents by using non dirty memory pages (when there is no more physical memory available) instead of just write to disk. That's the reason why I think that he should use FlushFileBuffers. The same goes to file mapping, if you are writing too many pages you should use FlushViewOfFile to prevent monopolization of the physical memory by that file mapping. I did a expirement writing sequentially over file mapping getting a tremendous system performance loss without FlushViewOfFile. Without FlushViewOfFile the program finishes its works very fast (only the first time), but all the other processes starts a lot of swapping because of this.

My conclusion is that is convinient to force file buffers writing to disk to make the memory occupied by those buffers ready to use for something else (but the buffers stills in memory to be used as cache, flushing just makes them more discardable).

09 Sep 2006, 15:16

f0dder

Joined: 19 Feb 2004
Posts: 3175
Location: Denmark

f0dder 09 Sep 2006, 15:20

Sounds like a good piece of advice - you just need to flush at strategically good times. Ie, better to do it after you're done writing, not after each 4kb write you do Smile

09 Sep 2006, 15:20

okasvi

Joined: 18 Aug 2005
Posts: 382
Location: Finland

okasvi 09 Sep 2006, 15:28

I think doing it strategically is not just about size, more like doing it between every 16mb's or every few minutes... I think ie. most torrent-apps(good example as it does alot of writing to file) uses something like that... but definately flushing the file every now and then is good practice

09 Sep 2006, 15:28

f0dder

Joined: 19 Feb 2004
Posts: 3175
Location: Denmark

f0dder 09 Sep 2006, 15:31

Windows does cache flushing "every now and then" already... It's got a background thread running for just that purpose. Look at "inside windows 2000".

09 Sep 2006, 15:31

LocoDelAssembly
Your code has a bug

Joined: 06 May 2005
Posts: 4624
Location: Argentina

LocoDelAssembly 09 Sep 2006, 15:35

Quote:

not after each 4kb write you do

Actually I'd call FlushViewOfFile at every 128 MB of written data Very Happy

Code:

format PE GUI 4.0
include 'win32axp.inc'

INVALID_HANDLE_VALUE equ -1
FILE_SIZE = 1024*1024*1024

start:
; Apertura del archivo
  invoke  CreateFile, 'test.dat',\                          ; lpFileName
                      GENERIC_READ or GENERIC_WRITE,\       ; dwDesiredAccess
                      0,\                                   ; dwShareMode
                      NULL,\                                ; lpSecurityAttributes
                      OPEN_ALWAYS,\                         ; dwCreationDistribution
                      FILE_ATTRIBUTE_NORMAL,\               ; dwFlagsAndAttributes
                      NULL                                  ; hTemplateFile
  cmp     eax, INVALID_HANDLE_VALUE
  mov     esi, eax

  invoke  SetFilePointer, esi, FILE_SIZE, 0, FILE_BEGIN
  invoke  SetEndOfFile, esi
@@:
; Creación del mapeo del archivo
  invoke  CreateFileMapping, esi,\                          ; hFile
                             NULL,\                         ; lpFileMappingAttributes
                             PAGE_READWRITE,\               ; flProtect
                             0,\                            ; dwMaximumSizeHigh
                             0,\                            ; dwMaximumSizeLow
                             NULL                           ; lpName
  mov     esi, eax

; Mapeo del archivo a memoria
  xor     ebx, ebx
  invoke  MapViewOfFile, esi,\                              ; hFileMappingObject
                         FILE_MAP_WRITE,\                   ; dwDesiredAccess
                         0,\                                ; dwFileOffsetHigh
                         ebx,\                              ; dwFileOffsetLow
                         FILE_SIZE                          ; dwNumberOfBytesToMap
  mov     edi, eax
  mov     esi, eax

.loop:

  mov     ecx, FILE_SIZE/8
  rep     stosb

  invoke  FlushViewOfFile, esi, 0

  add     ebx, FILE_SIZE/8
  cmp     ebx, FILE_SIZE
  jb      .loop

exit:
  invoke  ExitProcess, 0

.end start

f0dder is right, Windows already do that, the problem is when you write very fast Razz

[edit] I don't know why I used

Code:

  add     ebx, FILE_SIZE/8
  cmp     ebx, FILE_SIZE
  jb      .loop

Code:

  mov     ebx, FILE_SIZE/(FILE_SIZE/8)

.loop:

  mov     ecx, FILE_SIZE/8
  rep     stosb

  invoke  FlushViewOfFile, esi, 0

  dec     ebx
  jnz      .loop

That's a little better.[/edit]

09 Sep 2006, 15:35

< Last Thread | Next Thread >

Forum Rules:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum