When my program starts, it uses 2Mb of RAM, then I open the second dialog and use GetOpenFileName several times. After it my app occupies 5Mb!
When I close the second dialog it is still 5Mb. BUT: If I minimize and
maximize the main window again, the usage of RAM is 800Kb!!!
So my question is how can I free memory after closing various dialogs and so on? Why should I use 5Mb instead of 1Mb?

did you release the window handle after using them?

Don't worry about it, zhak - most of it is code and const data that are shared across multiple applications. If you want to see the "real" memory usage, you should use perfmon.msc (2k and XP) and check the amount of "private bytes" your app uses.