Funny thing about DLL code

Index > Windows > Funny thing about DLL code

Author

Thread

AlexP

Joined: 14 Nov 2007
Posts: 561
Location: Out the window. Yes, that one.

AlexP 05 Mar 2008, 22:56

I just experienced this, thought I'd post it.

My driver program (I mean program that uses my library) wasn't initializing correctly, so I opened it up in Olly and -:

Code:

7727A5F0   55               PUSH    EBP
7727A5F1   8BEC             MOV     EBP,ESP
7727A5F3   56               PUSH    ESI
7727A5F4   57               PUSH    EDI
7727A5F5   53               PUSH    EBX
7727A5F6   8BF4             MOV     ESI,ESP
7727A5F8   FF75 14          PUSH    DWORD PTR SS:[EBP+14]
7727A5FB   FF75 10          PUSH    DWORD PTR SS:[EBP+10]
7727A5FE   FF75 0C          PUSH    DWORD PTR SS:[EBP+C]
7727A601   FF55 08          CALL    NEAR DWORD PTR SS:[EBP+8]
7727A604   8BE6             MOV     ESP,ESI
7727A606   5B               POP     EBX
7727A607   5F               POP     EDI
7727A608   5E               POP     ESI
7727A609   5D               POP     EBP
7727A60A   C2 1000          RETN    10

That was Windows Vista Kernel32 code that calls the entry point in a DLL, I had never seen it before... I studied why it faulted (esp contained value 0x100), and noticed that the person who coded this apparently stored the stack register in esi, then restored it when the code returned.

So in a DLL entry, you can modify and mess with the stack and esp register as much as you want, but if you modify esi the code will fault.

Just thought it was something interesting, now I know I have to save the esi register on all entries.

05 Mar 2008, 22:56

asmrox

Joined: 19 Jan 2008
Posts: 160

asmrox 06 Mar 2008, 00:04

i always use this in dll:

entry $
pushad
code...
popad
retn 12

06 Mar 2008, 00:04

AlexP

Joined: 14 Nov 2007
Posts: 561
Location: Out the window. Yes, that one.

AlexP 06 Mar 2008, 00:08

? pushad? i'll check...

06 Mar 2008, 00:08

Goplat

Joined: 15 Sep 2006
Posts: 181

Goplat 06 Mar 2008, 01:23

AlexP wrote:

That was Windows Vista Kernel32 code that calls the entry point in a DLL, I had never seen it before... I studied why it faulted (esp contained value 0x100), and noticed that the person who coded this apparently stored the stack register in esi, then restored it when the code returned.

This code is to compensate for the fact that some broken DLLs return from their entry function with plain "ret" instead of the required "ret 12". Back when compiler optimization was slim to nil, this kind of bug often wouldn't actually affect anything, because the compiler just addressed everything relative to BP/EBP and didn't try any fancy stack tricks. My guess is an earlier version of Windows was compiled with such a non-optimizing compiler, and during this time the broken DLLs were made. Later, when Microsoft made their compiler better, the code broke so they put this wrapper in to keep those DLLs working.

Don't rely on this wrapper staying the same or even existing at all in the future; Microsoft doesn't care about backwards compatibility anywhere near as much as they used to (as you have surely seen if you're using Vista). To be safe, just follow the standard calling conventions.

Quote:

So in a DLL entry, you can modify and mess with the stack and esp register as much as you want, but if you modify esi the code will fault.

No. A DLL entry point follows the calling conventions like any other callback function. You can change EAX, ECX, and EDX if you want, but EBX, ESI, EDI, and EBP *all* must be preserved, and you must remove the parameters from the stack (with "ret 12").

asmrox wrote:

i always use this in dll:

entry $
pushad
code...
popad
retn 12

A DLL entry point must return a value to indicate success/failure. If it returns 0, the DLL is immediately unloaded and LoadLibrary returns NULL. With this code, you're just returning whatever was already in EAX, which could possibly be 0.

06 Mar 2008, 01:23

AlexP

Joined: 14 Nov 2007
Posts: 561
Location: Out the window. Yes, that one.

AlexP 06 Mar 2008, 03:53

Quote:

With this code, you're just returning whatever was already in EAX, which could possibly be 0.

So the correct procedure is to place a value other than 0 into eax before returning. Very well written, I will be sure to remember that point in the future.

06 Mar 2008, 03:53

revolution
When all else fails, read the source

Joined: 24 Aug 2004
Posts: 20404
Location: In your JS exploiting you and your system

revolution 06 Mar 2008, 04:22

It's in the doc's.

SDK - DllEntryPoint wrote:

Return Values

When the system calls the DllEntryPoint function with the DLL_PROCESS_ATTACH value, the function returns TRUE if it succeeds or FALSE if initialization fails. If the return value is FALSE when DllEntryPoint
is called because the process uses the LoadLibrary function, LoadLibrary returns NULL. If the return value is FALSE when DllEntryPoint is called during process initialization, the process terminates with an error. To get extended error information, call GetLastError.

When the system calls the DllEntryPoint function with any value other than DLL_PROCESS_ATTACH, the return value is ignored.

Do you have the SDK? If not it is available online at MSDN for anyone wishing to read it.

06 Mar 2008, 04:22

AlexP

Joined: 14 Nov 2007
Posts: 561
Location: Out the window. Yes, that one.

AlexP 06 Mar 2008, 04:28

Which SDK are you referring to? The driver SDK? Or do you mean the online documentation?

06 Mar 2008, 04:28

revolution
When all else fails, read the source

Joined: 24 Aug 2004
Posts: 20404
Location: In your JS exploiting you and your system

revolution 06 Mar 2008, 04:30

Platform SDK. It is available online for free, it has also been posted for download on the flatassembler.net website.

06 Mar 2008, 04:30

AlexP

Joined: 14 Nov 2007
Posts: 561
Location: Out the window. Yes, that one.

AlexP 06 Mar 2008, 04:31

I've been looking around microsft download, it's nowhere!!! I can't find it in FASM site.

06 Mar 2008, 04:31

asmrox

Joined: 19 Jan 2008
Posts: 160

asmrox 06 Mar 2008, 15:18

adding
xor eax,eax
inc eax will fix it?

06 Mar 2008, 15:18

revolution
When all else fails, read the source

Joined: 24 Aug 2004
Posts: 20404
Location: In your JS exploiting you and your system

revolution 06 Mar 2008, 15:22

AlexP wrote:

I've been looking around microsft download, it's nowhere!!! I can't find it in FASM site.

Hmm, I have just looked and it seems to have disappeared Sad

Sorry to put on the wrong track. It used to be there.

But the MSDN online version is always the most up-to-date anyway.

06 Mar 2008, 15:22

revolution
When all else fails, read the source

Joined: 24 Aug 2004
Posts: 20404
Location: In your JS exploiting you and your system

revolution 06 Mar 2008, 15:24

asmrox wrote:

adding
xor eax,eax
inc eax will fix it?

Sure, also "mov eax,1" and "or eax,1" and "or eax,-1" and "xor eax,eax|dec eax" and lots of other ways.

06 Mar 2008, 15:24

dap

Joined: 01 Dec 2007
Posts: 61
Location: Belgium

dap 06 Mar 2008, 16:03

Goplat wrote:

This code is to compensate for the fact that some broken DLLs return from their entry function with plain "ret" instead of the required "ret 12".

An interesting article BTW : http://blogs.msdn.com/oldnewthing/archive/2004/01/15/58973.aspx

_________________
(French only) http://dap.developpez.com

06 Mar 2008, 16:03

AlexP

Joined: 14 Nov 2007
Posts: 561
Location: Out the window. Yes, that one.

AlexP 06 Mar 2008, 21:45

hmm... I'll continue using the online MSDN, but I've found a new version of the SDK for "windows server 2008". I'll try it, just selected the options for only "Win32 examples" and "Win32 documentation".

06 Mar 2008, 21:45

< Last Thread | Next Thread >

Forum Rules:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum