flat assembler
Message board for the users of flat assembler.
 Home   FAQ   Search   Register 
 Profile   Log in to check your private messages   Log in 
flat assembler > Windows > 64Bit DLL & DLL_PROCESS_ATTACH

Goto page 1, 2  Next
Author
Thread Post new topic Reply to topic
jochenvnltn



Joined: 15 Jul 2011
Posts: 60
64Bit DLL & DLL_PROCESS_ATTACH
It seems (checked with a debugger) that a 64bit version of a DLL does not seem to understand DLL_PROCESS_ATTACH. Can someone here explain why that is ? When i use x86 code, everything works fine.


Code:

format PE64 GUI 4.0 DLL
entry DllEntryPoint

include 'win64ax.inc'

section '.code' code readable executable

proc DllEntryPoint hinstDLL,fdwReason,lpvReserved
        mov     eax,TRUE
        .if [fdwReason] = DLL_PROCESS_ATTACH
        invoke MessageBox,NULL,szText,szTitle,MB_OK
        .endif
        invoke ExitProcess,0

endp

proc DummyProc
     ret
endp


section '.data' data readable writeable

szText db '64Bit DLL!!',0
szTitle db 'Success',0



section '.idata' import data readable writeable
library user32,'user32.dll',kernel32,'kernel32.dll'


  import kernel32,\
            ExitProcess,'ExitProcess'

  import user32,\
         MessageBox,'MessageBoxA'


section '.edata' export data readable

  export 'Mydll.dll',\
         DummyProc,'DummyProc'



data        fixups
end         data 


Post 03 Dec 2017, 02:56
View user's profile Send private message MSN Messenger Reply with quote
fasmnewbie



Joined: 01 Mar 2011
Posts: 424
Try to simplify your DLLMain proc to basic initialization task only. Calling MessageBox and / or ExitProcess will call another DLLs. Replace ExiitProcess with ret. You probably need dummy fixups too.
Post 03 Dec 2017, 05:12
View user's profile Send private message Visit poster's website Reply with quote
jochenvnltn



Joined: 15 Jul 2011
Posts: 60

fasmnewbie wrote:
Try to simplify your DLLMain proc to basic initialization task only. Calling MessageBox and / or ExitProcess will call another DLLs. Replace ExiitProcess with ret. You probably need dummy fixups too.



nope.. that wasn't the problem .. Smile
Post 03 Dec 2017, 05:37
View user's profile Send private message MSN Messenger Reply with quote
jochenvnltn



Joined: 15 Jul 2011
Posts: 60
Got it working ..


Code:

format PE64 GUI 4.0 DLL
entry DllEntryPoint

include 'win64ax.inc'



proc DllEntryPoint hinstDLL:QWORD,fdwReason:QWORD,lpvReserved:QWORD

    mov QWORD  [rbp+10h], rcx
    mov QWORD  [rbp+18h], rdx
    mov QWORD  [rbp+20h], r8
    mov QWORD  [rbp+28h], r9

.if [fdwReason] = DLL_PROCESS_ATTACH


        invoke MessageBox,NULL,szText,szTitle,MB_OK
        .endif
        invoke ExitProcess,0

endp




szText db '64Bit DLL!!',0
szTitle db 'Success',0



section '.idata' import data readable writeable
library user32,'user32.dll',kernel32,'kernel32.dll'


  import kernel32,\
            ExitProcess,'ExitProcess'

  import user32,\
         MessageBox,'MessageBoxA'





data        fixups
end         data


Post 03 Dec 2017, 05:38
View user's profile Send private message MSN Messenger Reply with quote
jochenvnltn



Joined: 15 Jul 2011
Posts: 60
question now is .. Do i have to add


Code:

    mov QWORD  [rbp+10h], rcx
    mov QWORD  [rbp+18h], rdx
    mov QWORD  [rbp+20h], r8
    mov QWORD  [rbp+28h], r9 




In every proc i use in the DLL ?
If not then whats the reason for that ?
If yes then whats the reason for that ?
Post 03 Dec 2017, 05:42
View user's profile Send private message MSN Messenger Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 15303
Location: Bigweld Industries
fdwReason is in the register rdx. You can check the value in rdx directly instead of transferring it to the stack first.

Note that [rbp+18h] is the same as saying [fdwReason].
Post 03 Dec 2017, 11:49
View user's profile Send private message Visit poster's website Reply with quote
jochenvnltn



Joined: 15 Jul 2011
Posts: 60

revolution wrote:
fdwReason is in the register rdx. You can check the value in rdx directly instead of transferring it to the stack first.

Note that [rbp+18h] is the same as saying [fdwReason].



hi Revolution Smile


Can you give a small example of using Proc in 64bit this way ?
Im really interested in learning the logic behind it.
Post 03 Dec 2017, 17:36
View user's profile Send private message MSN Messenger Reply with quote
jochenvnltn



Joined: 15 Jul 2011
Posts: 60
Okay .. this seems to work


Code:

format PE64 GUI 4.0 DLL
entry DllEntryPoint

include 'win64ax.inc'

proc DllEntryPoint hinstDLL:QWORD,fdwReason:QWORD,lpvReserved:QWORD

.if rdx = DLL_PROCESS_ATTACH


        invoke MessageBox,NULL,szText,szTitle,MB_OK
        .endif
        invoke ExitProcess,0
endp

szText db '64Bit DLL!!',0
szTitle db 'Success',0

section '.idata' import data readable writeable
library user32,'user32.dll',kernel32,'kernel32.dll'


  import kernel32,\
            ExitProcess,'ExitProcess'

  import user32,\
         MessageBox,'MessageBoxA'

data        fixups
end         data


Post 03 Dec 2017, 18:08
View user's profile Send private message MSN Messenger Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 15303
Location: Bigweld Industries
It is not usual for a DLL to use ExitProcess. Normally this is left to the exe file to do that after it has cleaned up all its stuff.
Post 03 Dec 2017, 22:34
View user's profile Send private message Visit poster's website Reply with quote
jochenvnltn



Joined: 15 Jul 2011
Posts: 60

revolution wrote:
It is not usual for a DLL to use ExitProcess. Normally this is left to the exe file to do that after it has cleaned up all its stuff.


Can you give other examples besides leaving out ExitProcess ?
Post 04 Dec 2017, 01:49
View user's profile Send private message MSN Messenger Reply with quote
jochenvnltn



Joined: 15 Jul 2011
Posts: 60
for instance if i can still use rcx,rdx,r8,r9 inside the proc ?
do i have to push registers in the beginning of the proc and pop at the end like in x86 ?
if i use non volatile registers in the proc, do i have to save them also in the beginning ?
hints en tips on how to use Proc in x64 .. stuff like that would be welcome Smile
Post 04 Dec 2017, 01:57
View user's profile Send private message MSN Messenger Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 15303
Location: Bigweld Industries
MS have the fastcall convention for Windows. It is only applicable when calling the API functions, but you can still use it in your own internal functions also.
Post 04 Dec 2017, 02:04
View user's profile Send private message Visit poster's website Reply with quote
jochenvnltn



Joined: 15 Jul 2011
Posts: 60

revolution wrote:
MS have the fastcall convention for Windows. It is only applicable when calling the API functions, but you can still use it in your own internal functions also.



I found good info:

The registers RAX, RCX, RDX, R8, R9, R10, R11 are considered volatile and must be considered destroyed on function calls (unless otherwise safety-provable by analysis such as whole program optimization).

The registers RBX, RBP, RDI, RSI, RSP, R12, R13, R14, and R15 are considered nonvolatile and must be saved and restored by a function that uses them.
Post 04 Dec 2017, 02:23
View user's profile Send private message MSN Messenger Reply with quote
jochenvnltn



Joined: 15 Jul 2011
Posts: 60
So if i use RSI and RDI in my Proc, i must push them at start and pop at the end ?
Post 04 Dec 2017, 02:25
View user's profile Send private message MSN Messenger Reply with quote
jochenvnltn



Joined: 15 Jul 2011
Posts: 60
There should be a section on this forum that only talks about x64 ASM. Would be good for everyone to get resources from. Smile
Post 04 Dec 2017, 02:29
View user's profile Send private message MSN Messenger Reply with quote
donn



Joined: 05 Mar 2010
Posts: 68
If you want to conform to fastcall, then rsi/rdi:

push rbx rbp rdi rsi rsp r12 r13 r14 r15
...
pop r15 r14 r13 r12 rsp rsi rdi rbp rbx

for example (within a function). When moving on to x64, I found fastcall to contain some tricky quirks in the beginning.

MSDN's x64 calling conventions pages were the most helpful:

https://docs.microsoft.com/en-us/cpp/build/overview-of-x64-calling-conventions (think that's the main page).

The boards here have some good discussions on it also. I searched and found some good results (furs has good alignment tips, x64/fastcall searches should turn up results too). I'm not working with .dl's though.
Post 04 Dec 2017, 03:22
View user's profile Send private message Reply with quote
jochenvnltn



Joined: 15 Jul 2011
Posts: 60

donn wrote:
If you want to conform to fastcall, then rsi/rdi:

push rbx rbp rdi rsi rsp r12 r13 r14 r15
...
pop r15 r14 r13 r12 rsp rsi rdi rbp rbx

for example (within a function). When moving on to x64, I found fastcall to contain some tricky quirks in the beginning.

MSDN's x64 calling conventions pages were the most helpful:

https://docs.microsoft.com/en-us/cpp/build/overview-of-x64-calling-conventions (think that's the main page).

The boards here have some good discussions on it also. I searched and found some good results (furs has good alignment tips, x64/fastcall searches should turn up results too). I'm not working with .dl's though.

thank you for the link! I found some useful info here https://docs.microsoft.com/en-us/cpp/build/prolog-and-epilog
Post 04 Dec 2017, 03:31
View user's profile Send private message MSN Messenger Reply with quote
jochenvnltn



Joined: 15 Jul 2011
Posts: 60
Also found this
Examples for Win64 Iczelion tutorial
http://masm32.com/board/index.php?topic=4190.0
Would be nice if someone would make a FASM version & get some conversation going about 64bit ASM Smile
Post 04 Dec 2017, 03:44
View user's profile Send private message MSN Messenger Reply with quote
fasmnewbie



Joined: 01 Mar 2011
Posts: 424

jochenvnltn wrote:

fasmnewbie wrote:
Try to simplify your DLLMain proc to basic initialization task only. Calling MessageBox and / or ExitProcess will call another DLLs. Replace ExiitProcess with ret. You probably need dummy fixups too.



nope.. that wasn't the problem .. Smile



That's exactly the problem.

1) Calling ExitProcess at the start of a DLL initialization will render all other subsequent modules in your DLL inaccessible / invisible due to sudden exit of the entire DLL. Just use ret.

2) Calling other complicated modules such as MessageBox will clobber your EAX (=TRUE), which is supposed to be the return value of DLLEntryPoint/DllMain.

3) Microsoft strongly advised against calling other libraries in the DLLMain unnecessarily because they might be unsafe.

From https://msdn.microsoft.com/en-us/library/windows/desktop/dn633971(v=vs.85).aspx#general_best_practices

Things to avoid in DLLMain:

- Call LoadLibrary or LoadLibraryEx (either directly or indirectly). This can cause a deadlock or a crash.
- Call GetStringTypeA, GetStringTypeEx, or GetStringTypeW (either directly or indirectly). This can cause a deadlock or a crash.
- Synchronize with other threads. This can cause a deadlock.
- Acquire a synchronization object that is owned by code that is waiting to acquire the loader lock. This can cause a deadlock.
Initialize COM threads by using CoInitializeEx. Under certain conditions, this function can call LoadLibraryEx.
- Call the registry functions. These functions are implemented in Advapi32.dll. If Advapi32.dll is not initialized before your DLL, the DLL can access uninitialized memory and cause the process to crash.
- Call CreateProcess. Creating a process can load another DLL.
- Call ExitThread. Exiting a thread during DLL detach can cause the loader lock to be acquired again, causing a deadlock or a crash.
- Call CreateThread. Creating a thread can work if you do not synchronize with other threads, but it is risky.
- Create a named pipe or other named object (Windows 2000 only). In Windows 2000, named objects are provided by the Terminal Services DLL. If this DLL is not initialized, calls to the DLL can cause the process to crash.
- Use the memory management function from the dynamic C Run-Time (CRT). If the CRT DLL is not initialized, calls to these functions can cause the process to crash.
- Call functions in User32.dll or Gdi32.dll. Some functions load another DLL, which may not be initialized.
- Use managed code.
Post 04 Dec 2017, 03:57
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 15303
Location: Bigweld Industries

jochenvnltn wrote:

donn wrote:
If you want to conform to fastcall, then rsi/rdi:

push rbx rbp rdi rsi rsp r12 r13 r14 r15
...
pop r15 r14 r13 r12 rsp rsi rdi rbp rbx

for example (within a function). When moving on to x64, I found fastcall to contain some tricky quirks in the beginning.

MSDN's x64 calling conventions pages were the most helpful:

https://docs.microsoft.com/en-us/cpp/build/overview-of-x64-calling-conventions (think that's the main page).

The boards here have some good discussions on it also. I searched and found some good results (furs has good alignment tips, x64/fastcall searches should turn up results too). I'm not working with .dl's though.

thank you for the link! I found some useful info here https://docs.microsoft.com/en-us/cpp/build/prolog-and-epilog

PUSH/POP of RSP is not needed, and it might be dangerous to POP RSP anyway. Plus fastcall is more than just the volatile registers; the stack must also remain aligned to a multiple of 16 bytes, and a few other important points about the "shadow" space etc.
Post 04 Dec 2017, 06:46
View user's profile Send private message Visit poster's website Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  
Goto page 1, 2  Next

< Last Thread | Next Thread >

Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001-2005 phpBB Group.

Main index   Download   Documentation   Examples   Message board
Copyright © 2004-2016, Tomasz Grysztar.