format PE64 GUI 4.0 DLL
entry DllEntryPoint

include 'win64ax.inc'

section '.code' code readable executable

proc DllEntryPoint hinstDLL,fdwReason,lpvReserved
        mov     eax,TRUE
        .if [fdwReason] = DLL_PROCESS_ATTACH
        invoke MessageBox,NULL,szText,szTitle,MB_OK
        .endif
        invoke ExitProcess,0

endp

proc DummyProc
     ret
endp


section '.data' data readable writeable

szText db '64Bit DLL!!',0
szTitle db 'Success',0



section '.idata' import data readable writeable
library user32,'user32.dll',kernel32,'kernel32.dll'


  import kernel32,\
            ExitProcess,'ExitProcess'

  import user32,\
         MessageBox,'MessageBoxA'


section '.edata' export data readable

  export 'Mydll.dll',\
         DummyProc,'DummyProc'



data        fixups
end         data 
    

Try to simplify your DLLMain proc to basic initialization task only. Calling MessageBox and / or ExitProcess will call another DLLs. Replace ExiitProcess with ret. You probably need dummy fixups too.

format PE64 GUI 4.0 DLL
entry DllEntryPoint

include 'win64ax.inc'



proc DllEntryPoint hinstDLL:QWORD,fdwReason:QWORD,lpvReserved:QWORD

    mov QWORD  [rbp+10h], rcx
    mov QWORD  [rbp+18h], rdx
    mov QWORD  [rbp+20h], r8
    mov QWORD  [rbp+28h], r9

.if [fdwReason] = DLL_PROCESS_ATTACH


        invoke MessageBox,NULL,szText,szTitle,MB_OK
        .endif
        invoke ExitProcess,0

endp




szText db '64Bit DLL!!',0
szTitle db 'Success',0



section '.idata' import data readable writeable
library user32,'user32.dll',kernel32,'kernel32.dll'


  import kernel32,\
            ExitProcess,'ExitProcess'

  import user32,\
         MessageBox,'MessageBoxA'





data        fixups
end         data
    

    mov QWORD  [rbp+10h], rcx
    mov QWORD  [rbp+18h], rdx
    mov QWORD  [rbp+20h], r8
    mov QWORD  [rbp+28h], r9 
    

fdwReason is in the register rdx. You can check the value in rdx directly instead of transferring it to the stack first.

Note that [rbp+18h] is the same as saying [fdwReason].

format PE64 GUI 4.0 DLL
entry DllEntryPoint

include 'win64ax.inc'

proc DllEntryPoint hinstDLL:QWORD,fdwReason:QWORD,lpvReserved:QWORD

.if rdx = DLL_PROCESS_ATTACH


        invoke MessageBox,NULL,szText,szTitle,MB_OK
        .endif
        invoke ExitProcess,0
endp

szText db '64Bit DLL!!',0
szTitle db 'Success',0

section '.idata' import data readable writeable
library user32,'user32.dll',kernel32,'kernel32.dll'


  import kernel32,\
            ExitProcess,'ExitProcess'

  import user32,\
         MessageBox,'MessageBoxA'

data        fixups
end         data
    

It is not usual for a DLL to use ExitProcess. Normally this is left to the exe file to do that after it has cleaned up all its stuff.

MS have the fastcall convention for Windows. It is only applicable when calling the API functions, but you can still use it in your own internal functions also.

If you want to conform to fastcall, then rsi/rdi:

push rbx rbp rdi rsi rsp r12 r13 r14 r15
...
pop r15 r14 r13 r12 rsp rsi rdi rbp rbx

for example (within a function). When moving on to x64, I found fastcall to contain some tricky quirks in the beginning.

MSDN's x64 calling conventions pages were the most helpful:

https://docs.microsoft.com/en-us/cpp/build/overview-of-x64-calling-conventions (think that's the main page).

The boards here have some good discussions on it also. I searched and found some good results (furs has good alignment tips, x64/fastcall searches should turn up results too). I'm not working with .dl's though.

fasmnewbie wrote:

Try to simplify your DLLMain proc to basic initialization task only. Calling MessageBox and / or ExitProcess will call another DLLs. Replace ExiitProcess with ret. You probably need dummy fixups too.

nope.. that wasn't the problem ..

donn wrote:

If you want to conform to fastcall, then rsi/rdi:

push rbx rbp rdi rsi rsp r12 r13 r14 r15
...
pop r15 r14 r13 r12 rsp rsi rdi rbp rbx

for example (within a function). When moving on to x64, I found fastcall to contain some tricky quirks in the beginning.

MSDN's x64 calling conventions pages were the most helpful:

https://docs.microsoft.com/en-us/cpp/build/overview-of-x64-calling-conventions (think that's the main page).

The boards here have some good discussions on it also. I searched and found some good results (furs has good alignment tips, x64/fastcall searches should turn up results too). I'm not working with .dl's though.

thank you for the link! I found some useful info here https://docs.microsoft.com/en-us/cpp/build/prolog-and-epilog