flat assembler
Message board for the users of flat assembler.

flat assembler > Windows > FASM, DLL problem

Author
Thread Post new topic Reply to topic
pitchblack



Joined: 18 May 2019
Posts: 7
Good Day everyone

I have an issue with making a DLL that receive a pointer in a string.

Short story long; I'm developing some tool in Game Maker Studio, where I create a buffer and send the pointer of that buffer to the DLL function - as a string (don't ask why).

in this buffer I have 'poked' a few bytes of numbers, that I just change to some other byte numbers in the DLL function - all just for testing.
When returning to the caller (Game Maker) and 'peek' the buffer, NOTHING has changed - not a bit.

On the messageboard for Game Maker someone created, first a C++ to make a test, and then an assembly (flat) though I think he uses MASM (not sure, didn't ask).

Here's my Full code of the entire DLL:

Code:
format PE GUI 4.0 DLL
entry EntryPoint

include 'win32a.inc'

section '.code' code readable executable

proc EntryPoint hinstDLL,fdwReason,lpvReserved
        mov     eax,1
        ret
endp

proc Blow buf
local buf:DWORD
     mov eax,[buf]
     mov [eax],BYTE 71
     add eax,1
     mov [eax],BYTE 72
     add eax,1
     mov [eax],BYTE 73
    fld1
    ret
endp

section '.edata' export data readable

  export 'MyDLL.dll',Blow,'Blow'

section '.reloc' fixups data discardable
    


As everyone can see, nothing fancy and I've not changed anything in how FASM compile - almost just installed it.

Here's the other guy's code (which is working). It;s quite long cause he put up a messagebox to check if anything got there:

Code:
.model flat, stdcall
option casemap :none

includelib .\lib\user32.lib


.code
MessageBoxA             PROTO :DWORD,:DWORD,:DWORD,:DWORD


; Bare minimum required to construct a vaild DLL entrypoint
LibMain proc hInstDLL:DWORD, reason:DWORD, unused:DWORD

    mov eax, 1
    ret

LibMain endp


buffer$ = 8
dll_test proc
  
    ; Preserve EBP register.
    ; This is something to do with the calling address and the path back to the calling
    ; program. I don't really know the full details behind this one. Application crashes I
    ; don't do it.
    push ebp
    mov ebp, esp
  
    ; Preserve the EAX register
    ; This is done because the Windows Message Handler is contantly using the EAX
    ; register. Not preserving this will make the application behave screwy, if you
    ; use the EA registers in your program.
    push eax

    ; Display current contents of buffer to a windows message box.
    mov    eax, DWORD PTR buffer$[ebp]
    push 0
    push eax
    push eax
    push 0
    call MessageBoxA
  
    ; Re-write the first three characters in the buffer.
    mov    eax, DWORD PTR buffer$[ebp]
    mov bl, 68
    mov [eax+0], bl
    mov bl, 69
    mov [eax+1], bl
    mov bl, 70
    mov [eax+2], bl

    ; Restore the EAX register back to how it was when you started.
    pop eax

    ; Restore EBP register.
    pop ebp

    ; Return code '1' to the calling application.
    fld1
    ret    0

dll_test endp


end LibMain
    


Why is FASM's version not working - what am I missing??
EDIT: And yes, I've tried many things - for days now - I even checked that I actually got the pointer with a messagebox plus I checked if the bytes got changed to the new values, also with a messagebox, and it all worked.
But back in the buffer nothing was changed
Post 18 May 2019, 14:41
View user's profile Send private message Reply with quote
ProMiNick



Joined: 24 Mar 2012
Posts: 353
Location: Russian Federation, Sochi
Thou wil be laught, but thour enemy is one line of thour code:
Code:
local buf:DWORD    
Post 18 May 2019, 17:34
View user's profile Send private message Send e-mail Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 16707
Location: In your JS exploiting you and your system
Yes, You defined "buf" twice. The most recent definition will win and you are reading from an uninitialised local variable on the stack.
Post 18 May 2019, 19:33
View user's profile Send private message Visit poster's website Reply with quote
Tomasz Grysztar
Assembly Artist


Joined: 16 Jun 2003
Posts: 7312
Location: Kraków, Poland
BTW, when trying to assemble the same source with fasmg (and its fasm-compatible Windows headers), it does signalize the problem precisely.
Post 18 May 2019, 20:12
View user's profile Send private message Visit poster's website Reply with quote
pitchblack



Joined: 18 May 2019
Posts: 7
Hmmm..... thanks guys, I'll try it immediately, though I've already tried it without the initialization of 'buf' before - but just for "you never know".
Problem before was that if I didn't initialized 'buf', either as local or a global, I couldn't see that anything got transfered.

Anyway, will try again Smile
Post 19 May 2019, 06:31
View user's profile Send private message Reply with quote
pitchblack



Joined: 18 May 2019
Posts: 7
Well .... I removed the local buf:DWORD so it's just proc Blow buf -didn't work
I tried proc Blow buf:DWORD as well -didn't work.
Then I replaced eax with ebx (as buf probably is equal to eax), that didn't work either.
Also, the fld1 doesn't return anything but that's a minor concern at the moment. If I can get that buffer thing to work, I won't need to be able to return anything.
Post 19 May 2019, 07:33
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 16707
Location: In your JS exploiting you and your system
Show us how you are calling it.

The fld1 is useless, so removing that is okay.

Replacing eax with ebx might be bad if your calling function expects ebx to be preserved. Otherwise eax should be fine.
Post 19 May 2019, 09:00
View user's profile Send private message Visit poster's website Reply with quote
sinsi



Joined: 10 Aug 2007
Posts: 688
Location: Adelaide
What is your code for calling the Blow proc?
Post 19 May 2019, 09:10
View user's profile Send private message Reply with quote
pitchblack



Joined: 18 May 2019
Posts: 7
Well, there's not really much code to show - the call comes from the Game Maker Studio Extension, so I just put in the relevant info. I have no idea how it is doing the calling, but I assume it's more or less standard procedure, as others have no problem in either C++ or asm - in asm's case it was made in MASM.
Both tests were exactly like mine.

I'd have taken a screenshot of it, but no easy way to insert pics here, so I'll just tell it:
There are 6 inputs
1 the function name INSIDE Game Maker - the name we use when using the function in GML (Game Maker Language)

2. The exported name - the function name that GML will use to call the function in the DLL

3. An info text that shows when you type the function inside GML

4. Returning param of the function (Either a String or a Double)

5. The calling method (STDCALL or CDECL)

6. Parameters to hand over to the function - can be up to 7 - It's either a String or a Double - for a pointer we should use string.

So my inpust are just
1. blow
2. Blow
3. blow(ptr) - not relevant really in this
4. Double
5. STDCALL
6. String

So in the GML I just write a=blow(buffer_address)
And straight after I check the buffer's values and they are exactly the same as before I called the function. Even the 'a' value which is just zero.

Here's a link to the subject on the other forum, where you all can see both the C version and ASM version in more details.
https://forum.yoyogames.com/index.php?threads/assembler-gms-1-4-and-dll.62941/
Post 19 May 2019, 14:53
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 16707
Location: In your JS exploiting you and your system
I notice the original MASM code uses "RET 0" so that would suggest it is a CCALL function.

By default fasm uses STDCALL for functions unless you specify "ccall" in the proc definition.
Code:
proc ccall Blow buf
;...    
Post 19 May 2019, 15:05
View user's profile Send private message Visit poster's website Reply with quote
pitchblack



Joined: 18 May 2019
Posts: 7
revolution wrote:
I notice the original MASM code uses "RET 0" so that would suggest it is a CCALL function.

By default fasm uses STDCALL for functions unless you specify "ccall" in the proc definition.
Code:
proc ccall Blow buf
;...    


Right on it to do some testing though not sure it's the problem (but Far from expert on that subject Smile ). I've tried it before with changing call methods, but there probably was other things in the test code at that time. This time I just do it as simple as possible without messageboxes and so on.
But I noticed in the code that it states STDCALL at the beginning, so I assumed it was that - but but but ...... one never know Smile
Post 19 May 2019, 15:12
View user's profile Send private message Reply with quote
fasmnewbie



Joined: 01 Mar 2011
Posts: 549
Your DLL can't have empty fixups/pointers in the relocation section. Change it to something like this;
Code:
section '.reloc' fixups data readable discardable
    if $=$$
       dd 0,8  ;dummy values
    end if    

"readable" flag is required.
Post 19 May 2019, 18:23
View user's profile Send private message Visit poster's website Reply with quote
pitchblack



Joined: 18 May 2019
Posts: 7
fasmnewbie wrote:
Your DLL can't have empty fixups/pointers in the relocation section. Change it to something like this;
Code:
section '.reloc' fixups data readable discardable
    if $=$$
       dd 0,8  ;dummy values
    end if    

"readable" flag is required.


I will try that as well - the changing of call method in either fasm or game maker didn't help, which I expected really ..... from my knowledge all it does is letting the caller clean up the stack frame
Post 19 May 2019, 18:47
View user's profile Send private message Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 2792
Location: dank orb
All the abstraction and work arounds for the abstractions are causing undue confusion, imho. The routine is just:
Code:
dll_test:
    mov eax,[esp+4]

    ; do something with the address

    retn    

_________________
¯\(°_o)/¯ unlicense.org
Post 19 May 2019, 18:47
View user's profile Send private message Visit poster's website Reply with quote
pitchblack



Joined: 18 May 2019
Posts: 7
Ok, changing the '.reloc' Did provide some other result - now game maker crashes
Post 19 May 2019, 18:58
View user's profile Send private message Reply with quote
fasmnewbie



Joined: 01 Mar 2011
Posts: 549
That's good. That means you have other issues to attend to because having empty fixup doesn't crash your program. Try simple test case just to see how DLL works?

Code:
//test.c
//gcc -m32 test.c mydll.dll -o test.exe

#include <stdio.h>

extern int Blow();

int main()
{
        printf("Return value is %d\n",Blow()); //should return -3
        return 0;
}    


Your simple DLL... does nothing but to return -3 in EAX
Code:
;mydll.asm

format PE DLL
entry DLLEntryPoint
include 'win32a.inc'

section '.code' code readable executable

proc DLLEntryPoint hinstDLL,fdwReason,lpvReserved
        mov eax,TRUE
        ret
endp

proc Blow c
     mov eax,-3  ;test return value
     ret
endp

section '.edata' export data readable

  export 'mydll.dll',Blow,'Blow'

section '.reloc' fixups data readable discardable
    if $=$$
       dd 0,8   ;dummy values
    end if    
Post 19 May 2019, 19:27
View user's profile Send private message Visit poster's website Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  


< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2019, Tomasz Grysztar.

Powered by rwasa.