FASM, DLL problem

Index > Windows > FASM, DLL problem

Author

Thread

pitchblack

Joined: 18 May 2019
Posts: 7

pitchblack 18 May 2019, 14:41

Good Day everyone

I have an issue with making a DLL that receive a pointer in a string.

Short story long; I'm developing some tool in Game Maker Studio, where I create a buffer and send the pointer of that buffer to the DLL function - as a string (don't ask why).

in this buffer I have 'poked' a few bytes of numbers, that I just change to some other byte numbers in the DLL function - all just for testing.
When returning to the caller (Game Maker) and 'peek' the buffer, NOTHING has changed - not a bit.

On the messageboard for Game Maker someone created, first a C++ to make a test, and then an assembly (flat) though I think he uses MASM (not sure, didn't ask).

Here's my Full code of the entire DLL:

Code:

format PE GUI 4.0 DLL
entry EntryPoint

include 'win32a.inc'

section '.code' code readable executable

proc EntryPoint hinstDLL,fdwReason,lpvReserved
        mov     eax,1
        ret
endp

proc Blow buf
local buf:DWORD
     mov eax,[buf]
     mov [eax],BYTE 71
     add eax,1
     mov [eax],BYTE 72
     add eax,1
     mov [eax],BYTE 73
    fld1
    ret
endp

section '.edata' export data readable

  export 'MyDLL.dll',Blow,'Blow'

section '.reloc' fixups data discardable

As everyone can see, nothing fancy and I've not changed anything in how FASM compile - almost just installed it.

Here's the other guy's code (which is working). It;s quite long cause he put up a messagebox to check if anything got there:

Code:

.model flat, stdcall
option casemap :none

includelib .\lib\user32.lib


.code
MessageBoxA             PROTO :DWORD,:DWORD,:DWORD,:DWORD


; Bare minimum required to construct a vaild DLL entrypoint
LibMain proc hInstDLL:DWORD, reason:DWORD, unused:DWORD

    mov eax, 1
    ret

LibMain endp


buffer$ = 8
dll_test proc
  
    ; Preserve EBP register.
    ; This is something to do with the calling address and the path back to the calling
    ; program. I don't really know the full details behind this one. Application crashes I
    ; don't do it.
    push ebp
    mov ebp, esp
  
    ; Preserve the EAX register
    ; This is done because the Windows Message Handler is contantly using the EAX
    ; register. Not preserving this will make the application behave screwy, if you
    ; use the EA registers in your program.
    push eax

    ; Display current contents of buffer to a windows message box.
    mov    eax, DWORD PTR buffer$[ebp]
    push 0
    push eax
    push eax
    push 0
    call MessageBoxA
  
    ; Re-write the first three characters in the buffer.
    mov    eax, DWORD PTR buffer$[ebp]
    mov bl, 68
    mov [eax+0], bl
    mov bl, 69
    mov [eax+1], bl
    mov bl, 70
    mov [eax+2], bl

    ; Restore the EAX register back to how it was when you started.
    pop eax

    ; Restore EBP register.
    pop ebp

    ; Return code '1' to the calling application.
    fld1
    ret    0

dll_test endp


end LibMain

Why is FASM's version not working - what am I missing??
EDIT: And yes, I've tried many things - for days now - I even checked that I actually got the pointer with a messagebox plus I checked if the bytes got changed to the new values, also with a messagebox, and it all worked.
But back in the buffer nothing was changed

18 May 2019, 14:41

ProMiNick

Joined: 24 Mar 2012
Posts: 805
Location: Russian Federation, Sochi

ProMiNick 18 May 2019, 17:34

Thou wil be laught, but thour enemy is one line of thour code:

Code:

local buf:DWORD

18 May 2019, 17:34

revolution
When all else fails, read the source

Joined: 24 Aug 2004
Posts: 20454
Location: In your JS exploiting you and your system

revolution 18 May 2019, 19:33

Yes, You defined "buf" twice. The most recent definition will win and you are reading from an uninitialised local variable on the stack.

18 May 2019, 19:33

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8359
Location: Kraków, Poland

Tomasz Grysztar 18 May 2019, 20:12

BTW, when trying to assemble the same source with fasmg (and its fasm-compatible Windows headers), it does signalize the problem precisely.

18 May 2019, 20:12

pitchblack

Joined: 18 May 2019
Posts: 7

pitchblack 19 May 2019, 06:31

Hmmm..... thanks guys, I'll try it immediately, though I've already tried it without the initialization of 'buf' before - but just for "you never know".
Problem before was that if I didn't initialized 'buf', either as local or a global, I couldn't see that anything got transfered.

Anyway, will try again Smile

19 May 2019, 06:31

pitchblack

Joined: 18 May 2019
Posts: 7

pitchblack 19 May 2019, 07:33

Well .... I removed the local buf:DWORD so it's just proc Blow buf -didn't work
I tried proc Blow buf:DWORD as well -didn't work.
Then I replaced eax with ebx (as buf probably is equal to eax), that didn't work either.
Also, the fld1 doesn't return anything but that's a minor concern at the moment. If I can get that buffer thing to work, I won't need to be able to return anything.

19 May 2019, 07:33

revolution
When all else fails, read the source

Joined: 24 Aug 2004
Posts: 20454
Location: In your JS exploiting you and your system

revolution 19 May 2019, 09:00

Show us how you are calling it.

The fld1 is useless, so removing that is okay.

Replacing eax with ebx might be bad if your calling function expects ebx to be preserved. Otherwise eax should be fine.

19 May 2019, 09:00

sinsi

Joined: 10 Aug 2007
Posts: 794
Location: Adelaide

sinsi 19 May 2019, 09:10

What is your code for calling the Blow proc?

19 May 2019, 09:10

pitchblack

Joined: 18 May 2019
Posts: 7

pitchblack 19 May 2019, 14:53

Well, there's not really much code to show - the call comes from the Game Maker Studio Extension, so I just put in the relevant info. I have no idea how it is doing the calling, but I assume it's more or less standard procedure, as others have no problem in either C++ or asm - in asm's case it was made in MASM.
Both tests were exactly like mine.

I'd have taken a screenshot of it, but no easy way to insert pics here, so I'll just tell it:
There are 6 inputs
1 the function name INSIDE Game Maker - the name we use when using the function in GML (Game Maker Language)

2. The exported name - the function name that GML will use to call the function in the DLL

3. An info text that shows when you type the function inside GML

4. Returning param of the function (Either a String or a Double)

5. The calling method (STDCALL or CDECL)

6. Parameters to hand over to the function - can be up to 7 - It's either a String or a Double - for a pointer we should use string.

So my inpust are just
1. blow
2. Blow
3. blow(ptr) - not relevant really in this
4. Double
5. STDCALL
6. String

So in the GML I just write a=blow(buffer_address)
And straight after I check the buffer's values and they are exactly the same as before I called the function. Even the 'a' value which is just zero.

Here's a link to the subject on the other forum, where you all can see both the C version and ASM version in more details.
https://forum.yoyogames.com/index.php?threads/assembler-gms-1-4-and-dll.62941/

19 May 2019, 14:53

revolution
When all else fails, read the source

Joined: 24 Aug 2004
Posts: 20454
Location: In your JS exploiting you and your system

revolution 19 May 2019, 15:05

I notice the original MASM code uses "RET 0" so that would suggest it is a CCALL function.

By default fasm uses STDCALL for functions unless you specify "ccall" in the proc definition.

Code:

proc ccall Blow buf
;...

19 May 2019, 15:05

pitchblack

Joined: 18 May 2019
Posts: 7

pitchblack 19 May 2019, 15:12

revolution wrote:

I notice the original MASM code uses "RET 0" so that would suggest it is a CCALL function.

By default fasm uses STDCALL for functions unless you specify "ccall" in the proc definition.
Code:
proc ccall Blow buf
;...    

Right on it to do some testing though not sure it's the problem (but Far from expert on that subject Smile

). I've tried it before with changing call methods, but there probably was other things in the test code at that time. This time I just do it as simple as possible without messageboxes and so on.
But I noticed in the code that it states STDCALL at the beginning, so I assumed it was that - but but but ...... one never know Smile

19 May 2019, 15:12

fasmnewbie

Joined: 01 Mar 2011
Posts: 555

fasmnewbie 19 May 2019, 18:23

Your DLL can't have empty fixups/pointers in the relocation section. Change it to something like this;

Code:

section '.reloc' fixups data readable discardable
    if $=$$
       dd 0,8  ;dummy values
    end if

"readable" flag is required.

19 May 2019, 18:23

pitchblack

Joined: 18 May 2019
Posts: 7

pitchblack 19 May 2019, 18:47

fasmnewbie wrote:

Your DLL can't have empty fixups/pointers in the relocation section. Change it to something like this;
Code:
section '.reloc' fixups data readable discardable
    if $=$$
       dd 0,8  ;dummy values
    end if    
"readable" flag is required.

I will try that as well - the changing of call method in either fasm or game maker didn't help, which I expected really ..... from my knowledge all it does is letting the caller clean up the stack frame

19 May 2019, 18:47

bitRAKE

Joined: 21 Jul 2003
Posts: 4077
Location: vpcmpistri

bitRAKE 19 May 2019, 18:47

All the abstraction and work arounds for the abstractions are causing undue confusion, imho. The routine is just:

Code:

dll_test:
    mov eax,[esp+4]

    ; do something with the address

    retn

_________________
¯\(°_o)/¯ “languages are not safe - uses can be” Bjarne Stroustrup

19 May 2019, 18:47

pitchblack

Joined: 18 May 2019
Posts: 7

pitchblack 19 May 2019, 18:58

Ok, changing the '.reloc' Did provide some other result - now game maker crashes

19 May 2019, 18:58

fasmnewbie

Joined: 01 Mar 2011
Posts: 555

fasmnewbie 19 May 2019, 19:27

That's good. That means you have other issues to attend to because having empty fixup doesn't crash your program. Try simple test case just to see how DLL works?

Code:

//test.c
//gcc -m32 test.c mydll.dll -o test.exe

#include <stdio.h>

extern int Blow();

int main()
{
        printf("Return value is %d\n",Blow()); //should return -3
        return 0;
}

Your simple DLL... does nothing but to return -3 in EAX

Code:

;mydll.asm

format PE DLL
entry DLLEntryPoint
include 'win32a.inc'

section '.code' code readable executable

proc DLLEntryPoint hinstDLL,fdwReason,lpvReserved
        mov eax,TRUE
        ret
endp

proc Blow c
     mov eax,-3  ;test return value
     ret
endp

section '.edata' export data readable

  export 'mydll.dll',Blow,'Blow'

section '.reloc' fixups data readable discardable
    if $=$$
       dd 0,8   ;dummy values
    end if

19 May 2019, 19:27

< Last Thread | Next Thread >

Forum Rules:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum