flat assembler
Message board for the users of flat assembler.

Index > High Level Languages > ASM + VB

Author
Thread Post new topic Reply to topic
mr_orche



Joined: 10 Jun 2005
Posts: 4
Location: Yogyakarta, Indonesia
mr_orche
i tried to create a DLL using stdcall convention to be called from VB app. the DLL was modified
from EXAMPLES\ERRORMSG.ASM, and it contains a procedure named SubTest, a function named
FunctionTest (returning a long value via eax), and a few procedures existing from previous
source code (ERRORMSG.ASM).

when a call to SubTest was executed from VB app, it just works fine
as i expect, this also true when a call made to FunctionTest.
but if both SubTest and FunctionTest is used in VB app, an error occured saying "Bad DLL
calling convention". i really frustated about this and don't know why and how should i
resolve this. here is the source of the DLL (modified from ERRORMSG.ASM) and my VB module:


Code:
==========================================================================================
DLL source: myprocs.asm
==========================================================================================
; DLL creation example

format PE GUI 4.0 DLL
entry DllEntryPoint

include '%include%\win32a.inc'

section '.data' data readable writeable
result db "Return this", 0

section '.code' code readable executable
proc DllEntryPoint, hinstDLL,fdwReason,lpvReserved
  enter
  mov         eax,TRUE
  return

proc ShowErrorMessage, hWnd,dwError
  .lpBuffer dd ?
  enter
  lea         eax,[.lpBuffer]
  invoke  FormatMessage,FORMAT_MESSAGE_ALLOCATE_BUFFER+FORMAT_MESSAGE_FROM_SYSTEM,0,[dwError],LANG_NEUTRAL,eax,0,0
  invoke  MessageBox,[hWnd],[.lpBuffer],NULL,MB_ICONERROR+MB_OK
  invoke  LocalFree,[.lpBuffer]
  return

; VOID ShowLastError(HWND hWnd);
proc ShowLastError, hWnd
  enter
  ;invoke  GetLastError
  mov eax, 21
  stdcall ShowErrorMessage,[hWnd],eax
  return

proc SubTest, lngvalue
  enter
  mov edi, [lngvalue]
  mov eax, 99
  mov [edi+0], eax
  return

proc FunctionTest
  enter
  ;mov edi, [lngvalue]
  ;mov edi, eax
  mov eax, 99
  ;mov [edi+0], eax
  ;mov [edi+0], eax
  ;return
  return


section '.idata' import data readable writeable
  library kernel,'KERNEL32.DLL',\
          user,'USER32.DLL'
  import kernel,\
  GetLastError,'GetLastError',\
    SetLastError,'SetLastError',\
    FormatMessage,'FormatMessageA',\
         LocalFree,'LocalFree'
  import user,\
         MessageBox,'MessageBoxA'

section '.edata' export data readable
  export 'myprocs.dll',\
    FunctionTest,'FunctionTest',\
    ShowErrorMessage,'ShowErrorMessage',\
    ShowLastError,'ShowLastError',\
  SubTest,'SubTest'
section '.reloc' fixups data discardable




==============================================================================================
VB module source: 
==============================================================================================
Private Declare Sub SubTest Lib "myprocs.dll" (vLong As Long)
Private Declare Function FunctionTest Lib "myprocs.dll" () As Long
Sub Main()
   'both result& and dasdf& is a Long (4 byte signed numeric) variable
   
   ' case 1: this works
   ' only 1 call made to myprocs.dll
   SubTest result&
   
   
   ' case 2: this works
   ' only 1 call made to myprocs.dll
   'dasdf& = FunctionTest
   
   
   'case 3: not work  >>  Bad DLL calling convention
   'SubTest result&
   'dasdf& = FunctionTest
End Sub

    
[/code]
Post 04 Jun 2009, 01:43
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17476
Location: In your JS exploiting you and your system
revolution
Do you need to define in VB that a function is stdcall? Maybe you can try to modify your DLL functions to be ccall and see if VB is happy about that.

BTW: Why are there no endp's in your code? Are you using a custom set of macros? Also, enter and return are not used in the current fasm macros. Perhaps you can consider downloading the latest version?
Post 04 Jun 2009, 02:36
View user's profile Send private message Visit poster's website Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 2942
Location: vpcmipstrm
bitRAKE
I don't remember what the "&" in VB means? In the first case it appears that an address is being sent to SubTest, and then FunctionTest returns an address? Since your FASM code is consistent, I believe the error lies elsewhere. Yet, try the following code to bypass possible macro problems:
Code:
; store dword value 99 at address on stack
SubTest:
   mov eax,[esp+4]
     mov dword [eax], 99
 retn 4


; return dword value
FunctionTest:
 mov eax, 99
 retn    
Are you able to confirm the operation of each by printing the result? Not just that the code executes, but also that the intended consequences are achieved?
Post 04 Jun 2009, 03:39
View user's profile Send private message Visit poster's website Reply with quote
LocoDelAssembly
Your code has a bug


Joined: 06 May 2005
Posts: 4633
Location: Argentina
LocoDelAssembly
Shouldn't "Private Declare Sub SubTest Lib "myprocs.dll" (vLong As Long)" be "Private Declare Sub SubTest Lib "myprocs.dll" (ByRef vLong As Long)"?
Post 04 Jun 2009, 04:28
View user's profile Send private message Reply with quote
pal



Joined: 26 Aug 2008
Posts: 227
pal
LocoDelAssembly nah, in VB no predecessor to the argument means that it is ByRef.

The & at the end means the variable is a long variable. E.g. &H1000& is 0x1000 in long format.

I'll install VB later if you haven't got it sorted.
Post 04 Jun 2009, 08:53
View user's profile Send private message Reply with quote
mr_orche



Joined: 10 Jun 2005
Posts: 4
Location: Yogyakarta, Indonesia
mr_orche
revolution wrote:
Do you need to define in VB that a function is stdcall? Maybe you can try to modify your DLL functions to be ccall and see if VB is happy about that.

BTW: Why are there no endp's in your code? Are you using a custom set of macros? Also, enter and return are not used in the current fasm macros. Perhaps you can consider downloading the latest version?



as far as i know, stdcall is the default calling convention for external DLL call in VB. i dont use endp, enter, and return in this code because i compiled this on previous version of FASM. when i compiled this code using the latest version of FASM, i convert this to use endp, enter and return. but the problem still persist, VB still says "Bad DLL calling convention."
Post 10 Jun 2009, 02:51
View user's profile Send private message Visit poster's website Reply with quote
mr_orche



Joined: 10 Jun 2005
Posts: 4
Location: Yogyakarta, Indonesia
mr_orche
bitRAKE wrote:
I don't remember what the "&" in VB means? In the first case it appears that an address is being sent to SubTest, and then FunctionTest returns an address? Since your FASM code is consistent, I believe the error lies elsewhere. Yet, try the following code to bypass possible macro problems:
Code:
; store dword value 99 at address on stack
SubTest:
       mov eax,[esp+4]
     mov dword [eax], 99
 retn 4


; return dword value
FunctionTest:
 mov eax, 99
 retn    
Are you able to confirm the operation of each by printing the result? Not just that the code executes, but also that the intended consequences are achieved?



the "&" suffix means a long integer data type. ok i would try this code.
Post 10 Jun 2009, 02:55
View user's profile Send private message Visit poster's website Reply with quote
mr_orche



Joined: 10 Jun 2005
Posts: 4
Location: Yogyakarta, Indonesia
mr_orche
more suggestion plz
Post 10 Jun 2009, 02:58
View user's profile Send private message Visit poster's website Reply with quote
MSWarrior



Joined: 21 Mar 2008
Posts: 11
MSWarrior
Actually the problem comes from that you must save the edi register before changing it. Try changing this code only to test if it's working:

Code:
proc SubTest, lngvalue 
  enter
  mov ecx,edi
  mov edi, [lngvalue] 
  mov eax, 99 
  mov [edi+0], eax 
  mov edi,ecx
  return    


or use the bitRAKE's version, it is working as expected without problems.
Post 10 Jun 2009, 07:06
View user's profile Send private message Reply with quote
pal



Joined: 26 Aug 2008
Posts: 227
pal
If you are gonna use enter then don't you need to use leave otherwise you screw up the stack. Plus the proc macro will do an enter for you anyway and the end of the proc bit will do the leave, so enter is not needed either way.
Post 10 Jun 2009, 07:22
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17476
Location: In your JS exploiting you and your system
revolution
pal wrote:
If you are gonna use enter then don't you need to use leave otherwise you screw up the stack. Plus the proc macro will do an enter for you anyway and the end of the proc bit will do the leave, so enter is not needed either way.
But mr_orche never fully explained what the return macro does. It seems to already combine the endp functionality, so there is no reason why it can't also have leave functionality included.
Post 10 Jun 2009, 07:31
View user's profile Send private message Visit poster's website Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  


< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2020, Tomasz Grysztar. Also on YouTube, Twitter.

Website powered by rwasa.