DOS memory allocation and stack overwrite.

Index > DOS > DOS memory allocation and stack overwrite.

Author

Thread

BoraxMan

Joined: 10 Jul 2020
Posts: 3

BoraxMan 10 Jul 2020, 11:58

I am playing with DOS memory allocation for some retroprogramming, and have run into a problem where a retf call does not return to the callee. I have found that the stack is being overwritten, but I cannot figure out why.

The DOS calls to resize the EXE image and allocate memory success, and the location of the segments are as follows (in deciman paragraphs), at least on my system.

DATA2 : 3531
ENDSEG: 6093
STACK16: 3506
Newly allocated memory: 6094

Now the function 48h call returns a pointer in AX, which is 6094 decimal. I write 44 decimal in the first 40,000 spaces, and the following retf doesn't return. Using a debugger, I found that the last portion of the stack was overwritten with 0x2C, which is 44 decimal. Using a debugger, the retf call jumps to location 0x2c2c. Somehow I am overwriting the end of the stack.

Also, if I shrink the size of buffer to something small, say 4096 bytes, it may work.

Why is my stack being overwritten? It is located well before the data section.

Code:

        FORMAT MZ
        ; DOS 16-bit EXE format

        ENTRY CODE16:ShowMsg
        STACK STACK16:stackdata
        
SEGMENT CODE16 USE16
ShowMsg:

        call CODE4:ftest
        mov ax, DATA16
        mov ds,ax
        mov ah,09h
        mov dx, Msg
        int 21h
        mov ax,4c00h     ; Send exit code to dos
        int 21h         ; Send command to DOS

    
SEGMENT DATA16 USE16
    Msg db "Goodbye World!$"

SEGMENT STACK16 USE16
        stackdata rb 256
ENDSTACK:

SEGMENT CODE4 USE16
ftest:
        mov ax, DATA2
        call printAscifiedInt
        mov ax, ENDSEG
        call printAscifiedInt
        mov ax, STACK16
        call printAscifiedInt
        
        push ds
        pop ax

        mov bx, ENDSEG
        sub bx,ax
        mov ah,4ah
        int 21h  ; Deallocate memory from ENDSEG
        jc nomemm

        mov ax,DATA2
        mov ds,ax
        mov ah,48h
        mov bx,4000 ; Allocate 64K
        int 21h
        jc nomemm

        mov [off],ax 
        mov ds,[off] ; Move returned offset to DS  DS = 6094 decimal
        call printAscifiedInt
        mov cx,40000 ; Fill memory with 44's
        mov si,0
loopp:
        mov byte [ds:si],44 ; This iverwrites the stack?
        inc si
        loop    loopp
        retf ;  Because the stack is overwritten, this jumps somewhere undesirable.

end3:
        mov     ax,4c00h         ; Send exit code to dos
        int     21h             ; Send command to DOS

nomemm:
        mov ax,0900h
        mov dx,dood
        int 21h
        jmp end3

printAscifiedInt:
        ;initilize count
        pusha
        
        mov cx,0 
        mov dx,0 
    label1: 
        ; if ax is zero 
        cmp ax,0 
        je print1       
          
        ;initilize bx to 10 
        mov bx,10         
          
        ; extract the last digit 
        div bx                   
          
        ;push it in the stack 
        push dx               
          
        ;increment the count 
        inc cx               
          
        ;set dx to 0  
        xor dx,dx 
        jmp label1 
    print1: 
        ;check if count  
        ;is greater than zero 
        cmp cx,0 
        je exit
          
        ;pop the top of stack 
        pop dx 
          
        ;add 48 so that it  
        ;represents the ASCII 
        ;value of digits 
        add dx,48 
          
        ;interuppt to print a 
        ;character 
        mov ah,02h 
        int 21h 
          
        ;decrease the count 
        dec cx 
        jmp print1
exit:
        push ax
        push ds
        mov ax,DATA2
        mov ds,ax
        mov ah,09h
        mov dx, endl
        int 21h
        pop ds
        pop ax

        popa

        ret
        
SEGMENT DATA2 USE16
        dood db "Failed operation",10,13,24h
        off dw 4
        endl db 10,13,24h
buffer2:
        buffer rb 40960         ; Removing this line fixes it?


SEGMENT ENDSEG

10 Jul 2020, 11:58

revolution
When all else fails, read the source

Joined: 24 Aug 2004
Posts: 20299
Location: In your JS exploiting you and your system

revolution 10 Jul 2020, 12:07

What are the values of SP and SS?

10 Jul 2020, 12:07

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8351
Location: Kraków, Poland

Tomasz Grysztar 10 Jul 2020, 12:24

BoraxMan wrote:

Code:

        STACK STACK16:stackdata

The stack expands downwards, not upwards, so when you set up the initial stack pointer, you should point to the top of the area, not the bottom. It should be:

Code:

        STACK STACK16:ENDSTACK

Because "stackdata" in your case is offset 0, you had initial value of SP set to 0, and pushing anything to the stack was wrapping to 0FFFEh, which is further in memory (and ended up being overwritten).

10 Jul 2020, 12:24

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8351
Location: Kraków, Poland

Tomasz Grysztar 10 Jul 2020, 12:29

Also, additional bit of advice: instead of having to resize the program segment, you can make your program image not allocate excess memory in the first place. Do it with:

Code:

        HEAP 0

(you can put it next to your ENTRY and STACK settings).

This tells DOS to not allocate any additional memory on top of your defined segments, and leaves the rest of conventional memory available. Then you can safely get rid of that initial 4Ah system call. It was only needed for .COM programs, or maybe as a workaround for MZ linkers that had no option to make the program image not "greedy" (although I don't remember whether that was a real issue).

10 Jul 2020, 12:29

BoraxMan

Joined: 10 Jul 2020
Posts: 3

BoraxMan 10 Jul 2020, 14:45

Tomasz Grysztar wrote:

BoraxMan wrote:
Code:
        STACK STACK16:stackdata    
The stack expands downwards, not upwards, so when you set up the initial stack pointer, you should point to the top of the area, not the bottom. It should be:
Code:
        STACK STACK16:ENDSTACK    
Because "stackdata" in your case is offset 0, you had initial value of SP set to 0, and pushing anything to the stack was wrapping to 0FFFEh, which is further in memory (and ended up being overwritten).

That makes sense. My previous assembly experience is with DOS and Linux, but with MASM for DOS many years ago, and I don't recall having to define the stack in any more detail than simply using a directive. With Linux, the stack is set up by the OS.

I based my code off a tutorial here
https://www.codeproject.com/Articles/45788/The-Real-Protected-Long-mode-assembly-tutorial-for

But that has errors, and claims that SS and SP are automatically set. My error was assuming that SP would point to the top of the stack, and I think the example program in the tutorial has the same bug/assumption.

It works when I use
STACK STACK16:ENDSTACK

If I use in place of that, "STACK 100h", I note that the stack is placed right at the end, at location 0x1763. ENDSEG is at 0x1763, and DOS allocates memory at location 0x1764, which overwrites the stack again. So FASM must be putting the stack at the end, after ENDSEG and therefore I'm "freeing" the stack with my call to resize the program.

Removing the code which calls on DOS frees memory (Fuction 4Ah), and using "HEAP 0" as per your recommendation, results in correct behaviour.

Thanks for your help. Documentation was a little hard to come by with regards to proper memory management in DOS

10 Jul 2020, 14:45

< Last Thread | Next Thread >

Forum Rules:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum