flat assembler
Message board for the users of flat assembler.

Index > High Level Languages > How does malloc() look like in assembly?

Goto page 1, 2  Next
Author
Thread Post new topic Reply to topic
system error



Joined: 01 Sep 2013
Posts: 670
system error 14 Nov 2014, 02:22
What's the deal with malloc()?

Code:
i_am_a_malloc rb 30 ;is this a malloc?    


or is it something like .bss (like in NASM) or is it related to stack or something?
Post 14 Nov 2014, 02:22
View user's profile Send private message Reply with quote
system error



Joined: 01 Sep 2013
Posts: 670
system error 14 Nov 2014, 02:28
and then what's the deal with free()? What to free? Program exits anyway.
Post 14 Nov 2014, 02:28
View user's profile Send private message Reply with quote
system error



Joined: 01 Sep 2013
Posts: 670
system error 14 Nov 2014, 02:46
Mom??
Post 14 Nov 2014, 02:46
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 20483
Location: In your JS exploiting you and your system
revolution 14 Nov 2014, 03:21
rb is a reservation of bytes, in this case 30 bytes. It is a type of memory allocation that is done at compile time. You probably won't have any success trying to free that type of allocation it is baked into the exe file.

But most OSes provide a runtime method of memory allocation. Perhaps similar to:
Code:
invoke VirtualAlloc,...    
That type of allocation can be freed to return it to the memory pool for other applications to use.
Post 14 Nov 2014, 03:21
View user's profile Send private message Visit poster's website Reply with quote
redsock



Joined: 09 Oct 2009
Posts: 435
Location: Australia
redsock 14 Nov 2014, 04:29
In all *nix/*bsd environments, there are two ways that the "pool" is allocated:

Code:
man sbrk    


and

Code:
man mmap    


With mmap being the "preferred" modern-day flavour. Windows has its own similar methods for divvying up available virtual memory/resources.

The Mac OS X manpages used to say something like "sbrk is a historic curiosity..." haha
Post 14 Nov 2014, 04:29
View user's profile Send private message Reply with quote
redsock



Joined: 09 Oct 2009
Posts: 435
Location: Australia
redsock 14 Nov 2014, 04:31
Ahh I should have probably said: the real malloc then uses its own datastructures to break up the larger sections returned by mmap/sbrk/etc, and in the case of ptmalloc and the like, they vary greatly in complexity Smile
Post 14 Nov 2014, 04:31
View user's profile Send private message Reply with quote
system error



Joined: 01 Sep 2013
Posts: 670
system error 14 Nov 2014, 05:27
@revo
@redsock

So malloc is a dynamic stuff. Thanks. But how do they look like in assembly. Is it easy or difficult to simulate? Any specific x86 instruction to create similar dynamic allocation in assembly? I am clueless. One more question, is 'virtual' related to this thing?
Post 14 Nov 2014, 05:27
View user's profile Send private message Reply with quote
system error



Joined: 01 Sep 2013
Posts: 670
system error 14 Nov 2014, 05:36
redsock wrote:
Ahh I should have probably said: the real malloc then uses its own datastructures to break up the larger sections returned by mmap/sbrk/etc, and in the case of ptmalloc and the like, they vary greatly in complexity Smile
What if I want to create a version of malloc that doesn't map to virtual address space? Just physical flat. Or is it even possible to do in assembly alone? Or perhaps I got this wrong from the beginning...
Post 14 Nov 2014, 05:36
View user's profile Send private message Reply with quote
JohnFound



Joined: 16 Jun 2003
Posts: 3499
Location: Bulgaria
JohnFound 14 Nov 2014, 07:21
In assembly you have several options:

1. Use system libraries - simply import needed functions in your program and then call them, when needing memory block. This way is easy and universal. In many cases, when you want to import functions from other external libraries, this is the only way, because the external libraries uses the same mechanism of memory allocation.

2. Use your own heap manager - It is not easy, but possible. For example FreshLib has implemented OS independent heap manager, based on Vid's FASM lib: Source here
This solution makes your program highly OS independent. In FreshLib it was included, because of KolibriOS, where other heap manager is not available at all.
I personally use this solution in MiniMagAsm when it needs to run on 64bit Linux servers where 32bit compatible libraries are not installed.

3. For simple tasks, it is often better to simply get one big memory block using system functions (sys_brk or VirtualAllocate) and then to work in this memory, dispatching it on the program needs. This approach, for example, is used in FASM compiler with great success.
Post 14 Nov 2014, 07:21
View user's profile Send private message Visit poster's website ICQ Number Reply with quote
redsock



Joined: 09 Oct 2009
Posts: 435
Location: Australia
redsock 14 Nov 2014, 09:28
JohnFound wrote:
... In many cases, when you want to import functions from other external libraries, this is the only way, because the external libraries uses the same mechanism of memory allocation. ...


This isn't true, mainly because a valid address is valid across calls regardless of which memory manager gave it to you. Otherwise, you wouldn't be able to, say, mmap a file into your program's address space, and then pass that address to libc (or what is traditionally given to you via ptmalloc).

So, for a true "flat" address space, of course then you are bypassing the operating system entirely (and then the answer is yes, of course you can).

When running in linux as I do, this is functionally equivalent to calling sbrk (which extends the memory available to your program past its "end").

Each memory management strategy focuses on one of two things: 1) space efficiency, 2) speed (particularly when releasing a memory region). For what are commonly referred to as "general purpose allocators", they try to do a bit of both.

Your original question of: "What would it look like in assembler?" can be imagined just by thinking about how you would coalesce freespace, how you go about searching for available space, whether that is bin-based, or tree-based, or both. At what point do you _not_ do any of that, and instead just pass those off to the kernel/operating system? (ptmalloc for example in its default settings gives you mmap space above 1mb, but anything below that it does its very best to keep it space efficient, and coalesce freespace, etc.

In summary, most memory managers that are worth anything, including my own, go to a fair bit of trouble making sure that they work for a good many purposes/environments.

Smile $0.02 thusly
Post 14 Nov 2014, 09:28
View user's profile Send private message Reply with quote
redsock



Joined: 09 Oct 2009
Posts: 435
Location: Australia
redsock 14 Nov 2014, 09:59
system error wrote:
@revo
@redsock

So malloc is a dynamic stuff. Thanks. But how do they look like in assembly. Is it easy or difficult to simulate? Any specific x86 instruction to create similar dynamic allocation in assembly? I am clueless. One more question, is 'virtual' related to this thing?

Meant to answer this directly:

When your program is running from inside the constraints (read: friendly, comfortable, nice) environment of decent operating system, the address space your program physically resides in, as well as any calls you make are all virtual (as they must be).

If you boot bare metal and go to the trouble of getting it to boot in long and flat mode, then you have address 0x4000 really means 0x4000 offset into physical memory, though this sorta thing is rarely practical. Operating systems are nice.

Back to your Q re: is there instructions to do it? Of course. Atop the linux forum here on the board, there is a syscall reference. In it, you'll find the syscall number (in x86_64 mode) is decimal 12. man 2 brk will give you the arguments it wants. To use mmap, syscall number is 9. man 2 mmap will give you the arguments it wants. The same basic thing applies to all other kernel varieties, and windows of course has its own version of what accomplishes the same basic task, and that is: give your program some dynamic memory it can read/write(and execute from if you want).

By default of course, the data segment you declare with fasm directly is fixed size and there are no CPU instructions of course to manipulate it (it would defeat the purpose of all the fine benefits virtual address space gives us re: protection/resource sharing/etc).

Mmm, that excellent red wine after dinner made for a good ranting mood, hope no one minded Razz

Cheers
Post 14 Nov 2014, 09:59
View user's profile Send private message Reply with quote
JohnFound



Joined: 16 Jun 2003
Posts: 3499
Location: Bulgaria
JohnFound 14 Nov 2014, 10:34
redsock wrote:
JohnFound wrote:
... In many cases, when you want to import functions from other external libraries, this is the only way, because the external libraries uses the same mechanism of memory allocation. ...


This isn't true...


This is true, because very often the C libraries in Linux expect the dynamic memory objects to be allocated by malloc and this way to be releasable/resizeable by C heap manager. For example, how you will release some buffer returned to you by C library, if you don't import C heap functions?

_________________
Tox ID: 48C0321ADDB2FE5F644BB5E3D58B0D58C35E5BCBC81D7CD333633FEDF1047914A534256478D9
Post 14 Nov 2014, 10:34
View user's profile Send private message Visit poster's website ICQ Number Reply with quote
gens



Joined: 18 Feb 2013
Posts: 161
gens 14 Nov 2014, 15:27
i found the explanation from Doug Lea to be the most helpful (i didn't make an allocator yet)
http://gee.cs.oswego.edu/dl/html/malloc.html
Post 14 Nov 2014, 15:27
View user's profile Send private message Reply with quote
redsock



Joined: 09 Oct 2009
Posts: 435
Location: Australia
redsock 14 Nov 2014, 19:37
JohnFound wrote:
redsock wrote:
JohnFound wrote:
... In many cases, when you want to import functions from other external libraries, this is the only way, because the external libraries uses the same mechanism of memory allocation. ...


This isn't true...


This is true, because very often the C libraries in Linux expect the dynamic memory objects to be allocated by malloc and this way to be releasable/resizeable by C heap manager. For example, how you will release some buffer returned to you by C library, if you don't import C heap functions?

Your original statement, and the bit I said wasn't true, was that you said if you want to import functions from other libraries, that you _must_ use their method of memory allocation, simply because the external library may also.

Certainly, if you are using an external library that makes calls to malloc (whether via C++ new, or whatever), you'd then have to keep track of same, and return it via free (if you wanted that space to be reusable by the external library again).

My point was only that you aren't forced to do it that way, and your comment said specifically "this is the only way", when in fact it is not.

Even without using assembler at all, we can replace the general purpose allocator with whatever allocator we like, and the same is certainly true of mixing assembler also. My earlier point of being able to use two totally different memory managers at the same time as using external library calls stands.

Smile
Post 14 Nov 2014, 19:37
View user's profile Send private message Reply with quote
system error



Joined: 01 Sep 2013
Posts: 670
system error 16 Nov 2014, 13:02
redsock wrote:

Meant to answer this directly:

When your program is running from inside the constraints (read: friendly, comfortable, nice) environment of decent operating system, the address space your program physically resides in, as well as any calls you make are all virtual (as they must be).
Cheers
Can we bypass paging by using some facility in the kernel space? I mean like creating a contiguous physical space (no paging, no segmentation) which is beynd the control of the kernel, but not entirely independent of the kernel.
Post 16 Nov 2014, 13:02
View user's profile Send private message Reply with quote
system error



Joined: 01 Sep 2013
Posts: 670
system error 16 Nov 2014, 13:06
JohnFound wrote:
In assembly you have several options:
2. Use your own heap manager -


That's the term i been looking for, although not descriptive enough to this thing I am planning to do.
Post 16 Nov 2014, 13:06
View user's profile Send private message Reply with quote
JohnFound



Joined: 16 Jun 2003
Posts: 3499
Location: Bulgaria
JohnFound 16 Nov 2014, 13:21
system error wrote:
JohnFound wrote:
In assembly you have several options:
2. Use your own heap manager -


That's the term i been looking for, although not descriptive enough to this thing I am planning to do.


What thing you are planning to do?

_________________
Tox ID: 48C0321ADDB2FE5F644BB5E3D58B0D58C35E5BCBC81D7CD333633FEDF1047914A534256478D9
Post 16 Nov 2014, 13:21
View user's profile Send private message Visit poster's website ICQ Number Reply with quote
system error



Joined: 01 Sep 2013
Posts: 670
system error 16 Nov 2014, 13:31
JohnFound wrote:
system error wrote:
JohnFound wrote:
In assembly you have several options:
2. Use your own heap manager -


That's the term i been looking for, although not descriptive enough to this thing I am planning to do.


What thing you are planning to do?
some silly stuff without a HDD Razz
Post 16 Nov 2014, 13:31
View user's profile Send private message Reply with quote
cod3b453



Joined: 25 Aug 2004
Posts: 618
cod3b453 16 Nov 2014, 23:38
malloc can look however you want it to and free is indeed "optional" but anything remotely dynamic will quickly have problems since realloc/free won't return unused memory for other purposes but this might not be a problem.

While the usual implementation is to ask the OS for memory and carve it up using tree/map structures like others have detailed there's no reason you can't use, say, the stack in _start or program data like your example to give a simple implementation using just an incrementing pointer without even allowing a free, along the lines of (not tested or good code):
Code:
malloc:
        push ebx

        mov eax,dword [my_malloc_ptr]
        mov ebx,dword [esp+4] ; size

        ; Is ptr+size...
        add ebx,eax
        ; ...inside the buffer?
        cmp ebx,my_malloc_end
        jbe .ok ; (Branches can be removed using cmov's)
        ; No, null pointer
        xor eax,eax

        jmp .done

    .ok:
        ; Yes, save new ptr keep pointer
        mov dword [my_malloc_ptr],ebx

    .done:

        pop ebx

free:

        ret

my_malloc_ptr dd my_malloc_mem
my_malloc_mem:
rb 16*1024*1024
my_malloc_end:
    
This is only really useful for embedded/boot loader type stuff that has a known memory footprint and just needs some C(++) cotton wool provided without an (operational) underlying OS. It does at least give you an idea of the simplest it could be.
Post 16 Nov 2014, 23:38
View user's profile Send private message Reply with quote
system error



Joined: 01 Sep 2013
Posts: 670
system error 17 Nov 2014, 03:22
cod3b453 wrote:
malloc can look however you want it to and free is indeed "optional" but anything remotely dynamic will quickly have problems since realloc/free won't return unused memory for other purposes but this might not be a problem.

While the usual implementation is to ask the OS for memory and carve it up using tree/map structures like others have detailed there's no reason you can't use, say, the stack in _start or program data like your example to give a simple implementation using just an incrementing pointer without even allowing a free, along the lines of (not tested or good code):
Code:
malloc:
        push ebx

        mov eax,dword [my_malloc_ptr]
        mov ebx,dword [esp+4] ; size

        ; Is ptr+size...
        add ebx,eax
        ; ...inside the buffer?
        cmp ebx,my_malloc_end
        jbe .ok ; (Branches can be removed using cmov's)
        ; No, null pointer
        xor eax,eax

        jmp .done

    .ok:
        ; Yes, save new ptr keep pointer
        mov dword [my_malloc_ptr],ebx

    .done:

        pop ebx

free:

        ret

my_malloc_ptr dd my_malloc_mem
my_malloc_mem:
rb 16*1024*1024
my_malloc_end:
    
This is only really useful for embedded/boot loader type stuff that has a known memory footprint and just needs some C(++) cotton wool provided without an (operational) underlying OS. It does at least give you an idea of the simplest it could be.
This is the good stuff. It answers my "how" question. I didn't know we could do that "rb 16*1024*1024" thingy. It sent me that "out of memory" exception without the arithmetic.

Thanks.
Post 17 Nov 2014, 03:22
View user's profile Send private message Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  
Goto page 1, 2  Next

< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2025, Tomasz Grysztar. Also on GitHub, YouTube.

Website powered by rwasa.