Index
> Linux > ELF toolbox |
| Author |
|
|
sylware 08 Jul 2023, 14:57
I started to code an ELF interpreter.
It is far from finished and I'll have to code (more likely, copy code from other open source software) a libc to go with it. The idea is binary compat with the glibc to run the steam client and some major games. ELF is so much disgusting I started to write an alternative format: https://www.rocketgit.com/user/sylware/nyanlinux/source/tree/branch/master/blob/files/EDLF64.draft Implementation will require "re-working" many critical system components. I'll still work on the ELF interpreter/libc, but I switch priority with sueing my state administration about breaking noscript/basic (x)html interop (and I can become homeless anytime, I don't cry, this is the price to go against Big Tech). Yeah, my lawyer gave birth, she has been on "newborn leave" for a while, and it should be over... I think. https://www.rocketgit.com/user/sylware/nyanlinux/source/tree/branch/master/blob/files/nyanelf-20230707.tar.bz2 Oh, the licence is ofc Affero GPLv3 with a lesser exception for the lib parts and Affero GPLv3 for any application (I'll add a LICENSE file once I start to have something working). Last edited by sylware on 09 Jul 2023, 22:35; edited 2 times in total |
|||
08 Jul 2023, 14:57 |
|
|
sylware 08 Jul 2023, 17:19
@furs, a EDLF64 file is free to organize the way it wants those 64bits.
For "modules", which requires some executable symbols, a module init function would be provided the executable resolve function, or directly an array of the virtual addresses of those symbols. Something I am thinking about is to mandate the ability to unmap the executable itself. EDIT: I did update the draft for a clean munmap-ing of the executable itself (using an entry in the aux vector to provide the base address to munmap the executable). EDIT:I though also about the array of offsets from the start of the file. But I felt like letting the file code decides about that and I didn't like the idea of a sparse array instead of a sparse ID space (upon symbol deprecation/removal, think about a crypto TLS lib). |
|||
|
08 Jul 2023, 17:19 |
|
|
revolution 10 Jul 2023, 05:16
Furs wrote: ... (i.e. when importing a symbol, only the symbol name is needed and not what lib it's from, and then whichever library exports it first will have it resolved, which is insane). This allows, rightly or wrongly, the preloading of a library (LD_PRELOAD) to override functions in other libraries that the app is expecting to link with. But all of this is only true if the app uses a "standard" runtime linker like /lib/ld-linux.so.2 or similar. You can choose to ignore /lib/ld-linux.so.2 and use your own runtime linker to import whatever you want in any way that pleases you. |
|||
|
10 Jul 2023, 05:16 |
|
|
sylware 10 Jul 2023, 13:09
revolution wrote: /lib/ld-linux.so.2 and use your own runtime linker to import whatever you want in any way that pleases you. But you have to deal with ELF toxicity... erk... It is really, REALLY, hard to stay motivated to write that ELF interpreter (and actually the libc which basically has to come with it). |
|||
|
10 Jul 2023, 13:09 |
|
|
Furs 10 Jul 2023, 13:18
revolution wrote: But all of this is only true if the app uses a "standard" runtime linker like /lib/ld-linux.so.2 or similar. You can choose to ignore /lib/ld-linux.so.2 and use your own runtime linker to import whatever you want in any way that pleases you. Such ELF files literally do not provide information where the symbol it's from. All they do is provide: the libraries to load, and the symbol names to resolve. In PE (Windows DLL) for example, it provides the library to load, and then for this specific library, the symbols it wants. So it has no conflicts. I'm talking about how they store it in the actual format. |
|||
|
10 Jul 2023, 13:18 |
|
|
revolution 10 Jul 2023, 14:32
Then the "real" solution is to eschew dynamic linking completely.
Static link everything.  |
|||
|
10 Jul 2023, 14:32 |
|
|
sylware 10 Jul 2023, 17:17
revolution wrote: Then the "real" solution is to eschew dynamic linking completely. Oh... believe me, I would, but games do expect a set of binary interfaces with the system... and overkillingly require dynamic linking instead of dynamic loading. Another case: when you distribute open source binaries which are supposed to work on different hardware variants: you cannot have all the drivers compiled in an omega huge binary... unless we, as the humanity, manage to have one real standard per hardware type, namely we would need only 1 driver (for instance USB webcam). The mitigation would be excrutiatingly simple dynamic loading with hardcore stable in time as simple as possible interfaces, and avoid dynamic linking as a whole (you know... ELF...) |
|||
|
10 Jul 2023, 17:17 |
|
|
revolution 10 Jul 2023, 17:34
I'm not sure that ELF is the right thing to blame here.
It would be trivial to require that all imported functions must include the library name as a prefix. Similar to how I wrote above "Library1.initialise" instead of just "initialise". Naturally this will never happen because ... reasons. And this is the right place that deserves the blame, inertia, not the underlying format. Last edited by revolution on 10 Jul 2023, 20:11; edited 1 time in total |
|||
|
10 Jul 2023, 17:34 |
|
|
sylware 10 Jul 2023, 19:50
That reason alone, indeed, the format hardly can be blamed, but believe me, in the case of ELF, you will have many other reasons to honestly hate it. Not to mention GNU added kludge on top of it. I don't know what was the intent of those guys, but you know: mistakes are not malicious.
That's why I wrote the EDLF64 draft, while writing ELF handling code, it is so simple, and really does more than enough. But I need to acheive someting actually working *before*, coze I perfectly know that I am probably missing many things. |
|||
|
10 Jul 2023, 19:50 |
|
|
sylware 17 Jul 2023, 14:44
@Furs,
In the case of very stable ABIs, you are right, I am wrong: the ABI should be just offsets into the EDLF64 file. Because, to have a re-entrant resolution function, we require from the hardware architecture to let us just do that (a spinlock) WITHOUT any thread library loaded. But: For instance for really active ABIs like vulkan3d, this array of offsets would be quite sparse and of significant size. And on the long run, with holes which will be required to stay a veeeery long time. This is the same issue with some crypto libs with the life cycle of the various algorithms. The "free" array slots cannot be reused in order to let the user code know there is nothing here actually (or rely on a stable private mechanism to know that prior accessing those slots). I need to think about this reentrancy/threading-lib chicken and egg issue. Should I mandate the resolution function to be reentrant in some hardware architecture way, that independent of any threading libs? |
|||
|
17 Jul 2023, 14:44 |
|
|
Furs 18 Jul 2023, 13:19
sylware wrote: @Furs, That's exactly the nightmare you should aim to get rid of. Otherwise what's the point of making another format? ELF already suffers from it. Don't make the same mistake. If an ABI is so unstable by design and you should have "fallbacks" for it, then it should export a single API that allows you to query capabilities, and then another API that allows you to execute a capability (generic API). For Crypto APIs, for example, a generic API to encrypt/decrypt/whatever with some structure or parameters telling you which algorithm and its tuned parameters, and so on. Force people to not be lazy and they will. |
|||
|
18 Jul 2023, 13:19 |
|
|
sylware 18 Jul 2023, 21:22
I ate the bullet: I fixed the circular dependency with a high level threading dynamic library using a "hardware architecture dependent thread lock" (aka an atomic compare and exchange or kernel syscall).
I added a "EAGAIN" return code. I now need to let the modifications sink in for a little while. I have a good feeling at it being "finished"... for a first implementation and see what's missing. Furs wrote:
The symbol will dynamically fail to resolve and caller code will have a chance to deal with it. I removed "static loading" of dynamic libraries, only dynamic loading with dynamic symbol resolution is possible. furs wrote:
This is basically what vulkan3d does at several levels. |
|||
|
18 Jul 2023, 21:22 |
|
|
sylware 23 Jul 2023, 13:06
Allright, I have been letting it sink in, slept a few nights on that, and it came back to me: dead locks which can happen while initing a "service" and its dependencies. EDLF64 is not supposed to deal with that much of that.
For instance, you can have 2 differents implementations of the same service S, A and B, which can have really different initialization requirements. Of course, the client machine code of S must not change if it is A or B. Ok, I am stating the obvious. I am thinking of a "DPS", "Dynamic Process Service" somewhat building on top of EDLF64. A "DPS" runtime instance would have to track when the service is up and running, and try to deal with the initialization issues: circular dependencies/dead locks in a less worse way. Usually it means "standard" init/fini functions. Back to the drawing board. |
|||
|
23 Jul 2023, 13:06 |
|
|
sylware 24 Jul 2023, 11:46
Allright. An advice: don't try to think with a migraine, do something else.
First, a "standard" init called by EDLF64 runtime is really a bad idea (look at ELF). Components will have to define their own "specific" init functions for the EDLF64 plaform (or they can do lazy initialization, which seems to become the norm). Then in the initialization dependency calling graph, there is no magic around circular dependencies. The only mitigation is for those "init" functions to detect re-entrancy then to abort and to whine very loudly. I need to augment the specs with some loud warnings since I provide some "recommended" init/fini function prototypes. Now, I think we're good... for a first implementation and see what was missed. |
|||
|
24 Jul 2023, 11:46 |
|
< Last Thread | Next Thread > |
Forum Rules:
|