[bug] Wrong segment selected for [ebp*2] (and friends).

Index > Compiler Internals > [bug] Wrong segment selected for [ebp*2] (and friends).

Author

Thread

SeproMan

Joined: 11 Oct 2009
Posts: 70
Location: Belgium

SeproMan 15 Apr 2018, 13:56

FASM assembles these with no USE setting, so implied USE16:

Code:

mov ax, [ebp*1]   67 8B 45 00
mov ax, [ebp*2]   67 8B 44 2D 00
mov ax, [ebp*3]   67 8B 44 6D 00
mov ax, [ebp*4]   67 8B 04 AD 00 00 00 00
mov ax, [ebp*5]   67 8B 44 AD 00
mov ax, [ebp*8]   67 8B 04 ED 00 00 00 00
mov ax, [ebp*9]   67 8B 44 ED 00

The codes for [ebp*4] and [ebp*8] are correct.
The other encodings are wrong because by splitting the scaled-index-only address form into a base plus scaled-index address form (in order to avoid the 32-bit displacement component), EBP is now used as a base and so architecturally the default segment becomes SS.
When EBP is merely an index the default segment is DS.

The assembler can not make the assumption that SS=DS.

Personally I would prefer that if the programmer writes a scaled address form, it remains just that. Always correct an no trickery involved.
If on the other hand obtaining the shortest code has priority, then FASM should add a DS: override prefix.

Code:

mov ax, [ebp*1]   3E 67 8B 45 00
mov ax, [ebp*2]   3E 67 8B 44 2D 00
mov ax, [ebp*3]   3E 67 8B 44 6D 00
mov ax, [ebp*4]   67 8B 04 AD 00 00 00 00
mov ax, [ebp*5]   3E 67 8B 44 AD 00
mov ax, [ebp*8]   67 8B 04 ED 00 00 00 00
mov ax, [ebp*9]   3E 67 8B 44 ED 00

Of course when these scaled-index-only address forms are use with a LEA instruction, none of the above matters.

To see how NASM suffers the same problem click here.

_________________
Real Address Mode.

15 Apr 2018, 13:56

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8351
Location: Kraków, Poland

Tomasz Grysztar 15 Apr 2018, 15:42

SeproMan wrote:

The assembler can not make the assumption that SS=DS.

fasm makes this assumption by default, it is part of its "flatness" attribute. The standard mode of fasm's assembly is aimed at a fully flat model where DS=ES=SS, and if this is not the case then the instruction needs to explicitly state what segment should be accessed, fasm is going to optimize away the unneeded prefixes:

Code:

mov ax, [ds:ebp*1]  ; 3E 67 8B 45 00
mov ax, [ds:ebp*2]  ; 3E 67 8B 44 2D 00
mov ax, [ds:ebp*3]  ; 3E 67 8B 44 6D 00
mov ax, [ds:ebp*4]  ; 67 8B 04 AD 00 00 00 00
mov ax, [ds:ebp*5]  ; 3E 67 8B 44 AD 00
mov ax, [ds:ebp*8]  ; 67 8B 04 ED 00 00 00 00
mov ax, [ds:ebp*9]  ; 3E 67 8B 44 ED 00

mov ax, [ss:ebp*1]  ; 67 8B 45 00
mov ax, [ss:ebp*2]  ; 67 8B 44 2D 00
mov ax, [ss:ebp*3]  ; 67 8B 44 6D 00
mov ax, [ss:ebp*4]  ; 36 67 8B 04 AD 00 00 00 00
mov ax, [ss:ebp*5]  ; 67 8B 44 AD 00
mov ax, [ss:ebp*8]  ; 36 67 8B 04 ED 00 00 00 00
mov ax, [ss:ebp*9]  ; 67 8B 44 ED 00

mov ax, [ebp*1]     ; 67 8B 45 00
mov ax, [ebp*2]     ; 67 8B 44 2D 00
mov ax, [ebp*3]     ; 67 8B 44 6D 00
mov ax, [ebp*4]     ; 67 8B 04 AD 00 00 00 00
mov ax, [ebp*5]     ; 67 8B 44 AD 00
mov ax, [ebp*8]     ; 67 8B 04 ED 00 00 00 00
mov ax, [ebp*9]     ; 67 8B 44 ED 00

Note how you tell the assembler what the instruction is supposed to do and it then generates as short an opcode as possible that does exactly that. Unless you explicitly specify what segment the instruction has to operate on, fasm assumes the fully flat model and generates shortest possible opcodes under that assumption.

This is a general and fundamental paradigm of assembly languages as understood in fasm's design, that assembly language provides abstraction over the machine language such that the commands of assembly define exactly what the generated instruction is supposed to do and the choice of appropriate machine code is up to the assembler. This abstraction is especially apparent in case of x86 architecture, where even many basic instructions can be encoded in many ways (this also allows to the "fingerprinting" of assembler/compiler used to generate code).

On top of that, fasm's method is such that it seeks to generate smallest possible encoding for the requested instructions, as demonstrated by the choice of multi-pass assembly variant that was one of the fundamental blocks of fasm's design. This is the main reason why the above approach to segmented addressing was chosen (on this board you can find some older discussions of the problem from many years ago).

15 Apr 2018, 15:42

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8351
Location: Kraków, Poland

Tomasz Grysztar 15 Apr 2018, 16:18

Nevertheless, the above is not the full truth about the current state of fasm. A strict application of the principle mentioned above would lead to accepting things like LEA to MOV optimization, TEST optimization or changing the order of registers in expression to optimize [EBP+ESI] in flat mode into [ESI+EBP]. I personally have nothing against such optimization paths, but I recognize the reasoning of people that see this as a step too far, when the instruction they see in disassembly is too much removed from what they have written in the source (see the discussion in the thread about TEST optimization I linked to). Therefore fasm in its final iteration is more of a compromise and refrains from applying any more controversial optimizations. I really should make a more optimizing version of x86 macros for fasmg, though - there adding alternative variants of instruction encoders costs next to nothing.

15 Apr 2018, 16:18

SeproMan

Joined: 11 Oct 2009
Posts: 70
Location: Belgium

SeproMan 15 Apr 2018, 18:54

Tomasz Grysztar wrote:

Quote:

fasm makes this assumption by default, it is part of its "flatness" attribute. The standard mode of fasm's assembly is aimed at a fully flat model where DS=ES=SS

Would you believe I was not aware of the fact that this is what the "flat" in "flatassembler" stands for?
Under this assumption there's obviously no problem with how the code is currently generated.

Thanks for providing those links to previous discussions about the matter. I found it re-assuring to see that others already commented on this.
I did use the search engine before posting but couldn't find anything useful! Who would have known that "registers algebra" was the correct phrase to look for?

Thanks again for taking the time to answer (once more).

15 Apr 2018, 18:54

DimonSoft

Joined: 03 Mar 2010
Posts: 1228
Location: Belarus

DimonSoft 15 Apr 2018, 19:34

I’m surprised. Well, I never thought about the case since it not common to add something to EBP and vice versa, but I’m surprised FASM does it so (and NASM as well?).

Tomasz Grysztar wrote:

the commands of assembly define exactly what the generated instruction is supposed to do and the choice of appropriate machine code is up to the assembler

But this is definitely not the case. The commands of assembly do not state explicitly that DS = SS, thus I’d expect such code to work the way it is defined in SDM. And such behavior definitely violates the least astonishment principle: one generally expects Intel SDM to be the source of information by default and any behavior not explicitly stated in FASM help should conform to Intel SDM.

Of course, it is possible to define some rules that differ from the official documentation, for the purpose of opimizations. But I’m absolutely sure this should be stated explicitly in FASM help. Although a better solution would be to disable such behaviour and let the optimization find its way to assembly projects the same way other clever tricks do: “If you want to save a few bytes, you can do this and that but it comes with these consequences”.

This particular “optimization” (which it really is not since the reaulting behaviour differs from the original one) seems to be more like C/C++ undefined behavior stuff: it started as a set of opportunities for optimization and ended up as an ideology of the languages and a gun that lets you shoot your foot unexpectedly.

15 Apr 2018, 19:34

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8351
Location: Kraków, Poland

Tomasz Grysztar 15 Apr 2018, 20:08

DimonSoft wrote:

But this is definitely not the case. The commands of assembly do not state explicitly that DS = SS, thus I’d expect such code to work the way it is defined in SDM. And such behavior definitely violates the least astonishment principle: one generally expects Intel SDM to be the source of information by default and any behavior not explicitly stated in FASM help should conform to Intel SDM.

This may be the rightful critique of fasm's documentation, as there are still many details that I only "documented" on the forums and not in the official manual. But there is really no other way in fasm's syntax to "state explicitly that DS = SS" (while you can easily state that you need to use specific segment with a regular syntax). Making it default turned out very handy, since most of the operating systems moved to the flat model long time ago. Back in 1999 when I was writing fasm 1 it was already apparent and the time has only proven that it was the right choice. Turns out that the assembler originally created for niche purposes worked well for wide range of applications while retaining the flatness-related assumptions.

DimonSoft wrote:

Of course, it is possible to define some rules that differ from the official documentation, for the purpose of opimizations. But I’m absolutely sure this should be stated explicitly in FASM help. Although a better solution would be to disable such behaviour and let the optimization find its way to assembly projects the same way other clever tricks do: “If you want to save a few bytes, you can do this and that but it comes with these consequences”.

The way fasm handles register algebra is really outside of the scope of what Intel defines. You can compute addresses like this with fasm:

Code:

mov ax,[(ebp+ebx)*3-(ebx+100h)*2+ebp]

label a at ebp+ebx
label b at ebx+100h

mov ax,[ebp+a*3-b*2]  ; the same address, but its register components hidden

and Intel manuals cannot be of much assistance in how to interpret this with relation to segment registers - the computation of linear polynomials with register variables is a layer of abstraction added by fasm, thus the rules for it can only be defined in context of fasm and do not flow from Intel SDM.

15 Apr 2018, 20:08

DimonSoft

Joined: 03 Mar 2010
Posts: 1228
Location: Belarus

DimonSoft 15 Apr 2018, 20:39

Tomasz Grysztar wrote:

there is really no other way in fasm's syntax to "state explicitly that DS = SS" (while you can easily state that you need to use specific segment with a regular syntax)

Well, I can state explicitly that I want to use DS even though SS should be used by default. I mean, the optimization technique could be moved from the assembler to the programmer.

Tomasz Grysztar wrote:

The way fasm handles register algebra is really outside of the scope of what Intel defines. You can compute addresses like this with fasm: <…>

You’re right, I missed this part. And it makes things more complicated. Still, if the rules of the register algebra are consistent for different expressions, it is the part of FASM that is worth being documented and then choosing the base and index registers can also be defined by a few simple rules without breaking compatibility with Intel SDM.

What I’m trying to say is that it can be done this way or that but I vote for the solution to follow the least astonishment principle whenever possible, i.e. it is either well-documented behavior or works as expected without the need to know FASM is flat and optimizing assembler.

IMNSHO.

15 Apr 2018, 20:39

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8351
Location: Kraków, Poland

Tomasz Grysztar 15 Apr 2018, 21:03

DimonSoft wrote:

Well, I can state explicitly that I want to use DS even though SS should be used by default.

With fasm this should only be slightly rephrased: I can state explicitly that I want to use DS even though SS might be used by default.

DimonSoft wrote:

I mean, the optimization technique could be moved from the assembler to the programmer.

This would then make a different kind of assembler.

An assembler where it is the programmer who is choosing and optimizing the opcodes does not need multiple passes or register algebra. And is generally much easier to create.

Though there is also an option of making it as another alternative package of x86 macros for fasmg, relieved from handling these complexities they would probably become much simpler and faster compared to the fasm-compatible ones.

15 Apr 2018, 21:03

DimonSoft

Joined: 03 Mar 2010
Posts: 1228
Location: Belarus

DimonSoft 16 Apr 2018, 09:10

Tomasz Grysztar wrote:

DimonSoft wrote:
Well, I can state explicitly that I want to use DS even though SS should be used by default.
With fasm this should only be slightly rephrased: I can state explicitly that I want to use DS even though SS might be used by default.

Which is not obvious until FASM defaults are explicitly stated in the docs.

Tomasz Grysztar wrote:

DimonSoft wrote:
I mean, the optimization technique could be moved from the assembler to the programmer.
This would then make a different kind of assembler.

An assembler where it is the programmer who is choosing and optimizing the opcodes does not need multiple passes or register algebra. And is generally much easier to create.

Optimization already implies “no changes in code behavior”. I don’t see how being careful applying optimizations enforces the removal of optimizations other than this one. And we already have to know about, say, Allison’s algorithm and possibly prefer it over straightforward implementation. And to choose between movzx/movsx combinations of xor, cdq and mov. Avoiding optimizations that are not valid in certain conditions seem to be the right thing to do in general.

But I understand this would make some cool stuff in FASM useless, so I’ll say this stuff should just be expressed in the docs explicitly.

In fact, I don’t really care about this particular use case for my own projects and the assumption really seems perfectly valid for most cases except, maybe, some (?) OSDev projects. But since I’m teaching students and was the one who made FASM the assembler of choice in our university department, I’m surprised in a bad way: it was supposed to be used as the best tool that does everything the right way™ and now it turns out there might be lots of hidden surprises like this one.

16 Apr 2018, 09:10

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8351
Location: Kraków, Poland

Tomasz Grysztar 16 Apr 2018, 10:45

DimonSoft wrote:

Tomasz Grysztar wrote:

DimonSoft wrote:
Well, I can state explicitly that I want to use DS even though SS should be used by default.
With fasm this should only be slightly rephrased: I can state explicitly that I want to use DS even though SS might be used by default.
Which is not obvious until FASM defaults are explicitly stated in the docs.

You state it as if some other approach was unambiguously corrects based on the Intel documentation, but I do not see it that way. What Intel defines is the application of segments depending on the base register in the addressing fields, but when we move from the machine language to the assembly and the address is seen as an algebraic expression (unless we decide to define the address encoding fields explicitly like GAS does) it is no longer clearly defined what constitutes a base register except in the simple cases. If I write [ebx+esi+ebx], is then EBX a base register because it occurred first? One could try to define various arbitrary rules, but the point is that neither is clearly implied by the Intel specification which has more to do with actual machine encodings than with an assembly language.

When one writes something like [ebp*X], it is already not apparent whether this is going to be addressed through SS or not. It isn't obvious whether EBP is a base or index register then - when an address is simply an algebraic expression we have no universally obvious way of telling "there is no base register here". One could invent some rules like make [ebp] mean base register and [0+ebp] mean index register addressing, or even distinguish [ebp] from [ebp*1], but such rules would not be something that obviously follows from how the Intel instructions are defined, they would be specific quirks of a given assembly language. Again, because Intel manuals define things as related to machine encoding and assembly languages abstract many of these things away.

Whichever variant would get chosen, it would be surprising from some point of view. If one expects the address expression to actually reflect encoding structure like GAS does, then the fact that is is treated algebraically is going to be surprising. On the other hand having "A+0" mean something different than "0+A" or "A*1" would be just as surprising for anyone used to the rules for expressions in the most of existing languages. I would argue that the latter surprise would be much stronger, and the experience I had with fasm development seems to demonstrate it.

Therefore I always considered expressions like [ebp*2] ambiguous with respect to segment addressing, something that requires clarification, by writing either [ds:ebp*2] or [ss:ebp*2] if the distinction is really important in context (and leaving the bare address when the ambiguity does not matter, like in the flat mode). It is in some way similar to adding round brackets around sub-expression "just in case the operator precedence is different that I remember". It not only makes sure that things mean what they should, but also creates a better code, easier to correctly interpret by someone else reading it.

DimonSoft wrote:

Optimization already implies “no changes in code behavior”.

Yes, and this is one of the basic assumptions in my approach to assembly languages. Since the beginning of the development of fasm I focused on having a syntax that clearly defines what the instruction is supposed to do. Note that the old Intel manuals, like the ones for 80386, were not defining syntax for many specific instruction encoding variants that were important in places like OS development and with assemblers of the time it was sometimes very hard to enforce encoding of a specific variant needed, programmers often ended up writing machine codes by hand with DB. So with fasm I had chosen the approach of defining clearly through syntax what is the intended operation of the instruction and then letting the assembler find the correct encoding.

This is not inconsistent with the approach to the addresses we just discussed. For me it was obvious that since "ebp+ebp" and "ebp*2", and "ebp*3-ebp" are all algebraically equivalent, there is no clear definition of base register for them and therefore there is really no well-defined standard behavior that the assembler would have to preserve. Adding the segment prefix clearly states the instruction purpose, but the bare variant is up to the assembler to define.

Now, that fasm's manual failed to provide a detailed discussion of this is a different story. It may turn out that long after the fasm's development is finished I am still going to keep working on all these never-really-finished manuals.

16 Apr 2018, 10:45

DimonSoft

Joined: 03 Mar 2010
Posts: 1228
Location: Belarus

DimonSoft 16 Apr 2018, 11:03

Tomasz Grysztar wrote:

Therefore I always considered expressions like [ebp*2] ambiguous with respect to segment addressing, something that requires clarification, by writing either [ds:ebp*2] or [ss:ebp*2] if the distinction is really important in context (and leaving the bare address when the ambiguity does not matter, like in the flat mode). It is in some way similar to adding round brackets around sub-expression "just in case the operator precedence is different that I remember". It not only makes sure that things mean what they should, but also creates a better code, easier to correctly interpret by someone else reading it.

Well, having experience in real-mode programming I thought the fact of EBP occuring in such expression already means we use SS. Never really paid attention to the quirk in Intel’s addressing. So, still SS is less surprising but, like I said, I got your idea. And it makes sense.

Tomasz Grysztar wrote:

Since the beginning of the development of fasm I focused on having a syntax that clearly defines what the instruction is supposed to do.

Which is the best feature I’ve been looking for for ages until I’ve found FASM.

Tomasz Grysztar wrote:

Now, that fasm's manual failed to provide a detailed discussion of this is a different story. It may turn out that long after the fasm's development is finished I am still going to keep working on all these never-really-finished manuals.

I’d say a separate section related to FASM optimizations might be a good idea. It is the feature that influences the assembler output and is generally the feature to be proud of.

16 Apr 2018, 11:03

< Last Thread | Next Thread >

Forum Rules:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum