flat assembler
Message board for the users of flat assembler.

Index > Heap > Endianness?

Goto page 1, 2, 3, 4  Next
Author
Thread Post new topic Reply to topic
Artlav



Joined: 23 Dec 2004
Posts: 188
Location: Moscow, Russia
Artlav
I've recently got into an argument about endianness with a hardware engineer (who never programmed in x86 assembly beyond a university course, but did so extensively on big-endian CPUs from Motorola and his own), and it left me a little confused about a few things i took for granted.

What is the bit orientation in bytes and where/how does it matter?

His arguments were that on a little-endian system you need to manually load two bytes and swap them to extract the bits 6-10 for example, while i know by experience that it does not work that way, but takes one load, shift, and and.

After i told him so, the reply was that then the hardware has to swap the bytes upon loading an entire word into a register:
Image

When i load AX from offset 0, it comes out in such a way that after doing a shr ax, 6 i get bites 15 thru 6, not 7-0&15-14, as the image might suggest. And that was the main thing discussed.

So, how does it work?
I think the in-byte endianness is somehow separate from byte-order endianness, and he might be thinking about the first, while i was talking about the second?
Does the CPU logic swap bytes, or they come into the registers/etc as is simply the way it works anyway?
EDIT: If it doesn't, then why is it shift right that decreases the value, instead of the more consistent-to-the-byte-ordering shift left?
Post 22 Dec 2009, 18:54
View user's profile Send private message Visit poster's website Reply with quote
kohlrak



Joined: 21 Jul 2006
Posts: 1421
Location: Uncle Sam's Pad
kohlrak
When doing math within the register, you just think of AX as this:

Code:
+---------+
| AH | AL |
+---------+    


In all actuality, when taking bytes from memory, it looks like this:

Code:
+---------+
| AL | AH |
+---------+    


So if you load 0xFFEE into ax, AL holds 0xFF and AH holds 0xEE. I If you need to swap the registers, it's as easy as "bswap ax." It's a big endian to little endian conversion instruction. You could also use "xchg al, ah," but that wouldn't be as fast.

Obviously, he hasn't done any programming on x86 (don't know about other CPUs), but the idea is that aside from loading registers with a value, storing it in memory, or transmitting it, you don't have to worry about endianness. Your shl, shr, rol, and ror will *STILL* work. However, remember, that if you wanted to use a 1 byte register to store it (instead of in eax), that it'd be in AL yet. The idea isn't that the processor itself is backwords, but that the data send to and received from the processor is backwords.
Post 22 Dec 2009, 19:22
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger Reply with quote
Borsuc



Joined: 29 Dec 2005
Posts: 2466
Location: Bucharest, Romania
Borsuc
It's not backwards! It writes it starting with lowest byte, which is more logical than starting with the highest. If the memory is structured such that "increasing" is to the left, then it's not backwards -- who said memory has to go to the right?
Post 22 Dec 2009, 22:44
View user's profile Send private message Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 2915
Location: [RSP+8*5]
bitRAKE
Endianness only matters when speaking about how bytes are stored in memory. To use the term with regard to how the processor operates on data internally is incorrect - instructions work at various granularities which have nothing to do with endianness.

AL is always the least byte no matter if an instruction is operating on RAX, EAX, AX, or AL. In a rather consistent manner, the first byte of a memory location is the least byte - no matter the size of the data stored there.

No endianness is better than the other. It is just important to understand like Chirality.
Post 22 Dec 2009, 22:44
View user's profile Send private message Visit poster's website Reply with quote
kohlrak



Joined: 21 Jul 2006
Posts: 1421
Location: Uncle Sam's Pad
kohlrak
Quote:
It's not backwards! It writes it starting with lowest byte, which is more logical than starting with the highest. If the memory is structured such that "increasing" is to the left, then it's not backwards -- who said memory has to go to the right?


76543210FEDCBA98

That's the bit order. It doesn't have to, but it makes sense (to the programmer), because the bits are continuous.

Quote:
Endianness only matters when speaking about how bytes are stored in memory. To use the term with regard to how the processor operates on data internally is incorrect - instructions work at various granularities which have nothing to do with endianness.

AL is always the least byte no matter if an instruction is operating on RAX, EAX, AX, or AL. In a rather consistent manner, the first byte of a memory location is the least byte - no matter the size of the data stored there.

No endianness is better than the other. It is just important to understand like Chirality.


I don't agree entirely. To me it's not the difference between left hand or right hand, but fork or chopsticks. There are disadvantages to little endian (as already stated), but i'm sure big endian is much easier for internal register size management and probably for speed reasons.
Post 22 Dec 2009, 23:30
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17287
Location: In your JS exploiting you and your system
revolution
I think the "problem" stems from the cultural custom of writing numbers number in big-endian format. e.g. one thousand is 1000, not 0001.

If you look at code for bignum libraries, the big-endian code is overly complex, whereas the little-endian code is much clearer and easier to follow.

With little endian you can extend the size of the number by simply appending more bytes to the end and they automatically become the most significant. You can't do it that simply with big-endian.

Actually, I have yet to see where big-endian has an advantage.
Post 22 Dec 2009, 23:37
View user's profile Send private message Visit poster's website Reply with quote
ass0



Joined: 31 Dec 2008
Posts: 521
Location: ( . Y . )
ass0
So who decided to use little-endian in x86 was a jew?

_________________
Image
Nombre: Aquiles Castro.
Location2: about:robots
Post 23 Dec 2009, 00:21
View user's profile Send private message Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 2915
Location: [RSP+8*5]
bitRAKE
kohlrak wrote:
There are disadvantages to little endian (as already stated), but i'm sure big endian is much easier for internal register size management and probably for speed reasons.
Any examples of either statement? I have programmed in assember on 6502 and 680x0 as well as x86. The only disadvantage was my learning x86 afterward. As for the hardware: the bus width and caching puts both on equal footing.

_________________
¯\(°_o)/¯ unlicense.org
Post 23 Dec 2009, 00:51
View user's profile Send private message Visit poster's website Reply with quote
kohlrak



Joined: 21 Jul 2006
Posts: 1421
Location: Uncle Sam's Pad
kohlrak
Quote:
I think the "problem" stems from the cultural custom of writing numbers number in big-endian format. e.g. one thousand is 1000, not 0001.


On the countrary, within the byte, the least significant bit is still on the right, where the bytes themselves are in reverse order.

Quote:
If you look at code for bignum libraries, the big-endian code is overly complex, whereas the little-endian code is much clearer and easier to follow.


Never heard of them. Perhaps the code style has something to do with the arch (or are we talking about C here?).

Quote:
With little endian you can extend the size of the number by simply appending more bytes to the end and they automatically become the most significant. You can't do it that simply with big-endian.


Why not?

Quote:
Actually, I have yet to see where big-endian has an advantage.


As said, makes more sense to more people. Honestly, that's not a practical advantage, but an advantage nonetheless.

Quote:
Any examples of either statement? I have programmed in assember on 6502 and 680x0 as well as x86. The only disadvantage was my learning x86 afterward. As for the hardware: the bus width and caching puts both on equal footing.


The first claim i made was that it made less sense to more people, as shown by the fact this thread even exists (big endian made more sense to more people so we have CPUs with different endiens, which lead to even more confusion). The second claim is based on the idea similar to what revolution said; more bytes could be tacked on, and less hardware would be needed to work with size. You could simply shift and truncate when doing sign dependent operations like multiplication and division (in a temp register before actually storing it in the destination register).
Post 23 Dec 2009, 01:21
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17287
Location: In your JS exploiting you and your system
revolution
kohlrak wrote:
On the countrary, within the byte, the least significant bit is still on the right, where the bytes themselves are in reverse order.
But the individual bits are not in any order. It is just that we (humans) have a convention to write the MSb at the left.
kohlrak wrote:
Quote:
With little endian you can extend the size of the number by simply appending more bytes to the end and they automatically become the most significant. You can't do it that simply with big-endian.


Why not?
Because if you append bytes to the end of a BE number they become the LSBs. And now all of the other bytes have to change meaning to become a higher part of the number.

e.g.
LE number bytes: 01h 02h 03h 04h is 0x04030201
Now add some more to extend the precision.
LE number bytes: 01h 02h 03h 04h 00h 00h 00h 00h is still 0x04030201

For BE: 01h 02h 03h 04h is 0x01020304
And add some more bytes to extend the precision.
BE number bytes: 01h 02h 03h 04h 00h 00h 00h 00h is now 0x0102030400000000. A different value.

Bignum code often has a lot of special tricky code to deal with this type of problem in BE format.
Post 23 Dec 2009, 01:34
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17287
Location: In your JS exploiting you and your system
revolution
Oh yeah, I just remembered another of the horrors I encountered with BE hardware. You have to know the size of the hardware bus to access individual bytes.

In LE accessing the LSB: mov al,byte[number+0]
In BE accessing the LSB: mov al,byte[number+bus_size-1]

If you are on 64 bit hardware the bus_size is 8, 32bit is 4 etc.
Post 23 Dec 2009, 01:38
View user's profile Send private message Visit poster's website Reply with quote
kohlrak



Joined: 21 Jul 2006
Posts: 1421
Location: Uncle Sam's Pad
kohlrak
Quote:
But the individual bits are not in any order. It is just that we (humans) have a convention to write the MSb at the left.


in which case, using anything to bswap other than the bswap instruction would fail, no? If it was really like that, we could easily make the little endian concept entirely invisible to the programmer.

Quote:
e.g.
LE number bytes: 01h 02h 03h 04h is 0x04030201
Now add some more to extend the precision.
LE number bytes: 01h 02h 03h 04h 00h 00h 00h 00h is still 0x04030201

For BE: 01h 02h 03h 04h is 0x01020304
And add some more bytes to extend the precision.
BE number bytes: 01h 02h 03h 04h 00h 00h 00h 00h is now 0x0102030400000000. A different value.

Bignum code often has a lot of special tricky code to deal with this type of problem in BE format.


In such a situation, don't you typically either keep the same width or use a movsx-like or movzx-like instruction anyway?
Post 23 Dec 2009, 01:40
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17287
Location: In your JS exploiting you and your system
revolution
kohlrak wrote:
In such a situation, don't you typically either keep the same width or use a movsx-like or movzx-like instruction anyway?
Not usually. It is simpler to make code that just appends. You allocate some more memory, add zeros, and compute with a new precision. In BE format you need extra steps to move the existing bytes into the proper positions.

I have seen some code that uses LE format for the dwords to try to solve this problem. But the whole thing becomes very confusing with mixed formats (BE in each dword and LE in the whole number). But it doesn't scale when you translate the code to a 64bit system. You have to recode all the offsets for qwords. Lots of opportunity for errors.
Post 23 Dec 2009, 01:50
View user's profile Send private message Visit poster's website Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 2915
Location: [RSP+8*5]
bitRAKE
kohlrak wrote:
In such a situation, don't you typically either keep the same width or use a movsx-like or movzx-like instruction anyway?
No, because that would just push the problem to a higher grainularity and waste memory. Instead the largest machine size is used in a LE fashion to make appending possible without positional adjustments.

For example, (hex bytes, ascending memory addresses)

LE: 00 01 02 03 04 05 06 07
BE: 07 06 05 04 03 02 01 00

...and if the largest machine type was 32-bit, storage would be:

BE: 03 02 01 00 07 06 05 04

...and if the BIGINT grew past 64-bit...

BE: 03 02 01 00 07 06 05 04 0B 0A 09 08

...see the complexity for BE?
(reduces to LE equivalent)

_________________
¯\(°_o)/¯ unlicense.org
Post 23 Dec 2009, 02:04
View user's profile Send private message Visit poster's website Reply with quote
kohlrak



Joined: 21 Jul 2006
Posts: 1421
Location: Uncle Sam's Pad
kohlrak
bitRAKE wrote:
kohlrak wrote:
In such a situation, don't you typically either keep the same width or use a movsx-like or movzx-like instruction anyway?
No, because that would just push the problem to a higher grainularity and waste memory. Instead the largest machine size is used in a LE fashion to make appending possible without positional adjustments.

For example, (hex bytes, ascending memory addresses)

LE: 00 01 02 03 04 05 06 07
BE: 07 06 05 04 03 02 01 00

...and if the largest machine type was 32-bit, storage would be:

BE: 03 02 01 00 07 06 05 04

...and if the BIGINT grew past 64-bit...

BE: 03 02 01 00 07 06 05 04 0B 0A 09 08

...see the complexity for BE?
(reduces to LE equivalent)


Great for a hypothetical machine where you have no memory management. In any real situation, you can't just extend any old variable (without possibly writing over another) and you have to plan ahead, regardless of endianness (which is why we came up with PMODE afaik).

EDIT: Sorry, didn't see this at first...

Quote:
Not usually. It is simpler to make code that just appends. You allocate some more memory, add zeros, and compute with a new precision. In BE format you need extra steps to move the existing bytes into the proper positions.


So you're telling me that these machines don't have size conversion instructions?

Quote:
I have seen some code that uses LE format for the dwords to try to solve this problem. But the whole thing becomes very confusing with mixed formats (BE in each dword and LE in the whole number). But it doesn't scale when you translate the code to a 64bit system. You have to recode all the offsets for qwords. Lots of opportunity for errors.


They should've done it properly to begin with. Honestly, you should know what your supported width for the program is before you code it. Bad design, IMO.


Last edited by kohlrak on 23 Dec 2009, 04:05; edited 1 time in total
Post 23 Dec 2009, 03:50
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17287
Location: In your JS exploiting you and your system
revolution
kohlrak: For bignum code this is a real problem. For byte/word partial access it is a real problem. There ain't nothin' hypothetical about those two situations. Wink
Post 23 Dec 2009, 03:56
View user's profile Send private message Visit poster's website Reply with quote
kohlrak



Joined: 21 Jul 2006
Posts: 1421
Location: Uncle Sam's Pad
kohlrak
revolution wrote:
kohlrak: For bignum code this is a real problem. For byte/word partial access it is a real problem. There ain't nothin' hypothetical about those two situations. Wink


The problem is, you should know what you're going to support before you support it. You're going to have to set a limit on what your maximum number size is anyway to prevent other data from getting in the way. So I still don't see how bignum is a problem. For partial access, I find it more difficult to use little endian, because when working with partial bytes, i expect them to be in a format i'm used to (the way it's represented when i type in hex numbers). Having to work with numbers going one way but then bytes going the opposite way is really confusing, at least to me. I don't know about you, but i find it more error prone when you're doing something confusing than when it actually makes sense.
Post 23 Dec 2009, 04:11
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17287
Location: In your JS exploiting you and your system
revolution
For bignums, if you set a hard limit then the programs that use them would be almost useless.

As for partial bytes, if you think of the ordering scheme as numbered from the left towards the right then of course you will think LE is wrong. But if you put zero at the right and number successively towards the left then the numbering is nice. Like I mentioned above, the cultural custom of writing numbers in decimal as BE is creating the confusion. You might tend to think of the first digit at the MSD but you are also numbering that the zeroth digit. That is kind of like alice-in-wonderland style Razz. But, if we have the LSD numbered as zero and higher significant digits at higher addresses it seems very logical to me.
Post 23 Dec 2009, 04:24
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17287
Location: In your JS exploiting you and your system
revolution
For LE:
mov al,byte[number+0] is the LSB
mov ax,word[number+0] is the LSW
mov eax,dword[number+0] is the LSD
mov rax,qword[number+0] is the LSQ

For BE:
mov al,byte[number+0] is ?. It depends upon the bus width
mov ax,word[number+0] is ?. It depends upon the bus width
mov eax,dword[number+0] is ?. It depends upon the bus width
mov rax,qword[number+0] is ?. It depends upon the bus width
Post 23 Dec 2009, 04:27
View user's profile Send private message Visit poster's website Reply with quote
sinsi



Joined: 10 Aug 2007
Posts: 693
Location: Adelaide
sinsi
LE makes sense when you consider
byte[0] is 256^0
byte[1] is 256^1
etc.
Post 23 Dec 2009, 04:50
View user's profile Send private message Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  
Goto page 1, 2, 3, 4  Next

< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum


Copyright © 1999-2020, Tomasz Grysztar. Also on YouTube, Twitter.

Website powered by rwasa.