flat assembler
Message board for the users of flat assembler.

Index > Main > movdqu illegal operand

Author
Thread Post new topic Reply to topic
Andy



Joined: 17 Oct 2011
Posts: 55
Andy 01 May 2023, 23:36
I tried this basic code but when I compile I got an error that say: illegal operand.

Code:
format PE
entry start

section '.text' code readable executable
  start:
        movdqu xmm1, XMMWORD PTR [key]

section '.data' data readable writeable
        key db 0x2b, 0x7e, 0x15, 0x16, 0x28, 0xae, 0xd2, 0xa6, 0xab, 0xf7, 0x15, 0x88, 0x09, 0xcf, 0x4f, 0x3c    


I am quite noob but from my understanding movdqu moves a double quadword to xmm1 from another xmm register or 128-bit memory location like in my case. What I am missing and why I get this error?
Post 01 May 2023, 23:36
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 20414
Location: In your JS exploiting you and your system
revolution 01 May 2023, 23:45
The data size for 128-bit is dqword.
Code:
movdqu xmm0, dqword[$]    
Post 01 May 2023, 23:45
View user's profile Send private message Visit poster's website Reply with quote
Andy



Joined: 17 Oct 2011
Posts: 55
Andy 02 May 2023, 00:23
I was induced in error by the example code I follow. Actually I use a piece a code from Intel Advanced Encryption Standard (AES) New Instructions Set White Paper.

Code:
; Cipher key is stored in “Key”. For example,
; Key 0x0f0e0d0c0b0a09080706050403020100
; The key scheduled to be stored in the array Key_Schedule.
movdqu xmm1, XMMWORD PTR Key
movdqu XMMWORD PTR Key_Schedule, xmm1
mov rcx, OFFSET Key_Schedule+16
aeskeygenassist xmm2, xmm1, 0x1
call key_expansion_128
aeskeygenassist xmm2, xmm1, 0x2
call key_expansion_128
aeskeygenassist xmm2, xmm1, 0x4
call key_expansion_128
aeskeygenassist xmm2, xmm1, 0x8
call key_expansion_128
aeskeygenassist xmm2, xmm1, 0x10
call key_expansion_128
aeskeygenassist xmm2, xmm1, 0x20
call key_expansion_128
aeskeygenassist xmm2, xmm1, 0x40
call key_expansion_128
aeskeygenassist xmm2, xmm1, 0x80
call key_expansion_128
aeskeygenassist xmm2, xmm1, 0x1b
call key_expansion_128
aeskeygenassist xmm2, xmm1, 0x36
call key_expansion_128
jmp END;

key_expansion_128:
        pshufd xmm2, xmm2, 0xff
        vpslldq xmm3, xmm1, 0x4
        pxor xmm1, xmm3
        vpslldq xmm3, xmm1, 0x4
        pxor xmm1, xmm3
        vpslldq xmm3, xmm1, 0x4
        pxor xmm1, xmm3
        pxor xmm1, xmm2
        movdqu XMMWORD PTR [rcx], xmm1
        add rcx, 0x10
        ret
END:    


It works if I replace XMMWORD with dqword but I got other errors for lines mov rcx, OFFSET Key_Schedule+16 and add rcx, 0x10. I tried even more basic instructions with this register, like xor rcx, rcx and fails also with the same error "Illegal instruction".

This line is also strange in this example since rcx is 64 bits in size and xmm1 is 128 bits so I suppose this line will also throw an error eventually.
Code:
movdqu XMMWORD PTR [rcx], xmm1    


BTW what flavor of asm is used in these intel white papers?
Post 02 May 2023, 00:23
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 20414
Location: In your JS exploiting you and your system
revolution 02 May 2023, 00:27
Offset is not part of fasm. just delete it.

To access 64-bit instructions you need to put it into a 64-bit mode. Either use64 or with format ...
Code:
format pe64 ... ; 64-bit code    
Code:
use64 ; 64-bit    
Post 02 May 2023, 00:27
View user's profile Send private message Visit poster's website Reply with quote
Andy



Joined: 17 Oct 2011
Posts: 55
Andy 02 May 2023, 15:10
I almost succeeded to write the desired code but I encounter a strange behavior after the first call of aesenc.

Code:
format PE64 console
entry start

section '.data' data readable writeable
    msg db 0x32, 0x43, 0xf6, 0xa8, 0x88, 0x5a, 0x30, 0x8d, 0x31, 0x31, 0x98, 0xa4, 0xe0, 0x37, 0x07, 0x34
    key db 0x2b, 0x7e, 0x15, 0x16, 0x28, 0xae, 0xd2, 0xa6, 0xab, 0xf7, 0x15, 0x88, 0x09, 0xcf, 0x4f, 0x3c
    enc db 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00

section '.code' code readable executable
start:
    movdqu xmm0, dqword [msg]
    movdqu xmm1, dqword [key]
    pxor xmm2, xmm2

    movdqu xmm5, xmm1
    pxor xmm0, xmm1

    aeskeygenassist xmm6, xmm1, 0x01
    pshufd xmm6, xmm6, 11111111b
    shufps xmm2, xmm1, 00010000b
    pxor xmm1, xmm2
    shufps xmm2, xmm1, 10001100b
    pxor xmm1, xmm2
    pxor xmm1, xmm6
    aesenc xmm0, xmm1

    aeskeygenassist xmm7, xmm1, 0x02
    pshufd xmm7, xmm7, 11111111b
    shufps xmm2, xmm1, 00010000b
    pxor xmm1, xmm2
    shufps xmm2, xmm1, 10001100b
    pxor xmm1, xmm2
    pxor xmm1, xmm7
    aesenc xmm0, xmm1

    aeskeygenassist xmm8, xmm1, 0x04
    pshufd xmm8, xmm8, 11111111b
    shufps xmm2, xmm1, 00010000b
    pxor xmm1, xmm2
    shufps xmm2, xmm1, 10001100b
    pxor xmm1, xmm2
    pxor xmm1, xmm8
    aesenc xmm0, xmm1

    aeskeygenassist xmm9, xmm1, 0x08
    pshufd xmm9, xmm9, 11111111b
    shufps xmm2, xmm1, 00010000b
    pxor xmm1, xmm2
    shufps xmm2, xmm1, 10001100b
    pxor xmm1, xmm2
    pxor xmm1, xmm9
    aesenc xmm0, xmm1

    aeskeygenassist xmm10, xmm1, 0x10
    pshufd xmm10, xmm10, 11111111b
    shufps xmm2, xmm1, 00010000b
    pxor xmm1, xmm2
    shufps xmm2, xmm1, 10001100b
    pxor xmm1, xmm2
    pxor xmm1, xmm10
    aesenc xmm0, xmm1

    aeskeygenassist xmm11, xmm1, 0x20
    pshufd xmm11, xmm11, 11111111b
    shufps xmm2, xmm1, 00010000b
    pxor xmm1, xmm2
    shufps xmm2, xmm1, 10001100b
    pxor xmm1, xmm2
    pxor xmm1, xmm11
    aesenc xmm0, xmm1

    aeskeygenassist xmm12, xmm1, 0x40
    pshufd xmm12, xmm12, 11111111b
    shufps xmm2, xmm1, 00010000b
    pxor xmm1, xmm2
    shufps xmm2, xmm1, 10001100b
    pxor xmm1, xmm2
    pxor xmm1, xmm12
    aesenc xmm0, xmm1

    aeskeygenassist xmm13, xmm1, 0x80
    pshufd xmm13, xmm13, 11111111b
    shufps xmm2, xmm1, 00010000b
    pxor xmm1, xmm2
    shufps xmm2, xmm1, 10001100b
    pxor xmm1, xmm2
    pxor xmm1, xmm13
    aesenc xmm0, xmm1

    aeskeygenassist xmm14, xmm1, 0x1b
    pshufd xmm14, xmm14, 11111111b
    shufps xmm2, xmm1, 00010000b
    pxor xmm1, xmm2
    shufps xmm2, xmm1, 10001100b
    pxor xmm1, xmm2
    pxor xmm1, xmm14
    aesenc xmm0, xmm1

    aeskeygenassist xmm15, xmm1, 0x36
    pshufd xmm15, xmm15, 11111111b
    shufps xmm2, xmm1, 00010000b
    pxor xmm1, xmm2
    shufps xmm2, xmm1, 10001100b
    pxor xmm1, xmm2
    pxor xmm1, xmm15
    aesenclast xmm0, xmm1    


I use a tool that simulate each round so I tried to follow in debugger to see if I get the same results for each encryption round and it's pretty much the same except for the first 4 bytes (underlined with green color). I attached two screenshots with what I expect to get and what I get in debugger after the first round of encryption.


Description:
Filesize: 285.2 KB
Viewed: 2061 Time(s)

p2.jpg


Description:
Filesize: 153.5 KB
Viewed: 2061 Time(s)

p1.jpg


Post 02 May 2023, 15:10
View user's profile Send private message Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  


< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2024, Tomasz Grysztar. Also on GitHub, YouTube.

Website powered by rwasa.