Avx VPCMPB error show Ida pro

flat assembler
Message board for the users of flat assembler.

Index > Main > Avx VPCMPB error show Ida pro

Author

Thread

Roman

Joined: 21 Apr 2012
Posts: 1792

Roman 15 Aug 2023, 10:07

Code:

;fasmw 1.73 compiled fine
;32 bits program
VPCMPB k1 {k2}, ymm2, ymm3, 1

But Ida Pro show error: execute an illegal instruction
If I run my program I get crash.
Amd Ryzen 3500 avx supported

15 Aug 2023, 10:07

revolution
When all else fails, read the source

Joined: 24 Aug 2004
Posts: 20339
Location: In your JS exploiting you and your system

revolution 15 Aug 2023, 10:13

AFAICT fasm produces the correct encoding.

Code:

00000000  62F36D2A3FCB01    vpcmpltb k1{k2},ymm2,ymm3

15 Aug 2023, 10:13

Roman

Joined: 21 Apr 2012
Posts: 1792

Roman 15 Aug 2023, 10:15

VPCMPB this is avx 512
Maybe my ryzen 5 3500 not supported

15 Aug 2023, 10:15

Roman

Joined: 21 Apr 2012
Posts: 1792

Roman 15 Aug 2023, 10:18

Quote:

vpcmpltb k1{k2},ymm2,ymm3

Fasmw 1.73 compiled fine
But Ida Pro show
VPCMPB k1{k2}, ymm2, ymm2, 1

15 Aug 2023, 10:18

revolution
When all else fails, read the source

Joined: 24 Aug 2004
Posts: 20339
Location: In your JS exploiting you and your system

revolution 15 Aug 2023, 10:23

I used the nasm disassembler. I don't vouch for the correctness of the disassembly, but it shows the 7 bytes produced correctly.

15 Aug 2023, 10:23

Roman

Joined: 21 Apr 2012
Posts: 1792

Roman 15 Aug 2023, 10:42

Ok. Forget
I found
vpcmpeqb ymmdest, ymmsrc1, ymmsrc2
vpcmpgtb ymmdest, ymmsrc1, ymmsrc2

Ymmdest this is mask to YMM reg

All work fine in 32 bits

15 Aug 2023, 10:42

< Last Thread | Next Thread >

Forum Rules:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum