How can I use RC4 Encryption in FASM without a 3rd party .dll file?

RC4 is a variable-key-size stream cipher developed in 1987 by Ron Rivest for RSA Data Security, Inc. For seven years it was proprietary, and details of the algorithm were only available after signing a nondisclosure agreement.

In September, 1994 someone posted source code to the Cypherpunks mailing list—anonymously. It quickly spread to the Usenet newsgroup sci.crypt, and via the Internet to ftp sites around the world. Readers with legal copies of RC4 confirmed compatibility. RSA Data Security, Inc. tried to put the genie back into the bottle, claiming that it was still a trade secret even though it was public; it was too late. It has since been discussed and dissected on Usenet, distributed at conferences, and taught in cryptography courses.

RC4 is simple to describe. The algorithm works in OFB: The keystream is independent of the plaintext. It has a 8 * 8 S-box: S0, S1,..., S255. The entries are a permutation of the numbers 0 through 255, and the permutation is a function of the variable-length key. It has two counters, i and j, initialized to zero.

To generate a random byte, do the following:

i = (i + 1) mod 256
j = (j + Si) mod 256
swap Si and Sj
t = (Si + Sj) mod 256
K = St
The byte K is XORed with the plaintext to produce ciphertext or XORed with the ciphertext to produce plaintext. Encryption is fast—about 10 times faster than DES.

Initializing the S-box is also easy. First, fill it linearly: S0 = 0, S1 = 1,..., S255 = 255. Then fill another 256-byte array with the key, repeating the key as necessary to fill the entire array: K0, K1,..., K255. Set the index j to zero. Then:

for i = 0 to 255:
j = (j + Si + Ki) mod 256
swap Si and Sj
And that’s it. RSADSI claims that the algorithm is immune to differential and linear cryptanalysis, doesn’t seem to have any small cycles, and is highly nonlinear. (There are no public cryptanalytic results. RC4 can be in about 21700 (256! × 2562) possible states: an enormous number.) The S-box slowly evolves with use: i ensures that every element changes and j ensures that the elements change randomly. The algorithm is simple enough that most programmers can quickly code it from memory.

It should be possible to generalize this idea to larger S-boxes and word sizes. The previous version is 8-bit RC4. There’s no reason why you can’t define 16-bit RC4 with a 16 * 16 S-box (100K of memory) and a 16-bit word. You’d have to iterate the initial setup a lot more times—65,536 to keep with the stated design—but the resulting algorithm should be faster.

RC4 has special export status if its key length is 40 bits or under (see Section 13.. This special export status has nothing to do with the secrecy of the algorithm, although RSA Data Security, Inc. has hinted for years that it does. The name is trademarked, so anyone who writes his own code has to call it something else. Various internal documents by RSA Data Security, Inc. have not yet been made public [1320,1337].

So, what’s the deal with RC4? It’s no longer a trade secret, so presumably anyone can use it. However, RSA Data Security, Inc. will almost certainly sue anyone who uses unlicensed RC4 in a commercial product. They probably won’t win, but they will certainly make it cheaper for a company to license than fight.

RC4 is in dozens of commercial cryptography products, including Lotus Notes, Apple Computer’s AOCE, and Oracle Secure SQL. It is part of the Cellular Digital Packet Data specification [37].

include 'rc4.ash'

A:

        db 'some data',0

    some_code:

        mov esi,eax
        mov ebx,[esi]
        ret

B:

RC4Encode A,B,'the rc4 key'      

macro RC4Encode __rc4start,__rc4end,__rc4key
 {
        __rc4location = 0
        __rc4len = 0
        __rc4keylen = 0
        __rc4kboxchar = 0
        __rc4sboxchar = 0
        __rc4mix = 0
        __rc4swap1 = 0
        __rc4swap2 = 0
        __rc4counter = 0
        __rc4xor = 0
        __rc4data = 0

        __rc4location = $
        __rc4len = (__rc4end - __rc4start)


        virtual at 0
                db __rc4key
                __rc4keylen = $
        end virtual

        repeat 0x100
                virtual at 0
                        db __rc4key
                        load __rc4kboxchar byte from ((% - 1) mod __rc4keylen)
                end virtual

                db __rc4kboxchar
        end repeat

        repeat 0x100
                db (% - 1)
        end repeat

        repeat 0x100
                load __rc4kboxchar byte from (__rc4location + (% - 1))
                load __rc4sboxchar byte from (__rc4location + (% - 1) + 0x100)

                __rc4mix = ((__rc4mix + __rc4kboxchar + __rc4sboxchar) mod 256)

                load __rc4swap1 byte from (__rc4location + __rc4mix + 0x100)
                load __rc4swap2 byte from (__rc4location + (% - 1) + 0x100)

                store byte __rc4swap2 at (__rc4location + __rc4mix + 0x100)
                store byte __rc4swap1 at (__rc4location + (% - 1) + 0x100)
        end repeat

        __rc4mix = 0

        repeat __rc4len
                __rc4counter = (% mod 256)

                load __rc4swap1 byte from (__rc4location + __rc4counter + 0x100)

                __rc4mix = ((__rc4mix + __rc4swap1) mod 256)

                load __rc4swap2 byte from (__rc4location + __rc4mix + 0x100)

                store byte __rc4swap2 at (__rc4location + __rc4counter + 0x100)
                store byte __rc4swap1 at (__rc4location + __rc4mix + 0x100)

                load __rc4xor byte from (__rc4location + ((__rc4swap1 + __rc4swap2) mod 256) + 0x100)
                load __rc4data byte from (__rc4start + (% - 1))

                __rc4enc = (__rc4data xor __rc4xor)

                store byte __rc4enc at (__rc4start + (% - 1))
        end repeat

        repeat 0x200
                store byte 0x00 at (__rc4location + (% - 1))
        end repeat
 }
    

macro CryptRC4 __key , __start , __size
 {
        __rc4len = 0
        __rc4keylen = 0
        __rc4kboxchar = 0
        __rc4sboxchar = 0
        __rc4mix = 0
        __rc4swap1 = 0
        __rc4swap2 = 0
        __rc4counter = 0
        __rc4xor = 0
        __rc4data = 0

        virtual at 0
                db __key
                __rc4keylen = $
        end virtual

        rept 0x100 counter
        \{
                virtual at 0
                        db __key
                        load __rc4kboxchar byte from ((counter - 1) mod __rc4keylen)
                end virtual

                __rc4kboxchar.\#counter = __rc4kboxchar
        \}

        rept 0x100 counter
        \{
                __rc4sboxchar.\#counter = counter - 1
        \}

        rept 0x100 counter
        \{
                __rc4mix = ((__rc4mix + __rc4kboxchar.\#counter + __rc4sboxchar.\#counter) mod 256)

                rept 0x100 counter2
                \\{
                    if __rc4mix = ( counter2 - 1 )
                        __rc4swap1 =  __rc4sboxchar.\\#counter2
                    end if
                \\}
                __rc4swap2 =  __rc4sboxchar.\#counter

                rept 0x100 counter2
                \\{
                    if __rc4mix = ( counter2 - 1 )
                        __rc4sboxchar.\\#counter2 = __rc4swap2
                    end if
                \\}
                __rc4sboxchar.\#counter = __rc4swap1
        \}

        __rc4mix = 0

        repeat __size
                __rc4counter = (% mod 256)
                rept 0x100 counter2
                \{
                    if __rc4counter = (counter2 - 1)
                        __rc4swap1 =  __rc4sboxchar.\#counter2
                    end if
                \}

                __rc4mix = ((__rc4mix + __rc4swap1) mod 256)

                rept 0x100 counter2
                \{
                    if __rc4mix = ( counter2 - 1 )
                        __rc4swap2 =  __rc4sboxchar.\#counter2
                    end if
                \}

                rept 0x100 counter2
                \{
                    if __rc4counter = ( counter2 - 1 )
                        __rc4sboxchar.\#counter2 = __rc4swap2
                    end if
                \}

                rept 0x100 counter2
                \{
                    if __rc4mix = ( counter2 - 1 )
                        __rc4sboxchar.\#counter2 = __rc4swap1
                    end if
                \}

                rept 0x100 counter2
                \{
                    if ((__rc4swap1 + __rc4swap2) mod 256) = ( counter2 - 1 )
                        __rc4xor =  __rc4sboxchar.\#counter2
                    end if
                \}

                load __rc4data byte from (__start + (% - 1))

                __rc4enc = (__rc4data xor __rc4xor)

                store byte __rc4enc at (__start + (% - 1))
        end repeat
 }
    

macro copy name*, size*, address* {
  rept size/8 i:0 \{
    rept 1 j:i*8 \\{ load name\\#j qword from address+j \\}
  \}
  rept 1-((size) mod 8 - 1) shr 63 \{; something left
  local fragment
    rept 1 i:(size) and -8 \\{; calculate suffix
      name\\#i = 0; initialize accumulator
      rept (size) mod 8 j:0 \\\{
        load fragment byte from address+i+j
        name\\#i = name\\#i or fragment shl (8*j)
      \\\}
    \\}
  \}
}

macro paste name*, size*, address {
  rept (size)/8 i:0 \{
    rept 1 j:i*8 \\{
      match any, address \\\{ store qword name\\#j at address+j \\\}
      match    , address \\\{ dq name\\#j \\\}
    \\}
  \}
  rept 1 i:(size) and -8 \{; calculate suffix
    rept (size) mod 8 j:0 \\{; something left
      match any, address \\\{ store byte name\\#i shr (8*j) and 0xFF at address+i+j \\\}
      match    , address \\\{ db name\\#i shr (8*j) and 0xFF \\\}
    \\}
  \}
}    

virtual
  a db 6 dup 'a'
  copy A, 5, a; A is a name for copy of 5 bytes starting from label a
end virtual

virtual
  b db 10 dup 'b'
  paste A, 3, b+1; use our copy A to patch 3 bytes at b+1
  copy B, 10, b; B is another copy
end virtual

times 256 db '.'
paste A, 3, 16
paste B, 7, 32
paste A, 5; without address argument macro simply pastes specified number of bytes here
paste B, 10    

macro RC4 _key, start, len {
  copy plaintext, len, start
  virtual
    key db _key
    key.len = $-key
    s: times 256 db %-1
    i equ (%-1)
    j = 0
    repeat 256
      load s_i from s+i
      load k_i from key+i mod key.len
      j = (j+s_i+k_i) mod 256
      load s_j from s+j
      store s_i at s+j
      store s_j at s+i
    end repeat
    i equ (% mod 256)
    j = 0
    b: paste plaintext, len
    repeat len
      load s_i from s+i
      j = (j+s_i) mod 256
      load s_j from s+j
      store s_i at s+j
      store s_j at s+i
      load k from s+(s_i+s_j) mod 256
      load p from b+%-1
      store p xor k at b+%-1
    end repeat
    copy ciphertext, len, b
  end virtual
  paste ciphertext, len, start
}    

virtual
  repeat 256
    remainder = %-1
    repeat 8
      if remainder and 1
        remainder = remainder shr 1 xor 0xEDB88320
      else
        remainder = remainder shr 1
      end if
    end repeat
    dd remainder
  end repeat
  load crc_tab[$-$$] from $$
end virtual
; some code/data block
crc = 0
repeat $-$$
  load b from %-1
  load dword remainder from crc_tab[4*(b xor crc and 0xFF)]
  crc = crc shr 8 xor remainder
end repeat
dd crc; now the extended code/data block has zero remainder