challenges.re - RE 02#

The Challenges.re is a website with compilation of Reverse Engineering exercises, I’m writing this document for me only, to exercise my knowledge with assembly and reverse engineering. If you want to solve these exercises by yourself one day, please do not read my solution, try for yourself, my answer can be wrong, so don’t trust me, trust the process.

Question#

Reverse Engineering challenge #2. Tags: . What does this code do?

Assembly Code#

To clarify my mind into this exercise, I decided to comment each line with a longer description.

mov    eax,DWORD PTR [esp+0x4] ; 	MOVE value in the Address ESP+0x4 to EAX
bswap  eax ; 						SWAP the BYTES of a value ex: 0x123456 => 0x563412
mov    edx,eax ; 					Move EAX value to EDX
and    eax,0xf0f0f0f ;  			Make AND comparation with 0x0F0F0F0F and store to EAX
and    edx,0xf0f0f0f0 ; 			Make AND comparation with 0xF0F0F0F0 and store to EDX
shr    edx,0x4 ; 					Move the binary of EDX 4 Bits to the right (divide to 16)
shl    eax,0x4 ; 					Move the binary of EAX 4 Bits to the left (multiply 2⁴)
or     eax,edx ; 					Make an OR comparation and store into EAX
mov    edx,eax ; 					Move EAX value to EDX
and    eax,0x33333333 ; 			Make an AND comparation and store into EAX
and    edx,0xcccccccc ; 			Make an AND comparation and store into EDX
shr    edx,0x2 ; 					Move the binary of EDX 2 Bits to the right (divide to 4)
shl    eax,0x2 ; 					Move the binary of EAX 2 Bits to the left (multiply 2²)
or     eax,edx ; 					Make an OR comparation and store into EAX
mov    edx,eax ; 					Move EAX value to EDX (?)
and    eax,0x55555555 ; 			Make an AND comparation and store into EAX
and    edx,0xaaaaaaaa ; 			Make an AND comparation and store into EDX
add    eax,eax ;  					Add EAX with EAX (EAX += EAX || EAX*=2)
shr    edx,1 ; 						Move the binary of EDX 1 Bit to the right (divide to 2)
or     eax,edx ;					Make an OR comparation and store into EAX
ret ;								Return Operation

I’ve used the 0x123456 value as an example, it’s clearer for me to understand it using an example.

To explain what this code does, I’m going to track the values in EAX and EDX and see how they change, we can see:

; EAX: ; EDX: ;
; function f does a thing 
; And return
mov    eax,DWORD PTR [esp+0x4] ; MOVE value in the Address ESP+0x4 to EAX
	; EAX: 0x1234; EDX: 0x0;
bswap  eax ; SWAP the BYTES of a value ex: 0x123456 => 0x563412
	; EAX: 0x3412; EDX: 0x0;
mov    edx,eax ; Move EAX value to EDX
	; EAX: 0x3412; EDX: 0x3412;
and    eax,0xf0f0f0f ;  Make AND comparation with 0x0F0F0F0F and store to EAX
	; EAX: 0x402; EDX: 0x3412;
and    edx,0xf0f0f0f0 ; Make AND comparation with 0xF0F0F0F0 and store to EDX
	; EAX: 0x402; EDX: 0x3010;
shr    edx,0x4 ; Remove a Byte from the right
	; EAX: 0x402; EDX: 0x301;
shl    eax,0x4 ; Add a Byte to the right
	; EAX: 0x4020; EDX: 0x301;
or     eax,edx ; Make an OR comparation and store into EAX
	; EAX: 0x4321; EDX: 0x301;
mov    edx,eax ; Move EAX value to EDX
	; EAX: 0x4321; EDX: 0x4321;
and    eax,0x33333333 ;Make an AND comparation and store into EAX
	; EAX: 0x321; EDX: 0x4321;
and    edx,0xcccccccc ;Make an AND comparation and store into EDX
	; EAX: 0x321; EDX: 0x4000;
shr    edx,0x2 ; Move the binary of EDX 2 Bits to the right (divide to 4)
	; EAX: 0x321; EDX: 0x1000;
shl    eax,0x2 ; Move the binary of EAX 2 Bits to the left (multiply 2²)
	; EAX: 0xC84; EDX: 0x1000;
or     eax,edx ; Make an OR comparation and store into EAX
	; EAX: 0x1C84; EDX: 0x1000;
mov    edx,eax ; Move EAX value to EDX
	; EAX: 0x1C84; EDX: 0x1C84;
and    eax,0x55555555 ;Make an AND comparation and store into EAX
	; EAX: 0x1404; EDX: 0x1C84;
and    edx,0xaaaaaaaa ;Make an AND comparation and store into EDX
	; EAX: 0x1404; EDX: 0x880;
add    eax,eax ; Add EAX with EAX (EAX += EAX || EAX*=2)
	; EAX: 0x2808; EDX: 0x880;
shr    edx,1 ; Move the binary of EDX 1 Bit to the right (divide to 2)
	; EAX: 0x2808; EDX: 0x440;
or     eax,edx ; Make an OR comparation and store into EAX
	; EAX: 0x2C48; EDX: 0x440;
ret ;Return operation

If we separate the functions of this program, we could divide in 2, one part for inverting and other for scramble:

; EAX: ; EDX: ;
; function f does a thing 
; And return
invert_the_bytes:
	mov    eax,DWORD PTR [esp+0x4] 
	; EAX: 0x1234; EDX: 0x0;
	bswap  eax
	; EAX: 0x3412; EDX: 0x0;
	mov    edx,eax 
	; EAX: 0x3412; EDX: 0x3412;
	and    eax,0xf0f0f0f
	; EAX: 0x402; EDX: 0x3412;
	and    edx,0xf0f0f0f0
	; EAX: 0x402; EDX: 0x3010;
	shr    edx,0x4
	; EAX: 0x402; EDX: 0x301;
	shl    eax,0x4
	; EAX: 0x4020; EDX: 0x301;
	or     eax,edx
	; EAX: 0x4321; EDX: 0x301;

scramble:
	mov    edx,eax
	; EAX: 0x4321; EDX: 0x4321;
	and    eax,0x33333333
	; EAX: 0x321; EDX: 0x4321;
	and    edx,0xcccccccc
	; EAX: 0x321; EDX: 0x4000;
	shr    edx,0x2
	; EAX: 0x321; EDX: 0x1000;
	shl    eax,0x2
	; EAX: 0xC84; EDX: 0x1000;
	or     eax,edx
	; EAX: 0x1C84; EDX: 0x1000;
	mov    edx,eax
	; EAX: 0x1C84; EDX: 0x1C84;
	and    eax,0x55555555
	; EAX: 0x1404; EDX: 0x1C84;
	and    edx,0xaaaaaaaa
	; EAX: 0x1404; EDX: 0x880;
	add    eax,eax
	; EAX: 0x2808; EDX: 0x880;
	shr    edx,1
	; EAX: 0x2808; EDX: 0x440;
	or     eax,edx
	; EAX: 0x2C48; EDX: 0x440;
	ret

What this code do:#

This takes a message from the ESP (Stack Pointer) + 0x4, invert the bytes and call an AND operation with odds and even bits. This makes the message be scrambled.

But putting the scrambled message in the function, it will re-scramble to the same original message. Similar to an XOR.

Example:

input: 0x1234 => output: 0x2C48
input: 0x2C48 => output: 0x1234