Reverse Engineering (1/5): foundations — from source code to binaries and assembly

You are handed an executable with no source: a broken third-party library, an undocumented legacy device, or a suspected malware sample. To understand it, you have to run compilation backwards. This is part 1 of the 5-part Reverse Engineering for Developers series: the foundations for reading binaries — the compilation pipeline, executable formats, the minimum assembly you need, and calling conventions.

What does compilation throw away?

Reverse engineering is essentially compilation in reverse. Each stage destroys a layer of information you will later have to rebuild:

1. Preprocessing (gcc -E): macros and #include are expanded — macro names disappear.
2. Compilation (gcc -S): C becomes assembly — variable names, types and loop structure are lost; only registers and stack offsets remain.
3. Assembly (as): assembly becomes an object file with relocations.
4. Linking (ld): objects and libraries are combined. Static linking pulls in thousands of library functions you must learn to ignore.

Try it on a tiny program and compare optimisation levels:

gcc -O0 -g -o hello_O0 hello.c
gcc -O2 -s -o hello_O2 hello.c      # -s strips symbols
objdump -d -M intel hello_O0 | less

At -O2, functions get inlined, loops unrolled and variables live only in registers. That gap is exactly what makes real-world binaries hard to read.

Executable formats

Two maps before you read a single instruction: the import table (a binary calling connect and CryptEncrypt already tells a story) and the strings (strings -n 8 file): error messages, URLs and format strings point straight at interesting functions.

The minimum assembly you need

You do not need to memorise the Intel manual. About thirty instructions cover most compiler output:

• Data: mov, lea, push, pop, movzx, movsx
• Arithmetic: add, sub, imul, xor, and, or, shl, shr, sar
• Control flow: cmp, test, jmp, je/jne, jl/jg (signed), jb/ja (unsigned), call, ret

Patterns to spot instantly: xor eax, eax sets zero; lea rax, [rbx+rbx*4] is usually multiplication by 5, not a memory read; signed versus unsigned jumps reveal the original variable type.

Calling conventions

Recognising C constructs in assembly

• if/else: cmp + a conditional jump that skips a block.
• for/while: a backward jump to a label above.
• switch: a bounds check, then jmp [table + rax*8] (a jump table).
• struct: the same base register with fixed offsets [rdi+0x8], [rdi+0x10].
• C++ virtual call: mov rax, [rdi] (load the vtable), then call [rax+0x18].

Caveats

• Legal: only analyse software you own, are authorised to test, or that is published for practice (crackmes, CTFs). EU Directive 2009/24/EC (Art. 6) and DMCA §1201(f) allow decompilation for interoperability, but licences and local law still apply.
• Isolated lab: use a Linux VM with snapshots; never run unknown samples on your work machine.

How to apply it

• Dev: write five small C functions (if, loop, switch, struct, virtual), paste them into Compiler Explorer and compare -O0 with -O2 until you can predict the assembly.
• Tech Lead: make reading crash dumps and disassembly part of the team's debugging skills — some production bugs only show up at the binary level.
• PM: when integrating an undocumented device or legacy system, reversing the format is a legitimate option; budget time and a legal review for it.