From Text to Tokens: Lexing

To you, a program is text: lines, indentation, comments, prose-like structure. No language tool actually works on that text. The very first thing a compiler, interpreter, or obfuscator does is lex your file: scan it left to right and chop it into tokens, the atomic words of the language.

What a token is

A token is one meaningful unit: a name like price, a number like 42, a string like "hello", a keyword like while, or a symbol like + or {. The lexer also records what kind each token is, so everything downstream can ask is this a number without re-reading characters.

It is the same thing you do when reading a sentence. You do not process letters, you snap them into words and instantly know their roles. The lexer does that for code, and just like you ignore double spaces between words, it discards spacing that carries no meaning.

Try the live lexer

This is a real lexer for the academy's teaching language, running on every keystroke. Edit the code and watch the token stream react. Try adding spaces, blank lines, and a # comment, and notice what the stream does and does not care about.

What survives, and what disappears

• Names, numbers, strings, keywords, and symbols survive, as typed tokens.
• Spaces, indentation, and blank lines disappear. They only ever existed to help humans see structure.
• Comments disappear completely. The next stage of the pipeline never knows they were there.

This is why minified or mangled formatting is zero protection. A pretty-printer lexes the ugly file, then prints the same tokens back with clean layout, and the ugliness is gone in milliseconds. Any tool that consumes tokens is blind to formatting by construction. Real obfuscation has to change the tokens themselves, the names and values, or the structure they form, which is the subject of the next levels.

One subtlety worth respecting: a lexer must know exactly where a string starts and ends, because anything inside quotes is data, not code. A number of real-world obfuscator bugs, including in commercial tools, have come from text transforms that ignored string boundaries and rewrote characters inside literals. When you build or evaluate a transform, string extent is the first correctness question to ask.

The lexer defines what a language tool can see. From this stage on, your carefully formatted source is just a typed stream of tokens, and every transform and attack you will study operates on that stream or on trees built from it.

Frequently asked questions