When the compiler first looks at a source code file, it just sees a giant string of meaningless characters: i, n, t, , x, , =, , 1, 0, ;. The Lexer's job is to read this string character by character, discard useless information (like spaces and comments), and group the characters into logical words called Tokens.
TYPE_INT, TYPE_VOID, IF, RETURN).total, binarySearch).42, "Hello").PLUS, EQ, LAND, GT).=, ;, {, }, (, )).By converting the code into a stream of tokens, we make the next phase (the Parser) infinitely easier to write. The Parser no longer has to worry about whether a user put one space or ten spaces between int and x; it simply looks at the clean, organized list of tokens.
Type any VizLang code into the box below and click the button to watch the Lexer "chop" it into categorized tokens, exactly how your C code does it! Notice how it completely ignores comments.