Note that this is basically the same as the previous version, excluding the fact that it uses C++ idioms more and does a bit better in error checking.
Essentially a refactor of the C formed lexer into C++ style. I can already see some benefits from doing this, in particular speed of prototyping.