Lexical Analyzer

For those of you who do not know, I've been trying to create my own compiler. The first step in this is the lexical analyzer. Basically what a lexical analyzer does is separate everything in a source code and converts certain words/characters into tokens. The way that my lexical analyzer works in this GUI program is that it stores the tokens in an XML file in this format:

Code:

---

<source>
    <token>
        <name>token class name here</name>
        <value>token value here</value>
    </token>
    ....
    ....
</source>

---

Here is an image of the program in action:
Attachment 114945

The scanner scans in order, so some important ordering in the case above is that I have my number class above my decimal class. I also have the identifier very last in the order. The reason for the number before the decimal is because any decimal(in my case) can match a number, but not every number can match a decimal. The reason why I have the identifier last is because almost everything(in my case) can be interpreted as an identifier.

Here is the program(minus the binaries):
Attachment 114947

Here is the template that I use for my language in case you don't want to create the token classes:
Attachment 114949

Now this program is not practical in creating a custom compiler, it just simply shows a technique of a scanner.

Keep it up, very much interested to see this to completion.

Any updates on this?

It's funny that you mention something, because I've recently picked it back up about two/three days ago.

Actually, even though I have downloaded you example I don't know how to use it, could you provide a sample code that it can 'parse'?

Sorry for the late reply, busy weekend.

Quote:

---

could you provide a sample code that it can 'parse'?

---

This example is just of the lexical analyzer or lexer. The syntax analyzer or parser is a separate component of a compiler, but it parses the output of the lexical analyzer. I did have a working parser, but I was unhappy with it so I scrapped it. Now I'm working on a new one.