In the beginning...

The Void

Starter projects

This tutorial will take you through step-by-step on how to create literals and operators. If you'd prefer to skip ahead (though if this is your first project I'd recommend continuing), then you can download one of the starter projects which contain a set of default literals and operators to get you started here.

In the beginning there is nothing... or in our case null. Alas, since we haven't defined what null is we don't even have that yet! We can test this by typing null in our runner app:

Aardvark Language Test Utility
===============================
null
dev.larf.exception.ParserException: Unexpected token 'null' found in expression. Strict syntax checking is enabled
    at dev.larf.lexer.LARFLexer.analysePatterns(LARFLexer.java:98)
    at dev.larf.lexer.LARFLexer.tokenize(LARFLexer.java:57)
    at dev.larf.lexer.Lexer.tokenize(Lexer.java:49)
    at dev.larf.processor.LARFProcessor.process(LARFProcessor.java:82)
    at dev.larf.runner.LARFRunner.run(LARFRunner.java:39)
    at dev.larf.runner.LARFRunner.run(LARFRunner.java:10)
    at dev.larf.runner.Application.main(Application.java:10)

Token Class

As such, this will be our first task. Create a new package called tokens. Within that add another package called literals and add a new file called NullToken.java:

[src]
   [main]
       [java]
           [com.aardvark]
               [config]
                   AardvarkConfig.java
               [tokens]
                   [literals]
                       NullToken.java
               Application.java

Open up the new token class and add extends Token<Void> after the class name. As with the configuration file, this will inherit a number of methods to implement from the parent class. These can be seen here:

public class NullToken extends Token<Void> {

  @Override
  public Token<Void> createToken(String value) {
    return null;
  }

  @Override
  public PatternType getPatternType() {
    return null;
  }

  @Override
  public String getPattern() {
    return null;
  }

  @Override
  public Optional<String> getGuidance(String token, List<Integer> groupsCount) {
    return Optional.empty();
  }

  @Override
  protected List<Token<?>> process(LARFParser parser, LARFContext context, LARFConfig config) {
    return null;
  }
}

I'll briefly run through the features of this class before diving in. Firstly, you'll notice that the Token class we are extending always requires a generic type. This determines the type each token will represent / store as a value. In our case, since there is no Null type in Java, we can use Void. Here is a description of the other functions in the class:

createToken: Gets called when the lexer matches this tokens pattern against the beginning of the expression. It returns a new instance with the function parameter being the matched value. In this case it's null so we can just invoke the blank constructor as we won't store it.
getPatternType: This is an enum representing the pattern type for the current token. This has two possible values being either REGEX or GRAMMAR. For simple literals (Integer, Float, Double, String etc) these can be matched using regular expressions. For more complex structures like statements or collections, those use grammar.
getPattern: Dependent on the pattern type, this will either contain a regular expression or a grammar definition. For example, to define our null keyword we'll be using "^null" or "^nil" (whatever you prefer). As the expression is processed, its matched against what's left of the unprocessed expression String. The '^' before the keyword simply means that this is matched against the beginning of the String.
getGuidance: This is used when we start defining statements. If the expression is not valid and the current active statement is still waiting for a given token, we can provide them a message here to show them where they made the mistake and how to fix it.
process: During expression resolution, this gets called by the parser and this is where the bulk of our language code will go. For literals though like null, you will simply return the current token as-is.

Given the above, we can now populate our token with the following:

public class NullToken extends Token<Void> {

  public NullToken() {
    super("null", null);
  }

  @Override
  public PatternType getPatternType() {
    return PatternType.REGEX;
  }

  @Override
  public String getPattern() {
    return "^null";
  }

  @Override
  public Optional<String> getGuidance(String token, List<Integer> groupsCount) {
    return Optional.empty();
  }

  @Override
  public List<Token<?>> process(LARFParser parser, LARFContext context, LARFConfig config) {
    return Collections.singletonList(this);
  }

  @Override
  public Token<Void> createToken(String value) {
    return new NullToken();
  }
}

Configure and Run

We have also defined a constructor which calls the parent. This call sets a unique name for the token and specifies a value for the token, in this case it is just null. Congratulations, we have our first token! Now all we need to do is add it to our configuration:

public class AardvarkConfig extends LARFConfig {
  //...
  @Override
  protected void initTokenHandlers() {
    addTokenHandler(new NullToken());
  }
  //...
}

Go back to our runner, and...

Aardvark Language Test Utility
===============================
null
Result: null (Type: Null, Time taken: 19ms)

we get null being returned when it is typed in. Nothing too exciting, but something that is necessary to have in most languages. If you were wondering, don't worry too much about the performance for this first call. You'll notice repeated or different expressions revert to 1 millisecond or less. This is caused by the overhead of Java loading the classes and resources for the first time.

In the beginning...

The Void​

Token Class​

Configure and Run​

The Void

Token Class

Configure and Run