Boo compiler extensions in the Horn Dsl

In a previous post , I outined how horn build scripts where retrieved from the file system and compiled down into IL with the help of some wrapper or facade classes defined in the excellent Rhino.Dsl library.

I also mentioned that the technique used is to create what is known as an anonymous base class that is inserted into the boo compiler pipeline. This anonymous base class becomes the base class for the Dsl build script Dsl instances. The anonymous base class exists in the horn application code.

Let us now breakdown an example Dsl script file to see how we customise the boo compiler to achieve a nice syntax. The code below is the Dsl script used to describe the metadata that is required to build horn:

install horn:
    description "A .NET build and dependency manager"
    get_from svn("http://scotaltdotnet.googlecode.com/svn/trunk/")
    build_with msbuild, buildfile("src/horn.sln"), FrameworkVersion35
    output "Output"
    shared_library "."
                 
dependencies:
    depend @log4net >> "log4net"
    depend @castle  >> "castle.core"
    depend @castle  >> "Castle.DynamicProxy2"
    depend @castle  >> "castle.microKernel"
    depend @castle  >> "castle.windsor"

As we mentioned previously, the build script listed above is effectively a derived class of the BooConfigReader class that we have inserted into the boo compiler pipeline with the following code:

var factory = new DslFactory
            {
                BaseDirectory = packageTree.CurrentDirectory.FullName
            };
 
factory.Register<BooConfigReader>(new ConfigReaderEngine());

Below is a code snippet of the basic outline of the BooConfigReader anonymous base class we have defined in the horn application code to act as a base class for our Dsl scripts:

public abstract class BooConfigReader
{
    [Meta]
    public static Expression install(ReferenceExpression expression, BlockExpression action)
    {
        var installName = new StringLiteralExpression(expression.Name);
 
        return new MethodInvocationExpression(
                new ReferenceExpression("GetInstallerMeta"),
                installName,
                action
            );
    }
 
    public void description(string text)
    {
        Description = text;
    }

I will list code in both C# and boo for the BooConfigReader anonymous base class that will illustrate how we parse the code from Dsl script into our semantic or domain model which horn will use to build the required component.

Let us start at line 1 as that is a very good place to start:

install horn:

We have several extension points in boo when it comes to extending the language, from compiler steps to meta-methods, from AST attributes to AST macros. This means you can write extensions in the compiler when the boo code compiles.

One of these techniques is what is known in boo speak as a meta-method. A meta-method is a short cut to the compiler that accepts an AST node and returns an AST node.

It is important to comprehend what an AST node is. One of the steps in the boo compiler pipeline is known as the parsing stage which parses the source code stream into an abstract syntax tree. Subtrees or AST nodes will be passed to compiler extension points such as meta-methods or macros during compilation.

Meta-methods must be static and marked with the MetaAttribute for the compiler to recognise them as so. The BooConfigReader contains a meta-method named install that will take the AST node install horn: and transform it into an instance method of the BooConfigReader class. Because boo supports parenthisis-less method invocations we can call the install method as is listed above or it could be called with the parenthis install(horn).

Below is the C# listing of the install meta-method:

[Meta]
public static Expression install(ReferenceExpression expression, BlockExpression action)
{      
    var installName = new StringLiteralExpression(expression.Name);
 
    return new MethodInvocationExpression(
            new ReferenceExpression("GetInstallerMeta"),
            installName,
            action
        );
}

It is important to remember that the meta-method is invoked at compile time. When the compiler sees the call to the install meta-method during compilation of the Dsl script, it does'nt emit the code to call the method at runtime. Instead during compilation the meta-method is executed. Passed to the install meta-method is an AST of the arguments of the method code, the first being the horn part of the install horn: expression. The second being the anonymous block which is everything after the colon and is scoped by indentation. The code below is the anonyomous block.

    description "A .NET build and dependency manager"
    get_from svn("http://scotaltdotnet.googlecode.com/svn/trunk/")
    build_with msbuild, buildfile("src/horn.sln"), FrameworkVersion35
    output "Output"
    shared_library "."

Let us re-examine the method signature of the install meta-method code:

[Meta]
public static Expression install(ReferenceExpression expression, BlockExpression action)

The horn part of the install horn: expression is passed to the meta-method as a ReferenceExpression type. A ReferenceExpression type can be simply thought of as a token or block of text that is passed to the meta-method as an argument.

The anonymous method block is passed to the meta-method as a BlockExpression type.

Let us further examine the method body of the install meta-method:

    var installName = new StringLiteralExpression(expression.Name);
 
    return new MethodInvocationExpression(
            new ReferenceExpression("GetInstallerMeta"),
            installName,
            action
        );

I Will reiterate once more that the install meta-method is called at compile time. The meta-method takes an AST node and returns an AST node. In the code above we are creating the result of calling the meta-method that will replace the AST that was passed to it. In the above code, we are creating what is known as a MethodInvocationExpression that is creating a call to an instance method of the BooConfigReader class called GetInstallerName and the arguments that will be passed to it.

To sum up, the compiler replaces the install horn: expression with the following call to an instance of the BooConfigReader class GetInstallerMeta("horn", action);.

The GetInstallerMeta method is listed below for completeness.

public void GetInstallerMeta(string installName, Action installDelegate)
{
    InstallName = installName;
 
    installDelegate();
}

The same result can be achieved by defining the same meta-method in boo:

abstract class BooConfigReader(IQuackFu): 
    callable Action()
 
    [Meta]
    static def install(expression as ReferenceExpression, action as Expression):
        name = StringLiteralExpression(expression.Name)
 
        return [|
            self.GetInstallerMeta($name, $action)
        |]                 

The return statement of the install method introduces a concept that makes boo more suited to writing code for compiler manipulation. This strange expression creation is what is known as quasi quotation.

Quasi quotation is only accessible through boo and you will not be able to recreate this experience in any other .NET language. Writing the code to create the AST node in the C# version of the install method can quickly get tedious. Quasi quotation allows us to produce this AST code in a more concise manner.

When the compiler encounters this code, it does the usual parsing of the code, but instead of outputting the IL instructions that would execute the code, it outputs the code to build the required AST, much in the same way we built the MethodInvocationExpression in the C# example.

The most alluring part of quasi-quotation is that we are not limited to static binding. We can refer to external variables and the compiler will take care of creating the correct AST node. In the example above, we can access the name variable and the action argument in the quasi quotation. The experience is analagous to closures where you can access external variables that are not defined in the inner block.

We can now move swiftly through the rest of the Dsl:

get_from svn("http://scotaltdotnet.googlecode.com/svn/trunk/")

This line tells horn which source control management system to retrieve the source from and the uri of the source control expression.
This is painfully easy to handle and we use another meta-method to implement this that I have listed below:
We define another :

    [Meta]
    static def get_from(expression as MethodInvocationExpression):
        return expression

The boo compiler will pass in the svn("http://scotaltdotnet.googlecode.com/svn/trunk/") AST node in the form of a MethodInvocationExpression into the get_from meta-method. We will simply return this expression. If you remember, a meta-method takes an AST node as an argument and returns an AST node that will replace the meta-method in the IL.

The compiler will replace the meta-method with the call to svn("http://scotaltdotnet.googlecode.com/svn/trunk/"). This does of course rely on us having an svn instance method in our anonymouse base class which of course we do. I have listed it below for completeness.

    def svn(url as string):
          sourceControl = SCM.SVNSourceControl(url)

The last fragment of Dsl script code I want to examine in this post is listed below:

build_with msbuild, buildfile("src/horn.sln"), FrameworkVersion35

This line of code instructs horn which build engine we will use, the relative path to the build file and what version of the .NET framework horn will use to compile the source against.

It will come as no great suprise to learn that we have handled this yet again with a meta-method:

    [Meta]
    def build_with(builder as ReferenceExpression, build as MethodInvocationExpression, frameWorkVersion as ReferenceExpression):
 
        buildFile = build.Arguments[0]
        version = StringLiteralExpression(frameWorkVersion.Name) 
 
        return [|
            $builder($buildFile, $version)
        |]

The above code listing shows us how we can start to build more complex expressions using quasi quotation that will be returned from the meta-method.

You can see that we are using the builder argument to dynamically state which instance method to invoke. We are also referencing to external variables.

Below is the same method defined in C#.

[Meta]
public static Expression build_with(ReferenceExpression builder, MethodInvocationExpression build, ReferenceExpression frameWorkVersion)
{
    var targetName = builder.Name;
 
    return new MethodInvocationExpression(
            new ReferenceExpression(targetName),
            build.Arguments[0],
            new StringLiteralExpression(frameWorkVersion.Name)
        );
}

Both these methods will return a call to an instance method named msbuild which of course must be present for the compilation to succeed.

Below is the msbuild method:

    def msbuild(buildFile as string, frameworkVersion):
        version = System.Enum.Parse(typeof(Horn.Domain.Framework.FrameworkVersion), frameworkVersion)
        BuildEngine = BuildEngines.BuildEngine(Horn.Domain.MSBuildBuildTool(), buildFile, version)

If any of this is of interest to you then please join the Horn user group for updates or check out the source here.