You can listen and watch my talk on Dsls at Dsl DevCon I gave last week.
You cannot fail to be wooed by Irish accent here.
I am personally dismayed by my overuse of the expression "ahhhhhh" after every sentence.
Tuesday, 21 April 2009
Sunday, 12 April 2009
Boo compiler extensions in the Horn Dsl
In a previous post , I outined how horn build scripts where retrieved from the file system and compiled down into IL with the help of some wrapper or facade classes defined in the excellent Rhino.Dsl library.
I also mentioned that the technique used is to create what is known as an anonymous base class that is inserted into the boo compiler pipeline. This anonymous base class becomes the base class for the Dsl build script Dsl instances. The anonymous base class exists in the horn application code.
Let us now breakdown an example Dsl script file to see how we customise the boo compiler to achieve a nice syntax. The code below is the Dsl script used to describe the metadata that is required to build horn:
As we mentioned previously, the build script listed above is effectively a derived class of the BooConfigReader class that we have inserted into the boo compiler pipeline with the following code:
Below is a code snippet of the basic outline of the BooConfigReader anonymous base class we have defined in the horn application code to act as a base class for our Dsl scripts:
I will list code in both C# and boo for the BooConfigReader anonymous base class that will illustrate how we parse the code from Dsl script into our semantic or domain model which horn will use to build the required component.
Let us start at line 1 as that is a very good place to start:
We have several extension points in boo when it comes to extending the language, from compiler steps to meta-methods, from AST attributes to AST macros. This means you can write extensions in the compiler when the boo code compiles.
One of these techniques is what is known in boo speak as a meta-method. A meta-method is a short cut to the compiler that accepts an AST node and returns an AST node.
It is important to comprehend what an AST node is. One of the steps in the boo compiler pipeline is known as the parsing stage which parses the source code stream into an abstract syntax tree. Subtrees or AST nodes will be passed to compiler extension points such as meta-methods or macros during compilation.
Meta-methods must be static and marked with the MetaAttribute for the compiler to recognise them as so. The BooConfigReader contains a meta-method named install that will take the AST node install horn: and transform it into an instance method of the BooConfigReader class. Because boo supports parenthisis-less method invocations we can call the install method as is listed above or it could be called with the parenthis install(horn).
Below is the C# listing of the install meta-method:
It is important to remember that the meta-method is invoked at compile time. When the compiler sees the call to the install meta-method during compilation of the Dsl script, it does'nt emit the code to call the method at runtime. Instead during compilation the meta-method is executed. Passed to the install meta-method is an AST of the arguments of the method code, the first being the horn part of the install horn: expression. The second being the anonymous block which is everything after the colon and is scoped by indentation. The code below is the anonyomous block.
description "A .NET build and dependency manager"
get_from svn("http://scotaltdotnet.googlecode.com/svn/trunk/")
build_with msbuild, buildfile("src/horn.sln"), FrameworkVersion35
output "Output"
shared_library "."
Let us re-examine the method signature of the install meta-method code:
The horn part of the install horn: expression is passed to the meta-method as a ReferenceExpression type. A ReferenceExpression type can be simply thought of as a token or block of text that is passed to the meta-method as an argument.
The anonymous method block is passed to the meta-method as a BlockExpression type.
Let us further examine the method body of the install meta-method:
I Will reiterate once more that the install meta-method is called at compile time. The meta-method takes an AST node and returns an AST node. In the code above we are creating the result of calling the meta-method that will replace the AST that was passed to it. In the above code, we are creating what is known as a MethodInvocationExpression that is creating a call to an instance method of the BooConfigReader class called GetInstallerName and the arguments that will be passed to it.
To sum up, the compiler replaces the install horn: expression with the following call to an instance of the BooConfigReader class GetInstallerMeta("horn", action);.
The GetInstallerMeta method is listed below for completeness.
The same result can be achieved by defining the same meta-method in boo:
The return statement of the install method introduces a concept that makes boo more suited to writing code for compiler manipulation. This strange expression creation is what is known as quasi quotation.
Quasi quotation is only accessible through boo and you will not be able to recreate this experience in any other .NET language. Writing the code to create the AST node in the C# version of the install method can quickly get tedious. Quasi quotation allows us to produce this AST code in a more concise manner.
When the compiler encounters this code, it does the usual parsing of the code, but instead of outputting the IL instructions that would execute the code, it outputs the code to build the required AST, much in the same way we built the MethodInvocationExpression in the C# example.
The most alluring part of quasi-quotation is that we are not limited to static binding. We can refer to external variables and the compiler will take care of creating the correct AST node. In the example above, we can access the name variable and the action argument in the quasi quotation. The experience is analagous to closures where you can access external variables that are not defined in the inner block.
We can now move swiftly through the rest of the Dsl:
This line tells horn which source control management system to retrieve the source from and the uri of the source control expression.
This is painfully easy to handle and we use another meta-method to implement this that I have listed below:
We define another :
The boo compiler will pass in the svn("http://scotaltdotnet.googlecode.com/svn/trunk/") AST node in the form of a MethodInvocationExpression into the get_from meta-method. We will simply return this expression. If you remember, a meta-method takes an AST node as an argument and returns an AST node that will replace the meta-method in the IL.
The compiler will replace the meta-method with the call to svn("http://scotaltdotnet.googlecode.com/svn/trunk/"). This does of course rely on us having an svn instance method in our anonymouse base class which of course we do. I have listed it below for completeness.
The last fragment of Dsl script code I want to examine in this post is listed below:
This line of code instructs horn which build engine we will use, the relative path to the build file and what version of the .NET framework horn will use to compile the source against.
It will come as no great suprise to learn that we have handled this yet again with a meta-method:
The above code listing shows us how we can start to build more complex expressions using quasi quotation that will be returned from the meta-method.
You can see that we are using the builder argument to dynamically state which instance method to invoke. We are also referencing to external variables.
Below is the same method defined in C#.
Both these methods will return a call to an instance method named msbuild which of course must be present for the compilation to succeed.
Below is the msbuild method:
If any of this is of interest to you then please join the Horn user group for updates or check out the source here.
I also mentioned that the technique used is to create what is known as an anonymous base class that is inserted into the boo compiler pipeline. This anonymous base class becomes the base class for the Dsl build script Dsl instances. The anonymous base class exists in the horn application code.
Let us now breakdown an example Dsl script file to see how we customise the boo compiler to achieve a nice syntax. The code below is the Dsl script used to describe the metadata that is required to build horn:
install horn:
description "A .NET build and dependency manager"
get_from svn("http://scotaltdotnet.googlecode.com/svn/trunk/")
build_with msbuild, buildfile("src/horn.sln"), FrameworkVersion35
output "Output"
shared_library "."
dependencies:
depend @log4net >> "log4net"
depend @castle >> "castle.core"
depend @castle >> "Castle.DynamicProxy2"
depend @castle >> "castle.microKernel"
depend @castle >> "castle.windsor"
As we mentioned previously, the build script listed above is effectively a derived class of the BooConfigReader class that we have inserted into the boo compiler pipeline with the following code:
var factory = new DslFactory
{
BaseDirectory = packageTree.CurrentDirectory.FullName
};
factory.Register<BooConfigReader>(new ConfigReaderEngine());
Below is a code snippet of the basic outline of the BooConfigReader anonymous base class we have defined in the horn application code to act as a base class for our Dsl scripts:
public abstract class BooConfigReader
{
[Meta]
public static Expression install(ReferenceExpression expression, BlockExpression action)
{
var installName = new StringLiteralExpression(expression.Name);
return new MethodInvocationExpression(
new ReferenceExpression("GetInstallerMeta"),
installName,
action
);
}
public void description(string text)
{
Description = text;
}
I will list code in both C# and boo for the BooConfigReader anonymous base class that will illustrate how we parse the code from Dsl script into our semantic or domain model which horn will use to build the required component.
Let us start at line 1 as that is a very good place to start:
install horn:
We have several extension points in boo when it comes to extending the language, from compiler steps to meta-methods, from AST attributes to AST macros. This means you can write extensions in the compiler when the boo code compiles.
One of these techniques is what is known in boo speak as a meta-method. A meta-method is a short cut to the compiler that accepts an AST node and returns an AST node.
It is important to comprehend what an AST node is. One of the steps in the boo compiler pipeline is known as the parsing stage which parses the source code stream into an abstract syntax tree. Subtrees or AST nodes will be passed to compiler extension points such as meta-methods or macros during compilation.
Meta-methods must be static and marked with the MetaAttribute for the compiler to recognise them as so. The BooConfigReader contains a meta-method named install that will take the AST node install horn: and transform it into an instance method of the BooConfigReader class. Because boo supports parenthisis-less method invocations we can call the install method as is listed above or it could be called with the parenthis install(horn).
Below is the C# listing of the install meta-method:
[Meta]
public static Expression install(ReferenceExpression expression, BlockExpression action)
{
var installName = new StringLiteralExpression(expression.Name);
return new MethodInvocationExpression(
new ReferenceExpression("GetInstallerMeta"),
installName,
action
);
}
It is important to remember that the meta-method is invoked at compile time. When the compiler sees the call to the install meta-method during compilation of the Dsl script, it does'nt emit the code to call the method at runtime. Instead during compilation the meta-method is executed. Passed to the install meta-method is an AST of the arguments of the method code, the first being the horn part of the install horn: expression. The second being the anonymous block which is everything after the colon and is scoped by indentation. The code below is the anonyomous block.
description "A .NET build and dependency manager"
get_from svn("http://scotaltdotnet.googlecode.com/svn/trunk/")
build_with msbuild, buildfile("src/horn.sln"), FrameworkVersion35
output "Output"
shared_library "."
Let us re-examine the method signature of the install meta-method code:
[Meta]
public static Expression install(ReferenceExpression expression, BlockExpression action)
The horn part of the install horn: expression is passed to the meta-method as a ReferenceExpression type. A ReferenceExpression type can be simply thought of as a token or block of text that is passed to the meta-method as an argument.
The anonymous method block is passed to the meta-method as a BlockExpression type.
Let us further examine the method body of the install meta-method:
var installName = new StringLiteralExpression(expression.Name);
return new MethodInvocationExpression(
new ReferenceExpression("GetInstallerMeta"),
installName,
action
);
I Will reiterate once more that the install meta-method is called at compile time. The meta-method takes an AST node and returns an AST node. In the code above we are creating the result of calling the meta-method that will replace the AST that was passed to it. In the above code, we are creating what is known as a MethodInvocationExpression that is creating a call to an instance method of the BooConfigReader class called GetInstallerName and the arguments that will be passed to it.
To sum up, the compiler replaces the install horn: expression with the following call to an instance of the BooConfigReader class GetInstallerMeta("horn", action);.
The GetInstallerMeta method is listed below for completeness.
public void GetInstallerMeta(string installName, Action installDelegate)
{
InstallName = installName;
installDelegate();
}
The same result can be achieved by defining the same meta-method in boo:
abstract class BooConfigReader(IQuackFu):
callable Action()
[Meta]
static def install(expression as ReferenceExpression, action as Expression):
name = StringLiteralExpression(expression.Name)
return [|
self.GetInstallerMeta($name, $action)
|]
The return statement of the install method introduces a concept that makes boo more suited to writing code for compiler manipulation. This strange expression creation is what is known as quasi quotation.
Quasi quotation is only accessible through boo and you will not be able to recreate this experience in any other .NET language. Writing the code to create the AST node in the C# version of the install method can quickly get tedious. Quasi quotation allows us to produce this AST code in a more concise manner.
When the compiler encounters this code, it does the usual parsing of the code, but instead of outputting the IL instructions that would execute the code, it outputs the code to build the required AST, much in the same way we built the MethodInvocationExpression in the C# example.
The most alluring part of quasi-quotation is that we are not limited to static binding. We can refer to external variables and the compiler will take care of creating the correct AST node. In the example above, we can access the name variable and the action argument in the quasi quotation. The experience is analagous to closures where you can access external variables that are not defined in the inner block.
We can now move swiftly through the rest of the Dsl:
get_from svn("http://scotaltdotnet.googlecode.com/svn/trunk/")This line tells horn which source control management system to retrieve the source from and the uri of the source control expression.
This is painfully easy to handle and we use another meta-method to implement this that I have listed below:
We define another :
[Meta]
static def get_from(expression as MethodInvocationExpression):
return expression
The boo compiler will pass in the svn("http://scotaltdotnet.googlecode.com/svn/trunk/") AST node in the form of a MethodInvocationExpression into the get_from meta-method. We will simply return this expression. If you remember, a meta-method takes an AST node as an argument and returns an AST node that will replace the meta-method in the IL.
The compiler will replace the meta-method with the call to svn("http://scotaltdotnet.googlecode.com/svn/trunk/"). This does of course rely on us having an svn instance method in our anonymouse base class which of course we do. I have listed it below for completeness.
def svn(url as string):
sourceControl = SCM.SVNSourceControl(url)
The last fragment of Dsl script code I want to examine in this post is listed below:
build_with msbuild, buildfile("src/horn.sln"), FrameworkVersion35This line of code instructs horn which build engine we will use, the relative path to the build file and what version of the .NET framework horn will use to compile the source against.
It will come as no great suprise to learn that we have handled this yet again with a meta-method:
[Meta]
def build_with(builder as ReferenceExpression, build as MethodInvocationExpression, frameWorkVersion as ReferenceExpression):
buildFile = build.Arguments[0]
version = StringLiteralExpression(frameWorkVersion.Name)
return [|
$builder($buildFile, $version)
|]
The above code listing shows us how we can start to build more complex expressions using quasi quotation that will be returned from the meta-method.
You can see that we are using the builder argument to dynamically state which instance method to invoke. We are also referencing to external variables.
Below is the same method defined in C#.
[Meta]
public static Expression build_with(ReferenceExpression builder, MethodInvocationExpression build, ReferenceExpression frameWorkVersion)
{
var targetName = builder.Name;
return new MethodInvocationExpression(
new ReferenceExpression(targetName),
build.Arguments[0],
new StringLiteralExpression(frameWorkVersion.Name)
);
}
Both these methods will return a call to an instance method named msbuild which of course must be present for the compilation to succeed.
Below is the msbuild method:
def msbuild(buildFile as string, frameworkVersion):
version = System.Enum.Parse(typeof(Horn.Domain.Framework.FrameworkVersion), frameworkVersion)
BuildEngine = BuildEngines.BuildEngine(Horn.Domain.MSBuildBuildTool(), buildFile, version)
If any of this is of interest to you then please join the Horn user group for updates or check out the source here.
Saturday, 11 April 2009
Parsing the Horn Dsl with the help of a Rhino
In my previous post, I explained the purpose of the horn DSL. In the introduction I mentioned that the open source project named horn that I contribute to is based on the gentoo portage package management system. At the heart of both portage and indeed horn is the metaphor of the package tree. The package tree is conceptually a tree that contains leaves of package build instructions. In reality it is a directory structure that contains Dsl instance files of package build instructions.
In another previous post, I listed the reasons why boo was chosen as the language to host the internal Dsl.
The Dsl contains the build metadata that is required by horn in order to both retrieve an open source project from a remote soure control management system and details of how to build or compile the components.
I mentioned in the previous post that the purpose of the DSL is to create the in-memory domain model or semantic model that horn uses to retrieve and build open source software packages.
Below is the horn.boo file that tells the horn software system how to retrieve and build itself:
We like to think we are dog fooding by providing a build script for horn but others may differ. I am hoping that we did achieve our goal of constructing a semi English like prose of build instructions but I will have to leave that judgement to the interpretation of the reader, if you think it could be improved then please leave a comment.
I mentioned in the previous post that when horn receives a command line instruction such as horn -install:horn that horn will search through the horn package tree or directory structure of build files until it finds the requested package's build file. In this case the DSL script outlined above.
The main engine of horn or the core as it is known is a .NET assembly written in C#. We also have a thin client .NET console application also written in C# that accepts command line instructions before passing them onto the core for execution. The code to retrieve and search the package tree for the requested build file is non-interesting and standard so we will not mention it. Suffice to say that it retrieves the correct build file before parsing it. We will now take time to explain how we go about parsing a boo build file.
I must mention at this point that if I had not read Ayende's excellent book creating domain specific languages in boo then I do not think I would have been able to use boo as our host language of choice for the DSL. This book really is the best reference guide out there and trust me, I have looked.
One thing to bear in mind when writing a DSL in boo when compared to a more typical internal host language choice like python or ruby is that we do not have an interpeter to parse our scripts. The DSL is compiled down to IL before it is executed. Boo has an extensible compiler architecture that allows us to create a more palatable Dsl.
The very basic structure of the Boo compiler is the pipeline; it is how the compiler transforms bits of text in a file into executable code. The boo language allows you, the user to plug your own steps into the compiler pipeline. This allows you to add to the syntax of the language, just like the language designers did in C# 3.0 by adding query expressions. A compiler step can interact with the parsed code and change the abstract syntax tree. New pipelines and steps can be defined at will. The compiler pipeline is a collection of loosely coupled objects each with their own task to complete.
The technique we have chosen for horn is to create what is known as an anonymous base class that is the basis of our Dsl. The anonymous base class is a standard approach for building a Dsl in boo. It is composed of a base class in our application code, and a compiler step in boo that will turn the Dsl script into a class that is derived from the defined base class. It is known as an anonymous base class because there is no mention of the base class in the Dsl script. If you look at the horn.boo dsl script at the top of the page you can think of the keywords install, description, get_from etc. as methods on the base class that we are simply calling in our script.
Below is a code snippet of the basic outline of the BooConfigReader anonymous base class we have defined in the horn application code to act as a base class for our Dsl scripts:
Dsl script instances will inherit from this inherit from this class. Below is the first two lines of the horn.boo Dsl build script:
You can see that we have methods in the base classes that correspond to methods in the Dsl Script. We will explain how this works in the next post.
And below is a code snippet of the same anonymous base class written in boo:
As with before, I will explain this code in more detail in the next post. I want to concentrate on how to actually parse individual dsl scripts in this post.
It is at this point that I will introduce the excellent Rhino.Dsl library. I gave a mention earlier to Ayende who is the creator and probably the main contributor to the Rhino Tools open source project. I use many of these tools in my day job so it came as no suprise to discover that there is a Rhino.Dsl project that contains components that provide labour intensive functionality for building and running DSLs written in boo.
The Rhino.Dsl library provides a number of classes that will take care of a lot of the heavy lifting for the common Dsl tasks that we would otherwise have to code in horn ourselves. Examples of such tasks are parsing the Dsl files and shielding us from the repetitive boiler plate code that is need to add an anonymous base class to the boo compiler pipeline.
Let us examine the following code that uses 2 such Rhino.Dsl classes:
In the above code we are creating an instance of the Rhino.Dsl.DslFactory and telling it to load all the build scripts in a specific folder.
We then have the following line:
The Register method of the DslFactory allows us to register instances of another Rhino.Dsl class named DslEngine or as in the case above with the ConfigReaderEngine, classes that derive from DslEngine. The DslEngine class abstracts away a lot of the boiler plate code that is required to compile instances of Dsl scripts. The DslEngine contains a number of extension points. One of these extension points is a method called CustomizeCompiler. This method allows us to customise the Compiler pipeline. We mentioned earlier that the main mechanics for the horn Dsl was to create an anonymous base class and register it in the boo compiler pipeline. We are passing in the type of anonymous base class as a generic argument in the register method.
As mentioned earlier, the Rhino.Dsl.Engine class provides extension points for boo compiler pipeline extensibility. The point of extension we use in horn is the CustomizeCompiler method. Below is the code listing for ConfigReader class that inherits from DslEngine and we use in horn to add our own compiler modifications and that we registered in the DslFactory in the previous code listing.
The code above is showing how we can add our own custom steps into the compiler pipeline.
The important line of code to observe in the above example is the following:
This takes care of adding our anonymous base class to the pipeline which will mean that every build script we parse will inherit from this class. We use another Rhino class named ImplicitBaseClassCompilerStep that will again hide us from a lot of the boo infrastructure code to register an anonymous base class. The ImplicitBaseClassCompilerStep takes as arguments to it's constructor, the type of anonyomous base class, the abstract method that will be called to invoke the script and the namespace of the anonymous base class.
In horn we register the anonymous base class compiler class and we add another compiler step with the code below:
This allows us to transform the >> bitwise operators in the dependencies section of the Dsl into something else:
We will cover that in a future post.
Below is the code listing for how we get the DslFactory and DslEngine instances to work together to parse the Dsl script into our semantic model:
We call the Create method of the factory passing in the path to the Dsl script and the type of anonymous base as a generic argument. The prepare method is an abstract method on the anonymous base class. When we call this method, the Dsl Script is invoked which will in turn call methods or functionality on the anonymouse base class that is now the base class of the Dsl script.
Thanks to Rhino.Dsl, we can concentrate on authoring our Dsl and let it take care of the boiler plate boo plumbing code.
In the next post we will go line for line through the Dsl Script syntax to see how we achieved the result.
In the next post, we will get describe how the build script syntax was achieved.
If any of this is of interest to you then please join the Horn user group for updates or check out the source here.
In another previous post, I listed the reasons why boo was chosen as the language to host the internal Dsl.
The Dsl contains the build metadata that is required by horn in order to both retrieve an open source project from a remote soure control management system and details of how to build or compile the components.
I mentioned in the previous post that the purpose of the DSL is to create the in-memory domain model or semantic model that horn uses to retrieve and build open source software packages.
Below is the horn.boo file that tells the horn software system how to retrieve and build itself:
install horn:
description "A .NET build and dependency manager"
get_from svn("http://scotaltdotnet.googlecode.com/svn/trunk/")
build_with msbuild, buildfile("src/horn.sln"), FrameworkVersion35
output "Output"
shared_library "."
dependencies:
depend @log4net >> "lib"
depend @castle >> "castle.core"
depend @castle >> "Castle.DynamicProxy2"
depend @castle >> "castle.microKernel"
depend @castle >> "castle.windsor"
We like to think we are dog fooding by providing a build script for horn but others may differ. I am hoping that we did achieve our goal of constructing a semi English like prose of build instructions but I will have to leave that judgement to the interpretation of the reader, if you think it could be improved then please leave a comment.
I mentioned in the previous post that when horn receives a command line instruction such as horn -install:horn that horn will search through the horn package tree or directory structure of build files until it finds the requested package's build file. In this case the DSL script outlined above.
The main engine of horn or the core as it is known is a .NET assembly written in C#. We also have a thin client .NET console application also written in C# that accepts command line instructions before passing them onto the core for execution. The code to retrieve and search the package tree for the requested build file is non-interesting and standard so we will not mention it. Suffice to say that it retrieves the correct build file before parsing it. We will now take time to explain how we go about parsing a boo build file.
I must mention at this point that if I had not read Ayende's excellent book creating domain specific languages in boo then I do not think I would have been able to use boo as our host language of choice for the DSL. This book really is the best reference guide out there and trust me, I have looked.
One thing to bear in mind when writing a DSL in boo when compared to a more typical internal host language choice like python or ruby is that we do not have an interpeter to parse our scripts. The DSL is compiled down to IL before it is executed. Boo has an extensible compiler architecture that allows us to create a more palatable Dsl.
The very basic structure of the Boo compiler is the pipeline; it is how the compiler transforms bits of text in a file into executable code. The boo language allows you, the user to plug your own steps into the compiler pipeline. This allows you to add to the syntax of the language, just like the language designers did in C# 3.0 by adding query expressions. A compiler step can interact with the parsed code and change the abstract syntax tree. New pipelines and steps can be defined at will. The compiler pipeline is a collection of loosely coupled objects each with their own task to complete.
The technique we have chosen for horn is to create what is known as an anonymous base class that is the basis of our Dsl. The anonymous base class is a standard approach for building a Dsl in boo. It is composed of a base class in our application code, and a compiler step in boo that will turn the Dsl script into a class that is derived from the defined base class. It is known as an anonymous base class because there is no mention of the base class in the Dsl script. If you look at the horn.boo dsl script at the top of the page you can think of the keywords install, description, get_from etc. as methods on the base class that we are simply calling in our script.
Below is a code snippet of the basic outline of the BooConfigReader anonymous base class we have defined in the horn application code to act as a base class for our Dsl scripts:
public abstract class BooConfigReader
{
[Meta]
public static Expression install(ReferenceExpression expression, Expression action)
{
var installName = new StringLiteralExpression(expression.Name);
return new MethodInvocationExpression(
new ReferenceExpression("GetInstallerMeta"),
installName,
action
);
}
public void description(string text)
{
Description = text;
}
Dsl script instances will inherit from this inherit from this class. Below is the first two lines of the horn.boo Dsl build script:
install horn:
description "A .NET build and dependency manager"
You can see that we have methods in the base classes that correspond to methods in the Dsl Script. We will explain how this works in the next post.
And below is a code snippet of the same anonymous base class written in boo:
abstract class BooConfigReader(IQuackFu):
callable Action()
[Meta]
static def install(expression as ReferenceExpression, action as Expression):
name = StringLiteralExpression(expression.Name)
return [|
self.GetInstallerMeta($name, $action)
|]
def description(text as string):
desc = text
As with before, I will explain this code in more detail in the next post. I want to concentrate on how to actually parse individual dsl scripts in this post.
It is at this point that I will introduce the excellent Rhino.Dsl library. I gave a mention earlier to Ayende who is the creator and probably the main contributor to the Rhino Tools open source project. I use many of these tools in my day job so it came as no suprise to discover that there is a Rhino.Dsl project that contains components that provide labour intensive functionality for building and running DSLs written in boo.
The Rhino.Dsl library provides a number of classes that will take care of a lot of the heavy lifting for the common Dsl tasks that we would otherwise have to code in horn ourselves. Examples of such tasks are parsing the Dsl files and shielding us from the repetitive boiler plate code that is need to add an anonymous base class to the boo compiler pipeline.
Let us examine the following code that uses 2 such Rhino.Dsl classes:
var factory = new DslFactory
{
BaseDirectory = packageTree.CurrentDirectory.FullName
};
factory.Register<BooConfigReader>(new ConfigReaderEngine());
In the above code we are creating an instance of the Rhino.Dsl.DslFactory and telling it to load all the build scripts in a specific folder.
We then have the following line:
factory.Register<BooConfigReader>(new ConfigReaderEngine());
The Register method of the DslFactory allows us to register instances of another Rhino.Dsl class named DslEngine or as in the case above with the ConfigReaderEngine, classes that derive from DslEngine. The DslEngine class abstracts away a lot of the boiler plate code that is required to compile instances of Dsl scripts. The DslEngine contains a number of extension points. One of these extension points is a method called CustomizeCompiler. This method allows us to customise the Compiler pipeline. We mentioned earlier that the main mechanics for the horn Dsl was to create an anonymous base class and register it in the boo compiler pipeline. We are passing in the type of anonymous base class as a generic argument in the register method.
As mentioned earlier, the Rhino.Dsl.Engine class provides extension points for boo compiler pipeline extensibility. The point of extension we use in horn is the CustomizeCompiler method. Below is the code listing for ConfigReader class that inherits from DslEngine and we use in horn to add our own compiler modifications and that we registered in the DslFactory in the previous code listing.
public class ConfigReaderEngine : DslEngine
{
protected override void CustomizeCompiler(BooCompiler compiler, CompilerPipeline pipeline, string[] urls)
{
pipeline.Insert(1, new ImplicitBaseClassCompilerStep(typeof(BooConfigReader), "Prepare", "Horn.Core.Dsl"));
pipeline.InsertBefore(typeof(ProcessMethodBodiesWithDuckTyping), new RightShiftToMethodCompilerStep());
pipeline.Insert(2, new UnderscorNamingConventionsToPascalCaseCompilerStep());
pipeline.Insert(3, new UseSymbolsStep());
}
}
The code above is showing how we can add our own custom steps into the compiler pipeline.
The important line of code to observe in the above example is the following:
pipeline.Insert(1, new ImplicitBaseClassCompilerStep(typeof(BooConfigReader), "Prepare", "Horn.Core.Dsl"));
This takes care of adding our anonymous base class to the pipeline which will mean that every build script we parse will inherit from this class. We use another Rhino class named ImplicitBaseClassCompilerStep that will again hide us from a lot of the boo infrastructure code to register an anonymous base class. The ImplicitBaseClassCompilerStep takes as arguments to it's constructor, the type of anonyomous base class, the abstract method that will be called to invoke the script and the namespace of the anonymous base class.
In horn we register the anonymous base class compiler class and we add another compiler step with the code below:
pipeline.InsertBefore(typeof(ProcessMethodBodiesWithDuckTyping), new RightShiftToMethodCompilerStep());
This allows us to transform the >> bitwise operators in the dependencies section of the Dsl into something else:
dependencies:
depend @log4net >> "lib"
We will cover that in a future post.
Below is the code listing for how we get the DslFactory and DslEngine instances to work together to parse the Dsl script into our semantic model:
configReader = factory.Create<BooConfigReader>(buildFilePath);
configReader.Prepare();
We call the Create method of the factory passing in the path to the Dsl script and the type of anonymous base as a generic argument. The prepare method is an abstract method on the anonymous base class. When we call this method, the Dsl Script is invoked which will in turn call methods or functionality on the anonymouse base class that is now the base class of the Dsl script.
Thanks to Rhino.Dsl, we can concentrate on authoring our Dsl and let it take care of the boiler plate boo plumbing code.
In the next post we will go line for line through the Dsl Script syntax to see how we achieved the result.
In the next post, we will get describe how the build script syntax was achieved.
If any of this is of interest to you then please join the Horn user group for updates or check out the source here.
The Purpose of the Horn Dsl
In my previous post, I stated that we had chosen boo as the host language of our internal Dsl.
What I omitted to mention was the purpose of the Horn Dsl. I now run the risk of losing whatever interest that the readers (if there are indeed any) may have by still not mentioning the Dsl but instead mentioning what purpose it is supposed to serve. I feel I must further waiver from the technical details in order to give proper context to the Dsl's reason to exist.
As I mentioned in the introduction, horn is based on the gentoo portage package manager. At the heart of both portage and indeed horn is the metaphor of the package tree. The package tree is conceptually a tree that contains leaves of package build instructions. In reality it is a directory structure that contains Dsl instance files of package build instructions. Below is how the horn package tree looks at this time of writing.
If you examine the image above, you can see that we have one top level node that is rather unimaginatively named package_tree. Below the root package_tree node are child nodes that represent package categories like ORM's, IOC's, loggers etc. The rational being that the we can use these next level nodes as search criteria to list all the object relational mappers for example that horn could install. Contained in each of the category nodes are the individual package nodes like Nhiberate. In the example above you can see that Nhibernate is rightly positioned under a category parent node named orm. Contained within the indiviual Nhiberante package node is one build file named nhibernate.boo which contains the build metadata needed to install not only Nhibernate but any dependencies that are specified in the build file. The observant amongst you will notice that the package tree is persisted and retrieved from a subversion repository.
The package tree can be thought of a database of packages and their build instructions. When a user enters a command into the command prompt like the one below:
the horn software system will search through the package tree to try and find a node named the same as the value of the -install switch which in this case is Nhibernate. Upon finding the correct node, horn will then parse the contents of the build file which in this case is nhbiernate.boo into an semantic model or domain model that will be persisted in memory for the duration of the installation and more importantly contain all the information required to build and install the requested package. Martin Fowler called the domain model in this context the semantic model. In this article he states that from the domain model's point of view the DSL is just a fancy alternative way of creating it's objects and hooking them together.
The Dsl describes a BuildMetaData object that contains a SourceControl object and a BuildEngine object.
Typically, the source code for a particular package will be retrieved from a source control management system. This has been modelled as a SourceControl abstract class with particular implementations derived with their own specific implementation details. We currently have support for subversion only.
Once we have exported the source code to the client's file system, the next logical
step in the workflow is to build the source code. Horn has the concept of a BuildEngine object which is charged with building the source code. The Dsl has as part of it's definition a directive that tells horn which BuildTool to build the source from. Currently we have 3 BuildTool implementations, Nant, msbuild and rake. In most cases, the build of an open source system is complex and contrived. Typically the source code will contain a build script which orchestrates this complicated build process. We need to utilise these existing build scripts which in the .NET space are more times than not defined in Nant. The MSBuild build tool will generally work with single components.
Once we have constructed this semantic model filled with the values in the DSL then horn can build and install the requested package.
In the next article I will finally get to the boo DSL. I will explain how horn loads specific instances of the Dsl into memory and parses it's contents into the semantic model.
If any of this is of interest to you then please join the Horn user group for updates or check out the source here.
What I omitted to mention was the purpose of the Horn Dsl. I now run the risk of losing whatever interest that the readers (if there are indeed any) may have by still not mentioning the Dsl but instead mentioning what purpose it is supposed to serve. I feel I must further waiver from the technical details in order to give proper context to the Dsl's reason to exist.
As I mentioned in the introduction, horn is based on the gentoo portage package manager. At the heart of both portage and indeed horn is the metaphor of the package tree. The package tree is conceptually a tree that contains leaves of package build instructions. In reality it is a directory structure that contains Dsl instance files of package build instructions. Below is how the horn package tree looks at this time of writing.
If you examine the image above, you can see that we have one top level node that is rather unimaginatively named package_tree. Below the root package_tree node are child nodes that represent package categories like ORM's, IOC's, loggers etc. The rational being that the we can use these next level nodes as search criteria to list all the object relational mappers for example that horn could install. Contained in each of the category nodes are the individual package nodes like Nhiberate. In the example above you can see that Nhibernate is rightly positioned under a category parent node named orm. Contained within the indiviual Nhiberante package node is one build file named nhibernate.boo which contains the build metadata needed to install not only Nhibernate but any dependencies that are specified in the build file. The observant amongst you will notice that the package tree is persisted and retrieved from a subversion repository.
The package tree can be thought of a database of packages and their build instructions. When a user enters a command into the command prompt like the one below:
the horn software system will search through the package tree to try and find a node named the same as the value of the -install switch which in this case is Nhibernate. Upon finding the correct node, horn will then parse the contents of the build file which in this case is nhbiernate.boo into an semantic model or domain model that will be persisted in memory for the duration of the installation and more importantly contain all the information required to build and install the requested package. Martin Fowler called the domain model in this context the semantic model. In this article he states that from the domain model's point of view the DSL is just a fancy alternative way of creating it's objects and hooking them together.
The Dsl describes a BuildMetaData object that contains a SourceControl object and a BuildEngine object.
Typically, the source code for a particular package will be retrieved from a source control management system. This has been modelled as a SourceControl abstract class with particular implementations derived with their own specific implementation details. We currently have support for subversion only.Once we have exported the source code to the client's file system, the next logical
step in the workflow is to build the source code. Horn has the concept of a BuildEngine object which is charged with building the source code. The Dsl has as part of it's definition a directive that tells horn which BuildTool to build the source from. Currently we have 3 BuildTool implementations, Nant, msbuild and rake. In most cases, the build of an open source system is complex and contrived. Typically the source code will contain a build script which orchestrates this complicated build process. We need to utilise these existing build scripts which in the .NET space are more times than not defined in Nant. The MSBuild build tool will generally work with single components.Once we have constructed this semantic model filled with the values in the DSL then horn can build and install the requested package.
In the next article I will finally get to the boo DSL. I will explain how horn loads specific instances of the Dsl into memory and parses it's contents into the semantic model.
If any of this is of interest to you then please join the Horn user group for updates or check out the source here.
Thursday, 9 April 2009
The Decision to use Boo as the host language for the Horn internal DSL
In my previous post I introduced the horn package manager open source project that I have been contributing to and the fact that I am blogging these posts to help me prepare for my talk at the Dsl DevCon conference.
In my previous post, I mentioned that we would be basing the the horn package manager on the excellent gentoo portage package management system. At the heart of the portage system is the concept of ebuilds that contain information that allow the portage system to both compile and install software packages.
Our first step in the evolution of the horn was to author the Dsl of build and metadata instructions that we would model on portage's ebuild scripts.
I suppose the first question that a would be DSL author will face is, do I create an internal Dsl or an external Dsl? At the time of horn's inception, oslo had not been previewed and not wanting to get downd and dirty with something like yak or antlr an internal Dsl was the right choice. Even if oslo had been available, I probably would have gone with an internal Dsl anyway. I don't really get a chance to mess around with alternative languages in my day job so this was a chance to do some fun experimentation.
I have been reading Ayende's blog for a few years now and it was through this blog that I first heard the term ALT.NET. The blog alone has been a great source of new and interesting practices that I would have otherwise been oblivious to. Ayende is also the driving force behind the excellent Rhino tools. There are a multitude of excellent bits and pieces here that can help with common less granular programming tasks to full blown ETL transformations. Even getting familiar with the code base has been been enlightening and educational. I use parts of the excellent Rhino stack and it is the pain of upgrading such open source pacakages as Rhino that horn is trying to alleviate.
Rhino has a number of DSLs that are used to help give a higher level of abstraction to certain tasks. There is Binsor which is a DSL for initialising the Castle Windsor IoC container and Rhino.ETL which uses a DSL syntax to express transformations in an ETL workflow. In both cases the DSLs are internal DSLs with boo used as the host language. I have experience of using both these projects so the choice of our internal Dsl was an obvious one. I had also recently read the excellent Building Domain Specific Languages in Boo on a recent long haul flight from the UK to Vietnam. I will be paraphrasing from the aforementioned book, so it is only fair I mention both the tome and the author.
Boo is an object oriented language for the Common Language Infrastructure with a python inspired syntax and a special focus on language and compiler extensibility.
When I fist chose boo as the host language of our Dsl, I was not convinced of ironruby's maturity. Later, when I show how you can implement the same Dsl using ironruby, you will see this was wrong. I still think that boo offers more with respect to DSLs but this could be down to my limited experience of both languages.
I must take this time to both introduce boo to those in the audience who are unfamiliar with it and to speak gushingly about it's elegance. I see Boo as a slightly schizophrenic CLR language. I describe it this way because boo is a static language with many qualities that are more associated with dynamic languages.
Boo has the following characteristics that make it an ideal language for hosting DSLs:
- Significant Whitespace - Boo uses a colon to initiate a control block and the level of indentation to indicate the scope. When I came to experiment with rewriting the DSL in ironruby as we will see later, I found the do and end control block markers almost impalatable after working with boo.
- Type Inference and automatic type casting - You can liken this to C#'s var keyword but of course without the var.
- Duck Typing - You can by implementing the excellently named IQuackFu interface ask the compiler to relax the strong typing constraints and provide an experience that is very similar to constructs like method_missing in Ruby.
- Compiler Extensibility
As I will show later, boo has some stunning compiler extensibility points that set it truly apart from other languages. Meta methods and macro expansions are 2 techniques that can get you up and running very quickly with your own custom DSL.
In my next post, we will briefly list the purpose of the horn Dsl.
If any of this is of interest to you then please join the Horn user group for updates or check out the source here.
The Horn Package Manager - Introduction (The developer's cut).
I am writing a series of posts to help me prepare for my talk at Dsl DevCon.
I will be talking about the Dsl that is used to describe package definitions for the horn package manager open source project I have been contributing for.
You can read a series of introductions to the horn package manager here, here and here.
As is customary, I will briefly introduce myself. Compared to most of the speakers at the conference, I can best be classified as the average developer in the field. If any of you are familiar with the "hitch hiker's guide to the galaxy", then I am the developer who would get evaporated by the total perspective vortex if he actually seen himself positioned in his rightful place in the developer food chain. I have a keen interest in the ALT.NET space. I both agree and disagree with many of the exponents of ALT.NET in equal measures. I do find it a vibrant and fun place to share new ideas. Naturally such topics as the merits of DSLs are debated from time to time in this space.
In my day job, I just do not work on the kind of complex application that has deep, hard to fathom business rules that a DSL would help to abstract away the logic. The domain that empowers the development of my current software line is just not rich enough to be expressed in the gushing terms of Eric Evan's famous Domain Driven Design book. Indeed the time spent to create a semi English prose that would help unearth new models in our problem space is just not cost effective. Having just started my own business, the time spent on such a flight of fancy like this would hurt me where it hurts most, in the pocket. That said I am a passionate developer who not only codes for a living but codes in his spare time. More recently the focus of my hobbyist coding exploits has been to help start and contribute to an open source project that hopes to cure a genuine lacking in the .NET space. I can say without fear of recrimination that I have been the main contributor on this project.
As with most open source packages, the goal is to alleviate a genuine point of pain that clogs up a developers progress in his pursuit of delivering the software system. In the .NET space there is a persistent and genuine point of pain that it seems remarkable that nobody has tackled adequately yet.
The horn package manager as we have lovingly christened it sets out to ease the pain of building open source packages or indeed any project and it's dependencies. We are trying to plug a point of extreme pain that has been so effectively plugged in ruby by rubygems, by maven in Java and appget. Our initial goal is to issue an order at the command prompt and watch as our glorious software system takes charge of retrieving the source code of a project like Nhibernate from a source control management system and builds the binary packages as we watch the text of the output stream slide elegantly down the command prompt window. We want to not only build the source of the package but also any dependencies that the package is more than likely to have. Horn predominantly and initially wants to build packages from source code, we believe this will give us the maximum in flexibility and is the only way we can truly cure the pain of upgrading our open source packages. Allow me to give further context to this very real pain that I and countless others experience.
Our target audience is that of developers who like to live on the latest code version "trunk" or as we like to romantically think of it, the bleeding edge. The bleeding edge in this context is what is known as "working off the trunk". My current stack of open source packages that I use to help me with my daily software development tasks include such open source parts as:There is a deep and contrived dependency map between all these packages. A recent happening or event will better explain the treadmill of change that we face as developers if one of these loose house of cards decides to flip or upgrade itself.
If you are familiar with the .NET space then you will surely have noticed the release of the ASP.NET MVC framework. This release caused a lot of ALT.NET developers to quickly grab their svn client of choice and start rebuilding their software stack. Why? Because the upgrading of any dependency causes a domino effect of upgrades. Building this stack is a time consuming irritating, and error prone MANUAL process. Here is a brief synopsis of the build process.
When starting out in such a daunting project such as this, we made the very educated step of copying something existing. Package management in non compiled platforms such as ruby where you are dealing with only code files led us to not follow the excellent ruby gems. Other candidates for blatant plagarism such as maven seemed more to do with binary management than source code binary construction. One of the initial horn developers who namely was not me, suggested that we take a look at the gentoo portage package management system. Portage instantly seemed very interesting and resonated with our ways of thinking.
At the heart of portage is the metaphor of the portage tree. The portage tree can be thought of as a directory tree hive that contains categorised indiviual files of build instructions that are called ebuilds. An ebuild is a specialised bash script that automates the compilation and installation procedures of a software package. A user of portage will instruct portage via the command line as to which software package is required. Portage will then search the directory hive or portage tree for the file that contains the metadata or in portage speak, the ebuild that contains the instructions to get, compile and install the requested software package.
With such a fantastic and intriguing model to copy as portage, we were in business. We would replicate this model package tree database of build scripts in a .NET style. The first step was obviously to define our version of the ebuild. Probably the most obvious choice was Xml but as anyone who is as cool or as trendy as an ALT.NET developer could not take the social stigma off using such an archaic metadata format. In all seriousness having recently migragted my Nant build scripts to ruby's rake, the choice was an obvious one. Programming logic is ill placed in Xml, I think even the creator of Ant has later said that Xml was not a great choice for an otherwise excellent build language.
In my next post I will describe the decision to choose boo as the language to host our internal DSL.
If any of this is of interest to you then please join the Horn user group for updates or check out the source here.
I will be talking about the Dsl that is used to describe package definitions for the horn package manager open source project I have been contributing for.
You can read a series of introductions to the horn package manager here, here and here.
As is customary, I will briefly introduce myself. Compared to most of the speakers at the conference, I can best be classified as the average developer in the field. If any of you are familiar with the "hitch hiker's guide to the galaxy", then I am the developer who would get evaporated by the total perspective vortex if he actually seen himself positioned in his rightful place in the developer food chain. I have a keen interest in the ALT.NET space. I both agree and disagree with many of the exponents of ALT.NET in equal measures. I do find it a vibrant and fun place to share new ideas. Naturally such topics as the merits of DSLs are debated from time to time in this space.
In my day job, I just do not work on the kind of complex application that has deep, hard to fathom business rules that a DSL would help to abstract away the logic. The domain that empowers the development of my current software line is just not rich enough to be expressed in the gushing terms of Eric Evan's famous Domain Driven Design book. Indeed the time spent to create a semi English prose that would help unearth new models in our problem space is just not cost effective. Having just started my own business, the time spent on such a flight of fancy like this would hurt me where it hurts most, in the pocket. That said I am a passionate developer who not only codes for a living but codes in his spare time. More recently the focus of my hobbyist coding exploits has been to help start and contribute to an open source project that hopes to cure a genuine lacking in the .NET space. I can say without fear of recrimination that I have been the main contributor on this project.
As with most open source packages, the goal is to alleviate a genuine point of pain that clogs up a developers progress in his pursuit of delivering the software system. In the .NET space there is a persistent and genuine point of pain that it seems remarkable that nobody has tackled adequately yet.
The horn package manager as we have lovingly christened it sets out to ease the pain of building open source packages or indeed any project and it's dependencies. We are trying to plug a point of extreme pain that has been so effectively plugged in ruby by rubygems, by maven in Java and appget. Our initial goal is to issue an order at the command prompt and watch as our glorious software system takes charge of retrieving the source code of a project like Nhibernate from a source control management system and builds the binary packages as we watch the text of the output stream slide elegantly down the command prompt window. We want to not only build the source of the package but also any dependencies that the package is more than likely to have. Horn predominantly and initially wants to build packages from source code, we believe this will give us the maximum in flexibility and is the only way we can truly cure the pain of upgrading our open source packages. Allow me to give further context to this very real pain that I and countless others experience.
Our target audience is that of developers who like to live on the latest code version "trunk" or as we like to romantically think of it, the bleeding edge. The bleeding edge in this context is what is known as "working off the trunk". My current stack of open source packages that I use to help me with my daily software development tasks include such open source parts as:There is a deep and contrived dependency map between all these packages. A recent happening or event will better explain the treadmill of change that we face as developers if one of these loose house of cards decides to flip or upgrade itself.
If you are familiar with the .NET space then you will surely have noticed the release of the ASP.NET MVC framework. This release caused a lot of ALT.NET developers to quickly grab their svn client of choice and start rebuilding their software stack. Why? Because the upgrading of any dependency causes a domino effect of upgrades. Building this stack is a time consuming irritating, and error prone MANUAL process. Here is a brief synopsis of the build process.
- MVC Contrib has a dependency on the newly released ASP.NET MVC faramework. In order to use the newly released MVC framework, it will need to be rebuilt against the release candidate binaries.
- MVC Contrib has a WindsorControllerBuilder piece that gives easy IoC integration with the MVC framework. This is gives us a dependency with the Castle stack.
- I think I am correct in saying that MVC Contrib is built against the castle trunk version. You will need to rebuild against the Castle trunk verson. This will trigger other rebuilds from other open source projects
- Rhino has a dependency on the Castle stack.
- Nhibernate has a dependency on the castle stack.
- etc., etc.
- The treadmill of change has been triggered.
When starting out in such a daunting project such as this, we made the very educated step of copying something existing. Package management in non compiled platforms such as ruby where you are dealing with only code files led us to not follow the excellent ruby gems. Other candidates for blatant plagarism such as maven seemed more to do with binary management than source code binary construction. One of the initial horn developers who namely was not me, suggested that we take a look at the gentoo portage package management system. Portage instantly seemed very interesting and resonated with our ways of thinking.
At the heart of portage is the metaphor of the portage tree. The portage tree can be thought of as a directory tree hive that contains categorised indiviual files of build instructions that are called ebuilds. An ebuild is a specialised bash script that automates the compilation and installation procedures of a software package. A user of portage will instruct portage via the command line as to which software package is required. Portage will then search the directory hive or portage tree for the file that contains the metadata or in portage speak, the ebuild that contains the instructions to get, compile and install the requested software package.
With such a fantastic and intriguing model to copy as portage, we were in business. We would replicate this model package tree database of build scripts in a .NET style. The first step was obviously to define our version of the ebuild. Probably the most obvious choice was Xml but as anyone who is as cool or as trendy as an ALT.NET developer could not take the social stigma off using such an archaic metadata format. In all seriousness having recently migragted my Nant build scripts to ruby's rake, the choice was an obvious one. Programming logic is ill placed in Xml, I think even the creator of Ant has later said that Xml was not a great choice for an otherwise excellent build language.
In my next post I will describe the decision to choose boo as the language to host our internal DSL.
If any of this is of interest to you then please join the Horn user group for updates or check out the source here.
Subscribe to:
Posts (Atom)
Posts
