Dividing The Castle

For those unfamiliar with horn which I am guessing is pretty much everyone, horn is an effort to try and take the pain out of both getting and building open source projects. This also includes resolving the sometimes complex maze of dependencies that exists between projects.

In my previous post, I promised an update on the status of horn. I need to stay focused and have my eye on the prize this time round.

While developing horn recently we reached quite a puzzling impasse with the interesting dependency map that exists between Nhibernate and castle. The problem is that Nhibernate references castle and castle references Nhibernate.

When upgrading my OSS stack, these are the sequence of steps I perform in order to build this troublesome dependency map:

• Build Castle.DynamicProxy2 and copy the output to the Nhibernate bin directory


• Build Nhibernate and copy to the castle bin directory


• Build castle

The problem we have programmatically with this dilemma is that it is a circular reference. If we modeled this in the Dsl, we would have an infinite loop resulting in a stackoverflow.
note: Here is an overveiw of the horn DSL..

A long debate ensued on the Horn user group and we eventually reached a consensus that we had to somehow split the castle code base up in such a way that we could build individual parts of the castle source and thus avoid the circular reference. The castle code base is quite a beast as anyone who has ever performed an svn update or check out will testify, there are a lot of interwoven, inter-project dependencies between this behemoth of a solution. We needed someway of building the castle.DynamicProxy2.dll and any dependencies it might have on the code base. The default.build nant file that builds the castle source code was written to build all projects and has no hooks for building sub-projects. We would need some underhand tactics and some more Dsl trickery to somehow change the castle nant build script with changes of our own.

Here is the Dsl descriptor which houses the instructions that is needed to build the newly birthed castle.tools package that will produce the Castle.DynamicProxy2.dll as part of its output.

1install castle.tools:
2    description "Dynamic Proxy Generator for the CLR."
3
4    prebuild:
5        cmd "xcopy /s /y \"../Patch\" ."
6
7    include:
8         repository(castle, part("Tools"), to("Tools"))
9         repository(castle, part("Core"), to("Core"))
10        repository(castle, part("common.xml"), to("common.xml"))
11        repository(castle, part("common-project.xml"), to("common-project.xml"))
12        repository(castle, part("CastleKey.snk"), to("CastleKey.snk"))
13
14    build_with nant, buildfile("Tools/Tools.build"), FrameworkVersion35
15
16    switches:
17        parameters "sign=true","common.testrunner.enabled=false", "common.silverlight=false"
18
19    shared_library "SharedLibs/net/2.0"
20    output "build/net-3.5/debug"
21
22 package.homepage = "http://www.castleproject.org/"
23 package.forum = "http://groups.google.co.uk/group/castle-project-users?hl=en"

The first point of interest in this Dsl instance is at lines line 4 and 5

4    prebuild:
5        cmd "xcopy /s /y \"../Patch\" ."

A prebuild block was authored that gets parsed into the horn metamodel. Put simply, the code will attempt to execute any DOS commands that are placed in the prebuild block. This is a very temporary measure as the scope for danger is big if we let users enter their own DOS commands here. We will replace them with actual Dsl instructions at a later refactoring and when we know exactly what to create.

The intention of the xcopy is to patch the castle nant files with modifications we made that allow us to build this fragment of the castle code. We are acutally quite crudelly copying modified copies of the nant files over the freshly retieved real castle files. Crude as it is, it really was the only way bar petitioning the good people of castle to change their code or allow us to supply patches. This was quicker and more polite.

I then went leftfield and made a refactoring that is described in the include block of code:

7    include:
8         repository(castle, part("Tools"), to("Tools"))
9         repository(castle, part("Core"), to("Core"))
10        repository(castle, part("common.xml"), to("common.xml"))
11        repository(castle, part("common-project.xml"), to("common-project.xml"))
12        repository(castle, part("CastleKey.snk"), to("CastleKey.snk"))

The include block contains instructions of what parts of which project we want to build. We can retrieve directories or files from what has been christened a repository.

A repository in the context of horn is a node in the horn package tree that can be thought of as the whole of the code base. In this example it is the entirity of the castle trunk. All the parts are retrieved from the castle package tree node. The castle package tree node or repository will synchronise itself with the remote svn repository if the code is out of date. This opens up the opportunity of pulling together different parts of different repositories into more interesting and user defined builds.

The moral of this story is that although we were at first hit with a very difficult problem, the solution ended up opening up many opportunities for horn. I was quickly able to add a further build descriptor for castle.windsor.

The net result is that we are able to build Nhibernate and the relevant parts of castle. Next up, I want to tackle the infamous Rhino stack. Manually building Rhino is one of the reasons I was keen to tackle horn. I have had more problems building this stack than any other, much as I love the end result. This is a fight I intend to win by hook or by crook.

On a selfish note, I really like the way horn Dsl has evolved but then, I would.

The funny thing is, just after completing this work, the castle developers started debating splitting the castle code base as is discussed in this thread.

If anybody from castle is reading this thread then they should consider using horn as a means of pulling their split up stack together and building.

If any of this is of interest to you then please join the Horn user group for updates or check out the source here.

Open Source Adultery and Moral Dilemmas

I have been meaning to blog about the status of horn for some time now but have not got round to it for one reason or another.

For those unfamiliar with horn which I am guessing is pretty much everyone, horn is an effort to try and take the pain out of both getting and building open source projects. This also includes resolving the sometimes complex maze of dependencies that exists between projects.

You can read an introduction about the project here or here is a video of a talk I gave about horn at the Dsl DevCon conference.

As with anyone who contributes to OSS, time is the most precious and scarce commodity. A spare couple of hours here and a spare couple of hours there is all I have to offer. If you are engaged in an active relationship like I am then it is very much like committing adultery. You make pathetic excuses in order to steal a moment with your forbidden lust. I have lied through my teeth for a romantic interlude with my mistress. I have been writing mythical invoices or working overtime just to try and get some small feature done.

It is a very strange stop/start style of development and I do worry that the code base is suffering from this staccato style of development. I have tried to keep the code base as clean as is possible and there is currently over seventy percent test coverage on the main Horn.Core project. But is it the best code I have ever written? Definitely not but it never was going to be with this broken up development cycle.

This does bring up an interesting moral dilemma. Recently I spent some of these stolen moments refactoring the code and I am now questioning whether this was a good idea. Surely at this stage in the life cycle of such a fledgling OSS project like horn it is more important to just get it out there and get people using it? Getting interest and other contributors is the goal of any new OSS project and there must be many in my position struggling to push things forward on a limited budget of time. As I am a child of Belfast, I have recently been drawn to the idea of using some slightly more controversial and overt tactics to recruit committers. If a gang of baseball bat wielding paddies adorned in balaclavas break into your house one night shouting in strong Belfast accents "are you going to commit? Are ye?", I strongly advise you think long and hard before answering. We are masters of making you an offer you cannot refuse. I always wondered why so many sports shops in Belfast sold baseball bats but you never ever seen anybody playing baseball. Very strange indeed.

The thing is though, I believe horn's cause is just and right and it is amazing that nobody has set out to solve this difficult problem. In .NET we seem to be all hands to the pump when it comes to creating YIOC yet another inversion of control container, YATR yet another test runner or YAMF yet another mocking framework. But we really need synergy with other platforms by providing a .NET rubygems or a .NET appget. I did hear some rumblings of a rocks project or an ngems but that is all they were rumblings. They seemed to want to use binary dependencies which I don't think will prove very effective in .NET. You need the ability to build dependencies from the source code that borne them.

I have spent too many days knee deep in subversion and nant upgrading my whole stack just to apply a bug fix or acquire some new feature. This pain is very much a harsh reality for many who employ the use of tooling forged from open sourcery. If it is not horn then I hope it is something else. In fact if something else solves this then great, I can move onto some other ideas I have that might actually make me some money and are more selfishly driven.

I started this post with the intention of giving a horn update but have failed and got lost in my ramblings.

In my next post I will give that update and explain how we are able to split castle up with a bit of injenuity and brute force.

If any of this is of interest to you then please join the Horn user group for updates or check out the source here.

The Horn DSL Redux - Part II - Mixins and Blocks

In my last post, I described how I spiked a brief proof of concept that gave me a degree of confidence that I could rewrite the horn Dsl in ironruby.

At the time of writing, the horn Dsl was quite basic and looked like this:

    1 install horn:
    3        description "A .NET build and dependency manager"
    4        build_with msbuild, buildfile("src/horn.sln"), FrameworkVersion35
    5        shared_library "."
    6 
    7 dependencies:
    8         @log4net >> @lib

My mission was simple, take the above prose and translate it line for line into Ruby code.

My first bite of the cherry was to define the install method written in boo below:

    1 install horn:
    2        description "A .NET build and dependency manager"

In the boo version of the Dsl the install method gets added to an anonymouse base class via some quite remarkable compiler trickery that you can read about here. One thing we do not have at our disposable in Ruby or more importantly ironruby is a compiler. I would have to rely on the language constructs of the internal Dsl in order to create a fluent human readable syntax.

One way of parsing the install construct would have been to simply define the install method in a seperate code file. In Ruby any method that you define outside of a class or a module is appended onto the object base class that all ruby objects inherit from. This felt dirty and unsatisfactory. I digged deeper into the semantics of Ruby.

A better way soon appeared. I would define the install method in a Ruby module and append this method onto the object base class using a concept known as mixins.

A mixin is Ruby's way of adding the benefits of multiple-inheritance while sticking to a strict object hierarchy. Mixing in is the practice of grouping a set of methods and variables together into a module and inserting those modules as needed into a class. The class incorporates the module into its own capabilities and remembers that the module has been mixed in. Unlike inheritance, mixing in is always a dynamic process. It is not bound to the class definition itself.

A module in Ruby is a unit of code organization that is similar to a namespace in C#. Modules help you organize your code into a hierarchial namespace so that you can group logical bits of coding together.

I have to admit that I copied this idea from the RSpec source but plagarism is the highest compliment. I created the following module structure that I would use to mix in my install method into the object base class.

    1 module MetaBuilder
    2   module Dsl
    3     module Main
    4 
    5       def install(name, &block)       
    6         yield self if block_given?
    7       end
    8 
    9     end
   10   end
   11 end
   12 
   13 include MetaBuilder::Dsl::Main

I will explain the install method in more detail later. The important things to note are:

• There is an install method on line 5 that is contained in a nested module structure. This method is now available to mix in to other classes.


• The Ruby include statement on line 13 pulls all the module's regular methods into the class as instance methods.

With this in place I could start to create my first ironruby build script that would contain build instructions that horn can translate into build instructions for a particular component. Below is the start of an DSL instance script written in Ruby that will tell the horn runtime how to build horn itself.

    1 require 'hornbuild'
    2 
    3 install :horn do
    4   description "A .NET build and dependency manager"
    5 end

The require statement on line 1 is Ruby's reusable code mechanism that is used to load another code file and import all the class and method definitions and also more importantly any execution statements like the include statement that mixes in the include method. The hornbuild.rb file contains the module definition and install method outlined above.

Let us examine our meagre DSL thus far:

    1 require 'hornbuild'
    2 
    3 install :horn do
    4   description "A .NET build and dependency manager"
    5 end

Here we are calling the install method and passing in as arguments two important Ruby constructs that make Ruby a good candidate for an internal Dsl. The first argument is the :horn part which is what is known as a symbol in Ruby. A symbol is identified by the leading colon. With a symbol we've got a way to create a name that we can use in our Dsl and pass around, and it isn't a String. They are most commonly used in hashes which we will explain in a future post. For now think of it like a string constant that can help with readability in an internal Dsl.

The second argument we pass to the install method is what is known as a block in Ruby. A block in Ruby is a closure. The block that is passed to the install method is everything after the do keyword and everything before the end keyword. It is an anonymous code block that is analgous to anonymous delegates and lamdas in C#.

Now let us revisit the install method that will process the arguments that are passed to it:

    9 def install(name, &block)       
   10     yield self if block_given?
   11 end

The name parameter will accept the symbol and the second parameter will accept the block. Blocks are always passed at the end of a method call and are declared implicitly with an ampesand in front of it.

All that is left to do is explain the yield statement on line 10. The yield statement is analogous to the C# yield statement. Basically the yield statement is provided as a way to ease the creation of iterators. It gives control to a user-specified block from within a method's body. Here we are passing control back to the block which is everything after the do keyword and before the end keyword.

I had the beginnings of my ironruby Dsl now. I could progress with a greater confidence.

If any of this is of interest to you then please join the Horn user group for updates or check out the source here.

The Horn DSL Redux - Part I - The Spike

A couple of months ago while preparing for my talk at Dsl DevCon I was hit by a crushing fear or possibly a rush of reality. After seeing my name amongst the list of notable speakers I panicked. I felt I needed something pretty damn good to legitamise my place amongst such notable Dsl intelligencia.

After much soul searching, I decided to rewrite the horn Dsl in ironruby. I would give two renditions of the same story, one in boo and the other in ironruby. Like many, I have been swayed by the gushing reports of increased productivity and superlative elegance that many proclaim to have achieved through ruby. I had recently upgraded my Nant scripts to the ruby make equivalent rake. This also seemed like quite a fun thing to do.

I am not going to give an introduction to ironruby as much has been written on the subject. Suffice to say that this is Microsoft's ruby implementation. At the time of writing I was unsure of both the stability and the maturity of ironruby. The only way to gain some confience was to spike some initial requirements to ensure that this was worth my valuable time. The goal of the ironruby Dsl is exactly the same as the horn boo dsl which is namely to parse the Dsl into the horn metamodel. A previous posting outlines the objectives of the Dsl.

My first spike was to see whether I could parse an ironruby script from C# code that created a CLR object and returned it to the calling code.

Unfortunately the code I am about to narrate is a couple of months old and I have forgotten the vast quantities of profanity that went into making the following unit test pass.

First up, let us look at the very uncontroversial ironruby code that will return a CLR type. You will just have to take my word for it that I wrote this as the good lord intended, namely test first.

    1 class MetaDataFactory
    2     def return_meta_data()
    3         meta = Horn::Core::Dsl::BuildMetaData.new
    4         meta.Description = "A description of sorts"
    5         meta
    6     end
    7 end

The only line of any interest here is line 3 that creates an instance of the BuildMetaData class that represents the horn DSL. Line 5 illustrates the return symantics of ruby functions, namely without a return statement.

Below is the passing test that I finally ended up with:

   10 public class When_creating_a_clr_object_in_ironruby : Specification
   11 {
   12     private string buildFile;
   13 
   14     protected override void Because()
   15     {
   16         buildFile = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "rubyspike.rb");
   17     }
   18 
   19     [Fact]
   20     public void Then_the_object_should_be_accessible_in_csharp()
   21     {                      
   22         var engine = Ruby.CreateEngine();
   23 
   24         engine.Runtime.LoadAssembly(typeof (BuildMetaData).Assembly);
   25 
   26         engine.ExecuteFile(buildFile);
   27 
   28         var klass = engine.Runtime.Globals.GetVariable("MetaDataFactory");
   29 
   30         var instance = (RubyObject)engine.Operations.CreateInstance(klass);
   31 
   32         var metaData = (BuildMetaData)engine.Operations.InvokeMember(instance, "return_meta_data");
   33 
   34         Assert.Equal(metaData.Description, "A description of sorts");
   35     }
   36 }

I hope this syntax gets a little friendlier in .NET 4.0 but at the time of writing, this was my interpretation of the API. Here is a brief synopsis of the code:

• Line 22 creates an instance of the ironruby script engine.


• Line 24 loads the assembly that contains the BuildMetaData class definition into the Ruby script engine's AppDomain. This will allow the instantiation of any type defined in this assembly.


• Line 26 quite obviously parses the script.


• Line 28 and 30 contains the code I eventually stumbled across to create an instance of the class that I defined in the ruby code I outlined earlier. The passing of time has eased the pain of how difficult this discovery was for me.


• Line 32 invokes a member of the newly intantiated instance variable.

With this passing test, I had a modicum of confidence that it was worth my while to pursue my quest further.

If any of this is of interest to you then please join the Horn user group for updates or check out the source here.