Egypt trip in 2007 Nothing is more important than the software engineers in a software company. I just finished re-reading Joel Spolsky’s Smart and Get Things Done and it inspired this post. Not only do I admire his writing style, I share Joel’s vision of how a software company should be run. Pampering your programmers is the best decision a manager can make, especially when you’ve built a team that can hit the high notes.

One of Joel’s key elements to running a successful software company is to automate your build process up to the point where it only takes one operation. This minimizes the chance of error while enabling your developers to grey matter to something more complex than copying files. Although they might use the extra time to shop around for student loan consolidation plans (a practice called reverse telecommuting), in most cases they’ll return to writing fresh code or cleaning out existing bugs.

Today’s post is about one of the little things that made my life much easier as a developer: using NAnt to manage our software product line. I’ve come to realize that we encounter these little “sparks” every day, but we never talk about them. Sure, we’ve produce a number of complex software products and they are fun to describe, but I personally enjoy talking about the little things that save time, just like Henry Petroski verbosely describes common items in his books. Fortunately for you, I’ll keep the story short, unlike his description of the evolution of the paper clip in The Evolution of Useful Things (which is still an interesting read, by the way).

Background

We develop lots of ASP.NET websites. Our architecture includes database schemas and business objects shared amongst multiple projects and some common utility libraries.  Furthermore, instead of always inheriting from System.Web.UI.Page and System.Web.UI.UserControl, we have an object oriented inheritance tree as is good software engineering practice. We even have a shared user control library that gets copied over after a successful build. Furthermore, we use ASP.NET master pages and ASP.NET themes to structure our designs. As opposed to what you see in textbooks where themes can be chosen by the user according to their preferences (oh yes, please show me the pink background with fluffy kittens), we use themes to represent different franchise brands.

My point here is that we reusability is key to our solution. We build elements that we can use not only on the website but also in FranchiseBlast, the interactive kiosk, and the point of sale. However, the more you re-use, the more things get complicated. Indeed, the overhead caused by the added configurability we build into our reusable components is non-negligible. We're always on the lookout for new ways to keep things simple, while still reaping the benefits of reuse. We use the Strategy Design Pattern to encapsulate the behavioural changes in our systems and put our various configuration settings inside our Web.config file.

Hurdle #1: Different developers need different web.config files

Our configuration files have a few settings that we want to change on a per-user basis:

- Where should we email exception notifications?

- Database names & file paths

- Google API Keys

How do we manage this? If we put our Web.config file under source control, we'll end up with various conflicts when the developers change the configuration file to suit their tastes. I don't know about you, but I have better things to do than start memorizing API keys or digits of PI.

Solution #1

Our first solution wasn’t fantastic, but it was sufficient for a while. We simply removed the Web.config from source control and created new files, one for each developer (Web.config.jkealey, Web.config.etremblay, etc.) and one for the deployment server (Web.config.server1). When a change was to be made, we whipped out WinMerge and changed all the files. You can quickly understand that this process does not scale well, but it was sufficient for small projects with 2 to 3 developers.

Hurdle #2: Scaling to more than a couple machines

We deploy our point of sale software and kiosks via Subversion. It might be fun to use WinMerge to compare a couple Web.config files, but when you’ve got a hundred web applications to update to the new version, by comparing Web.config files, you’ve got a problem. Doing this by hand wasn’t very difficult but it was error-prone and time consuming. I don’t know if you have seen the Web.config additions that ASP.NET AJAX brought to the table, but upgrading from a release candidate of Atlas to the full release of ASP.NET AJAX was painful (we’re not talking about half a dozen settings in the AppSettings section).

Solution #2

1) Create a template Web.format.config that contains the general Web.config format, with certain placeholders for variables that vary on a per-developer or per-machine basis.

2) Create a web.default.properties that contains the default settings for the web.config

3) Create a web.developername.properties for each developer that simply overrides the default settings with other values when needed.

4) Write a script to replace the placeholders in the Web.format.config and generate your Web.config.developername files for you.

We implemented this strategy using NAnt. Our script does a bit more work because we’ve got interrelated projects, but I will describe the base idea here.

Examples:

Here is a portion of our web.format.config file:

[...]
<appSettings>
    <add key="GoogleMapsAPIKey" value="${GoogleMapsAPIKey}"/>
</appSettings>
<system.web>
   <healthMonitoring enabled="${healthMonitoring.enabled}">
       <providers>
           <clear/>
           <add type="System.Web.Management.SimpleMailWebEventProvider"  name="EmailWebEventProvider"
               from="${bugs_from_email}"
               to="${bugs_to_email}"
               subjectPrefix="${email_prefix}: Exception occurred"
               bodyHeader="!!! HEALTH MONITORING WARNING!!!"
               bodyFooter="Brought to you by LavaBlast Software Inc..."
               buffer="false" />
       </providers>
   </healthMonitoring>
</system.web>
[...]

Property files

Our default settings look something like the following:

<project>
    <property name="GoogleMapsAPIKey" value="ABQIAAAAkzeKMhfEKdddd8YoBaAeaBR0a45XuIX8vaM2H2dddddQpMmazRQ30ddddPdcuXGuhMT2rGPlC0ddd" />
    <property name="healthMonitoring.enabled" value="true"/>
    <property name="email_prefix" value="LavaBlast"/>
    <property name="bugs_to _email" value="info@test.com" />
    <property name="bugs_from_email" value="exception@test.com" />
</project>

 

Our per-developer files include the default settings, and override a few:

<project>
    <!-- load defaults -->
    <include buildfile="web.default.properties"   failonerror="true" />   
        
    <!-- override settings -->
    <property name="GoogleMapsAPIKey" value="ABQIAAAAkzeKMhfEKeeee8YoBaAeaBR0a45XuIX8vaM2H2eeeeeQpMmazRQ30eeeePecuXGuhMT2rGPlC0eee"/>
    <property name="bugs_to_email" value="jkealey@test.com" />
</project>

The NAnt script

We wrote a NAnt script that runs another NAnt instance to perform the property replacements, but the core code comes from Captain Load Test. It is a bit slow because we have to re-invoke NAnt, but it doesn’t appear like you can dynamically include a properties file at runtime. Feel free to comment if you find a way to make it more efficient. We don’t have our generated files under source control as we only version the property files.

<project name="generate configs" default="generate ">
    <property name="destinationfile"   value="web.config" overwrite="false" />  
    <property name="propertyfile"  value="invalid.file" overwrite="false" />  
    <property name="sourcefile"   value="web.format.config" overwrite="false" />
 
    <include buildfile="${propertyfile}"   failonerror="false"   unless="${string::contains(propertyfile, 'invalid.file')}" />   
    
    <target name="configMerge">    
        <copy file="${sourcefile}"  tofile="${destinationfile}" overwrite="true">
            <filterchain>
                <expandproperties />
            </filterchain>
        </copy>
    </target>
 
    <target name="generate ">
        <property name="destinationfile" value="web.config.${machine}" overwrite="true"/>
        <property name="propertyfile" value="web.${machine}.properties" overwrite="true"/> 
        <property name="sourcefile" value="web.format.config" overwrite="true"/>
        <echo message="Generating: ${destinationfile}"/>
        <!--<call target="configMerge"/>-->
        <exec program="nant">
            <arg value="configMerge"/>
            <arg value="-nologo+"/>
            <arg value="-q"/>
            <arg value="-D:sourcefile=${sourcefile}"/>
            <arg value="-D:propertyfile=${propertyfile}"/>
            <arg value="-D:destinationfile=${destinationfile}"/>
        </exec>
    </target>    
</project>

Hurdle #3: Software Product Lines

Egypt trip 2007 Up to now, we’ve talked about taking one project and making it run on a number of machines, depending on a few preferences. However, we’ve taken it one step further because our web applications are part of a software product line. Indeed, we have different themes for different brands. Different companies have different configuration settings and site maps files. Therefore, we needed to be able to generate configuration files for each brand AND for each machine. This also greatly increases the number of configuration files we need.

Solution #3

It wasn’t very difficult to expand to this new level of greatness thanks to the script presented in hurdle #2. We basically have default configuration files for each project (themes, sitemap, name, locale, etc) in addition to the files we’ve shown above. We simply have to load two configuration files instead of one.

We even wrote a batch file (SwitchToBrandA.bat) that generates the property file for the current machine only (via the machine name environment variable) and it replaces the current Web.config. By running one batch file, we switch to the appropriate product for our current machine.

Future work

Currently, it takes a couple minutes to create a new brand or add a new developer. It doesn’t happen often enough to make it worthwhile for us to augment the infrastructure to make it easier on us, but is a foreseeable enhancement for the future. I guess another future work item would actually be hire someone who is an expert in build automation, test automation and automatic data processing! :) These are skills they don't teach in university, but should!

kick it on DotNetKicks.com