Automatic packages from TFS: #1 - Overall plan

This is post 1 in an ongoing series titled Automatic packages from TFS

• Automatic packages from TFS: #1 - Overall plan
• Automatic packages from TFS: #2 – So how can I write a package file?
• Automatic packages from TFS: #3 – Pipelines and data transformation
• Automatic packages from TFS: #4 – Pipeline component internals
• Automatic packages from TFS: #5 – Wrap up

Recently I posted an idea for a PowerShell script to extract the set of changed items in TFS needing deployment for a Sitecore project. I left the script at the stage where it gave a list of the files to package, but didn't actually give you a package definition.

Having done a bit more thinking about how that might be achieved, I've decided that it's probably worth a series of posts as creating a sensible solution to the problem seems a bit too complex for a script and a single post.

So for this first post, I'll sketch out the problem I want to solve and look at some of the basic code if will require. I'll flesh out the code an implementation over the rest of the posts – but I'm not sure how many there will be at this point...

Requirements url copied!

I want to put together a tool which can be run either manually by a developer, or automatically by a build server. It should make use of the ChangeSet history in Team Foundation Server to determine the set of files, binaries and Sitecore Items which have changed and require deployment, and use that data to generate a standard Sitecore Package Definition file. It should be configurable, so that it can work with different projects that do not necessarily share the same underlying structure and processes. And it should avoid having any direct references to Sitecore itself, so it can be run without an instance of Sitecore available.

At this point, I'm thinking the tool needs to be implemented roughly as follows:

• Built as a command line executable
• Settings come from the parameters passed and/or a configuration file
• Read the changes from TFS as per the previous script
• Pass the set of changes through a pipeline process
• Pipeline components for "filter out unwanted files", "find files to deploy", "find binaries to deploy", "find items to deploy" and perhaps other tasks
• The output of the pipeline is data representing the package definition
• Save the package definition as an XML file

[My usual caveat that the code examples here miss out error handling and so on, in order to stay readable. Plus, in this case, I've not finished the code yet, so there's a good chance of it evolving a bit between posts]

So, to implementation:

Parsing the command line url copied!

If a tool is going to read command line parameters, it makes sense to re-use some pre-existing code for dealing with parsing them. A useful tool I've used for this before is this argument parser class from Peter Hallam @ Microsoft. Adding this class into the project allows us to define the arguments we want to parse in terms of a C# class that will represent the resultant data. We need to be able to specify the following options:

public class CommandLineParameters
{
    [Argument(ArgumentType.Required | ArgumentType.AtMostOnce, HelpText = "Starting changeset number", ShortName = "s", LongName = "start")]
    public int StartChangeSet;

    [Argument(ArgumentType.Required | ArgumentType.AtMostOnce, HelpText="The TFS working folder to process", ShortName="w", LongName="workingfolder")]
    public string WorkingFolder;

    [Argument(ArgumentType.AtMostOnce, HelpText = "Ending changeset number", ShortName = "e", LongName = "end", DefaultValue = -1)]
    public int EndChangeSet;

    [Argument(ArgumentType.AtMostOnce, HelpText = "The file name for the generated package", ShortName = "p", LongName = "package", DefaultValue = "GeneratedPackage.xml")]
    public string PackageFileName;

    [Argument(ArgumentType.AtMostOnce, HelpText = "The name for the package", ShortName = "n", LongName = "name", DefaultValue = "GeneratedPackage")]
    public string PackageName;
}

					

The Argument attribute marks these fields as being parameters which can be parsed from the command line. You can do the parsing using the following code:

static void Main(string[] args)
{
    CommandLineParameters cp = new CommandLineParameters();
    if (CommandLine.Parser.ParseArgumentsWithUsage(args, cp))
    {
        // parse succeeded - process your parameters
    }
}

					

The parser will compare the arguments passed to the program against the attributes defined on the CommandLineParameters class. If they can be matched up, then the instance of this class is populated with the data, and the if() will pass, allowing the app to continue working. But if the data doesn't match the attributes then a usage error is generated automatically. For example:

Getting the TFS changes url copied!

With the parameters parsed, the next thing to do is fetch the TFS changes. This can be done using a similar pattern to the approach from the PowerShell script – but using a more C# friendly way of executing the external tool. Plus, in this version we'll allow for both the first and last changeset to process (as per the configuration above) The class to manage the TFS interaction is as follows:

public class TFSCommandLine
{
    private string _toolPath = @"c:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\TF.exe";
    private string _argumentTemplate = @"history .\ /recursive /format:detailed /noprompt /version:{0}";

    public IDictionary<string, TFSActions> ProcessWorkItems(string workingFolder, int firstChange, int lastChange)
    {
        var output = runTool(workingFolder, firstChange, lastChange);
        var filteredOutput = filterOutput(output);

        return parse(filteredOutput);
    }

    private string runTool(string workingFolder, int firstChange, int lastChange)
    {
        string output;

        using (Process p = new Process())
        {
            p.StartInfo.UseShellExecute = false;
            p.StartInfo.RedirectStandardOutput = true;
            p.StartInfo.FileName = _toolPath;
            p.StartInfo.Arguments = string.Format(_argumentTemplate, formatVersionString(firstChange, lastChange));
            p.StartInfo.WorkingDirectory = workingFolder;
            p.StartInfo.CreateNoWindow = false;

            p.Start();
            output = p.StandardOutput.ReadToEnd();
            p.WaitForExit();
        }

        return output;
    }

    private string formatVersionString(int firstChange, int lastChange)
    {
        string first = "C" + firstChange.ToString();

        string last = string.Empty;
        if (lastChange != -1)
        {
            last = "C" + lastChange.ToString();
        }

        return first + "~" + last;
    }

    private IEnumerable<string> filterOutput(string output)
    {
        var items = output
            .Split('\n')
            .Select(i => i.Trim())
            .Where(i =>
                i.StartsWith("edit", StringComparison.InvariantCultureIgnoreCase) ||
                i.StartsWith("add", StringComparison.InvariantCultureIgnoreCase) ||
                i.StartsWith("delete", StringComparison.InvariantCultureIgnoreCase));

        return items;
    }

    private IDictionary<string, TFSActions> parse(IEnumerable<string> input)
    {
        char[] splitter = {'$'};

        Dictionary<string, TFSActions> results = new Dictionary<string, TFSActions>();

        var items = input.Select(i => i.Split(splitter, StringSplitOptions.RemoveEmptyEntries));

        foreach (var item in items)
        {
            string path = item[1];
            TFSActions action = (TFSActions)Enum.Parse(typeof(TFSActions), item[0].Trim(), true);

            if (results.ContainsKey(path))
            {
                results[path] = results[path] | action;
            }
            else
            {
                results.Add(path, action);
            }
        }

        return results;
    }
}

					

The entry point here is the ProcessWorkItems(). This calls the three private methods to extract the data from TFS, catch just the lines of that output which represent changes, and then parse these lines into the data that gets returned.

So first the call to runTool() runs the external tf.exe process. It sets up an instance of the .Net Framework's Process type as a wrapper to execute the external tool. At present the location of tf.exe is hard coded - but I plan to move that into a configuration settings later. The working directory for the process is set to the TFS Working Folder we want to process. And the code calculates the arguments to pass to the tool based on the first and last changes specified. The formatVersionString() method works out the correct string to pass to tf.exe to specify the version range required. And finally, it reads the Standard Output stream for this process into a string in order to capture the data we're going to process.

Then filterOutput is run over the string that's been captured. This does exactly the same thing that we did in PowerShell – breaking the string into lines, and keeping only the ones that represent a TFS change operation. And we end up with a list of strings representing changes.

Finally we call parse() to process those strings. Again, this works in much the same way that the PowerShell script did - but instead of storing the individual actions as strings the code is using an enumeration:

[Flags]
public enum TFSActions
{
    None = 0,
    Add = 1,
    Edit = 2,
    Delete = 4
}

					

So the code splits each of the lines received to get a path and an action, and it builds a dictionary keyed on the paths, and storing the set of actions associated with that path.

I'm thinking that as this code evolves, it may be useful to make the implementation of this class (specifically how it fetches the data) replaceable. I know it's possible to talk to TFS via an API rather than the command line. It may also be useful to be able to read test data to process without having a connection to TFS available – for automated tests perhaps. But that's a topic for a future post, perhaps.

Processing the data url copied!

The pattern I want to use for processing the data from TFS is a pipeline model. I'll talk about the implementation of this in a future post in this series, I think. But the overall process will be that the Pipeline will take the TFS data object as its input, it will pass this to each of the configured pipeline components and the result will be an object that represents a Package Definition. The code for running it will end up looking something like this:

CommandLineParameters cp = new CommandLineParameters();
if (CommandLine.Parser.ParseArgumentsWithUsage(args, cp))
{
    TFSCommandLine cmd = new TFSCommandLine();
    var data = cmd.ProcessWorkItems(cp.WorkingFolder, cp.StartChangeSet, cp.EndChangeSet);

    ProcessingPipeline pp = new ProcessingPipeline(cp.PackageName);
    pp.LoadPipelineComponentsFromConfiguration();
    var packageData = pp.Run(data);

    var xml = packageData.Serialise();

    var xws = new System.Xml.XmlWriterSettings();
    xws.OmitXmlDeclaration = true;
    xws.Indent = true;

    using (var xw = System.Xml.XmlWriter.Create(cp.PackageFileName, xws))
    {
        xml.Save(xw);
    }
}

					

The command line parameters get parsed, and on success they're passed in to the code to fetch data from TFS. A ProcessingPipeline object is created, and instructed to load the set of configured components. Then this is given the data from TFS, and after running all the pipeline components over it a package data set is returned. Each pipeline component gets a chance to modify the source data (for example to remove entries that aren't required for packaging, or to transform them into more useful data). The package data object that is generated can then be serialised to XML in the schema used by package definition files and saved to disk. The XMLWriterSettings is required here, as it turns out that Sitecore doesn't like loading a package definition if it starts with an XML declaration.

Next.. url copied!

I've got a bit more thinking to do before the second part of this series, as I want to nail down a prototype of the pipeline implementation and how it will get configured – but I think the next post will either be about the pipelines or about the object to represent and serialise the package data. Watch this space... ↑ Back to top