Getting pipelines from config

In my last post I was thinking about a more functional approach to defining pipelines, after having heard about some interesting new code that Sitecore had been working on. Since writing that I've had a few conversations where the topic "but what about if I want a pipeline to come from configuration?" has come up. I've been away from work for the last week doing my civic duty on Jury Service, but I've had some time in between court sessions that I've spent thinking about how last weeks ideas and configuration files might be combined.

So here's one way it could work:

First step is having some configuration... url copied!

<pipeline name="example">
  <step type="StronglyTypedPipelines.DoubleStep, StronglyTypedPipelines" />
  <step type="StronglyTypedPipelines.ToStringStep, StronglyTypedPipelines" />
  <step type="StronglyTypedPipelines.DuplicateStep, StronglyTypedPipelines" />
</pipeline>

					

It's just a root element for the pipeline, containing steps. And each <step> element then defines the type which must be instantiated to process that bit of the pipeline.

In Sitecore's world, the config for pipelines is commonly "patched" by add-on modules and the code you deploy for your site. Based on XML like it would be possible to implement some sort of config patching process that allows the same sort of "insert-before" or "replace" options as Sitecore does. I'm not going to go into the detail of that for this post, however. But for our purposes we can assume that code exists somewhere that can deal with doing any modification of the raw configuration before we try to instantiate the pipeline.

So the next task is to load the XML and get it ready for processing:

Loading the XML url copied!

public abstract class ConfigBasedPipeline<INPUT, OUTPUT> : Pipeline<INPUT, OUTPUT>
{
}

					

The functional difference between config and code based pipelines is basically how they're initialised, so the logic to do the loading can be put into the constructor of this type. Since the config will be written in XML, this code can be based on receiving the XML that describes the pipeline being loaded, and we can assume that any patching required has been done before the point that our class receives it.

So the constructor might look like:

public ConfigBasedPipeline(XElement pipelineXml)
{
    if (pipelineXml == null)
    {
        throw new ArgumentNullException(nameof(pipelineXml));
    }

    var pipelineSteps = parsePipelineSteps(pipelineXml);
    validatePipelineSteps(pipelineSteps);

    PipelineSteps = input => processPipelineSteps(pipelineSteps, input);
}

					

It checks that we got a valid XML Element, tries to parse it into pipeline step objects and then tries to validate that these meet the type requirements of the pipeline. Finally we can use those objects to initialise the Func<INPUT,OUTPUT> which actually runs the pipeline process that we saw described in the code-first constructor from the previous post.

Parsing and validating url copied!

private IList<IPipelineStep> parsePipelineSteps(XElement pipelineXml)
{
    var pipeline = new List<IPipelineStep>();

    foreach (var xStep in pipelineXml.Elements("step"))
    {
        string typeName = xStep.Attribute("type").Value;

        var type = Type.GetType(typeName);
        var ctr = type.GetConstructor(Type.EmptyTypes);
        var obj = (IPipelineStep)ctr.Invoke(Type.EmptyTypes);

        pipeline.Add(obj);
    }

    return pipeline;
}

					

The result of parsing is going to be a list of pipeline steps. In the previous post, the interface defining a step had generic parameters for the input and output. That makes it a bit difficult to handle here, as we don't know it's type parameters until after it's been created – which in turn makes defining a list to hold it harder. So to make this easier, I went back and added a base interface without type parameters that all the steps inherit from:

public interface IPipelineStep
{
}

public interface IPipelineStep<INPUT, OUTPUT> : IPipelineStep
{
    OUTPUT Process(INPUT input);
}

					

That allows creating a simple List<IPipelineStep> to store the result of the configuration.

The steps themselves can be generated by looping through each step element and creating an object from the type attribute that they have. In the real world this code probably needs some tests to ensure that the elements and attributes are all correct for parsing - but that's left out here for clarity.

But since last time we were talking about pipeline steps which can change the type of the input, just getting the right set of objects isn't really enough here. We need to be sure that the data type going in will be accepted, and the right result type will be generated. It seems better to test that before we try to run the pipeline – so hence some validation is required.

That's handled by the validatePipelineSteps() method:

private void validatePipelineSteps(IList<IPipelineStep> pipelineSteps)
{
    int stepNumber = 0;

    Type pipelineBaseInterface = this.GetType().GetInterface("IPipelineStep`2");
    Type currentInputType = pipelineBaseInterface.GenericTypeArguments[0];
    Type outputType = pipelineBaseInterface.GenericTypeArguments[1];
    foreach (var step in pipelineSteps)
    {
        stepNumber += 1;

        Type stepBaseInterface = step.GetType().GetInterface("IPipelineStep`2");
        Type stepInType = stepBaseInterface.GenericTypeArguments[0];
        Type stepOutType = stepBaseInterface.GenericTypeArguments[1];

        if (currentInputType != stepInType)
        {
            string msg = "Step #{0} {1} input type {2} does not match current type {3}.";
            throw new InvalidOperationException(string.Format(msg, stepNumber, step.GetType().Name, stepInType.Name, currentInputType.Name));
        }
        currentInputType = stepOutType;
    }
    if (currentInputType != outputType)
    {
        string msg = "Final step #{0} {1} output type {2} does not equal output of pipeline {3}.";
        throw new InvalidOperationException(string.Format(msg, stepNumber, pipelineSteps.Last().GetType().Name, currentInputType.Name, outputType.Name));
    }
}

					

This needs to iterate through each of the objects we generated from the XML and check it's inputs and outputs. It starts by looking at the type parameters for the overall pipeline object. Fetching the base generic pipeline step interface allows us to work out what the type parameters for the input and output of the overall pipeline is. So we can assume that the first step we encounter needs to have an input that is the same type as the overall pipeline.

Then the code can loop through each subsequent step, checking that the output type of one step matches the input type of the next. Again, it can use the type parameters of the base step interface to work this out. And finally, the output of the very last step must match the output of the overall pipeline.

If all of that matches up then all is well. Otherwise, the code raises exceptions that try to specify which step has been found to be incorrect and why.

And executing the steps... url copied!

private OUTPUT processPipelineSteps(IList<IPipelineStep> pipelineSteps, INPUT input)
{
    object output = input;

    foreach (IPipelineStep step in pipelineSteps)
    {
        MethodInfo mi = step.GetType().GetMethod("Process", BindingFlags.Public | BindingFlags.Instance);
        output = mi.Invoke(step, new[] { output });
    }

    return (OUTPUT)output;
}

						

public class ExampleConfigBasedPipeline : ConfigBasedPipeline<int, string>
{
    public ExampleConfigBasedPipeline(XElement pipelineXml) : base(pipelineXml)
    {
    }
}

						

var input = 13;

XDocument xd = XDocument.Load("ConfigBasedPipeline.xml");
//
// Patching the configuration data would go here
//
XElement pipelineXml = xd.Document.Element("pipeline");

var pipeline = new ExampleConfigBasedPipeline(pipelineXml);

var output = pipeline.Process(input);