XM Cloud: We're on a journey

There was a lot of interesting discussion at SUGCON NA and the MVP Summit towards the back-end of last year. I've got piles of notes I took about stuff that caught my attention over the course of those events. But out of all the sessions, one specific thing stuck out to me as a vision of our future as Sitecore developers. And it's a topic that's come up a few times in my conversations with people at work and in the general community. So it seemed like it was worth writing about...

I watched Andy Cohen from Sitecore speaking at both events about Sitecore's work to improve and extend XM Cloud, and how that's lead to changes to the best ways for us to work with it and approach our projects.

And while he was discussing that he made an interesting point: That despite us having the ability to customise our CM images for XM Cloud projects, Sitecore don't want us to do this any more. And it wasn't a point made quietly or subtly either.

This slide is not leaving any room for misunderstanding:

For people who've been working with Sitecore for a long time, that might seem like an odd or surprising statement. We've been used to having a lot of flexibility to customise how the innards of Sitecore works. We've had config patches, pipelines, events and other extension points to make use of - and most historic solutions have made use of these features to meet business requirements. For better or worse, a lot of integration and customisation for business purposes has happened on XP CM servers over the years.

So the idea that these options would go away with XM Cloud seems like a dramatic departure from the status quo. But in my head, I think Sitecore have good reasons for this, so let me explain my thinking...

We're part of the way through a transition url copied!

Deployment patterns for web technology have been on a bit of a journey over my career:

flowchart TB
  subgraph J[Our journey]
    OnPrem[We started out
with deployments
 On Prem]
    IaaS[That morphed into
Infrastructure as a Service]
    PaaS[Then we jumped to
Platform as a Service]
    SaaS[And now
we're adopting
 Software as a Service]

    OnPrem --> IaaS
    IaaS --> PaaS
    PaaS --> SaaS
  end

					

And one of the drivers for that journey has been how vendors want to provide their software to us.

For many years we've been used to working with Sitecore's on-prem or PaaS hosted product. That provides a lot of flexibility, because you're able to change whatever you like in your deployment. But it also means that pretty much every install is different, and you notice this most when you're struggling through upgrade cycles. To cover all the possibilities for change that the flexibility brings, the instructions for upgrades have to be complex and are largely manual.

While we (as customers and partners) have been handling most of the work for upgrades and patches that has been a challenge for us, but not a major issue for Sitecore. But now, as they're moving towards SaaS products, it becomes a very significant challenge for them.

You can't run software hosting at scale if every update to your product code needs a pile of manual work per-client to unpick and fix the unique customisations that client made. To build good SaaS software you need to architect your products in a way that means any custom code and logic that's specific to your clients' projects sits outside of the software stack you're selling and hosting. That way you can apply your updates to the SaaS product's codebase and infrastructure without needing to worry about anyone's customisations. (As long as they're not breaking changes - but that's a whole other discussion...)

So far, I think we've seen the first phase of Sitecore's transition towards SaaS. That was to move towards the "CM is a container" pattern we have right now with XM Cloud. This isn't pure SaaS, even though they host the XM Cloud containers for us. But this is a first step towards simplifying their hosting challenges. With this approach it's much simpler to automate updates - because our customisations are built on top of Sitecore's docker images.

When there's an update for the code in the CM in this model, Sitecore can build a new Docker Image for the updated role and then trigger a redeployment of each client's instance. The deploy-process magic which builds each client's custom docker images layers customisations back on top of Sitecore's new image. There's a bit of delay and effort associated with this, but it's orders of magnitude better than the PaaS or IaaS approach of 90-page instruction PDFs we'd had before.

So this gets us part of the way towards a more scalable approach. It's better, but updates still have some fragility. But importantly it still requires having separate infrastructure per client. And that is expensive - so there's more to do in the future.

So the reason for Sitecore's change of tack is about where we might end up... url copied!

A pure SaaS solution should have a few big deployments of Sitecore's product, which are shared by all the clients. Rather than separate copies of the software services per-client, we're likely to end up with something akin to different "client IDs" in shared databases, so we never see each other's data - but one big pile of servers that can run the code for everyone. These will be partitioned in some way to allow for phased roll-out of updates across clients, but we can think of it as one big deployment. It's definitely many fewer instances than the "separate thing per client" we have right now.

flowchart LR
  subgraph New[Our glorious future]
    subgraph r1[Code Region 1]
      direction LR
      d1[(Client data 1)]
      d2[(Client data 3)]
      dn[(Client data n)]
    end
    subgraph r2[Code Region 2]
      direction LR
      d3[(Client data 2)]
      d4[(Client data 4)]
      dn1[(Client data n+1)]
    end
  end

  subgraph Old[The old model]
    direction LR
    c1{{Client containers 1}}
    c2{{Client containers 2}}
    c3{{Client containers 3}}
    c4{{Client containers 4}}
    cn{{Client containers n}}
    cn{{Client containers n+1}}
  end

  Old -- The transition
path we're
on --> New

					

Some of Sitecore's products are in this state now (I think OrderCloud is an example - look how easy it is to sign up for a trial of that, because no new infrastructure needs provisioning) but XM Cloud isn't there yet.

One of the key requirements for achieving this is that a client's customisations cannot live inside Sitecore's big deployment. Both for the ease-of-update reasons I've mentioned and for a variety of fun security reasons too, this stuff needs to live outside Sitecore-owned infrastructure. It needs to get connected up by out-of-process techniques like WebHooks, Sitecore Connect and APIs instead of our traditional patches and in-process customisations.

And to set us off down that road we need to move away from changing code in the container images, and focus on the newer out-of-process ways to customise things instead. Alongside this we can put more focus on running our development tasks using the CMS in the cloud, and just our "head" website on our local infrastructure. That will make website build simpler (and probably faster for most people) and will mean front-end developers have lots less to worry about when setting up a solution. They should not need Docker at all most of the time. Something a lot of front-end devs will likely be very pleased by.

This change is likely the first step down the road towards a fundamental adjustment in how "Sitecore" as we know it runs. We stop customising containers now. We stop running them locally soon. The old bits of UI from XP will get retired, and over time I suspect we'll see the core of XM transition into a series of API services rather than a distinct piece of software. We'll have all the relevant features in Pages and the management portal to be our UI, so we won't need Experience Editor or the traditional Sitecore Desktop any more.

And once that happens, Sitecore will be in a place where they can run a pure-SaaS model. The "product" becomes just that API surface, and now we don't care what happens inside the closed-box which offers those APIs to us. It can all get re-written in Erlang if they want, but as long as the APIs stay the same it won't affect us and how we deliver our projects at that point. And this flexibility allows the back-end to change towards that "one pile of servers and databases rows for everyone" model, because we no longer care what's inside the box of APIs.

In theory that will dramatically lower the run-time costs Sitecore pay for XM Cloud, and it will allow the core CMS to change at the same speed as the other SaaS products. Someone at SUGCON NA suggested Sitecore have averaged a bit more than one change a day to their SaaS products in the last year. The core CMS can definitely benefit from that speed of evolution...

Scared now? url copied!

I've talked about this with a few developers of late, and chunk of them see this as scary change. "Will we even need back end developers any more?" is a common question.

Well on one hand, I think us back-end devs have to admit that our role in the Sitecore project ecosystem has to change. We're not going to be writing code that calls Sitecore.Kernel any more - because all of that stuff will be hidden inside the "closed box" of the future system. (Well, in reality the kernel as we currently know it probably won't exist - that code is very likely to be permanently retired once we get to the "just an API surface" part of this transformation)

But that does not mean we're useless!

You can write headless front-end code in modern .Net - ASP.Net Core is supported now and maybe things like Blazor will gain support in the future. Plus all those out-of-process integrations are going to require wiring up. All that complicated code we used to write to enable business rules to be enforced has to go somewhere. A good chunk of the approaches to achieving that can involve C# - Azure Functions and the like become more important in the composable world. And writing Typescript for your Headless SXA UI isn't a million miles away from C# either...

I think most of us back-end Sitecore devs have worked in a surprisingly constant development world for the last 10+ years. But despite that it's fair to say that the only constant in IT is change, and we're going to have a bit of change to deal with in the future.

Though I suspect we will have a bit of a long-tail of client projects that don't want to or can't migrate off XP for a while yet...

↑ Back to top