You can have “Solr as a Service” now...

Recently I was writing about the changes to Java licensing that Oracle are enforcing in 2019. It's not an uncommon reaction to the challenges that the new license introduces to start thinking about alternatives to how you might manage search for your Sitecore deployments. So what can you do?

Big picture – you have two choices:

1. Stop using Solr, and move on to a different search engine.
   Which is almost certainly either Azure Search or Coveo these days. Both of these have some technical differences from Solr, so you have some big config changes to make, and potentially some code changes may be required too.
   (It's been pointed out that you can't actually use Coveo for the analytics side of search in Sitecore, so it doesn't get you out of the "needing Solr" thing entirely – my bad)
2. You can get someone else to run Solr for you.
   You could push Solr up to Azure PaaS if you're feeling adventurous, and Microsoft can handle the Java licensing. Or you can go full SaaS and buy your Solr service off someone who specialises in running it. This way you change where Sitecore looks for Solr, but the rest of your code and config stays the same.

I'd been discussing the idea of "Solr as a Service" in the office recenly, so I was interested in finding out how it might work. Google tells me that SearchStax is one organisation who are selling this service, so I thought I'd give it a try and see what I could find out with some basic setup and tests.

Getting started with SearchStax url copied!

As with any SaaS service, the first thing you need to do is sign up for an account. Since I've been experimenting, I set up a free trial account via their "try it out with Sitecore" form. I noted that there are also "start a free trial" links elsewhere on their site, but there seems to be an issue with those, and I was unable to register correctly.

Whichever way you register it, that gets you a 14-day trial, which you can convert to a paid account at the end of the process if you want.

With that in place, the SearchStax "getting started" docs talk about creating a Solr instance in your new account. It's pretty trivally simple really – you choose which cloud provider you want to use, where in the world you want your instance to be, and what scale of instance you need. I chose to set up an AWS instance in the UK with a "3 replica" cluster.

Having made those choices you just drum your fingers on the desk for a few minutes while the cloud-magic happens and your Solr instances are created and configured. And after it completes you end up with:

You can click on your deployment, to drill down to a more detailed view:

Critically this page gives you two bits of data you'll need later: The Solr endpoint for your instance, and the Zookeeper Ensemble data you'll need for further configuration work.

There's one other thing to do here: In the documentation SearchStax provides, they include a link to a zip containing a customised version of the standard Zookeeper admin scripts. You'll need to download that zip, and extract the relevant bits. Assuming you're working on Sitecore 9.1, and using Windows, you're going to need:

Now, the other thing I realised at this point, is that while Sitecore document the idea of using SolrCloud for your Sitecore instances, they don't really talk much about the details of setting up from scratch. So I had to improvise...

Setting up Sitecore 9.1 url copied!

As it stands, I don't think you can run an install via SIF against a remote Solr instance you don't own. If it's your box, I think you can use some of SIF 2.0's "remoting" features to run the Solr bits of the install on a specific server. (Though I've not tried this myself) But if you don't have admin access to a remote box (as in the SaaS model) or you're just not sure what the "right" setup is (like me, looking at a clustered deployment) then SIF probably isn't the right answer, I think. (Though I'm entirely happy to be proved wrong – if there's some magic "SolrCloud setup" settings I'd missed, do let me know in the comments and I'll updated this post)

With the appropriate time and effort, you could adjust the standard SIF 2.0 install scripts to create an instance of Sitecore without Solr attached, but for the purposes of "just making stuff work" I chose to install an instance of Solr 7.2.1 locally using my "quick solr setup" script. And with that in place, I could use the default "developer install" setup behaviour with SIF to set up a working instance of Sitecore 9.1.

With that done, I checked it was all working ok, stopped IIS, Solr and the xConnect services and then cleared out the old log files so that I could re-configure everything to run against SearchStax.

Plugging the two together url copied!

There's a bit of confusion when you compare the SearchStax docs and the Sitecore docs for setting up a SolrCloud instance. SearchStax says "you need to use our Zookeeper scripts to upload the Solr configuration remotely", but Sitecore says "Run the standard Zookeeper scripts on the Solr server". Not being much of an expert on Solr, I chose to follow the SearchStax model, since it was obvious to me that this was designed to work when you have a replicated cluster of Solr instances in one "SolrCloud" deployment, and you don't have RDP access to any of them. It's not obvious from Sitecore's docs whether their approach will work remotely to configure all of the Solr servers in the Zookeeper Ensemble – especially since you never configure Sitecore with any data about the Zookeeper endpoints. (Again, I'm happy to be proved wrong here – comment below if you know more about this than me, and I'll update my post)

Since I'm not too confident about what might have changed in the Solr indexes for v9.1 I decided I'd just upload the config for each individual local index up to SearchStax, so I could use the "right" config against each index I ended up creating. That saves any messing about with diff tools looking to see which config files might be different...

Given there are 13 indexes in a 9.1 install, I decided that uploading all this config needed some automation...

# What did I name my Sitecore instance
$prefix = "sc910"

# What did SearchStax say my Zookeeper Ensemble was?
$zkEnsemble = "ss000000-1-eu-west-2-aws.searchstax.com:2181,ss000000-2-eu-west-2-aws.searchstax.com:2181,ss000000-3-eu-west-2-aws.searchstax.com:2181"

# Where did I put the SearchStax zkcli.bat file?
cd "C:\Users\Administrator\Desktop\solr-7\scripts"

# Grab all the solr indexes that SIF created
$folders = Get-ChildItem "C:\Solr\solr-7.2.1\server\solr\$prefix*"

# Iterate the solr indexes
$folders | % {

  # And for each one, upload its config to the SearchStax Zookeeper ensemble with a unique name
  .\zkcli.bat -zkhost $zkEnsemble -cmd upconfig -confdir C:\Solr\solr-7.2.1\server\solr\$($_.Name)\conf -confname cfg_$($_.Name)

}

					

That hacky bit of PowerShell pushed the config from each of my local Solr indexes up into the cloud, so it was available to attach to the correct collection in SearchStax. It takes a while to run (as each individual upload takes some time) but it completed ok.

So the next job was to create cores/indexes/whatever in my SearchStax solr instance. Checking Sitecores docs, the correct things to create for SolrCloud are actually "collections". I could probably automate this too, but it was a simple enough form to fill in to create them, so I just copied and pasted into the Solr UI:

You specify the name of the index, which I took from the list of indexes created locally. You pick a config to assign to the new collection, which should have the same name as the collection (with a prefix) because of how the script above uploaded the config. And then you specify the correct sharding and replication data, which for my instance was "one shard, three replicas" to match what you chose when you set up your SearchStax instance.

And with that done, I could reconfigure Sitecore to point at the new SearchStax instance. That just involved editing the ConnectionStrings.config files. Remember that there are a few instances of this file now, so you probably want to find them with a search under the web roots for Siteocre and xConnect. And also remember that the url has slightly different structure in different places. In the Sitecore connection strings it's just the overall Solr url, but in the xConnect files the collection name is included too.

Now, one small point to watch here is that Sitecore's docs specify that you need to add a parameter to the URL to say "this endpoint represents a SolrCloud instance". You have to put ;solrCloud=true onto the end of each connection string.

With that done you can start up IIS and the xConnect-related services again. And log in to the admin interface.

Sitecore's docs suggest you should use Control Panel's "Populate Managed Schema" option at this point, but I think this isn't relevant because of the upload of the individual configs above. So I went straight to the index building – which completed without any obvious errors.

So, did it work? url copied!

Now that's a good question. 😉

On the positive side, I can see that there are lots of documents in my Solr collections, and the search inside Sitecore's Content Editor works:

Plus I don't see any of the traditional issues associated with Solr being broken – like Experience Forms not showing forms you've created, or the Analytics report pages having cryptic "An error occurred" messages in a red banner across the top.

But I note there are some errors in the Sitecore logs.

In the crawling log, I've seen odd errors about computed fields failing:

3136 19:12:53 WARN  Could not compute value for ComputedIndexField: __solr_field_type for indexable: sitecore://core/{4255D463-DBA7-4E62-8B5D-68986A39BD96}?lang=da&ver=1
Exception: System.Exception
Message: Item "/sitecore/client/Business Component Library/version 2/Layouts/Renderings/Common/ActionControl/Action/__Standard Values" is not a template section. (method: Sitecore.Data.Items.TemplateFieldItem.op_Implicit(Item item)).
Source: Sitecore.Data.Items.TemplateFieldItem.op_Implicit(Item item)
   at Sitecore.Diagnostics.Error.Raise(String error, String method)
   at Sitecore.Data.Items.TemplateSectionItem.op_Implicit(Item item)
   at Sitecore.Data.Items.TemplateFieldItem.op_Implicit(Item item)
   at Sitecore.ContentSearch.SolrProvider.FieldNames.TypeResolving.Index.ComputedFields.TemplateFieldComputedIndexField.ComputeFieldValue(IIndexable indexable)
   at Sitecore.ContentSearch.AbstractDocumentBuilder`1.<>c__DisplayClass57_0.<AddComputedIndexFieldsInParallel>b__0(IComputedIndexField computedIndexField, ParallelLoopState parallelLoopState)

					

And in the main site logs I've also seen errors that appear to be "Sitecore could not see the Solr server" for some reason:

3472 19:04:02 ERROR Unable to connect to the remote server
Exception: SolrNet.Exceptions.SolrConnectionException
Message: Unable to connect to the remote server
Source: SolrNet
   at SolrNet.Impl.SolrConnection.Get(String relativeUrl, IEnumerable`1 parameters)
   at SolrNet.Impl.SolrCoreAdmin.Status(String coreName)
   at Sitecore.ContentSearch.SolrProvider.SolrCloudIndexSummary.GetReplicaIndex(Replica replica)
   at System.Linq.Enumerable.<>c__DisplayClass7_0`3.<CombineSelectors>b__0(TSource x)
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at System.Linq.Enumerable.Sum(IEnumerable`1 source)
   at Sitecore.ContentSearch.SolrProvider.SolrCloudIndexSummary.get_NumberOfDocuments()
   at Sitecore.ContentSearch.Client.Forms.IndexingManagerWizard.BuildIndexCheckbox(String name, String header, ListString selected, ListString indexMap)

Nested Exception

Exception: System.Net.WebException
Message: Unable to connect to the remote server
Source: System
   at System.Net.HttpWebRequest.GetResponse()
   at HttpWebAdapters.Adapters.HttpWebRequestAdapter.GetResponse()
   at SolrNet.Impl.SolrConnection.GetResponse(IHttpWebRequest request)
   at SolrNet.Impl.SolrConnection.Get(String relativeUrl, IEnumerable`1 parameters)

Nested Exception

Exception: System.Net.Sockets.SocketException
Message: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 10.0.X.XX:8983
Source: System
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)

					

In both cases these errors crop up a few times, but they're not flooding the logs.

Since I've not spent much time with v9.1 I'm not entirely sure if these are issues with my 9.1 install or issues with the config I've changed for SearchStax. When I get a chance I need to swap back to the local Solr instance and test whether the same errors crop up in that configuration.

There are also some errors in the Solr logs, but it's very difficult to tell if these are caused by Sitecore, or if they're a side-effect of the standard "I put a server on the internet, and random blackhats will port scan it and throw standard hacks every open port to see if anything sticks". I can see a collection of directory traversal attacks being tried against Solr, so I'm not at all sure if the odd "Early EOF" errors I see as well are related to this or not...

So, I think it's working – but I think I'm going to need to find time for some more formal testing to make sure I've not missed any issues here...

↑ Back to top