Development environments with PowerShell DSC – Coveo REST API & Coveo for Sitecore

This is post 7 of 7 in a series titled Development environments with PowerShell DSC

• Development environments with PowerShell DSC – Introduction to DSC
• Development environments with PowerShell DSC – Windows Features
• Development environments with PowerShell DSC – Mongo DB
• Development environments with PowerShell DSC – SQL Server
• Development environments with PowerShell DSC – Sitecore
• Development environments with PowerShell DSC – Coveo CES
• Development environments with PowerShell DSC – Coveo REST API & Coveo for Sitecore

After last week's work on installing the Coveo search service, this week we'll move on to how that script can be extended to install the Coveo REST API and the Coveo for Sitecore package.

Before I get going, couple of notes about the examples I'm including in this series of posts: They all have a call to `Start-DSCConfiguration` using the "-Force" flag to make DSC run this configuration immediately in "push" mode. However, the `Configuration` blocks declared should all work in pull configurations. Writing the scripts this way just makes them easier for people to try out without any other setup effort. Also the examples are all pretty much self-contained, so the overall configuration of a server using them would rely on many separate scripts. You don't necessarily need to have as many as I'm showing here. It's more to keep the examples clear and usable than to represent the "right" architecture. You're free to merge things together, or even break them apart further if that suits you.

Installing the Coveo REST API url copied!

Adding the REST service follows a similar pattern to last week's installations of prerequisites: Copy over the file described in the config data, and then use a `Script` resource to run the installation. Appending the following to last week's work can do this:

File CopyCoveoApi
{
    DependsOn = "[script]InstallCoveo"
    SourcePath = "$PackagePath\$($Node.Coveo.CESAPIInstaller)"
    DestinationPath = "$($Node.TempFolder)\$($Node.Coveo.CESAPIInstaller)"
    Ensure = "Present"
}

Script InstallCoveoApi
{
    Credential = $cred
    DependsOn = "[file]CopyCoveoApi"
    GetScript = { 
        @{ Feature = "Install Coveo REST API" }
    }
    TestScript = { $False }
    SetScript = {
        $tmp = "$($using:Node.TempFolder)"
        $inst = "$($tmp)\$($using:Node.Coveo.CESAPIInstaller)"
        $secret = "$($using:Node.Coveo.AppSecret)"

        &$inst /exelog "$($tmp)\coveo-api.log" /qn "APPLICATION_SECRET=$secret" | Out-Null
    }
}

					

After the file gets copied over, the API's installer can be run passing in a couple of parameters. The /qn tells it not to present any UI. The APPLICATION_SECRET parameter can be any string, but it needs to match what's stored in the config files we'll get to in a bit.

Installing the Coveo for Sitecore extensions url copied!

This bit is a little more challenging. In a manual install, after the Sitecore package has installed, post-install steps are run which present a UI wizard that gathers configuration settings. However when you install a package via automation, the post-install steps don't get run. I considered writing some code to automate the classes underlying this UI, but I found this difficult to achieve in the time I had available for research. Hence I've fallen back to a simpler method:

First off, we need a helpful prerequisite for later in the install: Powershell Extensions for Sitecore. This is a package we can install using the endpoint we added back in the original Sitecore installation.

To install the extension we need to copy over the package file (described in our config) and then call the remote install endpoint to trigger the installation:

File AddPSEPackage
{
    SourcePath = "$PackagePath\$($Node.Sitecore.PowerShellExtensions)"
    DestinationPath = "$($Node.WWWRoot)\$($Node.Sitecore.InstanceName)\Data\packages"
    Type = "File"
    Ensure = "Present"
}

Script InstallPSE
{
    DependsOn = "[file]AddPSEPackage"
    GetScript = {
        @{ Feature = "Install PowerShell Extensions" }
    }
    TestScript = {
        Test-Path "$using:Node.WWWRoot\$using:Node.Sitecore.InstanceName\Website\App_Config\Include\Cognifide.PowerShell.config"
    }
    SetScript = {
        $siteName = $using:Node.Sitecore.InstanceName
        $sitecoreFolder = "$($using:Node.WWWRoot)\$($siteName)\Data\packages"   
    
        $module = "$($using:Node.Sitecore.PowerShellExtensions)"

        $query = (Split-Path -Path $module -Leaf)
               
        Write-Verbose "Calling package upload tool for $using:query"
    
        $url = "http://$siteName/PackageDeploy.aspx?modules=$query"
        $result = Invoke-WebRequest -Uri $url -UseBasicParsing -TimeoutSec 600 -OutFile ".\$siteName-PackageResponse-$query.log" -PassThru
        
        if($result.StatusCode -ne 200) {
            Write-Verbose "StatusCode: $($result.StatusCode)"
            throw "Package install failed for $query"
        }
        else {
            Write-Verbose "Install ok"
        }
        
        $file = Join-Path $sitecoreFolder $query
        Remove-Item $file -Force
    }
}

					

This Script resource is the DSC translation of the remote package installation from previous posts.

PowerShell extensions allows for remoting - you can call run scripts from inside DSC which execute inside Sitecore. To enable this we have to add a module to PowerShell. This can be done by unzipping the remoting files from Marketplace into your PowerShell modules folder:

Archive EnableSPERemoting
{
    Path = "$PackagePath\$($Node.Sitecore.PowerShellRemoting)"
    Destination = "$Env:ProgramFiles\WindowsPowerShell\Modules\SPE"
}

					

With that in place, the code can copy over the Coveo for Sitecore package and use the same approach to install it:

File AddCoveoPackage
{
    DependsOn = "[script]InstallCoveoApi"
    SourcePath = "$PackagePath\$($Node.Coveo.CoveoForSitecore)"
    DestinationPath = "$($Node.WWWRoot)\$($Node.Sitecore.InstanceName)\Data\packages"
    Type = "File"
    Ensure = "Present"
}
       
Script InstallCoveoPackage
{
    DependsOn = "[file]AddCoveoPackage"
    GetScript = {
        @{ Feature = "Install Coveo for Sitecore" }
    }
    TestScript = {
        $folder = "$($using:Node.WWWRoot)\$($using:Node.Sitecore.InstanceName)\Website\Coveo"
        Test-Path -Path $folder
    }
    SetScript = {
        $siteName = $using:Node.Sitecore.InstanceName
        $sitecoreFolder = "$($using:Node.WWWRoot)\$($siteName)\Data\packages"   
    
        $module = "$($using:Node.Coveo.CoveoForSitecore)"

        $query = (Split-Path -Path $module -Leaf)
               
        Write-Verbose "Calling package upload tool for $using:query"
    
        $url = "http://$siteName/PackageDeploy.aspx?modules=$query"
        $result = Invoke-WebRequest -Uri $url -UseBasicParsing -TimeoutSec 600 -OutFile ".\$siteName-PackageResponse-$query.log" -PassThru
        
        if($result.StatusCode -ne 200) {
            Write-Verbose "StatusCode: $($result.StatusCode)"
            throw "Package install failed for $query"
        }
        else {
            Write-Verbose "Install ok"
        }
        
        $file = Join-Path $sitecoreFolder $query
        Remove-Item $file -Force
    }
}

					

So now the script needs to provide all the same configuration that the post-install steps. The first thing it needs to do is to create the "Top Results" folder. That can be done using the PowerShell extensions remoting:

Script CreateTopResults
{
    DependsOn = "[script]InstallCoveoPackage"
    GetScript = {
        @{ Feature = "Create Coveo Top Results folder" }
    }
    TestScript = {
        $False
    }
    SetScript = {
        Import-Module -Name SPE

        $siteName = $using:Node.Sitecore.InstanceName

        $session = New-ScriptSession -Username admin -Password b -ConnectionUri "http://$siteName"

        Invoke-RemoteScript -Session $session -ScriptBlock { 
            New-Item -Path "master:/sitecore/system/Modules" -Name "Coveo" -ItemType "/sitecore/templates/System/Node"
            New-Item -Path "master:/sitecore/system/Modules/Coveo" -Name "Top Results" -ItemType "/sitecore/templates/CoveoModule/Search/Top Results/Top Result Folder"
        }
    }
}

					

This imports the remoting module we installed later, creates a remote connection to our Sitecore instance, and then runs the two New-Item statements required to ensure the right folders exist in this remote session.

Next we need to provide the appropriate configuration for the Coveo extension. Having tried automating the code under the surface of the post-install steps wizard without much success, I've fallen back to taking the configuration files from a manual install and deploying them via file copy. Three files are required: The Coveo license data (which can be found next to the Sitecore license after the post-install steps have been completed) and then the Search Provider and Search Provider REST API config files, which are found in the App_Config/include folder. You can do a manual install with the settings you need, and then take the appropriate files from that.

One minor change is required to the default files before Coveo will work, however. When you use the post-install steps UI, it saves a password for Rabbit MQ into the configuration. By default it encrypts these credentials. If you copy the encrypted credentials to a different machine, these won't work. So you need to change the encrypted value to plain text. The default password is "guest", so you can change the <QueuePassword> in the Coveo.SearchProvider.config file before adding it and the others to the PackagePath folder.

They can then be deployed to the right places with File resources:

File CopyCoveoLicense
{
    DependsOn = "[script]InstallCoveoPackage"
    SourcePath = "$PackagePath\coveolicense.xml"
    DestinationPath = "$($Node.WWWRoot)\$($Node.Sitecore.InstanceName)\Data\coveolicense.xml"
    Type = "File"
    Ensure = "Present"
    Force = $True
}

File CopySearchProviderConfig
{
    DependsOn = "[File]CopyCoveoLicense"
    SourcePath = "$PackagePath\Coveo.SearchProvider.config"
    DestinationPath = "$($Node.WWWRoot)\$($Node.Sitecore.InstanceName)\Website\App_Config\include\Coveo.SearchProvider.config"
    Type = "File"
    Ensure = "Present"
    Force = $True
}

File CopySearchProviderRestConfig
{
    DependsOn = "[File]CopyCoveoLicense"
    SourcePath = "$PackagePath\Coveo.SearchProvider.Rest.config"
    DestinationPath = "$($Node.WWWRoot)\$($Node.Sitecore.InstanceName)\Website\App_Config\include\Coveo.SearchProvider.Rest.config"
    Type = "File"
    Ensure = "Present"
    Force = $True
}

					

With all the config in place, there's one more thing to do. The Sitecore part of Coveo configures the CES services we installed in the last post when you first build an index. Hence we need to trigger a build of the Coveo instances to finish off. PowerShell Extensions can achive this for us:

Script RebuildCoveoIndexes
{
    DependsOn = "[file]CopySearchProviderConfig", "[file]CopySearchProviderRestConfig", "[file]CopyCoveoLicense"
    GetScript = {
        @{ Feature = "Rebuild Coveo indexes" }
    }
    TestScript = {
        $False
    }
    SetScript = {
        $siteName = $using:Node.Sitecore.InstanceName

        $session = New-ScriptSession -Username admin -Password b -ConnectionUri "http://$siteName"

        Invoke-RemoteScript -Session $session -ScriptBlock { 
            Rebuild-SearchIndex -Name coveo_master_index
            Rebuild-SearchIndex -Name coveo_web_index
        }
    }
}

					

PowerShell extensions has a commandlet Rebuild-SearchIndex which generates the index.

You'll probably need to make other configuration changes to your Sitecore projects and config because Coveo has changed the names of the default indexes – but at this point Coveo should be good to serve results. You can look at the Diagnostics that have been added to the Sitecore Control Panel:

And with that, you're ready to do some development work...

PS: I have noted that occasionally, when you look at the diagnostics you will get a red error for the REST endpoint after installation. This didn't happen on the test virtual machine I developed these scripts on, but did sometimes happen on the VMs being used for actual development work. (I've not worked out why this happened, however) If you encounter this problem, I found that restarting the Coveo CES service before you rebuild the indexes seemed to resolve the issue. The DSC Resource for Windows services can be used to automate this.

↑ Back to top