Deploying Dianoga in developer containers

I bumped into an interesting issue recently, which I though others might come across. Trying to run a project with Dianoga in it didn't work properly in a developer's Docker container – it kept failing whenever it was asked to process an SVG image. Why didn't that work? Here's why:

Edited to add: Over on Twitter Mark Gibbons helpfully points out that this issue is specific to older versions of Dianoga. With V5 and newer the SVG optimiser now uses a pre-built single-file version, so it wouldn't have the problem below. But the overall issue remains for any other reason you might need to put `node_modules` (or other excluded file/folder types) inside a container.

The issue: url copied!

I've been working on a project which is using Sitecore's docker containers for development. When you build the solution in Visual Studio, it copies the build output to a "deploy" folder, which is mapped into the docker container using a volume. The project has Dianoga set up in the source tree – and all the files and dependencies are deployed by Visual Studio when you hit "publish".

Initially that all seemed to be working fine – but when an SVG image was added to a page log errors started to appear in the container:

1244 08:38:25 INFO  Dianoga: /1548975485 is not something that can be optimized, either because of its file format or because it is excluded.
2336 08:38:25 ERROR Dianoga: Unable to optimize /1548975485 due to a processing error! It will be unchanged.
Exception: System.InvalidOperationException
Message: C:\inetpub\wwwroot\App_Data\Dianoga Tools\SVGO\node.exe exited with unexpected exit code 1. Output: 
    at node.js:974:3
    at startup (node.js:139:18)
    at Function.Module.runMain (module.js:441:10)
    at Function.Module._load (module.js:276:25)
    at Function.Module._resolveFilename (module.js:325:15)
Error: Cannot find module 'C:\inetpub\wwwroot\App_Data\Dianoga Tools\SVGO\node_modules\svgo\bin\svgo'

    ^
    throw err;
module.js:327

Source: Dianoga
   at Dianoga.Optimizers.CommandLineToolOptimizer.ExecuteProcess(String arguments)
   at Dianoga.Optimizers.CommandLineToolOptimizer.ProcessOptimizer(OptimizerArgs args)
   at Dianoga.Optimizers.OptimizerProcessor.Process(OptimizerArgs args)
   at (Object , Object )
   at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
   at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain, Boolean failIfNotExists)
   at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain)
   at Dianoga.Processors.Pipelines.DianogaOptimize.ExtensionBasedOptimizer.ProcessOptimize(ProcessorArgs args)
   at (Object , Object )
   at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
   at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain, Boolean failIfNotExists)
   at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain)
   at Dianoga.MediaOptimizer.Process(MediaStream stream, MediaOptions options)

					

But at the same time, log messages for successful resizing were appearing for other image formats:

3052 08:38:25 INFO  Dianoga: optimized /Default Website/sc_logo.png [original size] (final size: 3141 bytes) - saved 940 bytes / 23.03%. Optimized in 346ms.
3028 08:38:25 INFO  Dianoga: optimized /Default Website/cover.jpg [original size] (final size: 113193 bytes) - saved 6526 bytes / 5.45%. Optimized in 663ms.

					

It's clearly running – so what's up with SVGs?

What's happening? url copied!

A bit of digging lead to an important clue in the logs above. In the middle of the error stack trace is the message `Error: Cannot find module 'C:\inetpub\wwwroot\App_Data\Dianoga Tools\SVGO\node_modules\svgo\bin\svgo'` - it's saying it cannot find the SVG optimiser code.

Looking at the deployment folder that Visual Studio has written to, that file does exist, however:

Since I can see it on my physical disk, but the code in the container is complaining it's not there, I looked at the files inside the container:

Now the error makes sense – the whole "node_modules" folder referenced in the error above is missing from the website folder. That certainly explains why the error happens. But why is the file missing? Cue a pile of googling that lead to no useful answers, and then a bit of head scratching.

It struck me that the proces of getting the files from your deployment folder on your physical disk to the website folder is a little more complex than you might first expect. The deploy folder on your physical disk is mapped into your container with a volume definition – but if you look at it that volume doesn't map to the wwwroot folder. As I understand it there's a bug or missing feature in Docker for Windows that means you can't map a volume to a folder in a container which already has files in it. In these containers, the wwwroot folder already has Sitecore in it - so your deployment volume gets mapped to c:\deploy inside the container instead - a folder which is empty in the container's image. And then some magic script (that's part of the entrypoint for the container) watches that folder and copies any changes to the wwwroot - getting around the limitation of Docker.

So looking at c:\deploy inside the container, what do we see?

Bingo - the files are in the container. They just didn't get copied over to wwwroot...

The solution: url copied!

So I dug into the container, and looked at the entrypoint script, which lives in `C:\tools\entrypoints\iis\Development.ps1`. It turns out that starts another script which does the file copying as a background job:

    # Start Watch-Directory.ps1 in background
    Start-Job -Name $watchDirectoryJobName -ArgumentList $WatchDirectoryParameters -ScriptBlock {
        param([hashtable]$params)

        & "C:\tools\scripts\Watch-Directory.ps1" @params

    } | Out-Null

					

This job is running another script: C:\tools\scripts\Watch-Directory.ps1. And the root of this whole issue was staring out of the parameters for that script:

[CmdletBinding(SupportsShouldProcess)]
param(
    [Parameter(Mandatory = $true)]
    [ValidateScript( { Test-Path $_ -PathType 'Container' })]
    [string]$Path,

    [Parameter(Mandatory = $true)]
    [ValidateScript( { Test-Path $_ -PathType 'Container' })]
    [string]$Destination,

    [Parameter(Mandatory = $false)]
    [int]$Sleep = 200,

    [Parameter(Mandatory = $false)]
    [int]$Timeout = 0,

    [Parameter(Mandatory = $false)]
    [array]$DefaultExcludedFiles = @("*.user", "*.cs", "*.csproj", "packages.config", "*ncrunch*", ".gitignore", ".gitkeep", ".dockerignore", "*.example", "*.disabled"),

    [Parameter(Mandatory = $false)]
    [array]$ExcludeFiles = @(),

    [Parameter(Mandatory = $false)]
    [array]$DefaultExcludedDirectories = @("obj", "Properties", "node_modules"),

    [Parameter(Mandatory = $false)]
    [array]$ExcludeDirectories = @()
)

					

The default value for on of the parameters explicitly excludes any folder called node_modules by default! That does make some sense - I think in most scenarios you wouldn't want that copied into your container – it's usually HUGE and it's also usually not required at runtime. But Dianoga's SVG optimiser is an exception here...

But luckily it takes that set of exclusions as a parameter, and the parent entrypoint script also allows you to pass through a hashtable of parameters for this file watching script.

So the solution to this whole thing is surprisingly simple. You can adjust the entrypoint entry for your container in your Docker Compose file, to pass through the value for $DefaultExcludedDirectories with no exclusion for node_modules. So for my CM container, that becomes:

  cm:
    image: ${REGISTRY}${COMPOSE_PROJECT_NAME}-xm1-cm:${VERSION:-latest}
    build:
      context: ./containers/build/cm
      args:
        BASE_IMAGE: ${SITECORE_DOCKER_REGISTRY}sitecore-xm1-cm:${SITECORE_VERSION}
        SPE_IMAGE: ${SITECORE_MODULE_REGISTRY}spe-assets:${SPE_VERSION}
        TOOLING_IMAGE: ${SITECORE_TOOLS_REGISTRY}sitecore-docker-tools-assets:${TOOLS_VERSION}
    environment:
      Sitecore_ConnectionStrings_Solr.Search: http://solr:8983/solr;SolrCloud=true
    volumes:
      - ${LOCAL_DEPLOY_PATH}\website:C:\deploy
      - ${LOCAL_DATA_PATH}\cm:C:\inetpub\wwwroot\App_Data\logs
      - ..\src:C:\unicorn
    entrypoint: powershell -Command "& C:\tools\entrypoints\iis\Development.ps1 -WatchDirectoryParameters @{DefaultExcludedDirectories=@('obj', 'Properties'); Path='C:\deploy'; Destination='C:\inetpub\wwwroot';}"

					

I discovered in trying to do this that if you override any parameters like this, you have to pass them all - hence the need to add the Path and Destination properties too. If you don't the sync script throws an error when you try to start the container.

But once that change was made, and the CM container was rebuilt, all the files for Dianoga appeared in the right places:

And Dianoga starts compressing SVGs:

1516 09:22:43 INFO  Dianoga: optimized /Default Website/sc_logo.png [original size] (final size: 3141 bytes) - saved 940 bytes / 23.03%. Optimized in 275ms.
2776 09:22:43 INFO  Dianoga: optimized /Default Website/cover.jpg [original size] (final size: 113193 bytes) - saved 6526 bytes / 5.45%. Optimized in 635ms.
2172 09:22:44 INFO  Dianoga: optimized /1548975485.svg [original size] (final size: 5822 bytes) - saved 12993 bytes / 69.06%. Optimized in 1461ms.

					

Problem solved!

↑ Back to top