Pay attention to the subtle details in your Devops pipeline YAML

Computers. Very useful when they work, but wildly frustrating when they don't. Recently I had one of those moments of frustration (well, two days actually) with Azure Devops and its YAML-based build pipelines. The root cause here seems like one of those things that could well bite others, so here's what happened to me...

The issue

I've been working with a client who had set up a DevOps build pipeline for their container-based website. Broadly their build process is split into "build the C# code" and "publish the Docker Images". And in between those two steps there's a bit of housekeeping to remove some stuff from the build which shouldn't end up in the images, and copy some bits and pieces into the right locations for the DockerFiles to find.

To give the simplest example possible, one of the things that should get tidied up are some .gitkeep files, which are used to make sure that empty directories can be committed to source control. The build process fills these with relevant files, but it's tidier to get rid of the "noise" before building the images. As a very basic example, the source tree might look like:

A few files, but including some .gitignore files which shouldn't end up in the output.

The client's DevOps pipeline was using the YAML style pipeline definitions. And they'd added a "Delete Files" task to do some of this tidying. The task was set to remove FolderOne/.gitkeep and FolderTwo/.gitkeep when it ran.

But when we looked at the output of the build, the files which should have been deleted were still present:

All the folders listed in the Delete task still had their .gitkeep files.

But the output of the task for deletion didn't show any errors:

So it was not at all obvious what might be wrong, and I spent quite a long time staring at the YAML file for the build, trying to work out what was causing this:

For Google's benefit, the YAML for that deletion task is:

- task: DeleteFiles@1
  inputs:
    SourceFolder: '$(Build.Repository.LocalPath)'
    Contents: >-
      FolderOne/.gitkeep
      FolderTwo/.gitkeep

					

The solution

In the end, the answer came to me because I went to my own DevOps instance and tried to recreate the issue in a new blank repo.

I used the "assistant" in the UI to create a new delete task, trying to replicate the problem. And it created this:

- task: DeleteFiles@1
  inputs:
    SourceFolder: '$(Build.Repository.LocalPath)'
    Contents: | 
      FolderOne/.gitkeep
      FolderTwo/.gitkeep

					

Do you see the difference? In the "broken" example, the "Contents" field in the YAML is defined using Contents: >- but in the working example it was defined as Contents: |. A really subtle difference, but once I corrected that in the client's build the results changed:

The .gitignore files had gone...

But why?

Well it turns out YAML's handling of multi-line fields is way more complicated (Flexible? Powerful? Confusing?) than I thought.

There are two things going on. One is the "style" of the block, which is determined by the > or the | character. The > in the original file meant "replace newlines with spaces", where the | character tells the parser to keep the newlines. And then the - from the original file meant "leave a single newline at the end of the block".

So the original file wasn't working because the delete command was receiving

FolderOne/.gitkeep FolderTwo/.gitkeep<newline>

					

rather than

FolderOne/.gitkeep<newline>
FolderTwo/.gitkeep<newline>

					

And that kind of makes sense that the delete operation might get confused...

In response to this post, my friend Ben Duguid points out that you can turn on the pipeline's debug logging as well - which can help provide details like what files are being searched for or processed. That's another useful tactic when debugging issues like the one above.

↑ Back to top