absurdum reductus

Valid YAML, Three Commands: The Folded Block That Reads Right and Runs Wrong

A deploy step that every reader and every parser agreed was one command reached the shell as three. What a YAML folded block actually folds, why no cheap check could see the problem, and the two guards we kept afterwards.


The command that never ran#

This site is built by a CI pipeline and handed to the web host over ssh. The deploy step is one pipeline: tar the build, pipe it to the host. Written the way anyone would write a long command they expect another human to read, it looked like this:

commands:
  - >-
    tar czf - -C site/dist .
    | ssh -T -i /tmp/key
      -o BatchMode=yes
      deploy@example.com

It is valid YAML. It reads correctly, and anyone reviewing it would be right about what it says. The first time it ran, ssh printed its usage message, tar died of a broken pipe, and the pipeline went red.

ssh had been started with no destination, because the destination was no longer part of its command line. It was two lines further down, being offered to the shell as a command in its own right.

Three lines of Python#

import yaml
step = yaml.safe_load(open("step.yaml"))
print(repr(step["commands"][0]))
'tar czf - -C site/dist . | ssh -T -i /tmp/key\n  -o BatchMode=yes\n  deploy@example.com'

Two newlines survived. The >- indicator means “folded”: line breaks become spaces, so a paragraph can be wrapped in the source and arrive as one line. But folding applies only between lines at the block’s own indentation. A line indented further than that is treated as preformatted text, and the line break in front of it is kept.

That rule exists for a good reason. It is what lets a folded paragraph contain an indented code sample or a list without having it reflowed into mush. The specification is doing exactly what it promises. It is also exactly the wrong behaviour for the one thing people most often put in a folded block in a CI file, which is a long shell command with its continuation lines indented so that a person can see they are continuations.

Run that string under a shell with ssh and tar replaced by stubs that print their arguments:

tar argv: czf - -C site/dist .
ssh argv: -T -i /tmp/key
sh: 2: -o: not found
sh: 3: deploy@example.com: not found

Three commands. The first is a pipeline into an ssh with nowhere to go.

The readable version is the broken one#

The same command, four ways, parsed with the same library:

how it is writtenwhat the shell receives
>-, continuation lines indented further3 lines
>-, every line at the same indentation1 line
|- (literal block)4 lines, the second starting with a pipe
plain scalar, simply continued on the next lines1 line

The second row works, and looks like a mistake waiting to be “tidied” by the next person who touches it. The fourth row works too, until the command contains a # or a : and the plain scalar ends somewhere nobody intended. The first row is the only one that looks deliberate, and it is the one that is wrong. Indentation added as a courtesy to the reader changed the program.

Why nothing caught it#

The defect is not in the file. It exists only in the parsed string, and that puts it out of reach of every inexpensive check there is. Parsing succeeds, because nothing is malformed. Review succeeds, and a careful reviewer succeeds hardest of all, because the source says precisely what was meant.

We had even been diligent. Before writing the step we had checked the build image and confirmed it carried ssh, tar and gzip and did not carry rsync, which is why the step ships a tarball through a pipe at all. That was real verification, aimed one layer too low. Checking that the ingredients of a command exist is not checking the command.

And the failure was quiet, by our own design. The site was mid-move from one backend to another, and the cutover had been arranged so that the old backend kept serving until a new build actually existed on the new host. A deploy that publishes nothing therefore changes nothing a visitor can see. The safety mechanism and the silence were the same mechanism. The only signal was an exit code.

The fix is one line#

Nothing clever. Each command becomes a single line, and the long option list is built up with the shell’s own positional parameters instead of the YAML’s line folding:

commands:
  - 'set -- -T -i /tmp/key -o BatchMode=yes'
  - 'tar czf - -C site/dist . | ssh "$@" deploy@example.com'

It is uglier than the folded block. It also means the same thing to YAML, to the shell and to the reader, which the folded block did not.

Two guards, and why there are two#

A style check, in the pipeline. The build image has a POSIX shell and awk and no YAML parser, so the first guard cannot ask what the YAML means. It checks the shape instead: under a commands: key, every line of content must begin a new list item. That bans multi-line commands outright without needing to understand folding. It is crude, it runs as the first command of the build, and it fails on the original file.

An executed check, on a workstation. The second guard parses the pipeline file, takes the deploy step’s commands as the strings the CI system will actually hand to a shell, and runs them with ssh and tar replaced by stubs that record their arguments. Then it asserts on the argument list ssh received: the destination is there, the key is there, the options arrived as options.

Only the second one is evidence. It does not read the command; it runs it. The first is what you use in the place where evidence is unaffordable, and it is honest about being a rule of style.

Both were watched failing before either was trusted. The executed check keeps the original folded block as a fixture that must fail, so that if the check ever stops being able to see the defect, it says so. A guard nobody has seen go red is a comment.

The rule that travels#

A shell command embedded in a configuration format is a rendered artifact. What you review is the source; what runs is the rendering; and the two can differ in ways the source cannot show you. This is not particular to one CI system. Anywhere YAML carries a command — pipeline steps, container command and args, configuration-management run blocks — the string that executes is the parser’s output, not the text on the screen.

So render it. Parse the file and look at the string, or better, parse the file and run the string against something harmless. It takes three lines of Python to see the problem, and one small test to never see it again.