Valid YAML, Three Commands: The Folded Block That Reads Right and Runs Wrong
A deploy step that every reader and every parser agreed was one command reached the shell as three. What a YAML folded block actually folds, why no cheap check could see the problem, and the two guards we kept afterwards.
The command that never ran#
This site is built by a CI pipeline and handed to the web host over ssh. The deploy step is one pipeline: tar the build, pipe it to the host. Written the way anyone would write a long command they expect another human to read, it looked like this:
commands:
- >-
tar czf - -C site/dist .
| ssh -T -i /tmp/key
-o BatchMode=yes
deploy@example.com
It is valid YAML. It reads correctly, and anyone reviewing it would be right
about what it says. The first time it ran, ssh printed its usage message,
tar died of a broken pipe, and the pipeline went red.
ssh had been started with no destination, because the destination was no
longer part of its command line. It was two lines further down, being offered
to the shell as a command in its own right.
Three lines of Python#
import yaml
step = yaml.safe_load(open("step.yaml"))
print(repr(step["commands"][0]))
'tar czf - -C site/dist . | ssh -T -i /tmp/key\n -o BatchMode=yes\n deploy@example.com'
Two newlines survived. The >- indicator means “folded”: line breaks become
spaces, so a paragraph can be wrapped in the source and arrive as one line.
But folding applies only between lines at the block’s own indentation. A
line indented further than that is treated as preformatted text, and the line
break in front of it is kept.
That rule exists for a good reason. It is what lets a folded paragraph contain an indented code sample or a list without having it reflowed into mush. The specification is doing exactly what it promises. It is also exactly the wrong behaviour for the one thing people most often put in a folded block in a CI file, which is a long shell command with its continuation lines indented so that a person can see they are continuations.
Run that string under a shell with ssh and tar replaced by stubs that
print their arguments:
tar argv: czf - -C site/dist .
ssh argv: -T -i /tmp/key
sh: 2: -o: not found
sh: 3: deploy@example.com: not found
Three commands. The first is a pipeline into an ssh with nowhere to go.
The readable version is the broken one#
The same command, four ways, parsed with the same library:
| how it is written | what the shell receives |
|---|---|
>-, continuation lines indented further | 3 lines |
>-, every line at the same indentation | 1 line |
|- (literal block) | 4 lines, the second starting with a pipe |
| plain scalar, simply continued on the next lines | 1 line |
The second row works, and looks like a mistake waiting to be “tidied” by the
next person who touches it. The fourth row works too, until the command
contains a # or a : and the plain scalar ends somewhere nobody intended.
The first row is the only one that looks deliberate, and it is the one that
is wrong. Indentation added as a courtesy to the reader changed the program.
Why nothing caught it#
The defect is not in the file. It exists only in the parsed string, and that puts it out of reach of every inexpensive check there is. Parsing succeeds, because nothing is malformed. Review succeeds, and a careful reviewer succeeds hardest of all, because the source says precisely what was meant.
We had even been diligent. Before writing the step we had checked the build
image and confirmed it carried ssh, tar and gzip and did not carry
rsync, which is why the step ships a tarball through a pipe at all. That
was real verification, aimed one layer too low. Checking that the
ingredients of a command exist is not checking the command.
And the failure was quiet, by our own design. The site was mid-move from one backend to another, and the cutover had been arranged so that the old backend kept serving until a new build actually existed on the new host. A deploy that publishes nothing therefore changes nothing a visitor can see. The safety mechanism and the silence were the same mechanism. The only signal was an exit code.
The fix is one line#
Nothing clever. Each command becomes a single line, and the long option list is built up with the shell’s own positional parameters instead of the YAML’s line folding:
commands:
- 'set -- -T -i /tmp/key -o BatchMode=yes'
- 'tar czf - -C site/dist . | ssh "$@" deploy@example.com'
It is uglier than the folded block. It also means the same thing to YAML, to the shell and to the reader, which the folded block did not.
Two guards, and why there are two#
A style check, in the pipeline. The build image has a POSIX shell and
awk and no YAML parser, so the first guard cannot ask what the YAML means.
It checks the shape instead: under a commands: key, every line of content
must begin a new list item. That bans multi-line commands outright without
needing to understand folding. It is crude, it runs as the first command of
the build, and it fails on the original file.
An executed check, on a workstation. The second guard parses the pipeline
file, takes the deploy step’s commands as the strings the CI system will
actually hand to a shell, and runs them with ssh and tar replaced by
stubs that record their arguments. Then it asserts on the argument list
ssh received: the destination is there, the key is there, the options
arrived as options.
Only the second one is evidence. It does not read the command; it runs it. The first is what you use in the place where evidence is unaffordable, and it is honest about being a rule of style.
Both were watched failing before either was trusted. The executed check keeps the original folded block as a fixture that must fail, so that if the check ever stops being able to see the defect, it says so. A guard nobody has seen go red is a comment.
The rule that travels#
A shell command embedded in a configuration format is a rendered artifact.
What you review is the source; what runs is the rendering; and the two can
differ in ways the source cannot show you. This is not particular to one CI
system. Anywhere YAML carries a command — pipeline steps, container
command and args, configuration-management run blocks — the string that
executes is the parser’s output, not the text on the screen.
So render it. Parse the file and look at the string, or better, parse the file and run the string against something harmless. It takes three lines of Python to see the problem, and one small test to never see it again.