George V. Reilly

Diffing a fragment of a file

A while back, I had extracted some code out of a large file into a separate file and made some mod­i­fi­ca­tions. I wanted to check that the dif­fer­ences were minimal. Let's say that the extracted code had been between lines 123 and 456 of large_old_­file.

diff -u <(sed -n '123,456p;457q' large_old_file) new_file

What's happening here?

A similar example: Diff continue.

Bash: echo success of previous command

C-like languages have a ternary operator, cond ? true_re­sult : false_re­sult. Python has true_re­sult if cond else false_re­sult. Bash doesn't have a ternary operator, but there are various workarounds.

I wanted to print succeeded or failed based on the exit code of the previous command in a shell script. In Unix, all programs exit with an integer status code. Successful programs exit with 0; all other values, positive or negative, indicate failure. In Bash, the status code of the previous program is held in $?.

some/command or-other fer example

STATUS="$([ "$?" == 0 ] && echo 'succeeded' || echo 'failed')"
echo "Results: $STATUS"

There are other ways to handle this.

Diff a Transformed File

I wanted to diff two files. One of them needed some seds on each line and sorting. I wanted to do that on the fly, without leaving a massaged in­ter­me­di­ate file lying around.

colordiff --unified <(cat orphaned_permalinks.txt
                        | sed 's@'
                        | sed 's/.aspx$/.html/'

Bash: Bulk Renaming

I had to rename several hundred thousand files today. Thanks to a botched invocation of Im­ageMag­ick, they all looked like unique_pre­fix.png.jpg, whereas we simply wanted unique_pre­fix.jpg.

I found a suitable answer at the Unix Stack­Ex­change. As one of the many variants of parameter sub­sti­tu­tion, Bash supports ${var/Pattern/Re­place­men­t}: “first match of Pattern within var replaced with Re­place­ment.”

for f in *.png.jpg;
    mv $f "${f/.png}"

The target expression could also have been written as "${f/.png.jpg/.jpg}"

Quoting in Bash

Various Bash guides recommend putting quotes around just about everything. I had a script that contained this line:

sudo apt-get install --yes $(cat "$BUILD/install_on_aws_ubuntu.txt")

While refac­tor­ing, I put another set of quotes around the $(cat ...) out of an abundance of caution:

sudo apt-get install --yes "$(cat "$BUILD/install_on_aws_ubuntu.txt")"

Several other changes later, I couldn't figure out why my script had stopped working.

Here's what happened:

$ cat $HOME/tmp/demo.txt

$ echo $(cat "$HOME/tmp/demo.txt")
foo bar quux

$ echo "$(cat $HOME/tmp/demo.txt)"

The new outer quotes retained the newlines; the original replaced newlines with spaces.

The Bash FAQ has more: specif­i­cal­ly Command Substition and Word Splitting.

Bash: Getting and Setting Default Values

Bash has some handy syntax for getting and setting default values. Un­for­tu­nate­ly, it's a collection of punc­tu­a­tion characters, which makes it hard to Google when you can't quite remember the syntax.

Getting a default value using ${var:-fallback}:

# set $LOGDIR to $1 if $1 has a value; otherwise set $LOGDIR to "/var/log"

# use $VERSION unless it's empty or unset; fall back to extracting someprog's version num
build_version=${VERSION:-$(someprog --version | sed 's/[^0-9.]*\([0-9.]*\).*/\1/')}

The colon-dash con­struc­tion is known as the dog's bollocks in typography.

Setting a default value, using ${var:=fallback}:

$ echo $HOME
$ echo ${HOME:=/tmp}
$ unset HOME
$ echo ${HOME:=/tmp}
$ echo $HOME
$ cd; pwd

Note: := uses the new value in two cases. First, when continue.

Checking minimum version numbers in Bash

I worked on a Bash script today that sets up various pre­req­ui­sites for our build. We need a recent version of Docker but our Bamboo build agents are running on Ubuntu 14.04, which has a very old version of Docker. The script upgrades Docker when it's first run. The script may be run more than once during the lifetime of the agent, so the second and subsequent calls should not upgrade Docker.

Basically, I wanted

if $DOCKER_VERSION < 1.9; then upgrade_docker; fi

Un­for­tu­nate­ly, it's not that easy in Bash. Here's what I came up with.

install_latest_docker() {
    if docker --version | python -c "min=[1, 9]; import sys; ↩