The Secret to Fast, Reliable Builds

Jun 14, 2012

Download Parallel build execution is a popular technique for reducing software build length, and for good reason. Today, multi-core computers are common, particularly for software developers, so there's horsepower to spare, and it seems seductively easy to dabble with parallel builds: just slap a "-j" onto your make command-line,sit back and enjoy the benefits of a build that's 2, 3 or 4 times faster than it used to be.

Unfortunately, with this simple approach you are guaranteed to run into parallel build problems. There will be incomplete dependencies in your makefiles, tools that don't adequately uniquify their temp file names, and a host of other things that can introduce race conditions into your parallel build.

Sometimes everything will work perfectly, and you get a nice, fast, correct build. Other times, the build blows up in spectacular fashion. Then there are the builds that appear to succeed, but in fact generate incorrect outputs, because a command ran too early and used files generated in a previous build instead of the current build.