SCons — build your software better
A few years ago I worked on a large (60+ man-year) software project, involving C/C++ and Java. The software stack originated from demonstrator software, and its make files had become unmanageable over time. Even worse, they were also unreliable. After evaluating a lot of options we opted for a radical approach: rewrite the entire build system from scratch, based on SCons.
A few weeks ago an ex-colleague contacted me with some questions how we approached things back then. Today, while attending a children's birthday party a discussion on make related issues popped up (can you imagine?). All of this reminded me of how elegant the SCons way of doing things is, which is what I would like to share with you today.
Consider a program that can be built in debug and release mode. Additionally, its sources are neatly organized in subdirectories. This is not an uncommon use case, but anybody who has ever written make files for such a use case knows it requires quite some discipline to keep things tidy (recursive makes files, passing down options, organizing build settings in top-level make files, ...).
This is how it is done in SCons. First, let's take an example directory layout:
example/SConstruct example/src/SConscript example/src/toolkit/SConscript example/src/toolkit/toolkit.c example/src/toolkit/toolkit.h example/src/program/SConscript example/src/program/program.c
So what do we have here?:
- The top level "make" file is called
SConstruct, which is intentionally different from lower level "make" files namedSConscript. You can invoke SCons usingscons -ufrom any directory, and SCons will automatically search directories upwards from your current path for the top levelSConstructfile. - All sources are neatly organized below the
srcdirectory. - There is a program called
programthat depends on a library namedtoolkit.
The top-level SConstruct file contains the following:
# Let's define a common build environment first...
common_env = Environment()
common_env.Append(CPPDEFINES={'VERSION': 1})
# Our release build is derived from the common build environment...
release_env = common_env.Clone()
# ... and adds a RELEASE preprocessor symbol ...
release_env.Append(CPPDEFINES=['RELEASE'])
# ... and release builds end up in the "build/release" dir
release_env.VariantDir('build/release', 'src')
# We define our debug build environment in a similar fashion...
debug_env = common_env.Clone()
debug_env.Append(CPPDEFINES=['DEBUG'])
debug_env.VariantDir('build/debug', 'src')
# Now that all build environment have been defined, let's iterate over
# them and invoke the lower level SConscript files.
for mode, env in dict(release=release_env,
debug=debug_env).iteritems():
env.SConscript('build/%s/SConscript' % mode, {'env': env})
The src/SConscript file merely takes the build environment that is passed to it and invokes the SConscript files from the lower levels:
Import('env')
for subdir in ['toolkit', 'program']:
env.SConscript('%s/SConscript' % subdir, {'env': env})
The toolkit library is built as follows (src/toolkit/SConscript):
Import('env')
# toolkit.h is located in this directory, add it to the include path
env.Append(CPPPATH=['.'])
# Let's declare a library named toolkit, using toolkit.c as its only source
env.Library('toolkit', ['toolkit.c'])
Finally, program is built using src/program/SConscript:
Import('env')
# We require toolkit.h
env.Append(CPPPATH=['../toolkit/'])
env.Program('program', ['program.c'], LIBS=['toolkit'], LIBPATH='../toolkit')
That's it. Let's build:
bash-3.2$ scons scons: Reading SConscript files ... scons: done reading SConscript files. scons: Building targets ... gcc -o build/debug/program/program.o -c -DDEBUG -DVERSION=1 -Ibuild/debug/program -Ibuild/debug/toolkit build/debug/program/program.c gcc -o build/debug/toolkit/toolkit.o -c -DDEBUG -DVERSION=1 -Ibuild/debug/toolkit -Ibuild/debug/toolkit build/debug/toolkit/toolkit.c ar rc build/debug/toolkit/libtoolkit.a build/debug/toolkit/toolkit.o ranlib build/debug/toolkit/libtoolkit.a gcc -o build/debug/program/program build/debug/program/program.o -Lbuild/debug/toolkit -ltoolkit gcc -o build/release/program/program.o -c -DRELEASE -DVERSION=1 -Ibuild/release/program -Ibuild/release/toolkit build/release/program/program.c gcc -o build/release/toolkit/toolkit.o -c -DRELEASE -DVERSION=1 -Ibuild/release/toolkit -Ibuild/release/toolkit build/release/toolkit/toolkit.c ar rc build/release/toolkit/libtoolkit.a build/release/toolkit/toolkit.o ranlib build/release/toolkit/libtoolkit.a gcc -o build/release/program/program build/release/program/program.o -Lbuild/release/toolkit -ltoolkit scons: done building targets.
As you can see:
- Both debug and release versions are built, for both the toolkit and the program.
- The
VERSIONis passed to both the debug and release builds, as it originates from the common build environment. - We did nothing special to indicate that
toolkitshould be built beforeprogram. This dependency was automatically handled by SCons.
But that is not all. How about cleaning up?:
bash$ scons -c scons: Reading SConscript files ... scons: done reading SConscript files. scons: Cleaning targets ... Removed build/debug/program/program.o Removed build/debug/toolkit/toolkit.o Removed build/debug/toolkit/libtoolkit.a Removed build/debug/program/program Removed build/release/program/program.o Removed build/release/toolkit/toolkit.o Removed build/release/toolkit/libtoolkit.a Removed build/release/program/program scons: done cleaning targets.
With SCons you do not need to write cleanup rules. SCons derives a dependency tree from all the declarations inside the SConstruct and SConscript, across all subdirectories in your project. Given that tree it is perfectly clear what files to throw away when cleaning.
But there is more. Ever attempted to parallelize a build? Sure, there is make -j N, but try combining that with a recursive make structure, and you will find yourself jumping through hoops to maintain a proper build order. Not so with SCons. Given a complete and correct dependency tree of your complete project across all subdirectories, parallellizing your build is really just a matter of invoking scons -j N.
Finally, SCons files are in fact Python files, allowing you to use proper software development techniques even for your build system. Compare this to hacking macro's, or even generating make files when using Make).
