Raymond's Blog

SCons — build your software better

A few years ago I worked on a large (60+ man-year) software project, involving C/C++ and Java. The software stack originated from demonstrator software, and its make files had become unmanageable over time. Even worse, they were also unreliable. After evaluating a lot of options we opted for a radical approach: rewrite the entire build system from scratch, based on SCons.

A few weeks ago an ex-colleague contacted me with some questions how we approached things back then. Today, while attending a children's birthday party a discussion on make related issues popped up (can you imagine?). All of this reminded me of how elegant the SCons way of doing things is, which is what I would like to share with you today.

Consider a program that can be built in debug and release mode. Additionally, its sources are neatly organized in subdirectories. This is not an uncommon use case, but anybody who has ever written make files for such a use case knows it requires quite some discipline to keep things tidy (recursive makes files, passing down options, organizing build settings in top-level make files, ...).

This is how it is done in SCons. First, let's take an example directory layout:

example/SConstruct
example/src/SConscript
example/src/toolkit/SConscript
example/src/toolkit/toolkit.c
example/src/toolkit/toolkit.h
example/src/program/SConscript
example/src/program/program.c

So what do we have here?:

  • The top level "make" file is called SConstruct, which is intentionally different from lower level "make" files named SConscript. You can invoke SCons using scons -u from any directory, and SCons will automatically search directories upwards from your current path for the top level SConstruct file.
  • All sources are neatly organized below the src directory.
  • There is a program called program that depends on a library named toolkit.

The top-level SConstruct file contains the following:

# Let's define a common build environment first...
common_env = Environment()
common_env.Append(CPPDEFINES={'VERSION': 1})

# Our release build is derived from the common build environment...
release_env = common_env.Clone()
# ... and adds a RELEASE preprocessor symbol ...
release_env.Append(CPPDEFINES=['RELEASE'])
# ... and release builds end up in the "build/release" dir
release_env.VariantDir('build/release', 'src')

# We define our debug build environment in a similar fashion...
debug_env = common_env.Clone()
debug_env.Append(CPPDEFINES=['DEBUG'])
debug_env.VariantDir('build/debug', 'src')

# Now that all build environment have been defined, let's iterate over
# them and invoke the lower level SConscript files.
for mode, env in dict(release=release_env, 
    	       	      debug=debug_env).iteritems():
    env.SConscript('build/%s/SConscript' % mode, {'env': env})

The src/SConscript file merely takes the build environment that is passed to it and invokes the SConscript files from the lower levels:

Import('env')

for subdir in ['toolkit', 'program']:
    env.SConscript('%s/SConscript' % subdir, {'env': env})

The toolkit library is built as follows (src/toolkit/SConscript):

Import('env')
# toolkit.h is located in this directory, add it to the include path
env.Append(CPPPATH=['.'])
# Let's declare a library named toolkit, using toolkit.c as its only source
env.Library('toolkit', ['toolkit.c'])

Finally, program is built using src/program/SConscript:

Import('env')
# We require toolkit.h
env.Append(CPPPATH=['../toolkit/'])
env.Program('program', ['program.c'], LIBS=['toolkit'], LIBPATH='../toolkit')

That's it. Let's build:

bash-3.2$ scons
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
gcc -o build/debug/program/program.o -c -DDEBUG -DVERSION=1 -Ibuild/debug/program -Ibuild/debug/toolkit build/debug/program/program.c
gcc -o build/debug/toolkit/toolkit.o -c -DDEBUG -DVERSION=1 -Ibuild/debug/toolkit -Ibuild/debug/toolkit build/debug/toolkit/toolkit.c
ar rc build/debug/toolkit/libtoolkit.a build/debug/toolkit/toolkit.o
ranlib build/debug/toolkit/libtoolkit.a
gcc -o build/debug/program/program build/debug/program/program.o -Lbuild/debug/toolkit -ltoolkit
gcc -o build/release/program/program.o -c -DRELEASE -DVERSION=1 -Ibuild/release/program -Ibuild/release/toolkit build/release/program/program.c
gcc -o build/release/toolkit/toolkit.o -c -DRELEASE -DVERSION=1 -Ibuild/release/toolkit -Ibuild/release/toolkit build/release/toolkit/toolkit.c
ar rc build/release/toolkit/libtoolkit.a build/release/toolkit/toolkit.o
ranlib build/release/toolkit/libtoolkit.a
gcc -o build/release/program/program build/release/program/program.o -Lbuild/release/toolkit -ltoolkit
scons: done building targets.

As you can see:

  • Both debug and release versions are built, for both the toolkit and the program.
  • The VERSION is passed to both the debug and release builds, as it originates from the common build environment.
  • We did nothing special to indicate that toolkit should be built before program. This dependency was automatically handled by SCons.

But that is not all. How about cleaning up?:

bash$ scons -c
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Cleaning targets ...
Removed build/debug/program/program.o
Removed build/debug/toolkit/toolkit.o
Removed build/debug/toolkit/libtoolkit.a
Removed build/debug/program/program
Removed build/release/program/program.o
Removed build/release/toolkit/toolkit.o
Removed build/release/toolkit/libtoolkit.a
Removed build/release/program/program
scons: done cleaning targets.

With SCons you do not need to write cleanup rules. SCons derives a dependency tree from all the declarations inside the SConstruct and SConscript, across all subdirectories in your project. Given that tree it is perfectly clear what files to throw away when cleaning.

But there is more. Ever attempted to parallelize a build? Sure, there is make -j N, but try combining that with a recursive make structure, and you will find yourself jumping through hoops to maintain a proper build order. Not so with SCons. Given a complete and correct dependency tree of your complete project across all subdirectories, parallellizing your build is really just a matter of invoking scons -j N.

Finally, SCons files are in fact Python files, allowing you to use proper software development techniques even for your build system. Compare this to hacking macro's, or even generating make files when using Make).

Attachments

Meta

Published: Aug. 9, 2011

Author: pennersr

Comments:  

Word Count: 751

Previous: Statetris Reloaded

Bookmark and Share

Tags

build development make scons stack

Comments powered by Disqus