One thing I like to do with my code is make sure that it's refactored into manageable pieces. However, when it comes to building the software, I find that whatever build automation software I end up using (lately it's been GNU Make or SCons) ends up becoming a complete mess. The input files look like long scripts that seem to defy easy refactoring. I'd like to be able to refactor them in some way, but the concept of "function" doesn't quite behave the same way in some build automation software as it does in a programming language, so I find it difficult to write manageable Makefiles or SConscript files for any projects that are moderately complicated.
Does anyone have any advice on writing manageable input files for build automation software? Software-agnostic advice would be best, but even advice about a specific build automation tool would be helpful, particularly Make or SCons, since that's what I've been using for projects.
Edit: As Thorbjørn has pointed out, I should add some context and usage examples. I'm finishing up my PhD in chemical engineering and I do research in computational science. (I'm a pro tem mod over at SciComp.SE, for those of you who visit.) My projects typically involve a mix of compiled languages (C, C++, Fortran) that do some of the heavy lifting, scripting languages (Python, Perl) for prototyping, and occasionally, domain-specific languages for prototyping or technical purposes.
I've added two examples below, roughly in the 250-line range. The problem for me is generally a lack of modularity. Some of these projects can be organized into modular units, and it'd be nice to abstract the parts of the build along those lines to make it easier for me and future maintainers to keep track of. Breaking each script up into multiple files has been one solution I've been toying around with in my head.
The second example is particularly important, because I'm going to have to a large number of files to it soon.
Here's what a 265-line Makefile
might look like for me, taken from an actual project, and organized as best I can:
#!/usr/bin/make
#Directory containing DAEPACK library folder
daepack_root = .
library = $(daepack_root)/lib
wrappers = $(daepack_root)/Wrappers/DSL48S
c_headers = parser.h problemSizes.h
f77_headers=problemSizes.f commonParam.f
f90_headers=problemSizes.f commonParam.f90
includes = -I. -Iinclude -I/usr/include/glib-2.0 \
-I/usr/lib/glib-2.0/include -I/usr/include/libxml2 \
-I/usr/include/libgdome -I/usr/include/gtest/
#Fortran 77 environment variables
f77=gfortran
fflags=-ggdb -cpp -fno-second-underscore --coverage -falign-commons \
-mcmodel=large -fbacktrace -pg
flibs=
#Fortran 90 environment variables
f90=gfortran
f90flags=-ggdb -cpp -fno-second-underscore --coverage -falign-commons \
-mcmodel=large -fbacktrace -pg
f90libs=
#C environment flags
cc=gcc
cflags=-ggdb --coverage $(includes) -mcmodel=large
clibs=
#Libraries for linking
libs=-L$(library) -ldaepack_sparse -lblas -llapack -ldl -lg2c \
-lgdome -lxml2 -lgtest -lcunit -lcholmod -lamd -lcolamd -lccolamd \
-lmetis -lspqr -lm -lblas -llapack -lstdc++ -lpcre
#Object files
objs=main.o $(dsl48sObjs) $(gdxObjs)
gdxObjs = gdxf9def.o gdxf9glu.o gamsglobals_mod.o
commonObjs=libdsl48s_model.sl cklib.o parser.o $(gdxObjs)
originalModelObjs=originalModel.o dsl48sChemkinModule.o $(commonObjs)
cspSlowModelObjs=cspSlowModel.o dsl48sChemkinModuleSlow.o cspModule.o \
$(commonObjs)
orthoProjModelObjs=orthoProjModel.o dsl48sChemkinModuleOrthoProj.o \
orthoProjModule.o basisModule.o spqrUtility.o $(commonObjs)
#Shell environment variable definitions for FUnit
FCFLAGS := $(f90flags)
LDFLAGS := libdsl48s_model.sl cklib.o gdxf9glu.o parser.o spqrUtility.o \
$(libs)
misc=*table *size.f
output=*.out
#Ftncheck flags for static analysis of Fortran 77 code
ftnchekflags= -declare -include=. -library -style=block-if,distinct-do,do-enddo,end-name,goto,labeled-stmt,structured-end
all: ckinterp.exe parserTest.exe originalModel.exe cspSlowModel.exe \
orthoProjModel.exe spqrUtilityTest.exe
#Check code style with lexical analyzer
@echo Checking program style...
ftnchek $(ftnchekflags) rhs.f
ftnchek $(ftnchekflags) resorig.f
ftnchek $(ftnchekflags) res.f
# ftnchek $(ftnchekflags) cklib.f
# ftnchek $(ftnchekflags) ckinterp.f
#Set up baseline coverage data file
@echo Set up baseline coverage data file
lcov -c -i -d . -o conpDSL48Sbase.info
#Run unit test on cspModule.f90
@echo Running unit tests on cspModule.f90...
funit cspModule
#Generate test coverage data for cspModule.f90
@echo Generating test coverage data from cspModule.f90 tests...
lcov -c -d . -o conpDSL48ScspTest.info
#Run unit test on orthoProjModule.f90
@echo Running unit tests on orthoProjModule.f90...
funit orthoProjModule
#Generate test coverage data for orthoProjModule.f90
@echo Generating test coverage data from orthoProjModule.f90 tests...
lcov -c -d . -o conpDSL48SgenProjTest.info
#Run unit tests on the parser C library
@echo Running unit tests on parser in C...
-G_SLICE=always-malloc G_DEBUG=gc-friendly valgrind -v --tool=memcheck \
--leak-check=full --show-reachable=yes --leak-resolution=high \
--num-callers=20 --log-file=parserTest.vgdump \
./parserTest.exe > parserTest.log
#Generate test coverage data for the parser wrapper C library
@echo Generating test coverage data for the parser in C...
lcov -c -d . -o conpDSL48SparserTest.info
#Run unit tests on the SparseQR C library
@echo Running unit tests on SparseQR library in C...
./spqrUtilityTest.exe
#Generate test coverage data for the SparseQR C library
@echo Generating test coverage data for the SparseQR C library...
lcov -c -d . -o conpDSL48SsparseTest.info
#Run unit test on basisModule.f90
@echo Running unit tests on basisModule.f90...
funit basisModule
#Generate test coverage data for basisModule.f90
@echo Generating test coverage data from basisModule.f90 tests...
lcov -c -d . -o conpDSL48SbasisMod.info
#Combine test coverage data
@echo Combine baseline and test coverage data...
lcov -a conpDSL48Sbase.info \
-a conpDSL48ScspTest.info \
-a conpDSL48SgenProjTest.info \
-a conpDSL48SbasisMod.info \
-a conpDSL48SparserTest.info \
-a conpDSL48SsparseTest.info \
-o conpDSL48Stotal.info
#Post-process to remove coverage statistics from automatically
#generated source code.
@echo Removing coverage statistics for automatically generated source...
lcov -r conpDSL48Stotal.info basisModule_fun.f90 \
ckinterp.f cklib.f cspModule_fun.f90 davisSkodjeAd.f90 \
davisSkodjeJac.f90 davisSkodjeRes.f90 davisSkodjeRhs.f90 \
davisSkodjeSp.f90 gdxf9def.f90 gdxf9glu.c orthoProjModule_fun.f90 \
jac.f jacorig.f resad.f resadp.f resorigad.f resorigadp.f ressp.f \
resorigsp.f senrhs.f senrhsorig.f TestRunner.f90 \
-o conpDSL48Stotal.info
#Generate HTML report of coverage data
@echo Generate HTML report of coverage data...
genhtml conpDSL48Stotal.info
@echo Open "index.html" in browser for coverage results!
originalModel.exe: $(originalModelObjs) $(f90_headers) $(f77_headers) \
$(c_headers)
$(f90) $(f90flags) -o originalModel.exe $(originalModelObjs) $(libs)
originalModel.o: dsl48sChemkinModule.o $(commonObjs) $(f77_headers) \
$(f90_headers) $(c_headers)
$(f90) $(f90flags) -c -o originalModel.o originalModel.f90
cspSlowModel.exe: $(cspSlowModelObjs) $(f90_headers) $(f77_headers) \
$(c_headers)
$(f90) $(f90flags) -o cspSlowModel.exe $(cspSlowModelObjs) $(libs)
cspSlowModel.o: dsl48sChemkinModuleSlow.o cspModule.o $(commonObjs) \
$(c_headers) $(f77_headers)
$(f90) $(f90flags) -c -o cspSlowModel.o cspSlowModel.f90
orthoProjModel.exe: $(orthoProjModelObjs) $(f90_headers) $(f77_headers) \
$(c_headers) resOrthoFast.o
$(f90) $(f90flags) -o orthoProjModel.exe $(orthoProjModelObjs) \
resOrthoFast.o $(libs)
orthoProjModel.o: dsl48sChemkinModuleOrthoProj.o orthoProjModule.o $(commonObjs) \
$(c_headers) $(f90_headers) $(f77_headers) resOrthoFast.o basisModule.o
$(f90) $(f90flags) -c -o orthoProjModel.o orthoProjModel.f90
dsl48sChemkinModule.o: dsl48sChemkinModule.f90 cklib.o problemSizes.h \
parser.o $(c_headers) $(f90_headers)
$(f90) $(f90flags) -c -o dsl48sChemkinModule.o dsl48sChemkinModule.f90
dsl48sChemkinModuleSlow.o: dsl48sChemkinModuleSlow.f90 cspModule.o cklib.o \
problemSizes.h parser.o $(c_headers) $(f90_headers)
$(f90) $(f90flags) -c -o dsl48sChemkinModuleSlow.o \
dsl48sChemkinModuleSlow.f90
dsl48sChemkinModuleOrthoProj.o: dsl48sChemkinModuleOrthoProj.f90 \
orthoProjModule.o basisModule.o cklib.o problemSizes.h \
parser.o $(c_headers) $(f90_headers)
$(f90) $(f90flags) -c -o dsl48sChemkinModuleOrthoProj.o \
dsl48sChemkinModuleOrthoProj.f90
basisModule.o: basisModule.f90 cklib.o spqrUtility.o commonParam.f90
$(f90) $(f90flags) -c -o basisModule.o basisModule.f90
spqrUtility.o: spqrUtility.h spqrUtility.c
$(cc) $(cflags) -c -o spqrUtility.o spqrUtility.c
spqrUtilityTest.exe: spqrUtilityTest.o spqrUtility.o
$(cc) $(cflags) -o spqrUtilityTest.exe spqrUtilityTest.o \
spqrUtility.o $(libs)
spqrUtilityTest.o: spqrUtilityTest.c spqrUtility.o
$(cc) $(cflags) -c -o spqrUtilityTest.o spqrUtilityTest.c
cklib.o: cklib.f ckstrt.f
$(f77) $(fflags) -c -o cklib.o cklib.f
ckinterp.exe: ckinterp.o
$(f77) $(fflags) -o ckinterp.exe ckinterp.o
ckinterp.o: ckinterp.f
$(f77) $(fflags) -c -o ckinterp.o ckinterp.f
#Recursive makefile inherited from previous graduate students
libdsl48s_model.sl: $(f77_headers) cklibDAEPACK.f
cp $(wrappers)/makefile model.mk
make -f model.mk
resOrthoFast.o: libdsl48s_model.sl
$(f90) $(f90flags) -c -o resOrthoFast.o resOrthoFast.f90
problemSizes.f: problemSizes.fpp problemSizes.h
cpp problemSizes.fpp problemSizes.f
perl -p -i.bak -we 's/# /! /;' problemSizes.f
commonParam.f90: commonParam.f
perl -p -i.bak -we 's/^#/!/;' commonParam.f
echo "commonParam t f t fpp" | pref77tof90
echo "commonParam /" | f77tof90
perl -p -i.bak -we 's/integer a/!integer a/;' commonParam.f
perl -p -i.bak -we 's/END //;' commonParam.f90
commonParam.f: commonParam.fpp problemSizes.h
cpp commonParam.fpp commonParam.f
perl -p -i.bak -we 's/^#/!/;' commonParam.f
cspModule.o: cspModule.f90
$(f90) $(f90flags) -c -o cspModule.o cspModule.f90
orthoProjModule.o: gamsglobals_mod.o gdxf9def.o gdxf9glu.o orthoProjModule.f90 \
formatLabels.f90
$(f90) $(f90flags) -c -o orthoProjModule.o orthoProjModule.f90
gdxf9def.o: gdxf9def.f90
$(f90) $(f90flags) -c -o gdxf9def.o gdxf9def.f90
gdxf9glu.o: gdxf9glu.c gdxf9def.o
#64-bit version of wrappers (with underscores)
$(cc) $(cflags) -DCIA_LEX -DAPIWRAP_LCASE_DECOR -c -o \
gdxf9glu.o gdxf9glu.c
#64-bit version of wrappers (without underscores, for C interoperability)
# $(cc) $(cflags) -DCIA_LEX -DAPIWRAP_LCASE_NODECOR -c gdxf9glu.c
#32-bit version of wrappers
# $(cc) $(cflags) -DAPIWRAP_LCASE_DECOR -c gdxf9glu.c -Iinclude
gamsglobals_mod.o: gamsglobals_mod.f90 gdxf9def.o gdxf9glu.o
$(f90) $(f90flags) -c gamsglobals_mod.f90
parser.o: parser.c $(c_headers)
$(cc) $(cflags) -c -o parser.o parser.c
parserTest.exe: parserTest.o parser.o
$(cc) $(cflags) -o parserTest.exe parser.o \
parserTest.o $(libs)
parserTest.o: parserTest.cpp parser.o
$(cc) $(cflags) -c -o parserTest.o parserTest.cpp
clean:
-rm *.bak
-rm *.f77
-rm *.log
-rm commonParam.f90
-rm problemSizes.f
-rm commonParam.f
-make clean -f model.mk
-rm model.mk
-rm *.o
-rm *.mod
-rm $(misc)
-rm *.exe
-funit --clean
-rm *.gcno
-rm *.gcda
-rm *.info
-rm *.png
-rm *.html
-rm *.css
-rm -rf html
-rm *.pyc
-rm *.lst
Here's a 245-line SConstruct
file that I'm currently trying to organize for a project that's roughly as complex:
## \file SConstruct
# \brief Compiles the library and compiles tests.
#
import SCons
## \brief Build up directory names of each COIN library from package names
# and versions.
#
## Overall SCons environment
#
env = Environment();
flags = []
## Compile using debug versions?
#
debug = True
debugString = '-debug'
debugFlags = ['-ggdb']
dynamicLinkFlag = '-Wl,-rpath,'
if debug:
flags += debugFlags
## Compile Google Test from scratch.
#
GTestVersion = '1.6.0'
GTestStem = 'gtest-' + GTestVersion
GTestBuildIncDir = [GTestStem,
GTestStem + '/include',
]
GTestAllLib = env.Library('lib/libgtest.a', 'gtest-1.6.0/src/gtest-all.cc',
CPPPATH = GTestBuildIncDir,
CXXFLAGS = flags)
GTestMainLib = env.Library('lib/libgtest_main.a',
'gtest-1.6.0/src/gtest_main.cc',
CPPPATH = GTestBuildIncDir,
CXXFLAGS = flags)
GTestIncDir = GTestStem + '/include/gtest'
GTestLibDir = 'lib'
GTestLibFlags = ['gtest', 'gtest_main', 'pthread']
## Armadillo matrix library
#
ArmadilloLibFlags = ['armadillo'];
## Quick reminder of SCons flags:
# CPPPATH = path of headers (include directories)
# LIBPATH = path of libraries
# LIBS = flags of libraries
# CXXFLAGS = C++ compilation flags
#
## Locations of libraries installed on system in standard locations
#
StdIncDir = '/usr/include'
StdLibDir = '/usr/lib'
## Configuration information for COIN libraries
#
CoinUtilsVersion = '2.6.4'
ClpVersion = '1.12.0'
OsiVersion = '0.103.0'
CbcVersion = '2.5.0'
## Some standard directory locations of COIN libraries, with slashes added for
# for convenience.
#
CoinLibLocation = '/usr/local/COIN/'
StdCoinIncDir = '/include/coin'
StdCoinLibDir = '/lib'
CoinUtilsStem = 'CoinUtils-' + CoinUtilsVersion
ClpStem = 'Clp-' + ClpVersion
OsiStem = 'Osi-' + OsiVersion
CbcStem = 'Cbc-' + CbcVersion
if debug:
CoinUtilsStem += debugString
CbcStem += debugString
ClpStem += debugString
OsiStem += debugString
## Build up include directory names for COIN projects from constituent parts.
#
CoinUtilsIncDir = CoinLibLocation + CoinUtilsStem + StdCoinIncDir
ClpIncDir = CoinLibLocation + ClpStem + StdCoinIncDir
OsiIncDir = CoinLibLocation + OsiStem + StdCoinIncDir
CbcIncDir = CoinLibLocation + CbcStem + StdCoinIncDir
## Build up library names from COIN projects from constituent parts
#
CoinUtilsLibDir = CoinLibLocation + CoinUtilsStem + StdCoinLibDir
ClpLibDir = CoinLibLocation + ClpStem + StdCoinLibDir
OsiLibDir = CoinLibLocation + OsiStem + StdCoinLibDir
CbcLibDir = CoinLibLocation + CbcStem + StdCoinLibDir
## CPLEX
#
CpxStem = '/opt/ibm/ILOG/CPLEX_Studio_Academic123/cplex/'
CpxIncDir = CpxStem + 'include/ilcplex'
CpxLibDir = CpxStem + 'lib/x86-64_sles10_4.1/static_pic'
## Gurobi
#
GrbStem = '/opt/gurobi460/linux64/'
GrbIncDir = GrbStem + 'include'
GrbLibDir = GrbStem + 'lib'
OsiLibFlags = ['Osi', 'CoinUtils']
ClpLibFlags = ['Clp', 'OsiClp']
CbcLibFlags = ['Cbc', 'Cgl']
OsiCpxLibFlags = ['OsiCpx']
OsiGrbLibFlags = ['OsiGrb']
CpxLibFlags = ['cplex', 'ilocplex', 'pthread', 'm']
GrbLibFlags = ['gurobi_c++', 'gurobi46', 'pthread', 'm']
milpIncDirs = [CoinUtilsIncDir,
ClpIncDir,
OsiIncDir,
CbcIncDir,
CpxIncDir,
GrbIncDir,
GTestIncDir,
]
milpLibDirs = [CoinUtilsLibDir,
ClpLibDir,
OsiLibDir,
CbcLibDir,
CpxLibDir,
GrbLibDir,
GTestLibDir,
]
milpLibFlags = [OsiCpxLibFlags,
OsiGrbLibFlags,
CbcLibFlags,
ClpLibFlags,
OsiLibFlags,
CpxLibFlags,
GrbLibFlags,
GTestLibFlags,
]
##milpSolver = env.Object('milpSolver.cpp',
## CPPPATH = milpIncDirs,
## LIBPATH = milpLibDirs,
## CXXFLAGS = flags)
milpSolverTest = env.Program('milpSolverUnitTest',
['milpSolverTest.cpp',
'milpSolver.cpp'],
CPPPATH = milpIncDirs,
LIBPATH = milpLibDirs,
LIBS = milpLibFlags,
CXXFLAGS = flags,
LINKFLAGS = ['-Wl,-rpath,' + OsiLibDir])
env.Depends(milpSolverTest, [GTestAllLib, GTestMainLib])
## Chemkin source directories and files
#
ChemkinSourceDir = '/mnt/hgfs/DataFromOldLaptop/Data/ModelReductionResearch/Papers/AdaptiveChemistryPaper/AdaptiveChemistry/NonOpenSource/ChemkinII/';
ChemkinSourceList = ['cklib.f', 'pcmach.f','tranlib.f']
ChemkinSourceList = [ChemkinSourceDir + FileName
for FileName in ChemkinSourceList]
env.Depends('cklib.f','ckstrt.f')
## Cantera include directorie
#
CanteraStem = '/usr/local/cantera'
if debug:
CanteraStem += debugString
CanteraIncDir = CanteraStem + '/include/cantera'
CanteraLibDir = CanteraStem + '/lib'
CanteraTestingFlags = ['kinetics', 'thermo', 'tpx', 'ctbase', 'm',]
CanteraLibFlags = ['user', 'oneD', 'zeroD', 'equil', 'kinetics', 'transport',
'thermo', 'ctnumerics', 'ctmath', 'tpx', 'ctspectra',
'converters', 'ctbase', 'cvode', 'ctlapack', 'ctblas',
'ctf2c', 'ctcxx', 'ctf2c', 'm', 'm', 'stdc++']
CxxFortranFlags = ['g2c', 'gfortran'];
chemSolverIncDir = [CanteraIncDir,
StdIncDir,
'/usr/local/include',
GTestIncDir,
]
chemSolverLibDir = [StdLibDir,
CanteraLibDir,
GTestLibDir,
]
chemSolverLibFlags = [GTestLibFlags,
CxxFortranFlags,
CanteraLibFlags,
ArmadilloLibFlags,
]
chemSolverTest = env.Program('chemSolverUnitTest',
['chemSolverTest.cpp',
'chemSolver.cpp',
'ckwrapper.f90'] + ChemkinSourceList,
CPPPATH = chemSolverIncDir,
LIBPATH = chemSolverLibDir,
LIBS = chemSolverLibFlags,
CXXFLAGS = flags,
FORTRANFLAGS = flags,
F90FLAGS = flags)
env.Depends(chemSolverTest, [GTestAllLib, GTestMainLib])
#env.AddPostAction(milpSolverTest, milpSolverTest[0].abspath)
testAlias = env.Alias('test', [milpSolverTest, chemSolverTest])
AlwaysBuild(testAlias)
ckInterp = env.Program('ckinterp', ChemkinSourceDir + 'ckinterp.f')
canteraGTestLibFlags = CanteraTestingFlags + GTestLibFlags
#canteraGTestLibFlags = ['kinetics', 'thermo', 'tpx',
# 'ctbase', 'm', 'gtest', 'gtest_main', 'pthread']
canteraGTest = env.Program('canteraGTest',
'canteraGTest.cpp',
CPPPATH = chemSolverIncDir,
LIBPATH = chemSolverLibDir,
LIBS = canteraGTestLibFlags,
CXXFLAGS = flags)
env.Depends(canteraGTest, [GTestAllLib, GTestMainLib])
canteraMemTestLibFlags = CanteraTestingFlags
canteraMemTest = env.Program('canteraMemTest',
'canteraMemTest.cpp',
CPPPATH = chemSolverIncDir,
LIBPATH = chemSolverLibDir,
LIBS = canteraMemTestLibFlags,
CXXFLAGS = flags)
Best Answer
Refactoring strategy
In addition to the comments by dietbuddha and ThorbjørnRavnAnderson, another way to refactor build scripts is to separate them into multiple files. How you do this depends on the build system.
For Make, it's as simple as using the
include
command, as recommended in "Recursive Make Considered Harmful". This directive works just like#include
in the C preprocessor and processes the included file as if it had been cut and pasted in place of theinclude
command. Using theinclude
command, it's possible to refactor your mainMakefile
by moving modular pieces into sub-Makefile
s.CMake has a similar command.
SCons requires a similar type of approach with different commands. The basic idea of splitting up the build script into a master script and several smaller sub-scripts stays the same, but instead of directly including the text of the smaller scripts in the master build script, SCons treats the smaller scripts like separate namespaces (because SCons uses Python instead of the shell). SCons
Environment
objects have a method calledSConscript()
that enables you to import objects from anSConstruct
file to subsidiary files calledSConscript
files that can be used to refactor your build script. The basic idea ofSConscript
files and theSConscript()
command can be found here on the SCons Wiki. Examples of how to useSConscripts
in hierarchical builds can be found here in the SCons User Guide.Extrapolating from these three examples, it seems like the general strategy is to refactor a build script by splitting it into a master script that calls several files. How one does this is idiomatic to the particular build automation software used.
SCons Example, Revisited
Taking the
SConstruct
file above, I moved all of the configuration information into a module calledbuild_config.py
. All literals live in the global namespace, which could be dangerous for Python, but that's easily (though somewhat tediously) fixable. I checked ahead to make sure that I had no name clashes with__builtin__
(the Python built-in namespace, so I'm not overwriting any important objects).The main
SConstruct
file calls a bunch ofSConscript
files to build modules containing the major features I have in my code. Moving most of the build commands toSConscript
files makes theSConstruct
file really simple:Then all of three
SConscript
files have the extension.scons
and break up the project into modules representing different functionality.Google Test
SConscript
file:Mixed-integer linear programming solver
SConscript
file:Chemistry engine
SConscript
file:The combination of these five files is probably a bit longer than the original long file, but it's easier for me to manage because I can separate the build into a configuration file, and a bunch of mostly uncoupled units, so I don't have to keep track of the whole build in my mind at one time.