Making directories in GNU Make

This article looks at a variety of ways to achieve directory creation in GNU Make and points out a common trap for the unwary.

A simple example

The following simple Makefile builds an object file /out/foo.o from foo.c using the GNU Make built in variable $(COMPILE.C) to make a .o file from a .c by running the compiler.

foo.c is in the same directory as the Makefile, but foo.o is placed in /out/.

.PHONY: all
all: /out/foo.o

/out/foo.o: foo.c
    @$(COMPILE.C) -o $@ $<

This example works fine as lone as /out/ exists, if it does not you\'ll get an error from the compiler along the lines of

Assembler messages:
FATAL: can\'t create /out/foo.o: No such file or directory
make: *** [/out/foo.o] Error 1

Obviously what you\'d like to do is have the Makefile create /out/ automatically if it is missing.   

What not to do

Since GNU Make excels at making things that don\'t exist when necessary it seems obvious to make /out/ a prerequisite of /out/foo.o and have a rule to make the directory.  That way if we need to build /out/foo.o the directory will get created.

Here\'s the reworked Makefile with the directory as a prerequisite and a rule to build the directory using mkdir.   To make things simple I\'ve stored the name of the output directory in variable OUT so that I don\'t have to keep repeating myself, and I\'ve specified the -p option on the mkdir command which will be all the necessary parent directories.  In this case the path is simple, it\'s just /out/ but -p means that mkdir could make a long chain of directories in one go.

OUT = /out

.PHONY: all
all: $(OUT)/foo.o

$(OUT)/foo.o: foo.c $(OUT)/
    @$(COMPILE.C) -o $@ $<

$(OUT)/:
    mkdir -p $@

This works well for this simple example, but there\'s a major problem.  Since the timestamp on a directory is typically updated when any of the files inside the directory are updated this Makefile does too much work.

For example, just touching a random file inside /out/ forces a rebuild of /out/foo.o.  In a complex example this could mean that many object files are rebuilt for no good reason, just because other files were rebuilt in the same directory.

Solution 1: build the directory when the Makefile is parsed

A simple solution to the problem above is to just create the directory when the Makefile is parsed.   A quick call to $(shell) can achieve that.  Before any targets are created or commands run the Makefile is read and parsed.  If you put $(shell mkdir -p $(OUT)) somewhere in the Makefile then GNU Make will run the mkdir every time the Makefile is loaded.

OUT = /out

.PHONY: all
all: $(OUT)/foo.o

$(OUT)/foo.o: foo.c
    @$(COMPILE.C) -o $@ $<

$(shell   mkdir -p $(OUT))

The disadvantage of the simplicty is that if there are many directories to be created this could be slow.  And GNU Make is doing unnecessary work since it will attempt to build the directories every time you type make.

Solution 2: build the directory when all is built

A related solution is to only build the directory when all is being built.  This means that the directories won\'t get created every time the Makefile is parsed (which could avoid unnecessary work when you type make clean or make depend).

OUT = /out

.PHONY: all
all: make_directories $(OUT)/foo.o

$(OUT)/foo.o: foo.c
    @$(COMPILE.C) -o $@ $<

.PHONY: make_directories
make_directories: $(OUT)/

$(OUT)/:
    mkdir -p $@

TODO use a %/ pattern rule

Another disadvantage of this technique is that you must specify make_directories as a prerequisite of any target that the user might specify after make.  If you don\'t you could run into the situation where the directories have not been built.

Solution 3: use a directory \'marker\' file

If you look back at the original broken solution there\'s one rather nice feature: it builds the directory needed for a specific target and only that directory.   In a more complex example (where there were many such directories to be built) it would be nice to be able to use something like that solution while avoiding the problem of constant rebuilds as the timestamp on the directory changes.

To do that you can store a special empty file, that I call a \'marker\' file in the directory and use that as the prerequisite.  Since it\'s a normal file, normal GNU Make rebuilding rules apply and its timestamp is not affected by changes in its directory.

If you add a rule to build the marker file (and to ensure that its directory exists), it\'s possible to specify a directory as a prerequisite, by specifying the marker file as a proxy for the directory and get the desired effect.

In this example the marker file is called .f

OUT = /out

.PHONY: all
all: $(OUT)/foo.o

$(OUT)/foo.o: foo.c $(OUT)/.f
    @$(COMPILE.C) -o $@ $<

$(OUT)/.f:
    mkdir -p $(dir $@)
    touch $@

Notice how the rule to build $(OUT)/.f both creates the directory, if necessary, and touches the .f file.  Since the target is a file (the .f) it can safely be used as a prerequisite in the $(OUT)/foo.o rule.

The $(OUT)/.f rule uses the GNU Make function $(dir FILE) to extract the directory portion of the target (which is the path to the .f file) and passes that directory to mkdir.

The only disadvantage here is that it\'s necessary to specify the .f files for every rule that builds a target in a directory that might need creating.

Solution 4: use an \'order-only\' prerequisite to build the directory

If you are using GNU Make 3.80 or above another solution is to use an \'order-only\' prerequisite.  An order-only prerequiste is a prerequisite of a target that will be built before the target, but whether it is rebuilt or not does not affect the rebuilding of the target.  So GNU Make will cause an order-only prerequisite to be rebuilt but it will only affect the order in which targets are handled, and not the relationship between targets that causes rebuilding.

This is exactly what we\'d like in the original broken example: make sure that the directory has been rebuilt, but if its timestamp changes don\'t rebuild the .o file. 

Order-only prerequisites are specified by prefixing them with the bar symbol | before the prerequisites (and after any normal prerequisites have been specified). 

In fact a single character change to the broken example makes it work correctly under GNU Make 3.80 and above:

OUT = /out

.PHONY: all
all: $(OUT)/foo.o

$(OUT)/foo.o: foo.c | $(OUT)/
    @$(COMPILE.C) -o $@ $<

$(OUT)/:
    mkdir -p $@

Solution 5: use pattern rules, second expansion and a marker file

In a typical Makefile (i.e. not the sort of simple example people come up with in articles like this) targets are usually built using pattern rules.  For example, the simple example would more likely be written using a pattern rule like this:

OUT = /out

.PHONY: all
all: $(OUT)/foo.o

$(OUT)/%.o: %.c
    @$(COMPILE.C) -o $@ $<

It would be nice to be able to change the pattern rule so that it builds the directory automatically using marker files.

In GNU Make 3.81 there\'s an exciting new feature called \'second expansion\' (which is enabled by specifying the .SECONDEXPANSION target in the Makefile).  With second expansion the prerequisite list of any rule under goes a second expansion (the first expansion happens when the Makefile is read) just before the rule is used.  By escaping any $ signs with a second $ it\'s possible to use GNU Make automatic variables in the prerequisite list.

Using a marker file for each directory and second expansion you can create a Makefile that automatically creates directories only when necessary with a simple addition to the prerequisite list of any rule:

OUT = /tmp/out

.SECONDEXPANSION:

all: $(OUT)/foo.o

$(OUT)/%.o: %.c $$(@D)/.f
    @$(COMPILE.C) -o $@ $<

%/.f:
    mkdir -p $(dir $@)
    touch $@

.PRECIOUS: %/.f

The pattern rule used to make .o files has a special prerequisite $$(@D)/.f which uses the second expansion feature to obtain the directory in which the target (from $@ using the D modifier) is to be built.

That directory will be built by the %/.f pattern rule that knows how to build a .f file (and the containing directory).   Notice that the .f files are marked as precious so that GNU Make will not delete them.  Without this line the .f files are considered to be useless intermediate files and would be cleaned up by GNU Make on exit.

Conclusion

That's enough ways to create directories in GNU Make.  I\'ll be back in August with a new column about GNU Make and parallelism.