alexrp’s blog

ramblings usually related to software

Introducing ExMake

It’s that time of the month. That time where some random person on the Internet announces a new, fancy build system that’s better than all the other build systems and will take over the world.

On a more serious note, I just finished version 0.3.1 of ExMake, a build tool similar in nature to the POSIX make tool, but generally more modern. This version is more or less usable for building actual things.

So what is this thing, why does it exist, and how is it better than all the other build systems?

As mentioned, ExMake is a make-like tool. It’s not something meant to replace monolithic build system suites like Autotools. It’s simply a dependency-driven build tool that executes recipes to produce output files. The main reason I created it was that I was sick and tired of make’s scripting ‘language’ which is, at best, a text processor with macros. Other things that bothered me were make’s lack of support for libraries, its broken recursion model, its lack of caching of any kind, its fragile and inefficient parallel job scheduler, and its nightmare-inducing, shell-based recipes.

ExMake is written in Elixir, a general-purpose programming language, and also uses it as its scripting language. This does mean that ExMake requires Erlang to work. This is unlikely to be a problem, however, as Erlang is available for the vast majority of operating systems and architectures.

The Basics

Let me introduce ExMake by example. A simple Exmakefile might look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
defmodule MyCProject.Exmakefile do
    use ExMake.File

    import File, only: [rm!: 1]
    import Path, only: [join: 2]

    phony "all", ["myprog"], _, _ do
    end

    phony "clean", [], _, _, dir do
        rm! join(dir, "myprog")
    end

    rule ["myprog"], ["myprog.c"], [src], [tgt] do
        shell "cc #{src} -o #{tgt}"
    end
end

To build myprog, you just invoke exmake which will invoke the all rule since no other target has been named. As with make, you can pass -j 4 to build up to 4 things in parallel (though it obviously won’t matter in this case, since we’re just building one rule). ExMake does one thing very differently from make, however: It produces no console output at all by default. This may seem odd, but this is in line with the Unix philosophy of shutting up unless you have something interesting to say. ExMake will of course produce output if an error occurs. You can ask ExMake to be loud (print all shell invocations) by passing -l:

1
2
$ exmake -l
cc ./myprog.c -o ./myprog

Recursion

You’re probably wondering about the explicit ./ prefixes added to that shell invocation. This happens because ExMake only invokes recipes relative to the directory it was started in. This may seem weird if you’re coming from good old make, but there’s a very good reason that ExMake does this: Its recursion model. If you’ve ever maintained a non-trivial application using make as a build tool, you’ve probably dealt with make’s insane approach to recursion. There’s even a famous paper explaining why it’s evil and broken. ExMake does away with this traditional approach to recursion and instead treats recursion as a first-class citizen.

Recursion in ExMake is done with a simple directive that tells ExMake where to go and pick up another Exmakefile. Let’s see an example Exmakefile:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
defmodule MyCProject.Exmakefile do
    use ExMake.File

    import File, only: [rm!: 1]
    import Path, only: [join: 2]

    recurse "utils"

    phony "all", ["myprog"], _, _ do
    end

    phony "clean",
          [join("utils", "clean")],
          _, _, dir do
        rm! join(dir, "myprog")
    end

    rule ["myprog"],
         ["myprog.c", join("utils", "stuff.o")],
         [myprog_c, stuff_o], [tgt] do
        shell "cc #{myprog_c} #{stuff_o} -o #{tgt}"
    end
end

And utils/Exmakefile:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
defmodule MyCProject.Utils.Exmakefile do
    use ExMake.File

    import File, only: [rm!: 1]
    import Path, only: [join: 2]

    phony "all", ["stuff.o"], _, _ do
    end

    phony "clean", [], _, _, dir do
        rm! join(dir, "stuff.o")
    end

    rule ["stuff.o"], ["stuff.c"], [src], [tgt] do
        shell "cc -c #{src} -o #{tgt}"
    end
end

There’s a bit to take in here. Let me explain:

  • The recurse directive tells ExMake to go into utils and pick up the Exmakefile that’s in there. ExMake then adds all of the rules in that file to the dependency graph.
  • The top-level clean target now depends on utils/clean. This notation may seem odd, but it’s just that ExMake uses path separators for phony targets too. This dependency ensures that a top-level exmake clean will also clean up in utils.
  • myprog now depends on utils/stuff.o, which is built in the Exmakefile inside utils. This means that you could cd utils && exmake and then cd .. && exmake to build utils/stuff.o separately if you wanted.
  • The argument list (or more accurately, argument pattern list) for myprog now contains two identifiers in the sources list. This just means that we’re expecting the two dependencies (myprog.c and utils/stuff.o) that we declared, and therefore pattern match on that list to conveniently get them.

We can now build:

1
2
3
$ exmake -l
cc -c ./utils/stuff.c -o ./utils/stuff.o
cc ./myprog.c ./utils/stuff.o -o ./myprog

Notice how ExMake didn’t cd into utils. But because we wrote our recipes to use the paths passed to them instead of writing the paths literally, everything worked out as ExMake simply passed the full paths to the recipes.

Now, why is this way of dealing with recursion a good idea? The reason is quite simple: Since ExMake does not invoke itself recursively – for performance reasons – changing the current directory would be dangerous. Sure, it would work fine when executing a build script serially, but if you add -j 2 to the mix, suddenly some rules will start executing in the wrong directories! One way to work around that would be to add locks around directory changes, but then the performance gained by parallelizing the build is undermined.

So, it’s a tradeoff. You have to write rules more carefully, but on the other hand, ExMake doesn’t have to jump through ridiculous hoops such as invoking itself recursively and using IPC to communicate between the processes. This means that ExMake’s parallel job scheduler is much more robust than that of, say, GNU make.

Build Caching

One performance-related thing that ExMake puts a lot of effort into is caching. To see just how much this matters, let’s try timing non-cached and cached builds of the recursion example above. We do this by passing the -t flag. This will print a bunch of stuff after the build is done, but we’re only interested in the total time.

1
2
3
4
5
6
7
8
9
10
11
12
$ exmake -l -t
cc -c ./utils/stuff.c -o ./utils/stuff.o
cc ./myprog.c ./utils/stuff.o -o ./myprog

    ===------------------------------------------------------------------------------------------===
                                          ExMake Build Process
    ===------------------------------------------------------------------------------------------===

        Time                                          Percent    Name
        --------------------------------------------- ---------- -------------------------------
        ... snip ...
        0d | 0h | 0m | 0s | 578ms | 578720us          100.0      Total

After this build, everything has been cached. Let’s try a cached build:

1
2
3
4
5
6
7
8
9
10
11
12
13
$ exmake clean
$ exmake -l -t
cc -c ./utils/stuff.c -o ./utils/stuff.o
cc ./myprog.c ./utils/stuff.o -o ./myprog

    ===------------------------------------------------------------------------------------------===
                                          ExMake Build Process
    ===------------------------------------------------------------------------------------------===

        Time                                          Percent    Name
        --------------------------------------------- ---------- -------------------------------
        ... snip ...
        0d | 0h | 0m | 0s | 211ms | 211277us          100.0      Total

So we went from 578 milliseconds to 211 milliseconds. This doesn’t seem like a whole lot given the small test case, but on large projects with huge dependency graphs, just loading the graph from disk is significantly faster than computing all of it on every ExMake invocation. Similarly, loading the compiled build scripts is way faster than lexing, parsing, analyzing, and emitting them over and over.

Reusable Libraries

I hate how, when I use make, I have to reinvent so much stuff to invoke the tools I need. There’s no standard way to have libraries that can be installed and used in makefiles. In ExMake, there are standard mechanisms to construct and use libraries.

Let’s suppose I want to build a simple C# console application. If I use the C# library that ships with ExMake, my Exmakefile will look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
defmodule MyCProject.Exmakefile do
    use ExMake.File

    load_lib CSharp

    import File, only: [rm!: 1]
    import Path, only: [join: 2]

    phony "all", ["myprog.exe"],
          _, _ do
    end

    phony "clean", [],
          _, _, dir do
        rm! join(dir, "myprog.exe")
    end

    cs ["myprog.cs"], "myprog.exe"
end

Let’s run it:

1
2
3
4
$ exmake -l
Located program '/opt/mono/bin/mcs' ('CSC')
C# compiler type: mcs
/opt/mono/bin/mcs   -nologo      /out:./myprog.exe -- ./myprog.cs

And that’s it. We can add source files to the sources list as we go along. The C# library picked up a C# compiler automatically, set up all the arguments as needed, and created the myprog.exe rule and its recipe (with all its somewhat complicated internal logic to handle a lot of different use cases).

Note that information such as the C# compiler to use is cached.

Of course, it doesn’t end with just the standard libraries. You can write your own libraries that can be installed globally (or locally) and included in build scripts.

For example, a super simple C library (c.ex) might look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
defmodule ExMake.Lib.C do
    use ExMake.Lib

    description "Support for the C programming language."
    license "MIT License"
    version {1, 0, 0}
    url "http://joe.average.com"
    author "Joe Average", "joe@average.com"

    precious "CC"

    on_load args, _ do
        put("CC", args[:cc] || find_exe(["clang", "gcc", "icc"], "CC"))

        list_put("CFLAGS")
    end

    defmacro cc_flag(flag) do
        quote do: ExMake.Env.list_append("CFLAGS", unquote(flag))
    end

    defmacro c(srcs, tgt, opts \\ []) do
        quote do
            @exm_c_opts unquote(opts)

            srcs = unquote(srcs)
            tgt = unquote(tgt)

            rule [tgt], srcs,
                 srcs, [tgt], _ do
                flags = Enum.join(@exm_c_opts[:flags] || [], " ")
                srcs = Enum.join(srcs, " ")

                shell "${CC} ${CFLAGS} #{flags} -o #{tgt} #{srcs}"
            end
        end
    end
end

You can use ExMake to build the library with an Exmakefile like so:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
defmodule ExMake.Lib.C.Exmakefile do
    use ExMake.File

    load_lib ExMake

    import File, only: [rm!: 1]
    import Path, only: [join: 2]

    phony "all", ["Elixir.ExMake.Lib.C.beam"],
          _, _ do
    end

    phony "clean", [],
          _, _, dir do
        rm! join(dir, "Elixir.ExMake.Lib.C.beam")
    end

    exm_lib "c.ex", ["Elixir.ExMake.Lib.C"]
end

Once compiled to Elixir.ExMake.Lib.C.beam, it can be copied to the system so that a load_lib directive will be able to load it. For example, you could drop it into /usr/lib/exmake.

Missing/Unsupported Features

ExMake does away with some features of GNU make that I personally consider either broken, insane, or actively harmful. Or any combination of those.

Last-resort rules and the .DEFAULT target are unsupported. I have never come across a sensible use case for these, and it seems to me that if you find yourself needing these, you’ve done something horrifically wrong.

ExMake can’t create and update archive files. This is an odd feature that is better left to explicit invocations of ar.

The .LOW_RESOLUTION_TIME target does not exist in any form. I don’t know of any remotely relevant operating system or file system that needs this. I suspect it made sense back when it was introduced, but highly doubt its value today.

The .EXPORT_ALL_VARIABLES feature is unsupported as I strongly view this as a glaring recipe for disaster - it could affect all sorts of things in the programs being executed. It’s much better to explicitly export variables with the System.put_env function.

There is no support for .NOTPARALLEL (or anything like it). This is another case of hiding build system bugs intentionally. If two rules must not execute in parallel, make one depend on the other. .NOTPARALLEL is a shotgun solution that works, but harms parallelism of the entire build script.

.ONESHELL makes little sense in ExMake since recipes are regular Elixir expressions, not shell invocations.

Conclusion

In short, ExMake is a make-like tool using a modern, general-purpose programming language as its scripting language. It features extensive caching, reusable libraries, sane recursion, and better parallel job scheduling.

If you’re interested, you can find more info on the ExMake wiki, though I have yet to write the manual. Still, this post and the API documentation for ExMake.File and ExMake.Lib should get you going easily.