Annapurna

Autark: Rethinking build systems – Integrate, Don’t Outsource

2025-11-04T00:00:00+00:00

I enjoy programming in C. Over the years, I’ve built a couple dozen projects of varying complexity – from small toy programs to game engines, key-value storage systems, and even algorithmic trading platforms. Each time I start a new project, before I can begin the actual work, I have to go through a somewhat tedious phase: setting up the build system and defining dependencies on other projects and libraries.

CMake

Until recently, I used CMake. For each new project, this usually meant copying the build rules from one of my previous projects and adapting them to the current needs. While that approach made it easier to get started, the process was always painful. The imperative, BASIC-like scripting logic of CMake is too cumbersome for simple reuse or quick adaptation to changing build requirements.

Another inconvenience is that CMake requires you to constantly keep up with new features and changes in its vast ecosystem. If you ignore them for too long, one day your project simply stops building with the current stable CMake version – and you’ll need to enable some CMake policy to restore compatibility. Even if we ignore the scripting complexity, there’s still the long-term issue: eventually, after the project is released, it will stop building exactly as described in its own documentation.

As we all know, CMake doesn’t actually build your project – it’s a transpiler that converts CMakeLists.txt scripts into rules for lower-level tools like Make or Ninja. In practice, that means you depend on two separate tools (and their compatible versions) just to get a working build, not to mention the debugging headaches in complex setups.

I won’t go into a detailed analysis of CMake’s pros and cons here – it’s a popular system, and it earned that popularity for a reason. It’s far from perfect, but as my colleagues like to say, “Bad. But there’s nothing better yet.” Personally, I decided to pursue a different concept for a build system – one that aligns more closely with my own values: simplicity, stability, and lightness.

Make

So why not just use plain Make or even autotools? Take a look at the folks from a bit religious suckless.org community. They write great software using pure POSIX mode Make. Though, amusingly, they forgot to include Make itself in their list of software that rocks. I have my own reasons for not being too fond of Make, and I’m pretty sure I’m not alone. My short-term memory isn’t great – if I don’t write Make scripts every day, I can never remember all the quirky mechanics and syntax variations. Consider constructs like:

%.d: %.c
        @set -e; rm -f $@; \
         $(CC) -M $(CPPFLAGS) $< > $@.$$$$; \
         sed 's,\($*\)\.o[ :]*,\1.o $@ : ,g' < $@.$$$$ > $@; \
         rm -f $@.$$$$

Or:

test: all
    @for exec in $(EXECS); do \
        echo "Running $$exec..."; \
        $(MEM_CHECKER) ./$$exec; \
    done

And let’s not forget the Automatic Variables list from section 10.5.3 of the GNU Make manual – the one you’re apparently not allowed to ignore if you dare to use Make. Please don’t tell me “it’s all simple”. I know there are people for whom it is, but as I said earlier, for me and many colleagues with short memory spans, it’s not.

There’s another point: I believe that when a professional uses a tool, they should understand and be able to use all of its features. But the more complex the tool, the harder it is to claim you truly master it. (That’s why I’ll never call myself a C++ expert.)

In practice, we all end up sticking to a small set of build recipes that feel comfortable and just copy-paste them between projects. Make, however, is too low-level to express build logic easily and gracefully, especially when it needs to account for things like:

Changes in a Makefile should trigger a rebuild of all dependent targets.
Changes in environment variables like $CC or $CFLAGS should cause dependent modules to rebuild.
Modifications in system library headers should trigger rebuilds of the affected parts of the project.
There are several flavors of Make: POSIX Make, GNU Make, BSD Make and you must decide what functionality to sacrifice in order to stay compatible across environments.

And that’s just the beginning. Sure, you can make Make handle all of this, but at what cost? I’ve yet to see a handmade Make project that does it all correctly.

Autark

Realizing that it’s impossible to create a perfect build system, I focused on defining a few key characteristics I wanted to see in the final product even if that meant sacrificing some functionality elsewhere.

Here’s what I needed:

A simple, memorable syntax for build scripts.
Deeper dependency tracking compared to Make (see above).
Portability – ideally, the project shouldn’t require any preinstalled build system. It should be able to build itself using only the most basic tools, like a C compiler and a system shell.
The ability to verify and adapt the project before building (a configure phase).

Dependencies

To get started, I needed some inspiration and I found it in the Redo build tool. There are several Redo implementations available on GitHub, though the concept never became particularly popular. I suspect that’s because it feels rather academic: a talented mathematician proposed a simple, elegant idea, but for it to work in real-world projects, it needed to be wrapped in something more developer-friendly.

While experimenting with Redo, I discovered that building a project often required a large set of shell scripts scattered across many directories. In many of them, I had to reimplement utility functions that really belonged inside the build system itself. However, I did take away one important concept: the idea of managing dependencies through individual files and developed it further.

Portability

To achieve true portability, I took a fairly radical approach: during the first build, the project compiles its own build system from C, and from then on, that build system compiles the project. The build system is a small C program embedded directly inside a build.sh script, which acts as its entry point. It only requires a C99-compatible compiler (flags compatible with clang or gcc). The entire system is about 10K lines of code, compiles in under a second, and caches its build artifacts. This way, everything related to the build process is already part of the project itself – no need to worry about toolchain versions or environment differences across machines.

Script Syntax

Here I had to deal with a rather contradictory set of requirements. On one hand, I wanted a powerful declarative syntax; on the other, it needed to be simple, and the parser had to remain minimal to allow fast compilation of the build system during a cold start. I eventually settled on the following syntax:

RULE:

  rule_name { RULE | LITERAL ... }

LITERAL:

  word | 'single quoted words' | "double quoted words"

A script consists of a set of rules and literals.
Rules and literals are separated by whitespaces.
Every rule has a body enclosed in curly braces {}.
Rules form lists, similar to syntactic structures in Scheme or Lisp. The AST of a script can change dynamically depending on conditions. I was amused to find that a trace of Scheme somehow made its way into this project without me realizing it.

A formal syntax description in PEG format can be found here: https://github.com/Softmotions/autark/blob/master/scriptx.leg. Special thanks to Ian Piumarta for his excellent PEG parser generator which I’ve used in many of my projects.

Here’s a build script example from the demo project

cc {
  hello.c 
  ${CFLAGS}
  ${CC}
  consumes {
    # hello.h is a product of the `configure` rule,
    # so we declare our dependency on it here.
    hello.h
  }
}

# Builds the static library libhello.a
run {
  exec { ${AR} rcs libhello.a ${CC_OBJS} }
  consumes {
    ${CC_OBJS}
  }
  produces {
    ${LIBHELLO_A}
  }
}

Current Status of Autark

The build system autark.dev was implemented some time ago, and since then I’ve migrated most of my C/C++ projects to it, fixing issues and polishing things along the way.

Here are a few open-source projects that have already been migrated:

And even a fork of the third-party project protobuf-c where I used macros to build and run test cases.

Conclusion

Autark started as an experiment – a personal attempt to make the build process less painful and more predictable. Over time, it grew into something I now use daily across nearly all my C projects. It’s not meant to compete with CMake or reinvent the industry standard. Instead, it tries to do one thing well: give developers a lightweight, self-contained, and reliable way to describe how their projects are built.

I don’t claim that Autark is the best build system it’s simply the one that aligns with my values: simplicity, transparency, and long-term stability. If you’ve ever struggled with build scripts that felt like they were fighting you, maybe you’ll find something familiar in this approach.

The project is still evolving, and feedback is very welcome. You can learn more or try it yourself at autark.dev

Where are engineers?

2020-06-15T00:00:00+00:00

Extremes and excesses

A great number of paradoxes and contradictions are born in businesses when trying to answer this question. Decisions become extremes: professional criteria are lowered and, primarily business tasks are solved. Another extreme is increased requirements taking no account of the market and often appearing because of paranoia and narrowmindedness. This leads to the destruction of the basic idea of a company and many of the decisions become excessive. Concepts become vague: product quality and value, practicability to use technologies and developer’s level. For example, the product was being produced but the main client had a lot of own needs, which led to a technological and functional adjustment. Many big players dream of doing their products but business hampers them.

But let excuses be excuses. There are good projects as well as excellent professionals! Softmotions are unknown but we try to open to people a new oasis amid the huge and motionless Enterprise-juggernauts. It’s a matter for us to be flexible and able to change.

IT-extravaganza

Staying aside of "hype-rays" and rarely being in the recruitment market, our team have always thought that IT environment has already educated engineers. Certainly, a great number of Java projects and endless PR-companies of different businesses make us think that all the developers are already educated and they should not be explained simple things.

But IT world largely develops by the laws of playwriting and that’s when the problems not connected with development appear:

The number of non-core and unprepared specialists, who evolved to developers, has achieved the tipping point. People are now not embarrassed to send CVs with "a machine gunner" as their experience.

Machine gunner!

The specialists who labelled themselves as "Senior" appear to be "juniors". And there are many reasons for that: "my company said I am a senior", "I can solve any problem on Java", "Yandex is waiting for me". Some are more honest: "Of course I am a junior but I am paid as a senior — what can I do?"

Engineers?

Let’s come to the roots and we might understand something:

Engineer: (French) ingénieur ← (Latin) ingenuim — abilities, creativity

It is all simple: not to live in the realm of illusion, a person must create something, no matter if they are a writer or an engineer. It means:

Engineers think projects not businesses, missions, etc., so they can define the entry point of their effort.
An engineer must know what context is and must not suffer from "vacuum syndrome", which is often manifested in obsessive urge to use concrete technologies. The solution quality is measured by its congruence to the context, which can be changed.
Between the routine and interesting tasks, an engineer sees the right connection. If there is none, he expands the context and finds the contradiction. Interesting tasks are not a whim but the meaning of activity. Without them an engineer turns into a coder or a framework user and the business starts to parasite on the market body.
An engineer has the correct understanding of fundamental knowledge as of an instrument enabling to create. To confirm this opinion a "Yandex" developer, who often holds job interviews, commented on the problem: "The point is that people often consider a very limited number of solutions and don’t have a clue about some real facilities of the language."

An engineer needs to move in these directions — if they are a Junior or a Senior. Unfortunately, hype-rays create radio noise among people and make it difficult to assemble an engineer team. But this is a great chance to launch new companies and projects for the people who are tired of paradoxes and contradictions.

The Story of the IT-depression, birds and EJDB 2.0

2019-09-19T00:00:00+00:00

Apart from the development, I also teach at the Department of Informational Systems in a university, and once during the exam I told one of my students that at the end of the course students should have at least a basic idea on how the stop-and-copy garbage collector works. Student argued that the intellect and analytical skills can always offset not knowing some facts, plus the topic itself looks minuscule in comparison to the fundamental laws of nature. Student pointed out:

— Birds found out about electromagnetic fields way before us, they use them to orientate in space, but does it make them smarter than humans?

I want to tell you a story about one small but useful software project with some thoughtful implication.

It is 2011: popularization of NoSQL

The question about birds was asked in far-off 2011, and I really started to think about it. My lazy stream of thoughts was interrupted by a mundane developer’s task: to make a convenient information storage and search system for audio files metatags for a small media player written in C++.

Taking on a task, just like any engineer, I started looking for the available solutions and, surprisingly, received quite poor results: the implementation was either too bulky (SQLite) or had stability issues (GigaBASE). “Bulky SQLite?” you might ask. Strangely enough, yes. My inner perfectionist was rebelling: the size of my whole project was one tenth of the single sqlite.c file. At that time I didn’t compare project size to the system kernel. 2011 was the golden age of different NoSQL solutions. I was taking a closer look at the MongoDB as a possible solution but we faced two essential limitations:

It was difficult to use MongoDB as an add-on DLL because it’s developers initially didn’t plan for such a use case, and the MongoDB was not technically ready for this.
MongoDB was under AGPL license which is, in my opinion, detrimental. AGPL doesn’t bring anything to the world of open software and usually just a convenient cover for the commercially-licensed products with a shareware code.

Looking for a better option of the lightweight data storage system for small projects, I defined the key qualities of the product:

Document-oriented with a possibility to store hierarchy of documents with an arbitrary structure.
Support JSON documents collections. Implemented in C99 language.
Free license which allows use in the closed sourced products.

Same trap twice

As you can guess, we created our own solution, which was inspired by the MongoDB: a database management system implemented as a shared library (EJDB 1.0).

EJDB 1.0 is based on a fascinating project TokyoCabinet, alas — abandoned now. EJDB became quite popular on github among developers from all over the world. At that time one of the 10gen managers contacted me to discuss the query compatibility between EJDB and MongoDB but it didn’t go any further than a discussion.

However, we took too much from MongoDB and that became a problem. For example, the bulky format of the search queries, e.g.:

{"name": "John", "$and": [ {"$or": [{"age": 38}, {"age": 39}]} ]}

Eventually we realized that such query representation suits robots not people.

Another controversial feature taken from the MongoDB was to keep document’s primary key in the document itself as a special _id field, which mixes the original structure of the stored document with the implementation of the storage system. Undoubtedly, the better approach would be to separate original documents from the data structures used to manage collections.

Despite the inner beauty of the TokyoCabinet, it imposed LGPL 1.x license on EJDB, which is freer than AGPL but still limits the use of the EJDB by many projects.

Being a bit disappointed I released a number of versions of EJDB and then left it inactive for a few years. From time to time I recalled this project when I thought about that very question of birds.

2018: inspiration gained

It took a few years to finally convince myself that birds are not smarter than humans. They instinctively use the laws of nature unable to comprehend them and to use according to their needs. But aren’t many developers doing the same? Instead of comprehending “the laws of nature” they thoughtlessly migrate to technologies and frameworks like IT-birds migrating with the seasons? A true story (from job interviews):

— You said you used TypeScript a lot. Does it mean you’re also familiar with JS and you understand how TypeScript improves it and what’s the difference?

— Well, I haven’t studied JS, so I can’t say.

— Why your tenure was so short in every company you worked for?

— The projects were generally boring. Wild outsourcing, you see? I want to develop real products!

After interviewing yet another candidate Andrew and I got a drink, discussed candidates and talked about the unnecessary complications of the modern software. Many specialists who came to the interview were surprised that we do not use Spring Boot in our projects: "because everybody does". But almost no one is surprised by the Slack-like desktop apps based on Eleсtron, which effectively are secretly packed full featured web-browsers eating up hundreds of megabytes of virtual memory to serve a simple chat.

During the discussion we started to understand what irritates us so much in the modern IT — the inappropriate use of the monstrous hype solutions.

— Stop complicating! — we decided, and used this as an inspiration we took on the second version of EJDB.

EJDB 2.0: philosophy of openness and simplicity

Open source can not be half-open, so we decided to use the MIT license for all the project components, completely getting rid of LGPL code from the TokyoCabinet.

Just for the EJDB2 we developed Key-value data storage IOWOW under the MIT license.

Simple

IOWOW can work with a lot of key-value data sub-bases, stored in one file, which simplified transferring the data between devices and making backup copies and it also reduces the chance of inconsistencies in the storage data.

IOWOW is based on the simple data structure — skip list, which allowed us to create a much more understandable implementation of the persistent storage with a much smaller code base (compared to B+tree and LSM tree) with high performance (according to benchmarks).

Maximum volume of a database file is 512G, which is a consequence of compromises in the implementation of the one-file database on skip lists.

Documents stored in the EJDB2 collections are serialized in a simple binary format — Binn.

Clear

During the work on EJDB2 we reconsidered the search queries format and we introduced a new, more intuitive (comparing to the 1.x version) XPath-like search queries — the JQL language.

JQL supports JSON Patch and JSON merge patch standards. Here are some examples:

/[firstName = Andy]

/[firstName = Silvio] and /[lastName = "De Sousa"]

Select all documents and then keep (project) only firstName and lastName fields in the resulting documents:

/* | /{firstName,lastName}

Easy

It is always better to have a choice than not, so, despite that the easiest and the most efficient way of accessing the data is to do it locally, with EJDB2 you can expose your database via HTTP/Websocket protocols so you can access it remotely over the network using a simple text protocol. This substantially broadens the number of architectures you can have with EJDB2. To enable remote access, built-in HTTP server can be configured and started using C API or a separate executable can be started. You can both query and manage your data using network API.

Aside from the hype-rays

Having finished with the birds, I got haunted by the thoughts about a spanner — this invention was not intended to tighten all possible kinds of screw-nuts but the number of screw-nuts existing just for this spanner is enough for it to have it’s purpose of life. On the other hand, it is weird to tighten all screw-nuts with one spanner. It’s quite common to promise developers to ride a spaceship but in reality they dig a tunnel with a pickaxe. We give them pickaxe right away because it is the only way to stop being a bird and understand which spanner will suit which screw-nuts.

EJDB2