imiric 3 hours ago | next |

Happy to see someone inspired by Nix, but wanting to carve their own path. Nix popularized some powerful ideas in the Linux world, but it has a steep learning curve and a very unfriendly UI, so there is plenty of room for improvement there.

I'm not sure if Lua is the right choice, though. A declarative language seems like a better fit for reproducibility. The goal of supporting non-deterministic builds also seems to go against this. But I'm interested to know how this would work in practice. Good luck!

hinkley 2 hours ago | root | parent | next |

If you design it like SCons, it'll look imperative but behave more declaratively.

If I understand the architecture correctly, the imperative calls in the config file don't actually run the build process. They run a Builder Pattern that sets up the state machine necessary for the builds to happen. So it's a bit like LINQ in C# (but older).

I have no idea how that plays out single-step debugging build problems though. That depends on how it's implemented and a lot of abstractions (especially frameworks) seem to forget that breakpoints are things other people want to use as well.

zombiezen 2 hours ago | root | parent |

That's accurate (unless the config file attempts to read something from the build process, that will trigger a build).

It's a good point about debugging build problems. This is an issue I've experienced in Nix and Bazel as well. I'm not convinced that I have a great solution yet, but at least for my own debugging while using the system, I've included a `zb derivation env` command which spits out a .env file that matches the environment the builder runs under. I'd like to extend that to pop open a shell.

skybrian 22 minutes ago | root | parent | next |

One thing I like to see is a 'dry run' like 'make -n'. Although, maybe that's not possible in all cases.

Another possibility might be to output a something like a shell script that would do a rebuild the same way, so you can see what it did and hack it when debugging.

pdimitar 10 minutes ago | root | parent | prev |

Surface-level feedback: get rid of the word "derivation". Surely there must be a better way to describe the underlying thing...

kortex an hour ago | root | parent | prev |

> The goal of supporting non-deterministic builds also seems to go against this.

I think this is actually a great escape hatch. Supporting non-deterministic builds means more folks will be able to migrate their existing build to zb. Postel's law and all that.

mikepurvis 3 hours ago | prev | next |

Whoa, nifty. Can you speak more to the interop issues with Nix? I've been working on a pretty large Nix deployment in the robotics space for the past 3ish years, and the infrastructure side is the biggest pain point:

* Running a bare `nix build` in your CI isn't really enough— no hosted logs, lack of proper prioritization, may end up double-building things.

* Running your own instance of Hydra is a gigantic pain; it's a big ball of perl and has compiled components that link right into Nix internals, and architectural fiasco.

* SaaS solutions are limited and lack maturity (Hercules CI is Github-only, nixbuild.net is based in Europe and last I checked was still missing some features I needed).

* Tvix is cool but not ready for primetime, and the authors oppose flakes, which is a deal-breaker for me.

Something that's a barebones capable of running these builds and could be wrapped in a sane REST API and simple web frontend would be very appealing.

zombiezen 2 hours ago | root | parent | next |

Tracking issue is https://github.com/256lights/zb/issues/2

The hurdles to interop I see are:

- Nixpkgs is not content-addressed (yet). I made a conscious decision to only support content-addressed derivations in zb to simplify the build model and provide easier-to-understand guarantees to users. As a result, the store paths are different (/zb/store instead of /nix/store). Which leads to... - Nix store objects have no notion of cross-store references. I am not sure how many assumptions are made on this in the codebases, but it seems gnarly in general. (e.g. how would GC work, how do you download the closure of a cross-store object, etc.) - In order to obtain Nixpkgs derivations, you need to run a Nix evaluator, which means you still need Nix installed. I'm not sure of a way around this, and seems like it would be a hassle for users.

I have experienced the same friction in build infra for Nix. My hope is that by reusing the binary cache layer and introducing a JSON-RPC-based public API (already checked in, but needs to be documented and cleaned up) for running builds that the infrastructure ecosystem will be easier.

Rucadi 3 hours ago | root | parent | prev | next |

Why are flakes such a deal-breaker? While not ideal, you can still tag your versions in the .nix file instead of the lockfile.

I even had to avoid flakes in a system I developed used by ~200 developers since it involved a non-nixos OS and it involved user secrets (Tokens etc...) So with flakes I had to keep track of the secrets (and was a pain point, since they obviously didn't have to push them into the git repo) but nix flakes doesn't handle well omitting files on git (it ignores them also on nix commands). In the end, the workarounds were too messy and had to drop flakes entirely.

laurentlb 3 hours ago | prev | next |

I'd like to know more about the "Support for non-determinism" and how that differs from other build systems. Usually, build systems rerun actions when at least one of the inputs has changed. Are non-deterministic targets rerun all the time?

Also, I'm curious to know if you've considered using Starlark or the build file syntax used in multiple other recent build systems (Bazel, Buck, Please, Pants).

zombiezen 2 hours ago | root | parent |

(Hi! I recognize your name from Bazel mailing lists but I forget whether we've talked before.)

I'm mostly contrasting from Nix, which has difficulty with poisoning cache when faced with non-deterministic build steps when using input-addressing (the default mode). If zb encounters a build target with multiple cached outputs for the same inputs, it rebuilds and then relies on content-addressing to obtain build outputs for subsequent steps if possible. (I have an open issue for marking a target as intentionally non-deterministic and always triggering this re-run behavior: https://github.com/256lights/zb/issues/33)

I'll admit I haven't done my research into how Bazel handles non-determinism, especially nowadays, so I can't remark there. I know from my Google days that even writing genrules you had to be careful about introducing non-determinism, but I forget how that failure mode plays out. If you have a good link (or don't mind giving a quick summary), I'd love to read up.

I have considered Starlark, and still might end up using it. The critical feature I wanted to bolt in from Nix was having strings carrying dependency information (see https://github.com/NixOS/nix/blob/2f678331d59451dd6f1d9512cb... for a description of the feature). In my prototyping, this was pretty simple to bolt on to Lua, but I'm not sure how disruptive that would be to Starlark. Nix configurations tend to be a bit more complex than Bazel ones, so having a more full-featured language felt more appropriate. Still exploring the design space!

Ericson2314 31 minutes ago | prev | next |

Nice to see Windows support. We/I are working on that with upstream Nix too.

Also I hope we can keep the store layer compatible. It would be good to replace ATerm with JSON, for example. We should coordinate that!

Iceland_jack 3 hours ago | prev | next |

I appreciate the link to https://dl.acm.org/doi/10.1145/3236774

o11c 2 hours ago | root | parent |

Definitely interesting, but it's flat-out wrong about the limitations of `make`.

In particular, the `release.txt` task is trivial by adding a dummy rule to generate and include dependencies; see https://www.gnu.org/software/make/manual/html_node/Remaking-... (be sure to add empty rules to handle the case of deleted dynamic dependencies). You can use hashes instead of file modification times by adding a different kind of dummy rule. The only downside is that you have to think about the performance a little.

I imagine it's possible for a project to have some kind of dynamic dependencies that GNU make can't handle, but I dare say that any such dependency tree is hard to understand for humans too, and thus should be avoided regardless. By contrast, in many other build tools it is impossible to handle some of the things that are trivial in `make`.

(if you're not using GNU make, you are the problem; do not blame `make`)

msvan 3 hours ago | prev | next |

As a current Nix user, what I would really like is a statically typed language to define builds. Recreating Nix without addressing that feels like a missed opportunity.

Rucadi 3 hours ago | root | parent | prev |

For me the killer feature is Windows Support, Ericsson is doing a great job bringing nix into Windows, but the process it's understandably slow, If this project is similar enough to nix that I can kind-off translate easily the zb derivations to nix derivations, I'm willing to use it in windows (It's not like nix has windows programs in the nixpkgs either way I have to bring them in my own).

The problem for me is that I see no benefit on using this over nix language (which I kinda like a lot right now)

droelf 2 hours ago | root | parent | next |

We're working on rattler-build (https://github.com/prefix-dev/rattler-build/) - which is a build system inspired by Apko / conda-build and uses YAML files to statically define dependencies. It works really well with pixi (our package manager) but also any other conda compatible package managers (mamba, conda).

And it has Windows support, of course. It can also be used to build your own distribution (e.g. here is one for a bunch of Rust utilities: https://github.com/wolfv/rust-forge)

greener_grass 2 hours ago | prev | next |

I'm excited by this!

Quick question: if the build graph can be dynamic (I think they call it monadic in the paper), then does it become impossible to reason about the build statically? I think this is why Bazel has a static graph and why it scales so well.

zombiezen an hour ago | root | parent |

According to Build systems à la carte, "it is not possible to express dynamic dependencies in [Bazel's] user-defined build rules; however some of the pre-defined build rules require dynamic dependencies and the internal build engine can cope with them by using a restarting task scheduler, which is similar to that of Excel but does not use the calc chain." (p6)

IME import-from-derivation and similar in Nix is usually used for importing build configurations from remote repositories. Bazel has a repository rule system that is similar: https://bazel.build/extending/repo

So to answer your question: yes from the strictest possible definition, but in practice, I believe the tradeoffs are acceptable.

steeleduncan 3 hours ago | prev | next |

Looks great, Nix-with-Lua that also supports Windows would be amazing. Two questions if I may

- Does this sandbox builds the way flakes do?

- What is MinGW used for on Windows? Does this rely on the MinGW userland, or is it just because it would be painful to write a full bootstrap for a windows compiler while also developing Zb?

Also, its great to see the live-bootstrap in there. I love the purity of how Guix's packages are built, and I like the idea Zb will be that way from the start

zombiezen 2 hours ago | root | parent |

Nix sandboxes derivation runs on Linux even without flakes, and I'm planning on implementing that, yes: https://github.com/256lights/zb/issues/29 and https://github.com/256lights/zb/issues/31

MinGW is used to build Lua using cgo. I'd like to remove that part, see https://github.com/256lights/zb/issues/28 I haven't started the userspace for Windows yet (https://github.com/256lights/zb/issues/6), but I suspect that it will be more "download the Visual C++ compiler binary from this URL" than the Linux source bootstrap.

Yeah, I'm happy with live-bootstrap, too! I tried emulating Guix's bootstrap, but it depended a little too much on Scheme for me to use as-is. live-bootstrap has mostly worked out-of-the-box, which was a great validation test for this approach.

droelf 2 hours ago | prev | next |

Cool that this space is getting more attention - I just came from the reproducible builds summit in Hamburg. We're working on similar low level build system tools with rattler-build and pixi. Would love to have a chat and potentially figure out if collaboration is possible.

packetlost 3 hours ago | prev | next |

One request that I would make of a project like this is to support distributed builds out of the box. Like, really basic support for identical builder hosts (this is much easier now than in the past with containers) and caching of targets. Otherwise, this looks great! Big fan of the choice of Lua, though the modifications to strings might make it difficult to onboard new users depending on how the modification was made.

zombiezen 3 hours ago | root | parent | next |

Yup, remote building and caching is on my radar. I expect it will work much in the same way Nix does now, although I'm being a bit more deliberate in creating an RPC layer so build coordinators and other such tools are more straightforward to build.

The string tweak is transparent to users broadly speaking. IME with Nix this thing works the way people expect (i.e if you use a dependency variable in your build target, it adds a dependency).

alxmng 2 hours ago | prev | next |

Did you consider writing a nicer language that compiles to Nix? A "friendly" tool on the outside with Nix inside.

zombiezen 2 hours ago | root | parent | next |

Yup, that was how I built the prototype: https://www.zombiezen.com/blog/2024/06/zb-build-system-proto...

The last commit using that approach was https://github.com/256lights/zb/tree/558c6f52b7ef915428c9af9... if you want to try it out. And actually, I haven't touched the Lua frontend much since I swapped out the backend: the .drv files it writes are the same.

The motivation behind replacing the backend was content-addressibility and Windows support, which have been slow to be adopted in Nix core.

Rucadi 2 hours ago | root | parent | prev |

I don't think nix is that awful, while there are some tasks that are more difficult or can be a little bit verbose (if you want to play a lot with the attribute sets / lists or string manip) When using nix most of the time you'll end up just writing bash or using it as a templating language.

kortex an hour ago | prev | next |

How do you pronounce "Zb"? Zee-bee?

zombiezen an hour ago | root | parent |

Heh, I think I need to add something to the README. I've been pronouncing it as "zeeb" in my head as in the first syllable Zebesian Space Pirate from Metroid, but TIL that that's canonically "Zay-bay-zee-uhn" so idk.

Naming is hard.

IshKebab 3 hours ago | prev |

Interesting. I feel like I would have gone with Starlark over Lua, but I guess it's good to have options.

Does it support sandboxing?