I had a fortune to interview the CTO of ListenField - David Hagege, who also happens to be a recent subscriber to my newsletter.

ListenField is a Japan-based Agricultural Technology Startup with the mission to help farmers increase their productivity and income while reducing their environmental impact.

ListenField leverages Nix and NixOS for development of ReactJS and Ruby on Rails applications, and Python AI pipelines.

Please find the full text below. Quoted sentences belong to Drake. Non-quoted belong to David.

The Interview

Are you writing your message from NixOS? :)

I do!

I'm using NixOS on all my machines. This is I would say the biggest problem with NixOS - when you start to use it, you can hardly go back to anything else. Nix is replacing so many tools and solving so many headaches. Instead of having to remember a dozen of configuration file formats and tools, I learn Nix once and can configure my whole environment the same way on all my machines. Nix makes stuff like rvm, nvm, Ansible, Docker, and many others a thing from the past. Every single configuration of my system is present in my NixOS configuration, down to the scrolling behavior of my touchpad (i.e libinput.touchpad.naturalScrolling = false;). It also makes experimentation on your system a breeze. I know that any change I make, I can revert in a second.

Could you please give a couple of words about ListenField? What you do, what's your tech stack, etc.

We are a small, fully-remote startup of around 40 people.

The goal of our product is to assist farmers and agronomists in optimizing the use of their farmland. We are using satellite images, sensor data and other sources to predict at which date their crops will reach maturity, how much yield they will get, how sweet their crops will be, how they can reduce their fertilizer use, and numerous other farming-related insights.

Right now we are mainly focusing on Thailand and Japan.

Our tech stack is boring in a good way, we are using React/NextJS for our frontends, and Ruby on Rails for the backends. All our ML is in Python.

So, how did the NixOS journey begin?

We started to switch more and more things to NixOS in the last year. I was the one to start the initiative. I was already a NixOS user for some time and it was a no-brainer. I knew that I could convince people to use it.

Initially, it was just devShells for our development environments, then we wanted to use Nix on CI. This also worked, albeit as a bit awkward setup.

"Awkward setup" - could you please tell more? I'll say, this awkwardness is the thing the readers are interested in the most.

Using Nix on GitLab-CI was challenging at the beginning. Everything you install with Nix goes into the /nix/store, which on GitLab is not something you would be able to put in a CI cache (at least, not easily, and copying/retrieving from cache each time you run the CI would also be a slow operation). If you want to use Nix on CI, as soon as you try to install anything, Nix will have to download a lot of dependencies (it's like starting a distribution from scratch), which can take an hour easily. We had to find a quick solution to fix that problem. And in the end what we did is pre-build an OCI image from our local machine with the excellent Nix2Container. That image would then include Nix and all our project dependencies.

We made a nix function looking like:

{
  nix2containerPkgs, # Pass a nix2container.packages.${system}
  pkgs,
}: {
  builder = {
    imageName ? null,
    extraPackages ? [],
  }: let
    ciSetup = pkgs.runCommand "setup" {} ''
      # Enable nix flakes
      mkdir -p $out/etc/nix
      echo "sandbox = false" > $out/etc/nix/nix.conf
      echo "experimental-features = nix-command flakes" >> $out/etc/nix/nix.conf
    '';
  in
    nix2containerPkgs.nix2container.buildImage {
      name = imageName;
      tag = "latest";
      maxLayers = 100;
      initializeNixDatabase = true;

      copyToRoot =
        [
          (pkgs.buildEnv {
            name = "ci-root";
            paths = with pkgs;
            with pkgs.dockerTools;
              [
                coreutils
                bash
                nix
                usrBinEnv
                binSh
                caCertificates
                fakeNss
                ciSetup
              ]
              ++ extraPackages;
            pathsToLink = ["/bin" "/etc" "/var" "/run"];
          })
        ];
      config = {
        Cmd =["${pkgs.bash}/bin/bash"];
        Env = [
            "NIX_PAGER=cat"
            "USER=nobody"
            "SSL_CERT_FILE=${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt"
          ];
      };
    };
}

This piece of code would be reusable in all our projects, we would just call it like:

let
  nix2containerPkgs = nix2container.packages.${system};
  pkgs = import nixpkgs {inherit system;};
  imageBuilder =
        (import ./images.nix {
          inherit nix2containerPkgs pkgs;
        })
        .builder
in
  images.ciBase = imageBuilder {
    imageName = "registry.gitlab.com/listenfield/our-project";
    extraPackages = [/* put your project depedencies here */];
  };

Then, just doing nix run .#images.x86_64-linux.ciBase.copyToRegistry would build an OCI image with Nix (you don't even need Docker at all), and would push that image to the GitLab registry of our project. Then in our gitlab-ci.yml we would just use that image as base with image: registry.gitlab.com/listenfield/our-project:latest. Which means quick CI, since all the dependencies would already be there when the nix commands get called.

But wowadays, instead of that setup we have our own GitLab-runner configured with the Docker executor, which shares the /nix/store of the host. This has the benefit of sharing the already downloaded dependencies between all our projects. That makes all the projects' CIs faster (see Nix-glab-runner at my GitHub for an example).

People like metrics and data. Do you have some? Especially stuff like money savings, engineering toil reduction.

Sadly, I don't think I have concrete metrics. I can see that we are spending less time on headaches like conflicting dependencies between python projects, or having people ready to code quickly in new projects as now their environment is automatically set up, but I don't have the actual metrics to prove it.

As CTO, you have a lot of flexibility to set the agenda. Still, was it hard to sell people on Nix/NixOS? Were there any pushbacks?

It was surprisingly not hard to sell because I'm lucky to have open-minded individuals on the team who are all very eager to learn new tech.

I also tried to do that progressively. I started by only pushing flake.nix files in the root of each project with the dependencies set up. It's not bothering anyone, and if someone wants to actually give Nix a try, they could. After setting up Direnv and Nix, they would be able to just get into the project folder and have their whole environment set up for them. Who wouldn't want something that convenient?

As Docker runs in a VM on OSX, not having to use Docker was also a boost in performance for some. Given that the Nix learning curve can initially be quite steep, it's crucial to have a designated Nix expert within your team who serves as a resource for colleagues to consult and learn from (in my case, I filled this role).

I highly recommend reading as much nix code as possible in the beginning. The Nix Mindmap was helpful to have a global view. In video format, And the Nix Hour is great.

It seems that there is a specific set of problems in the enterprise-y world, which is supposedly best solved by just Docker/Kubernetes. How well does Nix solve your particular set of problems?

I feel like Docker is often the one-size-fits-all approach. Sure, it quickly helps you to run any project, but in the end it often leads you to not even knowing what your project's dependencies really are. Are you sure you included every single dependency of your project in that Dockerfile? Or is there a good chance one of your project dependencies is something that was already built in the base image? Unless you are starting your image from scratch, it's quite difficult to make sure that's the case.

By contrast, creating Nix packages involves building them within a sandbox environment, which forces you to add all the dependencies explicitly.

One problem that is now completely solved with Nix is our packaging. We previously had multiple Python ML projects, that were all supposed to run on the same worker instances. These projects' dependencies kept clashing with each other, as project A required X version of numpy, project B required Y version, and project A would often depend on B, etc. Some people were using poetry, others just pip, etc. When you have 20 projects like this, it quickly becomes really difficult to maintain over time.

So, we have decided to make our own nixpkgs monorepo. And we started writing nix packages for each one of our projects. Instead of having each developer set the versions of each of their Python dependencies, we now would target a specific nixpkgs revision of NixOS stable (23.11). By aiming at a specific nixpkgs revision, you're utilizing a set of dependencies that have already been vetted by the nixpkgs community for optimal compatibility, verified through Hydra CI. So that stuff like the numpy and pandas versions is always working well together. We would then just use the packages from NixOS stable in our packages' dependencies. As long as our tests pass, it's all good. Our Ruby on Rails main API is also packaged there. It was a bit challenging as Rails expect to be able to write in the current folder but /nix/store is read only, and a few other quirks like this.

Tomorrow, if, say, we want to change the Python version of all our packages from 3.11 to 3.12, it's as simple as changing one line in the flake in our nixpkgs monorepo. Such a change used to be a nightmare in 30 different repos all having different package management systems. Now it takes just one line, and it trickles down to all the projects. Same for any security fix.

Even in the unlikely circumstances that necessitate a specific version of a dependency package, we can simply target another nixpkgs revision specifically for that dependency (see Nix-version-search-cli).

Nix also ensures complete reproducibility of our machine learning models' results by providing the absolute certainty that, for a specific nixpkgs revision, everyone is using the same dependencies, both on personal machines and in production environments, during both the model training and the inference.

We also had some challenges with secrets management, but in the end we found a good solution with age and sops. We basically just set an AGE_PKEY variable in the secrets manager in GCP or in the env, and we use it to decrypt secrets at runtime inside a wrapper of our binary.

Another interesting benefit we got from Nix is that now we can have non developers try our tools. It's just a matter of installing Nix on their machine (which is surprisingly easy), and then they can run any of our projects simply by doing: nix run https://gitlab.com/listenfield/nixpkgs.#projectName. And that makes sure they are always using the latest version of our code. With further simplification achievable through nix bundle. With nix bundle, a self-contained bundle can be created, permitting seamless sharing and execution without the need for Nix installation itself.

I consider introducing Nix to the company a success.

Were there any failures with Nix/NixOS? Something you tried, but it did not fly.

I don't think there was any real failure. There was always a solution to achieve our goals, it just sometimes needed some deeper research as the documentation tends to be lacking sadly. 2 things that I think could be improved are:

• The support for private repositories in Nix is not great. The nix fetchFromGitLab function does not seem to accept private repositories at the moment (there is a Draft PR for it). We have to use builtins.fetchGit instead, which uses the local git configuration for pulling. But builtins.fetchGit does not seem to support LFS at the moment, only pkgs.fetchGit does, but that one cannot currently fetch from private repositories.

• The expectation that nix-generated OCI images would be lean was not met in our case. This may be due to certain packages defining more dependencies than necessary, or incorrectly specifying them (e.g., using propagatedBuildInputs (i.e runtime deps propagating to downstream) instead of nativeBuildInputs (i.e build-time deps) for instance). The automatic layering was also not particularly impressive in our case, as of 100 layers, 99 were quite small, and one was of a few GBs. I suppose this could be improved if Docker did not have that 125 layers limit.

What remains non-nixified? What could be nixified later, what should stay as is?

We are using OCI images for our deployments as we are big users of Cloud Run (GCP's CaaS), our images are made with Nix, but that means we are not using Nix itself for the deployment still (things like NixOps or Deploy-rs), as that would force us to use more classic VM instances instead. We are using Terraform for configuring our cloud, and I think it would be interesting to convert our configurations to Terranix. I'm also quite interested in trying to set up Hydra for our personal nixpkgs repo instead of Gitlab CI.

How do you find the current Nix ecosystem? Do you wish for something nix-y to exist?

I find the current Nix ecosystem great overall. I sometimes encounter instances where the documentation could be more comprehensive, necessitating an exploration of the code to grasp intricate details. One area that can be improved is tilting the balance between configuration and convention more towards the convention side. I often wonder, "is it the right way to do this? Is there a better recommended way?", yet, instead of discovering a single recommended strategy, one often encounters numerous alternatives within different projects.

This abundance of choice can be both empowering and perplexing, as it demonstrates widespread experimentation without cohesive collaboration on unified solutions. Nix flakes, though considered experimental, exemplify this pattern, with many users adopting them despite the lack of official endorsement. But I'm sure this will change over time as Nix gets more popular, and I know the community is well aware of this problem.

Thank you very much, David, it was great pleasure interviewing you.

Closing Words

As I've mentioned, it was pleasure to interview David, and I also want to thank all my subscribers and readers for motivating me to write, as well as for having the opportunity to learn many new things.

And it's definitely great to see a startup with noble mission to adopt such a technology as NixOS.

If you have any stories about Nix/NixOS worth telling, for work use at employer, and for leisure at home, do not hesitate to contact me over email, twitter, or mastodon.