— and why that matters for open source science projects

There's a specific kind of frustration that hits when you're trying to do something genuinely good — contributing compute cycles to CERN's particle physics research, say — and it just quietly breaks. No dramatic error. No obvious culprit. Just tasks that run, fail, and disappear, taking your donated CPU-hours with them.

That's where this story starts: a home Linux workstation running LHC@home, the volunteer computing arm of CERN's Large Hadron Collider programme. After a routine Docker upgrade, ATLAS simulation tasks started failing silently with "computation error." Credits stopped accumulating. The tasks looked like they were running — then weren't. No log entry pointed clearly at a cause.

What followed over the next few weeks was equal parts sysadmin detective work, open source contribution, and an experiment in what AI-assisted technical work actually looks like in practice.

The Problem, Unwrapped

LHC@home runs ATLAS particle physics simulations on volunteer machines. These tasks use CVMFS — a distributed filesystem from CERN — to pull in the actual simulation software at runtime. Part of the task lifecycle involves launching a lightweight Docker container running lighttpd, which serves the completed simulation results back to the BOINC wrapper.

The failure came down to a single changed default in Docker CE 29.x: CAP_SYS_ADMIN was no longer included in the default container capability set. Without it, the container couldn't mount a tmpfs internally. Without the tmpfs, lighttpd couldn't set up its working directory. Without lighttpd, there was no way to deliver results. The task failed. BOINC logged "computation error" and moved on.

The error buried in the container logs was stark once you knew where to look:

mount: /var/www/lighttpd: cannot mount tmpfs read-only

But getting there required ruling out AppArmor, Landlock, seccomp, and a half-dozen other plausible culprits first. It also required understanding why the apparently obvious fix — adding --privileged to the Docker invocation via docker_container_options in BOINC's cc_config.xml — wasn't working.

The answer to that last one turned out to be the most interesting part.

The Field That Didn't Exist

BOINC's documentation references docker_container_options as a configuration directive for passing flags to Docker container invocations. It is, in fact, not implemented. The field is not parsed. It does nothing. No warning is logged. No error is produced. It simply doesn't exist in the code.

The initial bug report filed against BOINC (issue #6914) framed this as the field being "silently ignored." A maintainer's response clarified the actual situation: "cc_config.xml doesn't have this property, that is why it's not passed anywhere." The issue was closed as invalid.

That's a genuinely useful clarification — but it also means there's no supported mechanism in BOINC for a volunteer to pass additional flags to Docker containers spawned by the client. None. If your host-side Docker configuration requires flags the project didn't anticipate, you're on your own.

The workaround that actually works is a wrapper script deployed at /usr/bin/docker that intercepts all docker run and docker create calls and injects --privileged. The real Docker binary is protected from being overwritten by package updates using dpkg-divert. It's functional, it's robust, but it's also exactly the kind of thing that shouldn't be necessary.

Turning a Workaround into a Feature Request

Having a working workaround and having a solution are different things. The workaround requires every affected volunteer to independently discover the problem, diagnose it correctly, and implement binary interception — a fairly high bar for a project that depends on broad volunteer participation.

The right fix is upstream: implement docker_container_options (or equivalent) in BOINC so that volunteers can pass host-specific Docker flags through a supported configuration path. This is exactly what a second GitHub issue — #6931 — requested.

The feature request framed the problem clearly: here's the failure mode, here's the root cause, here's the exact configuration directive that would solve it, here's why it matters for volunteer retention. It included the full diagnostic documentation as an attachment.

Within two days, the issue had been triaged, labelled, assigned to both the primary BOINC client developer and the LHC@home project contact at CERN, and prompted an architectural discussion between maintainers about whether this should be volunteer-configurable or project-controlled. That's a meaningful response to a feature request from a single volunteer with no prior relationship with the project.

Where the AI Fits

This is the part that's easy to overstate in either direction, so let's be precise about it.

The person running the machine brought the irreplaceable parts: the actual failure, the persistence to investigate it rather than give up, and the judgment to recognise when something was a genuine upstream gap rather than local misconfiguration. Without someone sitting in front of a machine watching ATLAS tasks silently fail and deciding to understand why, nothing in this story happens.

What the AI collaboration — in this case, Claude — contributed was compression. The diagnostic process drew on knowledge of Docker's capability model, BOINC's process architecture, Linux cgroup behaviour, dpkg-divert mechanics, and systemd unit semantics simultaneously. For someone working alone, building that knowledge base would have taken considerably longer than the investigation itself.

More concretely: the AI helped produce documentation that captured the diagnostic chain accurately enough to be useful to other BOINC administrators, and a bug report that spoke the language of the upstream project clearly enough to be taken seriously immediately. Both of those outcomes depend on getting the technical framing right — not just describing symptoms, but connecting them to causes in a way that's actionable for someone who maintains the codebase.

The key dynamic is that the AI doesn't have the problem. It can reason about the problem, help investigate it, and help communicate about it — but only because a human is providing the actual system output, the actual failure behaviour, and the actual judgment calls about what matters. That division of labour is what makes it work.

Why This Matters for Volunteer Computing

LHC@home, like most BOINC projects, depends on volunteers who are willing to donate CPU cycles to science they believe in. The barrier to entry is intentionally low — download the client, attach to the project, let it run.

The failure mode described here is particularly damaging to that model because it's invisible. A volunteer who upgrades Docker, watches their tasks start failing, and sees no clear explanation will almost certainly conclude the project is broken and move on. They won't file a bug report. They won't investigate the container capability model. They'll just leave.

The number of volunteers who have quietly churned off LHC@home after a docker-ce 29.x upgrade is unknowable, but the upgrade has been the default for new installations for months. Every one of those silent departures represents lost compute capacity and a missed opportunity to retain someone who was already motivated enough to set up volunteer computing in the first place.

Getting the fix into BOINC upstream — even if it takes several release cycles — means the next person who hits this failure has a supported path that doesn't require them to understand binary interception.

The Practical Upshot

If you're running LHC@home ATLAS tasks on Linux with docker-ce 29.x and seeing computation errors, the immediate fix is the wrapper script approach documented at github.com/black-vajra/sable-boinc_admin in docs/LHC_DOCKER_COMPATIBILITY.md. The upstream feature request that would make this unnecessary is tracking at BOINC GitHub issue #6931.

If you're thinking about what AI-assisted technical work actually looks like in practice — not the hype version, not the dismissive version — this is a reasonably clean example. The human brings the problem, the judgment, and the accountability. The AI brings breadth and speed. The output is better than either would produce alone, and the work is still fundamentally the human's.

That seems like the right way to think about it.

This article documents work conducted on pepperpots, a Kubuntu 24.04 workstation contributing to BOINC distributed computing projects including LHC@home, Einstein@Home, and MilkyWay@home. Full technical documentation of the setup, diagnostics, and workarounds is available at github.com/black-vajra/sable-boinc_admin.