Making CERN Behave: ATLAS, CMS, Theory, and a Little CPU Affinity Surgery

Volunteer computing is supposed to be simple.

You install BOINC. You attach a project. You donate spare compute cycles. Somewhere, far away, a scientist gets useful work done. The machine sits quietly in the corner doing mathematics for civilization while you make coffee, write emails, or watch something morally unserious on YouTube.

That is the ideal.

The reality, at least for serious multi-application projects like LHC@home, can be more complicated. Once VirtualBox, Docker, native multithreaded jobs, GPU helper threads, project preferences, BOINC scheduling limits, Linux CPU scheduling, and desktop responsiveness all collide, the volunteer machine can become less like a charitable appliance and more like a small experimental systems lab.

Over the past few days I revisited my LHC@home BOINC setup and ended up doing a round of practical engineering work on it. The result is a much healthier local configuration: ATLAS is running, CMS has been confirmed running, Theory Docker tasks have been confirmed running, and Einstein@Home GPU work is also behaving alongside the LHC workload when allowed. The local CPU affinity manager has been repaired, improved, tested visually, and partially pushed back into the supporting GitHub repository.

This post is a technical recap of what changed, why it mattered, and what remains to be tested.

The starting point: ATLAS was working, but the broader stack was uncertain

After restarting LHC@home, the first thing I noticed was that I was mostly receiving ATLAS tasks. That was not necessarily bad news. In fact, ATLAS was behaving well. The tasks were running successfully, and the machine was not throwing the kind of obvious BOINC/VirtualBox tantrums that originally pushed me into building local management tooling.

But the absence of CMS was noticeable. CMS tasks had previously been more troublesome on this host, mainly because CMS is a VirtualBox workload. VirtualBox BOINC applications are often where the idealized “just donate spare cycles” picture starts to break down. Now the project has to rely not only on BOINC, but also on the local virtualization stack, kernel modules, filesystem permissions, wrapper behavior, task startup ordering, disk paths, and whatever else the desktop environment happens to be doing.

ATLAS, especially in native multithreaded form, is a different beast. It is still heavy, but it is more direct: a large compute process, visible CPU load, and fewer moving virtualization parts.

So the initial state was good but incomplete:

• ATLAS was running.
• CMS was not yet appearing.
• Theory was queued but not validated.
• Einstein@Home was also active, including GPU work.
• The local affinity manager existed, but it had not been reviewed in a while.

Then CMS showed up.

That was the first pleasant surprise. CMS simulations started running successfully without the old drama. That changed the nature of the session. This was no longer just a matter of watching ATLAS chew through work. The host was now receiving multiple types of BOINC work again, including the historically more fragile LHC workloads.

At that point, the question became: is the system healthy because the projects are now behaving better, or because my local management layer is quietly keeping the machine out of trouble?

The answer is probably: both.

Why a CPU affinity manager exists at all

The local affinity script was originally created to solve a practical problem: heavy BOINC workloads can camp on particular cores and make the system feel worse than it needs to feel. On a many-core Linux desktop, there is no reason to let long-running scientific compute jobs continuously hammer the same cores if the work can be rotated intelligently.

Linux already has a scheduler, of course. BOINC also has its own CPU usage preferences. But real-world BOINC workloads are not all equal. Some are native processes. Some are VirtualBox VMs. Some are Docker-based. Some use the GPU but still need CPU helper threads. Some present themselves as a single process while internally using multiple threads. Some are supervised by wrappers. Some appear and disappear as child processes under BOINC slot directories.

So the purpose of the script is not to outsmart Linux in some grand theoretical way. It is much humbler:

keep BOINC compute workers inside a controlled CPU pool, rotate their allowed cores over time, and leave enough system headroom that the desktop remains usable.

In practice, this means repeatedly detecting active compute workers above a CPU threshold, calculating a proportional share of available managed cores, applying taskset, and renicing the workers to low priority.

The original configuration looked roughly like this:

POLL_INTERVAL=10
CPU_THRESHOLD=20.0
MIN_CORES=2
ROTATION_STEP=2
TOTAL_CORES=$(nproc)

The script polled every ten seconds. Any descendant worker above 20 percent CPU was considered active. Workers received at least two cores. Each cycle advanced the rotation window by two cores. On this machine, nproc reports 14 logical cores.

In principle, that should have allowed hot workloads to move around the CPU instead of sitting on the same cores forever.

But there was a bug.

The wraparound bug: rotating by shrinking is not rotating

The first major discovery was that the script was definitely changing CPU affinity, but not always correctly.

A proof log showed the same process receiving affinity masks like:

0-13
2-13
4-13
6-13
8-13
10-13
12,13

At first glance, this looks like rotation. The start point moves forward. But it is not true circular rotation. It is a shrinking range. Near the top of the core list, the process was being confined to fewer and fewer cores instead of wrapping around from the end of the CPU list back to the beginning.

The flawed logic was simple:

if [ "$end" -ge "$TOTAL_CORES" ]; then
    end=$(( TOTAL_CORES - 1 ))
fi

That clamps the end of the range. It does not wrap.

So if a worker was supposed to get, say, eight cores starting at core 10 on a 14-core host, the correct allocation would be something like:

10,11,12,13,0,1,2,3

But the old logic could only express:

10-13

That is not rotation. That is accidental throttling.

The fix was to add a circular core-list generator:

make_core_list() {
    local start=$1
    local count=$2
    local total=$3
    local -a cores=()

    if [ "$count" -ge "$total" ]; then
        echo "0-$((total - 1))"
        return
    fi

    for ((j = 0; j < count; j++)); do
        cores+=( $(( (start + j) % total )) )
    done

    local IFS=,
    echo "${cores[*]}"
}

Then instead of calculating start and end as a simple range, the script now generates an explicit CPU set:

cpuset=$(make_core_list "$core_cursor" "$allocated" "$TOTAL_CORES")
taskset -cp "$cpuset" "$pid"
core_cursor=$(( (core_cursor + allocated) % TOTAL_CORES ))

Once this was in place, the log began showing proper circular allocations:

10,11,12,13,0,1,2,3
12,13,0,1
3,4,5,6,7,8,9,10

That was the first real repair: the rotation now actually rotates.

This fix was committed and pushed to the black-vajra/boinc-devel repository as:

3e9c85d Fix BOINC affinity core rotation wraparound

The second problem: BOINC said 11 CPUs, but the script saw 14 cores

After the wraparound fix, the script was technically correct but still a little too aggressive.

BOINC itself was configured to use about 80 percent of CPU time and showed a usable CPU count around 11. But the affinity script was still using the full nproc count of 14 logical cores as the managed pool.

That meant the script could assign BOINC workers across all 14 cores, even though the intent was to leave some breathing room for the desktop and system services.

This became visually obvious in the system monitor. ATLAS and Einstein were running, and the CPU graph looked active in a way that was technically productive but still too close to total system occupation. The machine was not failing, but the configuration did not match the intended policy.

The fix was to add an explicit managed-core cap:

MAX_MANAGED_CORES=11

Then replace the simple TOTAL_CORES=$(nproc) with:

SYSTEM_CORES=$(nproc)

if [ "$MAX_MANAGED_CORES" -gt 0 ] && [ "$MAX_MANAGED_CORES" -lt "$SYSTEM_CORES" ]; then
    TOTAL_CORES="$MAX_MANAGED_CORES"
else
    TOTAL_CORES="$SYSTEM_CORES"
fi

The startup banner now confirms both values:

System cores: 14
Managed BOINC cores: 11

That is exactly what I wanted. The host has 14 logical cores, but the BOINC affinity manager only rotates work across cores 0-10. Logical cores 11-13 are left outside the BOINC-managed pool for the desktop, browser, system services, and general sanity.

After restarting the service, the logs confirmed the new behavior:

Allocating 11 cores proportionally across 2 worker(s)
'python' → cores 0,1,2,3,4,5,6,7,8
'einstein_O4AS_2' → cores 9,10

Later cycles showed continued wraparound, but only inside the 0-10 pool:

'einstein_O4AS_2' → cores 10,0,1,2,3,4,5,6
'python' → cores 7,8

A direct thread check confirmed that nothing significant was running on cores 11-13:

=== THREADS CURRENTLY ON CORES 11-13 ===
PID TID PSR %CPU COMMAND COMMAND

In other words: empty.

That was the second major improvement. The script was no longer merely rotating correctly; it was rotating within the intended system policy.

The third improvement: applying affinity to all threads

The next issue came from the nature of ATLAS native work.

The hot ATLAS process appeared as:

python runargs.EVNTtoHITS.py

with CPU usage well above 100 percent, sometimes around 200-400 percent. That means the process is either multithreaded or aggregating significant worker activity under one visible process.

The script was using:

taskset -cp "$cpuset" "$pid"

That applies CPU affinity to a process ID, but for multithreaded workloads, it can be better to apply affinity to all tasks/threads associated with that PID. The taskset option for that is -a.

So the line became:

taskset -acp "$cpuset" "$pid"

That is a small change, but it matters. ATLAS native multithreaded jobs are exactly the kind of workload where controlling only the main task may not be sufficient. Applying affinity to all threads makes the script’s behavior more consistent with its intent.

After this change, the system monitor gave the confirmation I wanted: pegged cores were visibly rotating. Not evenly loaded at every instant, which is not the goal, but clearly moving from one group of cores to another over time.

The terminal logs and visual history lined up:

'python'          → cores 10,0,1,2,3,4,5,6
'einstein_O4AS_2' → cores 7,8

'python'          → cores 1,2,3,4,5,6,7,8
'einstein_O4AS_2' → cores 9,10

'python'          → cores 3,4,5,6,7,8,9,10
'einstein_O4AS_2' → cores 0,1

That is the desired behavior.

Workload validation: ATLAS, CMS, Theory, and Einstein

The best part of the session was not the script fix by itself. It was that the actual BOINC workloads began behaving.

ATLAS was already running. The active task showed:

ATLAS Simulation 3.01 (native_mt)
Running (11 CPUs)

That gave the affinity manager a real multithreaded native LHC workload to manage. The script detected the ATLAS Python worker and rotated it inside the managed pool.

CMS was confirmed earlier as well. CMS tasks appeared as VirtualBox workloads, and the machine successfully ran them without the previous obvious failure behavior. That matters because CMS had been one of the more troublesome workloads historically.

Then came the important final test: Theory.

Theory tasks were present but waiting. To test them directly, I temporarily suspended everything else: Einstein, non-Theory LHC tasks, CMS, and ATLAS. The point was not to abort work, but to force BOINC into a Theory-only condition and see whether the Docker workload would actually start.

It did.

The BOINC task list showed multiple Theory tasks running:

LHC@home Theory Simulation 302.10 (docker) — Running
LHC@home Theory Simulation 302.10 (docker) — Running
LHC@home Theory Simulation 302.10 (docker) — Running
LHC@home Theory Simulation 302.10 (docker) — Running

Einstein was suspended by user request. CMS was suspended. The GPU dropped to almost idle, as expected, because the Einstein GPU workload was no longer active.

That confirmed the most important operational point: this host is now capable of running the major LHC@home workload classes I care about:

• ATLAS native multithreaded simulation
• CMS VirtualBox simulation
• Theory Docker simulation

alongside Einstein@Home GPU work when the project is allowed to run.

That is a much healthier state than where this machine had been during the earlier troubleshooting period.

What this says about BOINC project usability

This work also reinforces a larger point about volunteer computing projects.

A scientifically serious project should not require every volunteer to become a part-time systems engineer. It is one thing for me to build local scripts because I enjoy Linux, BOINC, and finding out why things break. It is another thing for an ordinary volunteer to install BOINC, attach LHC@home, and then be expected to understand the scheduler implications of VirtualBox, Docker, multithreaded ATLAS, CPU caps, affinity masks, process descendants, and file ownership issues.

The project may be scientifically sophisticated. The volunteer interface should not require a quantum physics degree.

To their credit, the current workload behavior appears better than some of what I saw earlier. CMS is running. Theory is running. ATLAS is running. That is real progress on my host. But the local affinity manager still provides value. It keeps compute work inside a controlled pool, rotates hot cores, preserves desktop responsiveness, and gives me better observability into what BOINC is actually doing.

In other words, the script is not a substitute for BOINC. It is a local safety rail.

The long-term question is whether BOINC projects with complex workloads should expose more user-facing profiles. For example:

• low-risk native-only mode
• VirtualBox-enabled mode
• Docker-enabled mode
• laptop-safe mode
• high-memory workstation mode
• GPU-plus-CPU mixed workload mode
• leave-N-cores-free mode

BOINC has preferences, but they often do not map cleanly onto the practical failure modes a user sees. “Use 80 percent of CPUs” is useful. But it does not necessarily stop a particular wrapper, VM, or multithreaded process from behaving in ways that feel hostile to the local desktop.

A user should not have to reverse-engineer the project’s operational assumptions just to contribute reliably.

Current state

At the end of this round, the local configuration is much stronger.

The affinity manager now has:

Correct circular wraparound
MAX_MANAGED_CORES=11
All-thread taskset application with taskset -acp
Proportional allocation among active workers
Low-priority renice behavior
Active worker detection by CPU threshold
Support for ATLAS, CMS/VirtualBox, and related BOINC worker patterns

The host has demonstrated:

ATLAS running
CMS running
Theory Docker running
Einstein GPU running
CPU hot spots rotating
Cores 11-13 reserved from BOINC affinity assignment
Desktop responsiveness protected

The first wraparound fix has already been committed and pushed to GitHub. The later live-script refinements — managed-core cap and taskset -acp — should also be synchronized back to the repository if they continue to behave well under longer runtime.

Remaining work

There are still a few follow-up tasks worth doing.

First, I want to let Theory run longer and confirm completion, not merely startup. Starting a Docker Theory task is a major milestone, but a completed and validated task is the real endpoint.

Second, I want to observe whether the affinity manager catches all relevant Theory/Docker worker processes, or whether the visible hot process is still mostly ATLAS while Theory is running in a different way. Docker workloads may expose themselves through BOINC wrappers, container processes, Python, or other child processes. It is worth verifying the process tree during sustained Theory execution.

Third, I may eventually test LHC@home without the affinity manager at all. That would answer an interesting question: has the project/client stack improved enough that the local script is no longer necessary, or is the script still quietly preventing rough edges from becoming failures?

That test should be controlled. The right way to do it is to record a baseline with the script enabled, stop the affinity service, run BOINC for a fixed window, compare task success and system responsiveness, and then restore the service.

Finally, the repository should be updated with the latest live-script refinements after a little more runtime confidence.

Conclusion

This was a productive maintenance session.

What began as a simple observation — “ATLAS is running, but where are CMS and Theory?” — turned into a useful repair of the local BOINC management layer. The affinity script had a real bug. The wraparound logic was wrong. It was rotating by shrinking. That is fixed. The managed-core policy was too broad. It used all 14 logical cores when the intended BOINC pool was closer to 11. That is fixed. The taskset call was not ideal for multithreaded workers. That is fixed.

More importantly, the actual science workloads are now alive.

ATLAS runs. CMS runs. Theory runs. Einstein runs.

The machine is no longer merely donating compute cycles. It is doing so in a way that is observable, controlled, and much less abusive to the local desktop.

That is the real win. Volunteer computing should be boring once it is configured. The work should go out, the results should come back, and the machine should remain usable by the human sitting in front of it.

After this round of repairs, this host is much closer to that ideal.

You can find the full history at our github - https://github.com/black-vajra/boinc-devel

Jonathan Brown is a cybersecurity researcher and software engineer at bordercybergroup.com.

If you would like to support our work — useful, well-researched, ad-free cybersecurity intelligence — subscribe, comment, or buy us a coffee! Thanks.