• Nobody cares. "People care about money, they don't seem to care about environmental savings, even if they're faster."
• The cost of switching is too high. Even a one-line change was too much friction.
• AI makes CI look cute. Datacenters keep getting built on cheap land in hot places running on coal and gas.

Not "Hello world!" but "Sorry world!" they wrote. 😢🌍 "Sorry we couldn't make the internet a greener place in 2025." I don't know about you, but this one hit me hard.

To be fair, startup failures are rarely proof of market indifference. There are a hundred ways to miss product-market fit that have nothing to do with whether the underlying problem matters. But the question buried in their epitaph is still worth grappling with. The carbon case is real and already lands with a lot of engineers. The harder problem is getting it to survive contact with those engineers' boss or whoever controls the budget.

Meanwhile, in the research world, the state of the art keeps advancing. Researchers are publishing increasingly sophisticated multi-variable optimization frameworks. New papers demonstrate that optimizing for carbon intensity alone can, in specific scenarios, shift environmental burdens (ex: water stress, air pollution, grid congestion) to already-strained regions. The science is getting better, but adoption is flat-lining or getting worse.

The Distinction Is Real

Let's define a couple terms before we go much further.

Carbon-aware computing means responding to real-time carbon intensity signals when making scheduling decisions and running deferrable workloads when and where the grid is cleaner. This is what most tools in the space do today, including Compute Gardener, the Carbon Aware SDK and what CarbonRunner did before turning off the lights.

Grid- or sustainability-aware computing means optimizing for the broader health of the grid and the local environment, of which carbon intensity is one (often the primary) signal among several. Water stress in the datacenter's basin. Air quality near the generation source. Grid congestion and price spikes that indicate system strain. These are real dimensions that vary by region and time and they don't always move in the same direction as carbon intensity.

Consider what this framing implies: carbon-aware isn't competing with grid-aware. It's an instance of it; grid-aware with one signal wired in. You can't argue carbon-aware vs. grid-aware any more than you can argue Camry vs. Toyota. The debate, to the extent there is one, is about scope and signal maturity, not about whether the approach is fundamentally right.

Geerd-Dietger (Didi) Hoffmann and Verena Majuntke recently demonstrated this with Orca, in a paper accepted to ICT4S '26. In quick summary, Orca is a sustainability-aware scheduling framework that jointly optimizes across carbon, water stress, air pollution exposure and grid stress indicators. Their work is solid and the direction is true. In a three-region case study (Frankfurt, Tokyo, Northern California), they showed that CO2-optimal routing sometimes diverges from locally-optimal routing. This is observed especially when a low-carbon region happens to be water-stressed or experiencing poor air quality.

I recently had the chance to speak with Didi about this work and I genuinely think it moves the field forward. Optimizing for a single variable in a complex multivariate system can push you away from the global optimum. Any engineer who's ever tuned a system knows this intuitively. And yet...

The Correlation Is Strong With This One

Here's what Orca's own data shows. In their case study, the carbon-only baseline and the multi-criteria optimum agreed on the same region for roughly two-thirds of the hours evaluated. The divergences were real but not dramatic and they were driven primarily by one region's (Japan's) persistently high water stress penalty.

This matches what we've seen in practice. Carbon intensity is strongly correlated with most other grid health indicators most of the time. When the grid is clean, it's usually because renewables are abundant. That, in turn, means less combustion and particulate, less water-intensive thermal generation, less air pollution from fossil plants and lower wholesale prices. The signals move together far more often than they diverge.

Don't get me wrong, when they do diverge, it matters. But the scenarios where carbon-optimal scheduling meaningfully worsens local outcomes generally involve specific regional characteristics. Factors such as a low-carbon grid that happens to sit in a water-stressed basin or a clean-energy window that coincides with grid congestion from other demand. These are knowable, addressable edge cases. And more than that, they're a roadmap. The divergences tell us exactly which signals to integrate next: basin-level hydrology, weather forecasting, real-time air quality, grid congestion. Not reasons to abandon carbon-aware scheduling, but the agenda for how it evolves additional dimensions of awareness.

The Signal Maturity Problem

Here's the part that gets lost in the academic discussion: carbon intensity is really the only environmental signal we have that's reliable, real-time, API-accessible and globally available in a form that schedulers or routers can actually consume.

The additional signals that frameworks like Orca want to consider (basin-level water stress at sub-monthly resolution, real-time air quality per datacenter zone, grid congestion indicators in comparable units across jurisdictions, operator-disclosed Water Usage Effectiveness figures) are either incomplete, regional, paywalled, updated infrequently, require bespoke integration per datacenter operator or, most commonly, some combination of these.

Orca's own evaluation highlights missing data points throughout their air quality time series (OpenAQ gaps), simplified water modeling (WUE multiplied by a coarse stress band) and the limitation of average-mix rather than marginal emissions factors.

These are real concerns AND they're not yet addressable at the tooling layer because the signal infrastructure doesn't exist. Carbon intensity is the floor we can build from today. As water stress data, grid congestion signals and air quality APIs become similarly mature, they absolutely should and will be incorporated.

But at that point, it's a signal upgrade. Not a paradigm shift.

The Behavioral Change Is the Hard Part

This is the part that matters most. And it's the part that CarbonRunner's epitaph makes impossible to ignore.

CarbonRunner didn't fail because they were using the wrong signal. They didn't fail because carbon-only optimization might occasionally route a CI job to a water-stressed region. They failed because, in their words, "nobody really cares."

The bottleneck to carbon-aware computing (or awareness of any kind) has never been signal quality. It's always been adoption. The activation energy required is organizational and cultural: classifying workloads by flexibility, shifting institutional expectations (a batch job submitted at 2am doesn't need to start at 2am — it just needs to finish by 8am), and integrating a scheduling layer that actually acts on external signals. No longer is it a "fire-and-forget" operation, but a decision with ongoing environmental (and economic) consequences. That's hard. It's unfamiliar. It requires changing habits that have been optimized for developer convenience or costs alone, not environmental awareness.

But here's the thing: that behavioral change only really needs to happen once. Once an organization has classified its workloads by flexibility, once it has built the scheduling infrastructure to act on external signals, once the institutional muscle memory exists, upgrading the signal set is trivial. Swapping a carbon intensity value for a multi-variable sustainability score is only a configuration change. The hard work is building organizations that can respond to any external signal at all. Getting them to respond to one unlocks responding to five. The habits transfer. The tooling upgrades.

The research community is doing important work improving signal quality and optimization scope. But debating whether to optimize for carbon intensity alone or a Pareto-weighted vector of carbon, water stress, NOx exposure and grid congestion is like arguing over wine pairing for a dinner that no one has RSVP'd to.

The Excused Inaction Pattern

I keep seeing the same reasoning structure across different conversations in this space.

• "Carbon intensity doesn't capture marginal emissions accurately." → True in some contexts.
• "Carbon-only optimization might increase water stress." → Possible in specific region+time combinations.
• "We should fix energy markets so prices incorporate externalities, then we won't need carbon signals." → A worthy long-term goal.

Every one of these is a legitimate observation about a real limitation. And every one of them, in practice, gets elevated from "limitation" to "road block" along its journey to the all-too-common destination of excused inaction.

It's a pattern, and once you see it in one discussion, you begin to see it everywhere.

To be perfectly clear: the researchers aren't "the problem." Didi's paper advances the field in exactly the right direction. Tammy Sukprasert's work at UMass Amherst on marginal vs. average signals sharpens our understanding. These are scientists doing what scientists should be doing... pushing toward more accurate models of complex systems.

The problem is what happens when those nuanced findings propagate into industry conversations. Even as multi-variable optimization is the right direction, some forces would happily weaponize its complexity as an excuse to skip earlier steps entirely. "Necessary but not sufficient" gets heard as "not necessary." "Has limitations" gets heard as "doesn't work." And the conclusion is always the same: we need to understand the problem better before we act. Which, as we know, is a wonderful way to delay action.

If We Build it, Will They Come?

We're building at a time where environmental stewardship isn't just undervalued, in some markets, it's actively penalized. "Sustainability" has become a dirty word in some boardrooms which were publishing sustainability reports only two years ago. CarbonRunner didn't fail because their technology didn't work or provide the value stated. They failed because the market they were selling into doesn't yet feel sufficient pressure (whether regulatory, economic or social) to care.

The EU's Corporate Sustainability Reporting Directive doesn't care about the political winds in Washington. Electricity prices don't care about your feelings regarding climate change. And the physical constraints of water-stressed grids feeding power-hungry datacenters don't negotiate with anyone's politics.

When these pressures mount again (and they will, regardless of any single administration's or nation's posture), the question won't be "Should we optimize for sustainability?" It'll be "How fast can we start?" The organizations that already know how to flex their compute will adapt in weeks. The ones that don't will spend years trying to catch up.

The major cloud providers offer carbon footprint reporting. These tools are genuinely useful for compliance and retrospective accounting, but post-hoc and coarse-grained by design. They can tell you how much carbon your EC2 fleet generated last month. They cannot help you make operational decisions such as to defer your batch job four hours because the grid is dirty or electricity is cheaper in off-peak hours. That real-time, workload-level scheduling layer is precisely what's missing.

And this is why tools that work on economic signals alongside environmental ones matter. The rough correlation between "green" and "cheap" doesn't require a friendly regulatory regime to deliver value. At Compute Gardener, we call this our green↔cheap knob. It's a blending engine that lets organizations weigh carbon optimization against cost optimization. In environments where you can pitch sustainability, lean green. In environments where that's a dirty word and you can only pitch savings, turn the knob to cheap. The scheduling flexibility you build serves both goals, because the signals are correlated more often than not.

Where This Is Heading

The research community is converging on what we might call "sustainability-aware" computing. That is, multi-variable optimization that considers carbon, water, air quality, grid health and cost as composable signals with configurable weights. That is the right destination.

At Compute Gardener, we're similarly on the journey from carbon-aware toward additional layers of awareness. Our architecture is built around a modular signal ingestion and blending engine. We use carbon intensity and electricity pricing today, with a modular design that allows incorporation of additional signals as they mature. Ultimately, the scheduler doesn't care whether its decision was based on optimizing for one variable or five. The organizational capability of classifying workloads and scheduling flexibly is the foundation. Everything built on top of that is an incremental lift.

Getting organizations to think about workload flexibility, to classify jobs by deferrability, to build scheduling infrastructure that responds to external signals; that's the expensive, culture-changing work. But, fortunately, it only needs to happen once. And it produces value on day one, even with carbon intensity as the only signal and even in a political environment that can't say the word "sustainability" out loud.

Ultimately, there's no debate between carbon-aware and grid-aware computing. If one exists, it's between awareness and unawareness. And closing that gap is where every bit of energy in this space should be focused. At the very least, it's where we continue to see value and will keep contributing.

Then, one day soon, new, upgraded and more reliable signals will become available. The research will keep improving. The optimization scope will broaden. But none of that matters if nobody cares.

You care. Start somewhere. Start now.

Interested in what sustainability-aware computing patterns could look like for your workloads? Whether you're running ML training, CI/CD pipelines, batch processing or any deferrable compute, we could help you quantify savings opportunities. Check out our open-source Kubernetes scheduler or reach out about a consulting engagement and we can continue the conversation considering your particular infrastructure.

Dave Masselink builds carbon/grid/sustainability-aware scheduling and routing tools at Compute Gardener. He's pretty sure the hardest part of sustainable computing is spurring action and has little to do with deciding the correct signal(s) to consider.

Why Write This?

I keep having the same conversations... with smart engineers, genuinely trying to understand carbon-aware computing. But sometimes they raise concerns/objections that sound reasonable yet rest on flawed assumptions about how the grid actually works.

The objections sometime sound like:

• "If you use clean energy, someone else gets pushed to dirty energy"
• "Shifting load will destabilize the grid"
• "Marginal emissions are what matter, not average intensity"
• "This only works if we have 90%+ renewables. And then, does it even matter?"
• "Wait, you want my $$$$ GPUs to sit idle?"

These aren't dumb people. I think they're trying hard to understand. They're voicing sophisticated sounding concerns that happen to be wrong. Or at least, misguided in ways that would end up confounding practical decision-making.

The problem: technologists think about grids like they think about compute clusters. More-or-less fixed capacity. Zero-sum allocation. Deterministic scheduling.

But grids aren't clusters. They're economic systems with markets, slack, planning horizons and operators who adapt to ever-changing patterns.

Fallacy 1: Musical Chairs 🪑

The claim: "Clean energy is limited. When you take it, someone else gets pushed onto fossil power. Net effect: zero... or worse"

Why it sounds right: In a datacenter, if you take a CPU core, someone else can't use it. Fixed resources, zero-sum allocation.

Why it's wrong: Grids aren't fixed-size resource pools.

When you move load from Virginia's fossil heavy grid (~500 gCO2eq/kWh) to Quebec's hydro powered grid (~50 gCO2eq/kWh):

• You don't "take" someone's hydro electrons
• Quebec doesn't tell existing customers "sorry, you get coal now"
• Quebec responds by... continuing to run hydro, because that's what it always has
• At worst, Quebec's marginal intensity bumps slightly — maybe to 55 gCO2eq/kWh
• Your compute still runs at ~1/10th the intensity compared to Virginia

The better mental model: Markets, not resource pools.

Buying more EVs doesn't force someone else to buy a gas car. It shifts demand curves and production incentives over time. It's the same with clean electricity; sustained demand in clean regions signals investment in more clean capacity.

Fallacy 2: The Fragile Grid 🩹

The claim: "Sudden load shifts trigger fossil backup, raise prices and risk grid instability."

Why it sounds right: Grids must balance supply and demand in real-time. Imbalances cause blackouts. Therefore, moving load = dangerous.

Why it's wrong: Grids are designed for variable load. Your compute job is not even a rounding error.

Scale check — what actually stresses grids:

Remember, here are just a few things the grid can support every day, without incident:

• Millions of air conditioners cycling on/off
• "it's half-time" → everyone opens their fridge and starts the [microwave] oven within a few seconds
• Cloud cover rolls across solar farms
• Morning demand ramps as cities wake up (or in the afternoon on solar-heavy grids)

Even if the largest data centers are relevant loads, individual compute runs/jobs hardly ever are. What is your batch job running over a couple hours? Even if it were meaningful in magnitude, grid operators would see it as predictable baseload. Practically a gift.

Rule of thumb: If your facility isn't comparable to an aluminum smelter (500MW+ constant draw), you're not a grid stability concern. You're a rather simple customer.

Fallacy 3: The Thundering Herd 🐏

The claim: "If everyone adopts carbon-aware scheduling, all load shifts to the same clean windows and crashes the grid."

Why it sounds right: Classic distributed systems problem. Everyone retries at the same time → system overload.

Why it's wrong: Three reasons.

1. Optimization spreads load, it doesn't synchronize it.

Carbon-aware schedulers don't fire every possible job in a single millisecond. They spread deferrable work across clean windows that are typically hours long. The "thundering herd" assumes everyone has identical deadlines and zero flexibility; but that's not the case for large swathes of deferrable compute.

2. The Invisible Hand steps in.

If too much load clusters in clean windows, wholesale prices rise and some load shifts back. This is how markets work. The equilibrium point isn't "everyone stampedes to 2pm." It's "2pm is more clean, but only carbon-sensitive loads may care."

3. Grid operators aren't NPCs.

Indepentent System Operators (ISOs) observe patterns and adapt. Load clustering at low-carbon hours? They schedule more generation for those hours. This is literally their job. The "optimization breaks everything" scenario assumes operators don't respond to changing demand; but theirs is a profession built on forecasting.

Fallacy 4: The Idle GPU 🤑

The claim: "Carbon-aware scheduling means expensive hardware sits unused during dirty windows."

Why it sounds right: If you defer work during high-carbon periods, the GPUs must not be earning their keep.

Why it's wrong: Deferral ≠ idleness. In practice, it becomes a prioritization signal and mechanism.

In a world with carbon-aware scheduling:

• Urgent, high-value work runs whenever it wants, including during dirty windows (at an implicit carbon premium)
• Deferrable, speculative work shifts to cleaner windows
• Total energy utilization may stay the same — but total system emissions drop!

Nobody's $$$$ GPUs are less than fully amortized! Market participants are simply influenced differently by the same signals. Think of congestion pricing on the roads. Rush hour doesn't mean empty highways. Discretionary trips shift; essential trips don't.

Fallacy 5: "Just Build More Renewables" 🌞

The claim: "Carbon-aware scheduling is a distraction. We just need more clean generation."

Why it sounds right: When the grid is 100% clean, timing shouldn't matter. Focus on the supply side.

Why it's incomplete: It's a classic "yes, and..." situation. Even with abundant clean generation, you need:

• Transmission capacity to move it
• Storage to buffer intermittency
• Grid stability services (frequency response, voltage support)

Flexible load (demand that can shift to match supply) is valuable infrastructure. It helps integrate renewables by absorbing variability. Carbon-aware compute isn't just about finding clean energy; it's about being a good grid citizen, facilitating more renewables coming online more quickly.

Clean capacity buildout and demand flexibility aren't competing strategies. They stack and complement each other.

Fallacy 6: Marginal Absolutism 💯

The claim: "Average grid intensity doesn't tell you your actual impact. Only marginal emissions matter."

Why it sounds right: Economically, marginal analysis is often correct. Your load is (mostly) served by whatever generator ramps up to meet it, not the grid average.

Why it's incomplete: We're still talking about meager loads. And especially when it comes to spatial shifting, marginal vs. average differences barely matter.

• Quebec's marginal source (hydro or imports from hydro-heavy neighbors) is (still much) cleaner than Virginia's marginal source (gas)
• The 10-15x intensity gaps between clean and dirty grids outweigh most marginal nuances

When it can matter more: Temporal shifting within a single grid with high renewable variability. The "cleanest" hour by average intensity might not be the hour with lowest marginal impact.

For most practical decisions: If you're choosing between Region A or B... Or you're choosing when, within a single grid, to run... average intensity is usually, at least, directionally correct. Don't let perfect be the enemy of good.

What Grid Operators Actually Care About ⚙️

They need to know:

• New large facilities (100+ MW sustained) — capacity planning
• Major industrial additions (smelting furnaces, fabs, large datacenters) — transmission planning
• Demand response program participation — operational planning

They like to know:

• Regional datacenter expansion trends
• EV charging pattern shifts
• Large commercial load forecasts

They care about:

• Reliability
• Reliability
• Reliability - this isn't a joke

They don't care about:

• Your 100 GPU cluster's schedule
• Whether your training job runs at 2am, 2pm or whenever
• Individual workload optimizations

But WE are allowed to care. Afterall, we all have lungs.🍃

The Bottom Line

Grids are resilient, adaptive, market-driven systems operated by the most responsible professionals you can possibly imagine. They've been balancing variable load for a century. The grid handles unexpected weather, sporting event demand synchs and large-scale industrial swings daily.

Your carbon-aware practices aren't going to break anything. ❤️ They will:

• Send demand signals toward clean regions → investment follows
• Shift flexible load to times/places with abundant clean supply → better utilize what's already clean
• Make you a more predictable, valuable customer → grid operators will like you 🎉💃🏽

Barriers to carbon-aware computing are real: operational complexity, data residency, latency requirements. "Grid stability" shouldn't be one of them.

Stop worrying about breaking the grid. Instead, optimize away carbon emissions today!

Let's figure out your carbon optimization opportunity →

Compute Gardener is an open-source project focused on making carbon-aware computing simpler. Join us in moving sustainable computing forward.

The Story So Far

In Part 1, we showed that shifting ML training jobs to cleaner time windows could reduce carbon emissions by ~30%. In Part 2, we scaled up to a real hyperparameter search, 21 LoRA fine-tuning experiments orchestrated with Ray, and achieved 40% reductions in carbon emissions using a tiered scheduling strategy.

Both experiments used Compute Gardener's scheduler to make temporal decisions: not whether to run a job, but when. And both ran on hardware in our lab — GPUs we own, in a single location, drawing from a single grid region.

Part 3 asks a different question: what if we could choose where to run?

Why Spatial Shifting Matters

Temporal shifting works because grid carbon intensity varies throughout the day. But it also varies across grids — dramatically. At any given moment, Oregon's hydro-heavy grid might be running at 80 gCO2/kWh while Virginia's gas-heavy grid sits at 350. That's more than a 4x difference, available instantly, no waiting required.

For most ML scientists and engineers, this is probably more accessible than temporal shifting. You may not have a dedicated lab with GPUs and a Kubernetes scheduler you control. But you can (sometimes) choose a cleaner cloud region to spin up your training job in. The question is whether anyone actually does — and whether the savings justify the effort.

Why Another Experiment?

The academic literature is full of simulations showing 20-40% potential savings from spatial shifting. Could a simulation predict that clean grids beat dirty ones? Sure. But simulations don't tell you whether Vast.ai actually maintains GPU availability across regions, whether Tailscale works reliably across continents, whether the Electricity Maps API is stable enough for real-time routing or whether a prototype can be setup in an afternoon.

Part 3 extends our real-world testing to cloud infrastructure; where most ML training actually happens. The experiment validates operational feasibility, not just theoretical carbon math.

An Experiment in Routing

Compute Gardener's scheduler currently handles temporal shifting within a single cluster. Spatial shifting is on the roadmap, but rather than wait for the perfect design, we experimented with the simplest approach: a routing script that checks carbon intensity across three regions and sends each job to the cleanest one.

This isn't what a production spatial shifting system would look like. (For anyone curious, we'll probably build on top of a gitops or hub+spoke multi-cluster strategy.) But it's small enough to build quickly and honest enough to teach us where the complexity actually lives.

The Setup

We kept the workload familiar: LoRA fine-tuning of Qwen2.5-Coder-7B, the same model from Part 2. But we stripped the infrastructure down to essentials: no Kubernetes, no Ray orchestration. Just three GPU instances, SSH and a routing script.

Three regions, three grid profiles:

• Quebec: Hydro-Quebec grid, one of the cleanest grids in North America. A viable clean option that's geographically close for US-based teams.
• Ohio: PJM grid, gas/coal-heavy, typically the highest carbon intensity of our three options. Representative of the "default" US East/Virginia/Midwest data centers where most workloads already run.
• Norway: Nordic grid, among the cleanest in the world due to hydro dominance. The "what if we could route to fundamentally different energy infrastructure" option.

We chose these regions to represent a practical range: the status quo (Ohio/Virginia), a clean North American alternative (Quebec) and an aspirational clean option (Norway). This creates a stark contrast — that's intentional. The question isn't whether clean grids exist, but whether dynamic routing into such a pool is accessible or necessary.

Hardware: NVIDIA RTX 4090 GPUs (24GB VRAM) rented via Vast.ai in Quebec, Ohio and Norway.

Network: Tailscale VPN connecting all instances to the router script running on a local machine, simplifying SSH access without exposing instances to the public internet.

Routing logic: At each scheduled interval, the script queries Electricity Maps for current carbon intensity in each region, picks the lowest, SSHs into that instance and kicks off training. No complex orchestration — just python train_lora_cloud.py --r X --alpha Y --lr Z executed remotely.

The script waits for each job to complete (~45 minutes), then waits 2 hours before the next submission. This gives roughly 2.75-hour intervals between job starts, allowing us to sample different times of day across all three grids.

The Infrastructure

Router script (runs locally in our California lab):

def pick_cleanest_region():
    """Query Electricity Maps, return cleanest region."""
    intensities = {}
    for region, config in REGIONS.items():
        response = requests.get(
            "https://api.electricitymap.org/v3/carbon-intensity/latest",
            params={"zone": config["zone"]},
            headers={"auth-token": ELECTRICITY_MAPS_TOKEN}
        )
        intensities[region] = response.json()["carbonIntensity"]
    
    return min(intensities, key=intensities.get), intensities

def submit_job_ssh(region, config, intensities):
    """SSH into region and kick off training."""
    ip = REGIONS[region]["ip"]  # Tailscale IP
    
    remote_cmd = (
        f"CARBON_REGION={region} "
        f"CARBON_INTENSITY={intensities[region]} "
        f"python3 train_lora_cloud.py "
        f"--r {config['r']} --alpha {config['alpha']} "
        f"--lr {config['lr']} --dropout {config['dropout']}"
    )
    
    subprocess.run([
        "ssh", "-i", SSH_KEY_PATH, f"root@{ip}", remote_cmd
    ])

The entire "routing layer" is a few hundred lines of Python.

The Submission Schedule

We ran 21 jobs over approximately 2.5 days. Each job trains the same Qwen2.5-Coder-7B model with LoRA, using different hyperparameter configurations from a predefined sweep (varying rank, alpha, learning rate and dropout).

The jobs weren't identical — we varied hyperparameters across the sweep to make the training genuinely useful:

Each experiment trains Qwen2.5-Coder-7B on 5,000 samples from the HelpSteer2 dataset. Training time: roughly 0.76 hours per job on 4090 GPUs.

Results

Where Did Jobs Actually Run?

Of the 21 jobs submitted over ~55 hours:

• Norway: 21 jobs (100%)
• Quebec: 0 jobs (0%)
• Ohio: 0 jobs (0%)

[Was planning to have a visualization here but... umm. Norway ate the whole pie! Can you imagine a circle?]

Every. Single. Job. went to Norway.

This wasn't quite what we expected. We anticipated some distribution — Quebec catching jobs during Norway's peaks, maybe Ohio winning the odd mid-day window. Instead, Norway's hydro-dominated grid was so consistently clean (33-39 gCO2/kWh) that it won every routing decision by a wide margin.

Quebec's hydro grid is objectively clean (40-63 gCO2/kWh, avg 51), but Norway's was cleaner. Ohio's fossil-heavy grid (472-580 gCO2/kWh, avg 523) never came close to Norway's baseline.

What this teaches us: Spatial shifting doesn't require sophisticated runtime optimization. The cleanest region is often so much cleaner that dynamic balancing barely matters. Just pick Norway (or Quebec or Pacific Northwest) instead of Ohio/Virginia and you're getting at least 80% of the carbon saving benefits.

Carbon Intensity Comparison

The energy consumption is essentially identical — we're doing the same computational work either way. The difference is purely in where that energy came from.

The Counterfactual: What If We'd Just Used Ohio?

The real-world baseline is Ohio/Virginia, because that's "where my team runs things." To establish this, we calculated what emissions would have been if all 21 jobs had run in Ohio at the times they were submitted and run, instead, in Norway.

Ohio's grid averaged 523 gCO2/kWh during our experiment window, while our routed jobs (all Norway) averaged 35 gCO2/kWh — a 93% reduction from a single routing decision: "use Norway instead of Ohio."

To put it another way: Ohio's grid was nearly 15x dirtier than Norway's throughout our experiment. Even Ohio's best hour (472 gCO2/kWh) was over 12x worse than Norway's worst (39 gCO2/kWh).

Time-of-Day Patterns

The pattern was remarkably stable:

• Norway: Consistently 33-39 gCO2/kWh across all hours. Hydro-dominated grids don't vary much with time of day.
• Quebec: Ranged 40-63 gCO2/kWh. Also hydro-heavy, also stable, just slightly higher baseline.
• Ohio: Ranged 472-580 gCO2/kWh. Even Ohio's "best" hours couldn't compete with Norway's "worst" hours.

The key insight: When one grid is fundamentally cleaner by 15x, temporal variations don't matter.

What We Learned

The Simplicity Thesis: The 80/20 of Carbon Reduction

The headline result — 93% carbon reduction — exceeds what many simulations predicted. But the more important finding is how we achieved it: not through sophisticated dynamic optimization, but through a single decision: "use Norway instead of Ohio."

Norway won all 21 routing decisions because it's fundamentally cleaner (33-39 gCO2/kWh) than the alternatives. Quebec is clean (~51), but not as clean. Ohio is dirty (~523) and no amount of temporal variation brings it close to Norway's baseline.

The 93% number isn't a discovery; anyone who's looked at Electricity Maps for 10 minutes could predict Norway beats Ohio. The finding is that 93% is achievable with very simple tools and minimal setup friction. The experiment proved you can actually route to Norway, not just that you theoretically should.

This is an 80/20 leverage point. The actionable advice isn't "build a complex multi-region orchestration system." It's "check Electricity Maps for your available cloud regions, pick the cleanest one as your new default, done."

What Actually May Require Complexity

Region-to-grid mapping can take a bit of homework. Cloud regions and GPU rental locations don't always map cleanly to electrical grid zones. In Ohio, exact location determines whether you're on PJM or MISO. "Norway" on Vast.ai could technically mean different parts of the Norwegian grid, though Norway's grid is remarkably homogeneous due to hydro dominance. Ultimately, we spent as much time verifying our instances were in the right grid zones as we did building the routing logic.

When using public cloud providers, Electricity Maps API will even do that mapping for you.

Spatial vs. Temporal: Different Problems, Different Solutions

Spatial shifting can be about defaults. For many workloads, "just always use Norway" is enough. The clean region wins so consistently that dynamic routing adds complexity without much benefit.

But spatial shifting has thorny constraints:

• Data residency and sovereignty (GDPR, healthcare, financial regulations)
• Data transfer costs and latency (moving TB-scale datasets between regions)
• Regional availability and quota limits

Temporal shifting avoids most of these. Running the same job at 2pm instead of 2am doesn't move data across borders or incur egress fees. For many organizations, temporal shifting within a single region may be more practical than spatial shifting across borders.

The layers stack. The real opportunity is combining a few:

1. Pick the cleanest region you can use (spatial)
2. Within that region, run jobs during cleanest hours (temporal; ex: Compute Gardener)
3. Use infrastructure sleep strategies (ex: KubeGreen) to shut down idle resources
4. Use carbon-aware autoscaling (ex: KEDA with carbon signals) for variable loads
5. Application layer optimization

Each layer compounds the benefit.

Versus the Simulations

Academic papers often report 20-40% potential savings from spatial shifting. Our 93% result significantly exceeds that range because we deliberately chose a stark contrast: one of the cleanest grids on Earth (Norway hydro) versus a dirty, fossil-heavy grid (Ohio coal/gas). Simulations typically model more moderate contrasts or average across diverse workloads.

We're not claiming 93% is universally achievable — it's not. But using an extreme example makes the point obvious: the leverage exists and the workflow to access it is surprisingly simple. The carbon cost of proving this? About 0.21 kgCO2 for the Norway runs — less than driving 2 miles. The difference from running this experiment versus just simulating it is negligible; the difference between running in Norway versus Ohio for real workloads at scale is enormous.

Where Compute Gardener Fits

This experiment taught us that routing and scheduling are different problems:

Routing (which region): Can often be a pre-deployment decision. "Should this job run in Ohio or Norway?" This can be as simple as a policy: "all deferrable ML training goes to the generally cleanest grid."

Scheduling (when to run): Can be a runtime decision within a cluster. This is where Compute Gardener excels. For deferrable batch workloads in Kubernetes, Compute Gardener (CG) provides carbon intensity thresholds per pod, maximum delay enforcement, energy budget tracking and price-aware scheduling.

Why temporal shifting still matters: Even in a dirty grid like Ohio, intensity varies 472-580 gCO2/kWh throughout the day. A job scheduled to run during midday (472) instead of overnight (580) achieves ~19% reduction — without crossing borders, moving data or dealing with compliance headaches.

The complete picture for carbon-aware ML training:

1. Spatial layer: Pick the cleanest region your constraints allow
2. Temporal layer: Within that region, defer jobs to cleaner hours
3. Infrastructure efficiency: Sleep idle resources, scale based on carbon signals
4. Application efficiency: Make the work itself more efficient

For organizations that can't use Norway's grid due to data residency, temporal shifting becomes even more critical.

What About US-Only Spatial Shifting?

An interesting middle ground we didn't test: routing between US regions only. This avoids most data residency concerns while still leveraging grid variation (Pacific Northwest: 80-150, Ohio/Virginia: 400-600, Texas: 200-500).

Does spatial shifting within America behave the same way? Set-it-and-forget-it? Or is that an environment when the "dynamic optimization matters" thesis from academic papers rings true? Our hypothesis: somewhere in between, but likely still favoring "just default to Quebec or Pacific Northwest" for most jobs.

Limitations and Honest Caveats

Our experiment showed dramatic results (93% carbon reduction), but it's important to understand what this doesn't capture and when the "just use Norway" heuristic breaks down.

Data Transfer Costs and Latency

We assume training data magically appears wherever we need it. In reality, moving large datasets between regions costs money and adds latency. For our 5,000-sample dataset (~few hundred MB) and ~15GB model, this was negligible. For TB-scale datasets, it's a real constraint that fundamentally changes the economics.

Data Sovereignty and Compliance

Many organizations face legal requirements: GDPR restricts where EU citizen's data can be processed, healthcare data often can't leave national boundaries, financial services may prohibit cross-border processing. Our "just use Norway" advice only works if you legally can use Norway. For many orgs, US-only or EU-only spatial shifting is the only option.

No Temporal Shifting in This Experiment

We deliberately excluded temporal shifting to isolate spatial effects. Jobs ran immediately in whichever region was selected. Combining spatial + temporal could compound savings (95%+ total reduction), but adds coordination complexity.

Measurement Uncertainty

Unlike Parts 1 and 2, we don't have actual power meter data. We're using observed GPU utilization (~320W at P2 power level) plus an estimate for system overhead (~50W). Vast.ai instance locations aren't usually as precisely defined as hyperscaler regions. Our carbon intensity numbers are real (from Electricity Maps API), but total emissions calculations have ~10-20% uncertainty.

That said, even with 20% measurement error in both directions, the fundamental result holds: Norway's grid (33-39 gCO2/kWh) is so much cleaner than Ohio's (472-580 gCO2/kWh) that the magnitude of the difference isn't in question, only the precise percentage.

What's Next

For Compute Gardener's roadmap:

• Temporal optimization within regions: CG already handles this for Kubernetes workloads
• Integration with spatial routing policies: Organizations can set "use Pacific Northwest for batch jobs" policies, CG optimizes when within that region
• Hybrid on-prem + cloud: If you have local GPUs and cloud access, CG can help you weigh more nuanced trade-offs (ex: run now locally or defer and burst to cloud during clean hours).

We're not waiting for perfect solutions to start reducing emissions. The necessary tools exist today.

Reproducibility

All code, logs and data from this experiment are available in our GitHub repo.

To run your own spatial shifting experiment:

1. Rent GPU instances in multiple regions (we used Vast.ai RTX 4090s)
2. Set up Tailscale or similar VPN for easy instance access
3. Install Python ML dependencies on each instance
4. Get an Electricity Maps API key (free tier works for experimentation)
5. Configure the routing script with your instance IPs and grid zones
6. Run: python spatial_router.py --run-experiment

No Kubernetes required. Total compute cost: approximately $50 over 2.5 days. Could be much less with a more active provisioning strategy (we had all nodes reserved).

Is This Right for Your Workloads?

Every discussion of carbon-aware computing comes with the same caveat: results depend heavily on your specific usage profile. Your default region, your job duration, your flexibility constraints; they do all matter.

The experiments in this series demonstrate what's possible under specific conditions. Of course, your conditions are different. Maybe you're already multi-region and spatial routing is low-hanging fruit. Maybe your workloads are latency-sensitive and can't move. The only way to know is to look at your actual usage patterns and do the analysis.

Whether you're exploring carbon-aware scheduling for the first time or trying to quantify the opportunity for your specific infrastructure, we offer consulting engagements designed to answer: what would this look like for us and our wacky X stack?

Let's figure out your carbon optimization opportunity →

Compute Gardener is an open-source project focused on making carbon-aware computing practical, not just theoretical. Join us in moving sustainable computing from white papers to production.

But here's the thing about theoretical knowledge: it lives in your head, where it can't really hurt you, at least not emotionally. The possibility seems safe enough at first consideration. Comfortable, even. You can sit in a company cafeteria, sipping free coffee, thinking about all that flexibility and upside, and the theoretical risks feel... manageable. Abstract. Like something that happens to other people.

Months into this journey, I'm constantly being reminded that imagining these realities is a completely different beast than experiencing them.

So this post is my attempt to put some of those feelings into words; not because I have answers, but because I suspect I'm not alone in being surprised by how these things actually hit.

The "Right Thing" Trap

Every morning, I face a version of the same question: Am I doing the right thing today?

Not in a moral or philosophical sense. But in a brutally practical one: Should I be writing code or should I be reaching out to potential users? Should I refine the product or should I be building community? Should I dig into that technical problem I find fascinating or should I force myself to do the marketing work that makes my skin crawl?

The trap is this: I can easily convince myself that what I want to be doing is what I should be doing. I'm technical. I love solving technical problems. It feels productive. It feels like progress. And maybe it is! Or maybe I'm just hiding in my comfort zone while the things that actually matter (the uncomfortable work of putting myself out there, of building relationships, of doing marketing as a decidedly non-marketing person) languish on my todo list.

Am I sure I'm laying the groundwork for innovative projects to come or am I simply yak shaving our lab setup? Am I REALLY sure about that?

Before venturing out, I knew these things would be challenging. It turns out they're challenging not because the tasks themselves are insurmountable, but instead because every single day you have to choose discomfort over comfort, uncertainty over certainty, vulnerability over control. You have to do the things you're worst at first, the things that feel awkward and forced, knowing full well it might not even matter.

And here's what makes it worse: you're never quite sure you're making the right call. Maybe the technical work is the priority right now. Maybe I am overthinking the marketing stuff. Maybe, maybe, maybe.

The theoretical knowledge that "founders need to do uncomfortable things" does not prepare you for the daily, grinding reality of doing them anyway while second-guessing yourself the entire damn time.

The Impact Paradox

Let's talk about the math that doesn't quite add up, the calculation we all make when we decide to start something.

At a large company, there's a floor and a ceiling on your impact. You're probably not going to directly change the world from inside Google or Microsoft, but you're also not going to have zero impact. You'll ship features, help colleagues, contribute to products that reach millions of people. The impact is real, if bounded.

But that's not why we become entrepreneurs. We do it because we want to break through that ceiling. We want the upside: the possibility of having massive impact, of building something that genuinely changes how people work, of seeing our ideas actually matter in the world.

Of course, in reaching for that possibility, we've also removed the floor.

The most likely outcome of any startup isn't moderate success or even graceful failure. It's zero impact. Or close to it. You can pour years of your life into something, do everything "right," and still end up with a product nobody uses, a blog nobody reads, a company nobody's paying.

I knew this, statistically, before I started. Startups fail. Most ideas don't find product-market fit. That's just how it works. What I didn't fully appreciate was how this knowledge would feel in the middle of the journey. When you're months in, when you're questioning whether you've even identified the right metrics to track progress, when you're not sure if you're making traction or just moving in circles, the possibility of zero impact stops being a statistic and starts being a weight you carry.

The irony is that entrepreneurs are precisely the people drawn to situations where they can amplify their impact. We want the leverage and the possibility of outsized returns on our effort. But that same leverage cuts both ways. The structure that lets you 100x your impact also allows you to 0x it.

And we go in with eyes open! We tell ourselves we understand the risks. We probably even believe it.

But sitting in that Google cafeteria, contemplating the leap, imagining the freedom and flexibility and potential... I don't think I really felt what it would be like to worry that all of this might amount to nothing. That the conversations I'm having, the code I'm writing, the posts I'm publishing; all of it might just be noise that nobody hears.

Of course, there are silver linings. Learnings. Connections. Skills developed. These aren't nothing. But they also aren't the goal. The goal is impact. The goal is building something that matters.

"But It Might Work for Us"

There's an old Arrested Development bit where Tobias and Lindsay are talking about having an open relationship. They acknowledge that such relationships never work for anyone, people are only deluded into thinking they will. And then Tobias says, cheerfully: "But it might work for us."

I know I often feel like Tobias. And if you're the kind of person considering starting something, or in the middle of your own journey, you're probably thinking: "Sure, but this doesn't apply to me." You're different. Your idea is different. You'll do the uncomfortable work. You won't fall into the traps.

And you know what? Maybe you're right! Of course, many entrepreneurs do succeed. The statistics don't apply equally to everyone. Maybe you are the exception. I really hope you are.

But here's my warning, delivered with as much vulnerability and hard-earned wisdom as I can muster while maintaining some professional credibility: you probably won't believe you're in the trap until you're already caught in it. You may not realize you've been hiding in comfortable technical work until months have passed. You likely won't feel the weight of potential zero-impact until you realize it's already around your neck.

I'm not writing this to discourage you. Honestly, I don't believe anything I write could discourage the kind of person who I'm writing this for. If you're built for it, you're going to do it anyway, warnings be damned.

I'm saying it because maybe, just maybe, having someone mention these feelings makes them slightly less isolating when (not if) you encounter them. Maybe knowing that the gap between theoretical understanding and lived experience is shared by others makes the surprise of it sting a tiny bit little less.

Where I Am Now

I'm still figuring this out. I don't have a neat conclusion or a five-point action plan. I'm not always sure I'm doing the right things on any given day. I don't know if Compute Gardener will gain the traction it needs to survive, let alone thrive. I don't know if I'm measuring the right metrics or if I'm even asking myself the right questions.

What I am doing is sitting in the discomfort. Doing the marketing work even though it feels painfully awkward. Building in public even though it's vulnerable. Writing posts like this one even though it would be easier to just stick to technical content.

And I'm trying to remember that the people who succeed at this aren't the ones who avoid these feelings. They're the ones who feel them and keep going anyway.

If you're in this with me, whether you're months ahead or months behind: I see you. This is harder than they told us. But even if we heard the warnings, we know we were probably going to do it anyway.

Because despite everything I've written here, despite the uncertainty and the discomfort and the very real possibility that this might not work out... I still think it just might work for us!

Compute Gardener is a project aimed at simplifying carbon-aware computing in Kubernetes. If you want to follow along with this journey; the technical wins, the strategic pivots, the occasional existential crisis... you can find details about the project on our website, github repo or reach out directly (dave@elevated-systems.com). I'm always happy to compare notes with fellow founders, especially over coffee. The kind we pay for ourselves now.

Continuing from Part 1

In Part 1, we demonstrated how Compute Gardener (CG) could reduce carbon emissions by ~30% for simple, recurring ML training jobs. The setup was straight-forward: daily ResNet50 training jobs, shifted to cleaner energy windows.

But that raised important questions that have come up in related discussion:

If everyone delays to 2pm, haven't we just moved the problem?

This works for toy examples, but what about real ML workflows?

These are fair questions. So for Part 2, let's build a more complex and realistic scenario: hyperparameter optimization for LLM fine-tuning using Ray on Kubernetes. And let's do it with intelligent, carbon-aware scheduling that distributes load rather than creating new stampedes.

Real-World Scenario: LLM Fine-Tuning Hyperparameter Search

Qwen2.5-Coder was one of the more pleasantly surprising open-weights models I've worked heavily with in the past year. This model made me realize that open models aren't more than a year (probably more like just 6 months) behind the capabilities of closed frontier models (of similar sizes).

Today, Qwen2.5-Coder is not the "latest and greatest" of open models BUT it does hold a soft spot in my heart. And perhaps more importantly, the newer Qwen3-Coder models are all too large to train on immediately available GPUs (even with some quantization).

So, we decided to fine-tune Qwen2.5-Coder-7B to create a specialized coding assistant. Like most ML projects, we didn't know the optimal training configuration upfront, so we needed to test multiple LoRA (Low-Rank Adaptation) hyperparameter combinations:

• Rank (r): [16, 32] — size of adapter (r=32 is the practical limit for 7B models at bfloat16 on a 24GB GPU)
• Alpha (a): [32, 64, 128] — scaling factor for LoRA weights
• Learning rate (lr): [1e-4, 5e-5, 1e-5] — how aggressively to update weights
• Dropout (d): [0.05, 0.1, 0.15] — regularization to prevent overfitting

A full factorial sweep would be 54 combinations (2 × 3 × 3 × 3), too many for this experiment. Instead, we designed a targeted sample of 21 experiments that systematically explore the hyperparameter space:

• 1 baseline and 7 related variations (Tier 1, r=16): Testing learning rates, alpha values and dropout values around the proven baseline rank
• 4 promising directions (Tier 2, r=32): Stepping up model capacity with standard hyperparameters
• 9 experimental long shots (Tier 3, r=32): Aggressive/unconventional hyperparameter combinations (high alpha, extreme dropout, very conservative learning rates)

This is a hyperparameter sweep; training the same model architecture multiple times with different configuration settings, then comparing results to find the best combination. It's one of the most common workflows in ML research and production model development.

Note on scope: The following approach works best for batch/deferrable workloads (ex: hyperparameter searches, model evaluations, periodic retraining, research experiments). Production inference requires different carbon-awareness strategies like geographic load balancing and efficient model architectures.

Training Dataset

We're fine-tuning on nvidia/HelpSteer2, an open-source dataset of 21k prompt-response pairs with human annotations for helpfulness, correctness, coherence, complexity, and verbosity. It's NVIDIA's recommended dataset for training helpful assistants.

For this experiment, we use 5,000 samples split 80/20 into training (4,000) and validation (1,000). This keeps training times reasonable while providing enough data to demonstrate the carbon-aware scheduling approach and compare hyperparameter configurations.

Setup and Timeline

Training times vary by rank due to the increased LoRA parameter count:

• r=16: ~1.4 hours (8 experiments)
• r=32: ~1.6 hours (13 experiments)

This is a total of ~32 GPU hours.

We used two 24GB GPUs (one NVIDIA RTX 3090, one NVIDIA RTX A5000) on separate nodes in our Kubernetes cluster. The experiment ran for just over 2 days (51.3 hours), 28-30Oct2025. The consumer RTX 3090 is ~5% faster at completing these jobs but uses ~80% more energy compared to the enterprise A5000. This presents a trade-off in carbon-aware scheduling: slightly faster completion vs much lower energy consumption.

The Traditional Approach: Random Submission

The typical workflow looks like this:

# Monday morning: Try a baseline
kubectl apply -f rayjob-r16-a32-lr1e4.yaml

# Monday afternoon: Results look good, try variations
kubectl apply -f rayjob-r16-a64-lr5e5.yaml
kubectl apply -f rayjob-r32-a32-lr1e4.yaml

# Tuesday morning: More experiments based on what worked
kubectl apply -f rayjob-r32-a64-lr5e5.yaml
# ... and so on

Predictable Result: Jobs run whenever submitted and resources are available, scattered somewhat randomly across the carbon intensity spectrum. Some at 350 gCO2eq/kWh (peak intensity hours), others at 120 gCO2eq/kWh (solar valleys).

Note: For the remainder of this article, we will simplify the unit as gCO2/kWh. But in all places, (k)gCO2eq or gCO2eq/kWh are technically accurate.

The Naive Carbon-Aware Approach (And Why It Fails)

Our first instinct was simple: "Just delay everything until carbon intensity is low!"

The problem: With limited GPU resources (2 GPUs) and many experiments, forcing everything to wait for the absolute lowest carbon window creates a stampede. When conditions are finally met (or max-delay expires), all jobs compete for just 2 GPUs. The first few might run during optimal windows, but the tail-end jobs likely don't start until carbon intensity climbs back to 250+ gCO2/kWh.

The lesson: We need to distribute load across multiple "good enough" windows rather than clustering everything at the absolute minimum.

Better Solution: A Tiered Strategy

Not all experiments are created equal. Some configurations need fast feedback (baseline adjacent validations), while others are speculative and can wait for optimal conditions (long shots).

Our strategy: Distribute experiments across multiple "good enough" carbon thresholds using three tiers, each with different urgency profiles:

• Tier 1 (225 gCO2/kWh, 24h max delay): Baseline adjacent variations with r=16 for fast feedback
• Tier 2 (175 gCO2/kWh, 48h max delay): Promising r=32 configurations with standard hyperparameters
• Tier 3 (125 gCO2/kWh, 96h max delay): Experimental r=32 long shots with aggressive hyperparameters

This is essentially a priority queue pattern with carbon constraints as the primary gate.

California's grid (CalISO) has a clear daily pattern: carbon intensity is highest overnight and early morning (300-400 gCO2/kWh), then drops dramatically during the midday solar peak (typically 100-200 gCO2/kWh around noon to 2pm), before climbing again in the evening as solar generation fades.

Our three-tier strategy doesn't try to hit the single "perfect" window throughout the day. Instead, it solves two practical problems when you have limited GPU resources (2 GPUs) and many experiments (21 jobs):

Preventing the Stampede

If all 21 experiments had the same low threshold (say, 100 gCO2/kWh), they would all wait behind CG's carbon gate until either:

1. Carbon intensity drops below 100 OR
2. They hit their max delay (say, 24h) and CG stops blocking them

But after CG releases the jobs, they compete for just 2 GPUs through normal Kubernetes resource scheduling. The initial few jobs can run immediately during optimal conditions. However, the deeply queued jobs likely won't be able to execute until a less optimal time. In the worst case, such a stampede can make carbon intensity worse for many of the jobs.

But by using 225/175/125 gCO2/kWh thresholds, we'll create a natural flow:

• Tier 1 jobs (threshold 225, 8 jobs) start running in early morning as intensity drops below 225
• Tier 2 jobs (threshold 175, 4 jobs) join in late morning when it dips below 175
• Tier 3 jobs (threshold 125, 9 jobs) run during the deepest part of the solar valley in the early afternoon

This spreads the 21-job workload across a handful of low-carbon periods rather than clustering everything at the absolute minimum.

Intelligent GPU Allocation

This is the priority queue pattern with carbon constraints. When a GPU finishes one experiment and becomes available, the scheduler picks the next job that:

1. Meets its carbon threshold - current intensity is low enough
2. Has the highest priority - Tier 1 > Tier 2 > Tier 3 within acceptable jobs
3. Has been waiting longest - within that tier

The separation of concerns makes this clean:

• Ray: Manages experiment orchestration and dependencies
• Kubernetes: Allocates GPU resources and handles pod lifecycle
• Compute Gardener: Gates scheduling based on carbon intensity threshold
• Result: Each layer handles what it's best at

The key insight: We're not optimizing to "run everything at minimum carbon intensity." Instead we're optimizing with a blended consideration, "distribute the queue intelligently so experiments run when carbon is acceptably low AND GPU resources are available."

Implementation: Automating Tier Assignment

Rather than manually categorizing each experiment, we automated tier assignment based on hyperparameter characteristics, using a python script.

Note: Please checkout this post's supporting materials in our Github repo.

The workflow

# 1. Generate targeted sample of 21 manifests with carbon tiers
python generate_sweep.py --output-dir sweep_manifests --core-subset
# Output: 21 YAML files with intelligent tier assignments and annotations

# 2. Submit to Kubernetes (can submit all at once, individually or by tier)
kubectl apply -f sweep_manifests/

# 3. Ray coordinates job execution, CG handles carbon-aware scheduling
# No further intervention needed - jobs run when conditions are favorable

What happens under the hood:

1. Ray creates pods ("head" and GPU workers) for each training job
2. Compute Gardener scheduler evaluates each pod:
   • Checks current carbon intensity via Electricity Maps API
   • Compares against pod's threshold annotation
   • If intensity is acceptable, schedules immediately
   • Otherwise, delays and re-evaluates periodically
3. Jobs complete, then save models and metrics to persistent storage
4. Repeat until all experiments finish

Carbon Intelligent Scheduling vs Simple Batch Submission

We ran all 21 experiments from October 28-30, 2025, capturing actual carbon intensity data for the California grid (CalISO) via Electricity Maps API. Here's what we found:

Three-Tier Strategy (Our Approach)

• Average carbon intensity: 142 gCO2/kWh
• Total emissions: 1.81 kgCO2
• Total energy consumption: 12.2 kWh across all 21 experiments
• Wall-clock completion time: ~50.3 hours
• Distribution: Naturally spread across low-carbon periods over 2 days
• Research velocity: Maintained — Tier 1 baseline results available within the first day

The Alternative Universe: No Carbon Awareness

To estimate relative impact, we calculated what would have happened if the same 21 experiments had run immediately upon submission, executing two at a time continuously through the first night:

• Average carbon intensity: 226 gCO2/kWh (62.7% higher than our approach)
• Total emissions: 3.03 kgCO2 (67% higher than our approach)
• Total energy consumption: 12.2 kWh (same — same work done either way)
• Wall-clock completion time: ~16.2 hours (2 GPUs running continuously)
• Distribution: Heavy overnight execution during fossil-fuel heavy grid hours

Key insight: We achieved more than 40% carbon reduction by intelligently distributing experiments across cleaner time windows. The same computational work (12.2 kWh) produced dramatically different emissions based purely on when it ran.

Assessing Training Quality

After all experiments completed, we needed to determine which hyperparameter configuration produced the best model. This requires both quantitative metrics and qualitative assessment.

Quantitative Metrics

Each training run logged both the final training loss as well as the average validation loss.

Validation loss (not training loss) is the primary quality metric. It measures how well the model generalizes to unseen data. Lower is better, but only to a point (too low can indicate overfitting).

We established rank=16, alpha=32, learning rate=1e-4, dropout=0.1 as our baseline configuration (validation loss: 0.7899). Here are the top performers relative to that baseline:

Top 5 configurations by validation loss:

Interesting findings:

• The top 3 models all used alpha=128 with r=32, achieving 9-12% improvement over baseline
• Tier 3 experiments (the "long shots" designed to wait for optimal carbon windows) dominated the top results
• Higher alpha values (128 vs 32) had more impact on performance than rank increases (r=32 vs r=16)

Qualitative Testing

Metrics only tell part of the story. We also deployed the top 3 candidates with vLLM and tested them on real coding tasks:

# Deploy top candidates for interactive testing
kubectl apply -f vllm-deployment-r32-a128.yaml
kubectl port-forward svc/vllm-lora 8000:8000

# Test with prompts
curl http://localhost:8000/v1/chat/completions -d '{
  "model": "Qwen/Qwen2.5-Coder-7B-Instruct:qwen-lora-r32-a128",
  "messages": [{
    "role": "user",
    "content": "Write a Python function for binary search with edge case handling"
  }]
}'

What we evaluated:

• Code correctness and handling of edge cases
• Code style and readability
• Ability to follow instructions and successfully use tools
• Response to follow-up questions ("ex: what else can be improved in this codebase?")

The winner: rank=32, alpha=128, learning rate=1e-4, dropout=0.05; this configuration combined the best quantitative performance (lowest validation loss) with seemingly strong code and assistance quality. This was a Tier 3 experiment which waited for near-optimal carbon conditions.

Recap

• 40% carbon reduction is achievable with intelligent scheduling: By shifting the same computational work (12.2 kWh) from high-carbon overnight hours to solar-powered daytime windows, we reduced emissions from 3.03 kgCO2 to 1.81 kgCO2. This saved 1.22 kgCO2 across the 21 experiments.

• Three-tier distribution prevents stampede: By using multiple thresholds (225/175/125 gCO2/kWh), we distributed load across the solar valley rather than clustering everything at the absolute minimum. All 21 jobs completed successfully without hitting max-delay timeouts.

• The trade-off is real but manageable: Carbon-aware scheduling stretched wall-clock time from 16 hours to 51 hours (3.2× longer). But for batch workloads like hyperparameter sweeps, this is often acceptable; we don't need results in 16 hours if we can get them in 2 days, having emitted 40% less carbon along the way.

• Priority queue based on carbon intensity signal works: Not all experiments are equal. Fast feedback on baseline adjacent (Tier 1) configurations, a bit more waiting for long shots (Tier 3). Tier 1 results were available within the first day, maintaining research velocity.

• The best model came from a Tier 3 experiment: Our top performer (r=32, alpha=128, lr=1e-4, dropout=0.05) was a "long shot" that waited for optimal carbon conditions (123 gCO2/kWh). It achieved 12% better validation loss than our baseline.

What's Next

Future work, and a potential part 3 in this series, might explore intelligent scheduling with carbon intensity forecasts as well as scaling up to cloud GPU providers where carbon data varies by region.

Interested in implementing something similar in your organization?

If automated carbon awareness for your ML infrastructure sounds appealing, we're available to help. Whether you're looking to deploy Compute Gardener, customize it for your use case or explore new or advanced carbon optimization strategies, don't hesitate to reach out. Let's work together to advance sustainable computing practices.

Compute Gardener is an open-source Kubernetes scheduler that enables carbon-aware workload scheduling across hybrid clouds. Join us in making carbon-aware computing the default, not the exception.

The Problem: ML's Hidden Carbon Cost

Every day and all across the world, thousands of machine learning (ML) engineers and data scientists hit "submit" on their training or fine-tuning jobs. Whether "locally" or in a data center somewhere, graphics processing units (GPUs) spin up immediately, regardless of how clean the energy mix on the grid is at that place and time. Without intervention, execution time is dictated more so by the eccentricities and daily working patterns of teams than an optimal, carbon-aware strategy.

In this post, we'll document a recent experiment to limit model training carbon emissions, one that didn't require any human intervention after initial setup. For our purposes, a training job will involve training the ResNet50 model on the CIFAR-100 dataset for 30 epochs.

Truthfully, this isn't the most cutting edge nor demanding task in 2025... but it should provide a somewhat representative training workload which is "bite sized" and reproducible for demonstrating carbon-aware scheduling principles broadly.

A few details:

• A single ResNet50 training run 🤖 on one of our consumer grade cards (RTX 3090) consumes ~0.475 kWh of energy (~57 minutes at ~500W total system power). 🔥⚡
  • That's enough energy to charge my partner's Electric SUV 🔋🛞 with about 1.5 miles of range.
• At higher grid intensity times (usually overnight in CalISO region) we can expect to see 300 or more gCO2eq/kWh vs 100 or fewer gCO2eq/kWh at lower intensity times (usually early afternoon). That crazy unit is "grams of carbon dioxide equivalent per kilowatt-hour".
  • That can mean hundreds of grams of CO2 difference per run, solely due to execution time. I find it hard to make sense of this value, until I remember that I measure out only 15g of coffee beans to grind each morning. ☕🫘

While this experiment represents one engineer's jobs for a week, imagine it extrapolating out to hundreds of runs per team per month across many organizations... and the true magnitude of the opportunity to optimize training for carbon, with minimal hassle, starts to come into focus. 🔎🌐

Enter Temporal Shifting

What if deferrable training jobs could wait for cleaner energy? By "deferrable," we mean workloads where no human is waiting for immediate results and no downstream system is blocked on completion (ex: batch retraining, experimental model iterations, scheduled data processing pipelines, research experiments). These are fundamentally different from customer-facing inference services, APIs or real-time applications where milliseconds matter.

Meanwhile, the grid's carbon intensity varies by 2-3x throughout the day in many regions. Time shifting as a strategy is nothing new. It's been discussed and adopted for years, but there wasn't as much tooling to simplify and automate as there ought to have been. In particular, we were interested in a flexible and extensible approach using the Kubernetes scheduler plugin framework.

Key insight: Deferrable workloads + Time flexibility = Carbon savings. We're not talking about delaying production API responses, but rather the training jobs that could wait hours or even days for optimal conditions... but only if the friction and overhead of delaying them isn't too high.

Compute Gardener: A Secondary Scheduler for Kubernetes

Rather than asking engineers to change their workflows, what if we made the infrastructure itself carbon-aware? That's exactly what our open-source Kubernetes scheduler does.

What It Is

• A carbon-aware Kubernetes scheduler that runs alongside the default scheduler
• Evaluates real-time grid carbon intensity before scheduling
• Defers non-critical workloads to cleaner time windows
• Ensures jobs run within some maximum delay (configurable, defaults to 24hrs)

What It Isn't

• A replacement for your existing default scheduler; only workloads that explicitly opt-in are affected
• An approach requiring code modification (whether training, inference or batch processing code)
• A strategy that asks you to sacrifice reliability for sustainability

Real-World Testing: A Week of ML Training

To validate this approach, we designed a week-long experiment with semi-realistic training patterns, measuring both the carbon impact and operational effects of carbon-aware scheduling. This occurred 22nd-26th Sept '25.

The Workload

• Model: ResNet50 (25M parameters)
• Dataset: CIFAR-100 (100 classes, 50K training images)
• Hardware: Single RTX 3090 (24GB VRAM)
• Duration: ~1 hour per run
• Pattern: Each day we simulated a typical ML engineer's daily workflow with 4 job submissions representing common trigger points throughout the workday and overnight automation:
  • Morning iteration: caffeine-fueled submission at 10:19am
  • Afternoon collab: "last chance" before EOD submission at 5:41pm
  • Midnight warrior: poor work-life balance induced submission at 11:12pm
  • Overnight auto-infra: re-training triggered as part of data ingestion pipeline at 3:55am

The Setup

We know not everyone has ready access to a Kubernetes cluster, but very similar patterns could be applied in other cloud/HPC orchestrated environments or even on local workstations.

For those who are responsible for such cluster(s) (DevOps teams, lead engineers, other tech decision makers), the aim was to ensure our scheduler would be easy to get up and running quickly. After install, it should be ready to start mitigating real-world emissions immediately with minimal-to-no further configuration. Compute Gardener uses the Electricity Maps API for real-time carbon intensity data. The free tier works well for this experiment but is limited to accessing live data for only a single grid region.

# standard k8s Job requesting secondary scheduler
apiVersion: batch/v1
kind: Job
metadata:
  name: ml-training-experiment
spec:
  template:
    spec:
      # The only change needed ↓
      schedulerName: compute-gardener-scheduler
      containers:
      - name: pytorch-training
        image: pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
        command:
          - python
          - /train.py
        resources:
          requests:
            nvidia.com/gpu: 1

Note: This example is simplified for clarity. A real job would include volume mounts for your training script and data, possibly a ConfigMap for hyper-parameters and other standard Kubernetes configuration.

The Configuration

By default, the Compute Gardener (CG) Scheduler uses a 24hr maximum delay and 200 gCO2eq/kWh intensity threshold. These can both be easily overridden with custom pod annotations. But in the spirit of keeping the experiment simple and reproducible, we stuck with the defaults.

What happens when jobs queue up? If grid intensity stays high long enough, CG enforces a maximum delay (default 24hrs), so jobs will run even at higher intensity rather than miss their deadline. Think of it as a carbon budget with a safety valve, not a hard gate that could block critical work.

The Results

The results are promising, ~30% reduction in emissions... but they are also quite dependent on grid intensity and workload schedules. In CalISO, last week, with this pattern of jobs, we'd argue, the opportunity was large enough to warrant pursuing.

Note on geographic variation: These savings reflect CalISO's grid characteristics. While many grids now exhibit daily carbon intensity variation, the magnitude differs by region. Check Electricity Maps to assess your grid's potential. You'll likely see some real savings if your grid region has daily intensity swings of >100 gCO2eq/kWh and a threshold is chosen based on local grid region history.

Week Overview

• Work Week 22nd-26th of September 2025
• Total Jobs: 20 (4 each day) training runs simulating a few data science workflows
• Without CG: All would have run immediately at submission time
• With CG: Strategic delays based on real-time grid carbon intensity

Carbon Impact

Since only 55% of the jobs encountered were delayed, those were the only ones reducing emissions. This value and the average delay duration per job are greatly influenced by how aggressively the intensity threshold and max delay are set.

Remember, all of this can scale up from workstations to data centers and beyond. The real goal is to show that compute can operate flexibly, in ways aligned to the grid and related signals.

Quick View

Figure 1: Grid carbon intensity (top), ComputeGardener shifted jobs (middle) and unshifted jobs (bottom) over nearly a day. Blue represents the midnight job, green the automated job, orange the morning job and yellow the end of day job.

Key Findings

Just considering this one week's worth of data, in CalISO region, we can say:

1. Peak intensity hours (3-6 AM): Carbon intensity averaged ~350 gCO2eq/kWh
2. Optimal windows (12-4 PM): Carbon intensity averaged ~120 gCO2eq/kWh
3. Automation jobs: The best savings usually came from the overnight jobs, most often being delayed around 6 or 7 hrs, until the morning. For example, the greatest carbon savings for a job was 90.1 gCO2eq. It executed at 158 gCO2eq/kWh, 8.9 hours after it first appeared when the intensity was 351 gCO2eq/kWh.
4. No failed jobs: 100% completion rate despite delays and no jobs hit their max delay

There's still a lot of low hanging fruit to pick here. Whereas this experiment, and Compute Gardener by default, uses a simple carbon intensity threshold, we'll soon be launching more sophisticated intensity forecast based logic. Essentially, rather than simply letting jobs execute as soon as intensity falls below a threshold, actually try to find the lowest intensity times, in the near future, to delay until.

Reproducibility

All experiment data, code and training scripts are available in our GitHub repo. We encourage you to reproduce these results in your own environment and share your findings.

Adoption: A couple minutes to Carbon Action

Step 1: Install Compute Gardener (One-time, 3 minutes)

# Add the Helm repository
helm repo add compute-gardener https://elevated-systems.github.io/compute-gardener-scheduler
helm repo update

# Standard installation - Requires Electricity Maps API key but works with FREE tier
helm install compute-gardener-scheduler compute-gardener/compute-gardener-scheduler \
  --namespace compute-gardener \
  --create-namespace \
  --set carbonAware.electricityMap.apiKey=YOUR_API_KEY

Step 2: Leverage It (add per pod spec, 1 minute each)

spec:
  schedulerName: compute-gardener-scheduler # That's it!

Step 3: [Optional] Fine-tuning

• Adjust carbon thresholds based on your grid region and workload urgency
• Set maximum delays that match your SLAs and persnickety engineers' expectations
• Configure time-of-use pricing awareness, if applicable in your environment

Beyond Simple Training: Production ML Pipelines

This sets the foundation for Part 2, where we'll explore a more complex and real-world training scenario

While ResNet training demonstrates the concept, this is just the beginning. In 2025, many organizations and enthusiasts are fine-tuning LLMs for domain-specific tasks and the carbon impact is massive. We're talking multi-GPU clusters running for hours or days... each job.

Next post: LLM LoRA Fine-tuning with Ray + Compute Gardener—how we made LLM training carbon-aware, handling workloads that represent the cutting edge of today's ML infrastructure. If you're fine-tuning models with (Kube)Ray, you won't want to miss this!

Interested in implementing something similar in your organization?

If automated carbon awareness for your ML infrastructure sounds appealing, we're available to help. Whether you're looking to deploy Compute Gardener, customize it for your use case or explore advanced carbon optimization strategies, our consulting services team can guide you through the process. Don't hesitate to reach out - let's work together to advance sustainable computing practices.

Have questions about implementation? Check our FAQ page for answers to common questions about installation, configuration, and best practices.

Compute Gardener is an open-source Kubernetes scheduler that enables carbon-aware workload scheduling across hybrid clouds. Join us in making carbon-aware computing the default, not the exception.

This should be cause for celebration. The ecosystem is maturing. Tools are becoming more sophisticated. Adoption barriers are falling.

And yet... latest projections show data center energy consumption (at least) doubling between now and 2030. GenAI workloads alone are expected to consume as much electricity as entire countries. We're in a race where the finish line keeps moving further away, faster than we're running.

This is the dual reality of carbon-aware computing in 2025: We're making real progress, but it's not enough. We're building better tools, but adoption isn't keeping pace with growth. We have solutions that work today. They are proven, tested and ready to deploy but they remain underutilized while energy demands soar.

Fortunately, the patterns emerging in our community show us a path forward. Not through any single tool or approach, but through an ecosystem where each solution serves its users best while amplifying the impact of others. Where a Python developer can add carbon awareness with a decorator, a DevOps engineer can enable it cluster-wide with a scheduler configuration and a cloud architect can optimize placement decisions through intelligent APIs.

Let me show you what this looks like in practice, why it matters more than ever and how you can start reducing your compute responsible carbon footprint today. Not someday, but today, with tools that already exist and patterns that already work.

While the challenge is growing faster than our solutions are currently being adopted, every kilowatt-hour shifted to a cleaner time, every workload deferred during peak carbon intensity, every optimization deployed—they all count. And they count faster when we work together.

Emerging Community Patterns

The carbon-aware computing space is crystallizing around three complementary patterns, each serving somewhat different users and use cases. Understanding these patterns (and how they work together) is key to accelerating adoption.

Developer-First Approaches: Code-Level Integration

Ryan Singman and the team at CarbonAware have pioneered what might be the most elegant entry point for developers: a decorator that makes any Python function running in Prefect/Airflow carbon-aware. Their approach is beautifully simple:

@carbon_aware(max_intensity=100) # Everything but this line remains unchanged
def train_model():
    model.fit(x_train, y_train)

This pattern shines because it meets developers where they are. No infrastructure changes. No DevOps coordination. Just add a decorator and your function automatically runs during lower intensity periods.

The limitation? It requires (minor) code changes. Legacy applications, vendor software or containerized workloads can't easily benefit without modification... which is not always practical or supported by others in an org. But for greenfield development and Python-heavy workflows, it's an immediate win!

Infrastructure-Level Solutions: Intelligent APIs

Electricity Maps' recent announcement represents another crucial pattern: moving complexity to specialized services. Their API now accepts queries like "What's the carbon intensity for AWS us-east-1?" and handles all the messy details of mapping cloud regions to electrical grids.

This is more than a convenience, it's a fundamental shift in how we think about carbon-aware infrastructure. Instead of every tool maintaining its own mapping tables (guilty as charged; we have one in Compute Gardener codebase), we can now rely on a constantly maintained and authoritative source.

Another of the CarbonAware projects, cloudinfo project, has been instrumental in pushing this direction, creating open mappings between cloud providers and grid regions. The fact that Electricity Maps integrated similar functionality shows how community ingenuity and collaboration drive real improvements.

Orchestration-Layer Tools: Zero Code Change Impact

This brings us to the pattern we've focused on at Compute Gardener: orchestration-level carbon awareness. By implementing carbon-aware scheduling at the Kubernetes layer, we can affect existing workloads without any code modifications.

Kubernetes already has a pluggable scheduler architecture. By deploying Compute Gardener as a secondary scheduler, teams can opt workloads into carbon-aware scheduling with a simple "annotation":

apiVersion: batch/v1
kind: Job
metadata:
  name: my-analysis-job
spec:
  template:
    spec:
      # This is all a job needs to specify in order to delegate to a carbon aware scheduler
      # (with default settings; custom thresholds, max delays, etc. can be further annotated as needed)
      schedulerName: compute-gardener-scheduler
      containers:
      - name: my-analysis-job
        image: existing-build-train-etl-etc:latest  # No changes to containers or related config needed!

No application changes. No risk to existing workloads. Just carbon-aware scheduling for any containerized workload that could leverage it.

The Collaborator's Dividend

Here's where it gets interesting (though not unexpectedly so, for those who believe in the value of open source software and data). These patterns aren't competing, they work in concert. And that composition is accelerating progress for everyone.

Take our recent decision to remove region mapping code from Compute Gardener. We spent some real engineering time towards building and maintaining cloud-to-grid mappings. But with Electricity Maps' new API endpoints and Ryan's open cloudinfo data, we're about to delete that code entirely.

Software engineers know that removing lines of code can OFTEN feel even better than adding new, especially when it allows for doing more with less. Less maintenance work for us, better data for our users and more time to focus on what we do best: making Kubernetes scheduling smarter.

This is the collaborator's dividend in action. When CarbonAware focuses on developer experience, Electricity Maps on data accuracy and accessiblity and Compute Gardener on orchestration, the whole ecosystem gets stronger. Users can choose tools that fit their constraints while benefiting from shared infrastructure.

Why This Matters Now: The Urgency Gap

Let's talk about the gassy elephant in the room: we're not moving fast enough.

Data center electricity consumption is projected to double by 2030, reaching 1,000 TWh annually; roughly the entire electricity consumption of Japan. GenAI workloads are accelerating this trend, with training runs consuming megawatt-hours and inference at scale beginning to dwarf traditional computing workloads.

Meanwhile, our carbon budgets are shrinking. To stay under 1.5°C of warming, we need to cut emissions by 45% by 2030. That's five years out. Not five years to commission a panel, five years to achieve.

This is why every deferred workload matters. When Compute Gardener delays a batch job from running at 500 gCO2eq/kWh to 100 gCO2eq/kWh, that's an 80% reduction in carbon emissions for that workload. Multiply that by thousands of jobs across hundreds of clusters and the impact becomes substantial.

But it's still not enough. Not when adoption remains limited to the sustainability-conscious few. Not when most organizations still view carbon awareness as a "nice to have" rather than a business imperative.

Practical Integration Patterns

So how do we accelerate? By making adoption so easy that NOT using carbon-aware computing becomes the exception. Here are patterns we're seeing work:

A Layered Approach

Start with orchestration-level tools for immediate impact to existing infra and layer in developer tools for new applications:

1. Week 1: Deploy Compute Gardener for all deferrable workloads
2. Month 1: Add carbon-aware decorators to data pipeline code
3. Quarter 1: Integrate Electricity Maps API for multi-region deployment decisions

Quick Wins

Focus on the easiest, highest-impact workloads first:

• Batch jobs: Usually perfect for time-shifting
• CI/CD pipelines: Run builds and tests during cleaner hours
• Data processing: ETL and analysis jobs are often ideal for carbon-aware scheduling
• Model training: Ridiculously high energy use makes carbon awareness critical

The Enterprise Pattern

For larger organizations, create a carbon-aware platform layer:

1. Deploy orchestration tools cluster-wide
2. Provide and encourage developer libraries and decorators
3. Integrate carbon metrics into existing dashboards
4. Set carbon budgets alongside compute budgets

Building the Future We Need

The patterns are clear. The tools exist. The urgency is undeniable. What we need now is adoption at scale.

This is where you come in. Whether you're a developer who can add a decorator to your next function, a DevOps engineer who can enable carbon-aware scheduling or a leader who can mandate carbon metrics alongside performance metrics... YOU have a role to play.

Start where you are:

• Have a Kubernetes cluster? Deploy Compute Gardener today
• Writing Prefect/Airflow pipeline code? Try CarbonAware's decorator for your next function
• Architechting multi-region deployments? Use Electricity Maps' API for placement decisions
• Building something new? Design carbon consideration in from the start

Share what you learn. The community grows stronger with every blog post, every GitHub issue and every conference talk about real-world carbon-aware computing.

And if you need help along the way (whether it's implementing these patterns, measuring impact or building a carbon-aware strategy), please reach out! We're all in this together and the clock is ticking.

The carbon-aware computing ecosystem is more than mature enough to make a difference. The patterns work. The tools exist. The only question is: will we deploy them fast enough?

Every kilowatt-hour shifted to cleaner times matters. Every optimized workload counts. We know the time to act is now.

Compute Gardener is an open-source Kubernetes scheduler that enables carbon-aware workload scheduling across hybrid clouds. Join us in making carbon-aware computing the default, not the exception.

The Signal Question

If, like me, you're new to the concepts:

Average carbon intensity tells us the emissions associated with the overall electricity mix on the grid at a given moment — essentially, what's the carbon footprint of a kWH (kilowatt-hour) of electricity right now, considering all generation sources currently active.

Marginal carbon intensity attempts to predict which generation sources would respond to a new load being added to the grid — in other words, what's the carbon footprint of the next kWh if we were to consume it right now.

The difference initially seems subtle, but it can be substantial. In many grids, the marginal generator (the one that ramps up to meet new demand) might be a natural gas plant, even when solar and wind make up a significant portion of the current mix.

The Community Conversation

This topic has sparked lively debates in the climate tech community. I first became aware of it through a conversation including Tammy Sukprasert, a researcher at UMass Amherst, on the Environment Variables podcast. In that episode, she highlighted the theoretical differences between average and marginal emissions signals and mentioned that organizations like WattTime have been exploring the use of marginal signals. She was kind enough, a few weeks after the episode, to talk with me and help me begin to grok the distinction.

During the recent San Francisco Climate Week, I had the opportunity to briefly discuss this with Olivier Corradi, CEO of Electricity Maps API, and a few of his colleagues. Their perspective is that average signals are more reliable, accessible and ultimately more actionable for most use cases.

WattTime has been a prominent voice in the space examining the potential benefits of marginal emissions. Their argument is that marginal emissions might more accurately reflect the true impact of new electrical loads on the grid. For what it's worth, I haven't had the chance to talk nearly as much with their team or others who follow this school of thought.

The Case for Marginal Signals

WattTime presents a compelling economic and operational case for marginal intensity signals. They argue that while average emissions tell us about the overall grid composition, they don't necessarily reflect what happens when we decide to consume electricity at a specific moment. Grid operators dispatch generation sources in order of cost (known as merit order dispatch), which means the generators responding to new demand often differ from the average mix. From an economic perspective, this is analogous to the difference between average cost and marginal cost - what matters for decision-making is the impact of your next action, less so the previous state of the system.

For carbon-aware computing specifically, this distinction becomes meaningful because our goal isn't just to measure or attribute emissions, but to actively reduce them through timing decisions. If a data center chooses to run a batch workload now versus in three hours, the real-world emissions impact of that decision is determined by which generators respond to that load - the marginal sources - not the average grid mix. Studies by researchers at Rocky Mountain Institute (Dyson & Mandel et al., 2017) and the University of Michigan (Li et al., 2017) have validated that timing decisions based on marginal emissions can lead to greater carbon reductions than those based on average emissions, particularly in grids with diverse generation sources.

The Complexity of Marginal Signals

The theory behind marginal signals is compelling. However, several practical challenges have informed our current approach with Compute Gardener:

1. Hyperlocal Dependencies: Marginal signals can vary significantly based on precise grid connection points. Without extremely granular location data (beyond what most cloud providers expose), the signal's accuracy degrades.

2. Load Scale Assumptions: Many marginal intensity models assume relatively large loads that can actually influence dispatch decisions. For individual computing workloads, even substantial clusters, this assumption may not hold.

3. Coordination Challenges: While marginal signals aim to reduce emissions by capturing the impact of new loads, research has identified scenarios where they could backfire. Research on load synchronization issues has shown that when many distributed actors independently respond to the same marginal emissions signal without coordination, they can create new demand peaks that could potentially trigger high-carbon peaker plants (Radovanovic et al., 2021).

4. Data Availability and Reliability: Average intensity data is more widely available, standardized and accessible across different regions, making it easier to implement globally consistent policies. Further, it is likely better understood and considered more reliable for the purposes of analytical emissions mitigation validation.

Olivier Corradi of Electricity Maps API has emphasized the coordination challenge in conversations and in their blog series "Marginal signals explained", noting that the theoretical benefits of marginal signals can be undermined when applied at scale without proper grid coordination. This synchronization effect is particularly concerning in grids with high renewable penetration that already face steep evening ramps when solar generation decreases, such as California's famous "duck curve" scenario, where rapid ramping requirements in the evening can necessitate firing up more carbon-intensive peaker plants.

Our Implementation Choice

For Compute Gardener's initial release, we've chosen to focus on average carbon intensity signals, specifically through the Electricity Maps API. This decision wasn't made lightly and attempts to strike a pragmatic balance between building useful tools now and engaging with the debate as it rages on and we accumulate more data.

Our focus has been on validating the core hypothesis: we can meaningfully reduce emissions by temporally shifting computing workloads based on carbon intensity signals. Starting with average intensity allows us to:

1. Implement widely: The scheduler can work across diverse geographic regions with consistent signal quality.

2. Validate quickly: We can measure and demonstrate emissions reductions in real-world scenarios.

3. Iterate responsibly: As our understanding evolves, we can incorporate more sophisticated signals without disrupting existing installations.

4. Avoid overcomplicated abstractions: By starting with a clearer signal, we build interfaces and patterns that make sense to users and avoid premature optimization.

Looking Forward

While we've started with average intensity signals, Compute Gardener's architecture is designed to accommodate future signal sources. The modular implementation of our carbon-aware scheduling allows for:

1. Multiple signal sources: Organizations with access to high-quality, locationally relevant marginal signals could plug them into the scheduler.

2. Hybrid approaches: Combining average and marginal signals based on workload characteristics and regional grid dynamics.

3. Spatial shifting: As we expand beyond temporal optimization to consider moving workloads between regions, we can incorporate both signal types for more sophisticated decision-making.

The Bigger Context

This discussion exists within a broader question of how distributed systems should respond to grid carbon signals. Should every individual system independently optimize for carbon, or should there be coordination? Do we need carbon-aware load balancing at the grid level? How might pricing mechanisms better align individual incentives with grid-level outcomes?

These are the questions that keep us climate-focused nerds up at night and they don't have simple or one-size-fits-all answers.

Your Thoughts?

I'm curious what you think about this debate. Have you implemented carbon-aware systems using intensity signals? What has your experience been? For those with particular expertise in data center and grid operations, what critical perspectives might I be missing?

This is an exciting and ever evolving conversation and one that will shape how computing adapts to our always constrained world. I look forward to continuing it with you.

Want to take action now?

Check out our Kubernetes scheduler if you're interested in learning more or are looking for a way to take some real steps forward in your sustainable compute game today!

The Challenge

Standard Kubernetes scheduling doesn't consider energy implications - pods are placed to optimize for resource utilization, spread, affinity and other traditional metrics. But what if we want to:

1. Delay non-urgent workloads until carbon intensity of the grid is lower[3]
2. Run batch jobs during off-peak electricity pricing hours
3. Track energy consumption for workloads and enforce budgets
4. Optimize for energy-efficient hardware when available[4]

Overview

We built our solution using Kubernetes' scheduler framework, which allows custom plugins to influence scheduling decisions.[7] The core components include:

# Main components of the scheduler
api/      # Various data providers' API clients 
carbon/   # Carbon-aware scheduling implementations (currently Electricity Map API)
config/   # Configuration types and validation
metrics/  # Prometheus metrics and hardware profiling
price/    # Price-aware scheduling implementations (currently time-of-use)

Our scheduler implements two key extension points in the framework:

1. PreFilter: Initial validation of scheduling constraints and annotations related to carbon intensity and price
2. Filter: Main decision logic that evaluates efficiency and suitability of potential nodes to bind

Decision Flow

PreFilter Stage

The PreFilter stage/method handles initial validation of scheduling constraints and annotations related to carbon intensity and pricing.

// PreFilter implements the PreFilter interface (simplified)
func (cs *ComputeGardenerScheduler) PreFilter(ctx context.Context, state *framework.CycleState, 
                                              pod *v1.Pod) *framework.Status {
    // Check pricing constraints, if enabled
    if cs.config.Pricing.Enabled && cs.pricingImpl != nil {
        if status := cs.pricingImpl.CheckPriceConstraints(pod, cs.clock.Now()); !status.IsSuccess() {
            return status
        }
    }

    // Check carbon intensity constraints, if enabled
    if cs.config.Carbon.Enabled && cs.carbonImpl != nil {
        // Get threshold from configured threshold
        threshold := cs.config.Carbon.IntensityThreshold
        // Override with annotated value, if present
        ...

        // Get current carbon intensity
        intensity, err := cs.carbonImpl.GetCurrentIntensity(ctx)
        if err != nil {
            // If we can't determine carbon intensity, allow pod thru
            return framework.NewStatus(framework.Success, "")
        }

        // Make scheduling decision based on threshold
        if intensity > threshold {
            msg := fmt.Sprintf("Current carbon intensity (%.2f) exceeds threshold (%.2f)", 
                              intensity, threshold)
            return framework.NewStatus(framework.Unschedulable, msg)
        }
    }

    // Default to letting pod thru
    return framework.NewStatus(framework.Success, "")
}

Filter Stage

PreFilter occurs without any node awareness. Then, in the Filter stage, we gain node information and can consider our knowledge of node energy use and efficiency in scheduling decisions.

// Filter implements the Filter interface (simplified)
func (cs *ComputeGardenerScheduler) Filter(ctx context.Context, state *framework.CycleState, 
                                          pod *v1.Pod, nodeInfo *framework.NodeInfo) *framework.Status {
    // Logic for hardware profiling and node selection
    if cs.config.HardwareProfiling.Enabled && cs.hardwareProfileImpl != nil {
        profile := cs.hardwareProfileImpl.GetNodeProfile(nodeInfo.Node().Name)
        if profile == nil {
            return framework.NewStatus(framework.Unschedulable, "No hardware profile available")
        }

        // Apply node selection based on hardware capabilities
        if !cs.nodeSelectionImpl.IsNodeSuitable(profile, pod) {
            return framework.NewStatus(framework.Unschedulable, "Node not suitable for workload")
        }
    }

    // Default to letting pod thru
    return framework.NewStatus(framework.Success, "")
}

This implementation follows the Kubernetes scheduler pattern of returning Unschedulable status when conditions aren't met, which causes the pod to wait in the scheduling queue until conditions change.

Real-Time Carbon Intensity Data

The scheduler consumes carbon intensity data from external APIs and makes scheduling decisions based on configurable thresholds. Our solution:

1. Supports pod-level configuration through annotations, allowing different thresholds per workload
2. Uses caching and failure backoff to tolerate API failures and maintain scheduler reliability
3. Makes it easy for operators to dynamically override scheduling decisions as preferred

Hardware Profiling and Estimating Node Energy Use

Accurately modeling power consumption, estimating energy usage, and selecting suitable nodes based on hardware capabilities enables efficient scheduling. It seems like it SHOULD be easy, but most computing hardware wasn't designed with real-time power monitoring as a primary feature.

Fortunately, for Graphical Processing Units (GPUs), we can leverage direct power metrics through NVIDIA's DCGM, which provides reasonably accurate real-time power consumption data to which we can apply a PUE (Power Usage Effectiveness) factor. However, for CPUs and memory, we must rely on a combination of heuristics, frequency scaling information, and utilization patterns to estimate power consumption.

Our approach follows and extends the Green Software Foundation's Software Carbon Intensity (SCI) methodology[8], adding enhancements for CPU frequency scaling and dynamic power profiling. We believe our estimates are within 5% of actual consumption for calibrated nodes - likely sufficient accuracy for carbon accounting and potential carbon credit validation.

// calculatePodPower estimates power consumption (simplified)
func calculatePodPower(nodeName string, cpu float64, memory float64, gpuPower float64) float64 {
    // Get hardware profile for the node
    profile := getNodeHardwareProfile(nodeName)
    
    // Apply frequency scaling adjustment
    cpuFreq := getCurrentCPUFrequency(nodeName)
    if cpuFreq > 0 && profile.BaseFrequency > 0 {
        freqRatio := cpuFreq / profile.BaseFrequency
        // Power typically scales with square of frequency
        powerRatio := freqRatio * freqRatio
        profile.MaxPower *= powerRatio
    }

    // Power-law model for CPU power considering utilization
    cpuPower := profile.IdlePower + 
                (profile.MaxPower - profile.IdlePower) * 
                math.Pow(cpu, 1.4)  // Common exponent used in datacenter modeling
                
    // Add GPU power with datacenter overhead (PUE)
    totalPower := cpuPower
    if gpuPower > 0 {
        totalPower += gpuPower * profile.GPUPUE
    }

    return totalPower
}

// Calculate energy using numerical integration
func CalculateTotalEnergy(records []PodMetricsRecord) float64 {
    totalEnergy := 0.0
    
    for i := 1; i < len(records); i++ {
        // Integrate power over time, summing totalEnergy
        ...
    }

    return totalEnergy
}

This allows us to model power consumption accurately with dynamic frequencies, and then calculate energy usage over time using numerical integration.

Performance Considerations

A key concern with any scheduler is adding latency to scheduling decisions. We implemented several optimizations:

1. Caching: Both carbon API responses and hardware profiles are cached, ensuring API hiccups never derail a scheduling decision
2. Parallel Processing: API requests and metrics collection happen asynchronously wrt core scheduler flow
3. Bounded Memory: If capturing metrics, downsampling is used for long-running pods to limit memory usage
4. Prometheus Integration: Leverage existing monitoring infrastructure; don't reinvent or duplicate

Most importantly, all of this runs as a secondary scheduler, which means:

1. System-critical components continue using the default scheduler; your 24/7 HA services will never even know there's another scheduler in the cluster
2. Only workloads that opt-in are subject to carbon/price decisions; can opt-in at pod level or namespace level using webhook admission controller
3. Scheduling delays (capped at a configurable max) cause temporal workload shifting strategies that have shown promising results in reducing carbon emissions[5]

Pod Configuration

Workload creators can opt into carbon/price-aware scheduling by requesting schedulerName: compute-gardener-scheduler in pod spec and then can apply individual job/pod overrides with annotations:

metadata:
  annotations:
    # Set custom carbon intensity threshold
    compute-gardener-scheduler.kubernetes.io/carbon-intensity-threshold: "200.0"
    # Enable price awareness using configMap'd schedules
    compute-gardener-scheduler.kubernetes.io/price-enabled: "true"
    # Set energy budget
    compute-gardener-scheduler.kubernetes.io/energy-budget-kwh: "5.0"
    # Limit any potential carbon/price delays to a maximum of 12 hrs.
    compute-gardener-scheduler.kubernetes.io/max-scheduling-delay: "12h"
spec:
  schedulerName: compute-gardener-scheduler

Projected Results and Validation Needs

Based on our simulations and initial measurements, the scheduler shows promising potential benefits:

1. Carbon reduction: Testing suggests potential for 50%+ reduction in carbon emissions for time deferrable workloads (mitigations measured 03-17Mar25 with intensity data in CalISO region cycling between common daily lows of 50 and highs of 200+ gCO2eq/kWh)
2. Cost savings: Testing indicates 20%+ reduction in electricity costs when leveraging time-of-use (TOU) rates (mitigations measured 03-17Mar25 and considering PGE's E-TOU-B pricing schedule)
3. Energy tracking: Previously invisible energy consumption becomes visible and actionable (priceless, right? ;D)
4. Performance impact: Testing shows minimal increase in scheduling latency (< 100ms per decision; shouldn't be significant considering k8s orchestration timescales)

For a medium-large cloud data center with ML training workloads, our simulations project approximately 18 metric tons of CO2 equivalent emissions that could be avoided per month. These estimates align with recent research showing substantial potential for carbon reduction in ML workloads through intelligent scheduling.[6]

We're actively seeking partners to help validate these projections in production environments. If you're interested in deploying carbon and/or price-aware scheduling in your Kubernetes clusters, we'd love to collaborate to measure and refine real-world impact. Please reach out via our website or GitHub repository.

Future Developments

As the field of carbon-aware computing continues to evolve, several exciting developments are on the horizon:

1. Integration with Advanced Forecasting: Leveraging ML, along with weather and various other signals, to predict future carbon intensity and optimize scheduling decisions in real-time.
2. Enhanced Hardware Profiling: Developing more accurate models for power consumption across different common cloud hardware configurations, including CPUs, GPUs, and memory.
3. Global Carbon Markets: Integrating the scheduler with global carbon markets to enable workloads mitigating carbon emissions, in a calibrated, validated way, to recieve carbon credits.
4. Cross-Cluster Scheduling: Extending the scheduler's capabilities to manage carbon emissions across multiple Kubernetes clusters and thereby space.

These developments will further enhance the ability of the Compute Gardener to reduce carbon footprints and optimize costs in Kubernetes environments.

Conclusion

Building a carbon and price-aware Kubernetes scheduler involves considered integration with Kubernetes core components, external APIs, and hardware-specific optimizations. By creating as a secondary scheduler with an opt-in model, we've made it easy to deploy without disrupting critical workloads while still providing substantial, and immediate, environmental and cost benefits.

Try it out today with your next training or build job!

Compute Gardener Scheduler is open source and available on GitHub.

Related Reading

Our work builds on the thoughts and patterns from those in the community who have pioneered research in carbon-aware computing, in particular with a k8s perspective. We're grateful for the foundation they've established, which has made our approach practical and possible. Beyond the sources cited above, [9], [10], and [11] have each influenced our designs, as well.

[1] Reuters. Data center build-out stokes fears of overburdening biggest US grid. March 13, 2025.

[2] James, A., & Schien, D. (2019). A Low Carbon Kubernetes Scheduler. In Proceedings of the 6th International Conference on ICT for Sustainability (ICT4S 2019), Vol. 2382. CEUR Workshop Proceedings.

[3] Philipp Wiesner, Ilja Behnke, Dominik Scheinert, Kordian Gontarska, Lauritz Thamsen. Let's Wait Awhile: How Temporal Workload Shifting Can Reduce Carbon Emissions in the Cloud

[4] Zhenyu Wen, Renyu Yang, Albert Zomaya. Energy-Aware Kubernetes Scheduler: Opportunities and Challenges

[5] Randall Ross, Matt Rutherford, David Grunwald. Carbon-Aware Computing And Scheduling for CPU-Intensive Workloads in Heterogeneous Clusters

[6] James Downton, Antonios Katsarakis, Juliana Franco, et al. Scheduling machine learning training jobs on heterogeneous compute with Carbon signal

[7] Piontek, T., Haghshenas, K. & Aiello, M. Carbon emission-aware job scheduling for Kubernetes deployments. J Supercomputer 80, 549–569 (2024).

[8] Green Software Foundation. Software Carbon Intensity (SCI) Specification

[9] UMass Solar. Carbon-aware-dag. GitHub repository.

[10] C3Lab. K8s-carbon-aware-scheduler. GitHub repository.

[11] Zemdom. Carbon-aware-scheduling. GitHub repository.

The Unseen Challenges of Modern Infrastructure

Today's reality:

• Complex Hybrid Environments make consistent optimization nearly impossible
• Siloed Decision Making separates cost and sustainability goals
• Missing Data forces teams to operate on approximations
• Signal Integration Gaps leave scheduling systems blind to carbon intensity and pricing fluctuations
• Optimization Guilt haunts teams who know they should optimize carbon but lack actionable tools
• Accelerating Growth in data centers means emissions will only increase without intervention

Introducing Compute Gardener

Within hours, your workloads can begin responding to real-world conditions - automatically shifting to times when your region's carbon intensity is lower or when electricity prices drop during off-peak hours.

This delivers immediate impact on both carbon emissions and costs without requiring you to rearchitect your systems or compromise performance. Our open-source scheduler integrates seamlessly with your existing Kubernetes infrastructure, allowing for controlled, gradual adoption.

Empowering Teams with Real-Time Insights

With Compute Gardener, infrastructure teams gain:

• Visibility into energy consumption patterns across workloads and nodes
• Control through namespace-level energy budgets with automated notifications
• Confidence from data-driven scheduling decisions based on methodologies aligned with Green Software Foundation guidelines
• Flexibility to optimize workloads based on their specific characteristics
• Value Creation potential through energy savings as carbon credits

From Complexity to Action

What makes Compute Gardener different:

1. Immediate Impact: Begin optimizing workloads within hours
2. Start Where You Are: Integrate with existing Kubernetes clusters
3. Gradual Adoption: Test with namespace-level policies
4. Automatic Adaptation: Respond dynamically to changing conditions

Our Path: Local Optimizations to Global Stewardship

Compute Gardener is open source because sustainable computing requires collective innovation. While our scheduler delivers immediate value with single-region carbon data, our forthcoming Advanced Signals API synthesizes a comprehensive global dataset across:

• Worldwide carbon intensity variations
• Dynamic electricity pricing
• Real-time cloud infrastructure costs

This enables both temporal optimization (when to run workloads) and spatial optimization (where to run them). Beyond optimization, we're also exploring how to transform saved energy—"nega-watts"—and mitigated carbon emissions—"nega-tons"—into validated carbon credits. This represents a fundamental shift: from sustainability as a cost center to sustainability as a value creator and potential revenue stream.

Join the Movement Toward Sustainable Computing

The path to sustainable infrastructure doesn't need to be through massive one-time changes—it can begin with accessible tools that enable continuous improvement. Compute Gardener provides these tools today, whether you're taking your first steps toward carbon-aware computing or ready to transform your entire approach to infrastructure.

<> GitHub Repo | ⚡ Explore Use Cases & Docs | 💃🏾 Connect with Our Team

Whether you're looking to reduce operational costs, optimize infrastructure efficiency, or prepare for future carbon markets, we're here to help. Our consulting services team can guide you through implementation, from initial assessment to ongoing optimization.

Let's build more efficient computing infrastructure together—where cost savings and sustainability go hand in hand.

Photo by Robert Koorenny on Unsplash