Kevin Simback

22 min read

Decentralized Training: the stakes have never been higher

TLDR; this is the companion to my decentralized inference piece where we look at the other half of the AI equation - training. We cover what’s at stake, what makes it hard, and the projects solving it.

The decentralized inference report made the argument that you actually need to solve 3 really hard technical problems - sharding, verification, and privacy - and any one of those could be a full product on its own. Decentralized training inherits versions of those same 3 hard technical problems but with a different, and more consequential output - build a large model across machines that no single party owns, and do it over the public internet.

The core question with decentralized training: can this even work at a scale that matters, and if so, who cares enough to pay for it?

The interesting part is the real progress the industry has made so far this year on addressing that question. Spoiler alert: it is showing strong signs that it can actually work.

What’s at stake

Centralized vs. decentralized AI is not merely a technical debate about architecture or efficiency. It is a high-stakes contest over who controls the most powerful general-purpose technology humanity has ever developed. Let that sink in.

AI will be increasingly used to mediate knowledge, make decisions, define economies, influence culture, broker power, and even change biology. So yeah, we need to carefully think through who controls it.

At its core centralized AI concentrates training data, model weights, compute infrastructure, and often inference in the hands of a small number of big tech corporations.

Decentralized AI (DeAI) distributes some or all of these elements across many independent nodes/participants, often using blockchain incentives or peer-to-peer protocols, such that no single entity owns or fully controls the system.

Decentralized training - the actual development of foundation model weights - is generally considered the last frontier for DeAI due to the technical challenges associated in making it work at model sizes that are useful relative to centrally trained models that exist today.

Right now the ability to train a frontier model lives inside a small number of firms in the US and China. Training a frontier model takes a tightly-connected datacenter with tens of thousands of GPUs wired together and often costs nine-to-ten-figures. Those are resources that very few firms have and thus not nearly as easy to pull off in a decentralized format.

Decentralized training attempts to train a competitive model across thousands of scattered, potentially mismatched and independently-owned GPUs, talking over ordinary internet links. If that can be done then the ability to build frontier AI stops being a moat owned by just a handful of firms and becomes something a community or an open-source project can do.

That’s the stake. Not a cheaper solution but a different distribution of who gets to build the most important technology of the decade.

Now the hard part - building it!

Distributed is not decentralized

The single most important distinction, and the one I see most conflated in the public discourse is using distributed and decentralized interchangeably - they are not the same.

Distributed training is many machines, with one owner and central control over the output and IP.

Decentralized training is many machines across different owners, locations, and trust boundaries, connected over the internet, often permissionless. No master node or central control points.

The decentralized model has to solve for:

  • a network between nodes that’s four to five orders of magnitude slower than an interconnected datacenter

  • machines that are mismatched and possibly unreliable

  • no reason to trust that any of the nodes did the work they claim

Every project featured in this article is an attempt to solve those problems.

Aren’t open weights models good enough?

The fact that we have an open weights ecosystem building SOTA models is great, albeit I’d like to see more geographic diversity at the top of the ranks beyond the Chinese firms.

But this doesn’t solve the core issue in the centralized vs. decentralized debate. Open weights models are still trained and owned by a centralized entity, and as we are starting to see - this can be throttled or restricted at the whim of a government or the decision of a company.

What if the top open weights labs decide to not publish the weights for new releases, right at a time when the intelligence is becoming most useful?

The way I like to frame it is that decentralized training is the supply side of the open-weight ecosystem’s independence. Without it, the open weights community is totally dependent on these (mostly Chinese) labs continuing to release the weights.

Who cares enough to pay for decentralized models?

Like anything else that is decentralized, there is a tax involved - by definition decentralized is going to be slower and requires more work.

So who pays the decentralization tax on the training side? Good question, and one that we’ve not fully answered. A few potentials:

  • Open-weight / open-source communities that want models no lab can pull. There’s real demand here, but they mostly consume the output (the open weights) without much regard to how the weights were generated, and there are plenty of viable open weights models (HuggingFace lists nearly 3 million models as of publication) so it’s unclear if there’s a willingness to pay.

  • Sovereign and quasi-sovereign buyers - nations or large institutions that want intelligence without hyperscaler or lab dependence. This seems most logical to me, but these are also buyers with potentially large resources that could train / post-train their own models without paying the decentralization tax. I’m frankly surprised we’ve not seen countries begin to train their own models to mitigate the dependence on the US and China for intelligence.

  • The privacy cohort - those who want to train across data you’re not allowed to send to centralized labs (hospitals, banks, regulated industries). Similar to the sovereign buyers, but less about the dependence and more about “just can’t use them” so need a good alternative.

  • Contributors with idle compute who want to get paid for it. Not really demand for the models themselves, but often a community of like-minded (sovereign biased) individuals who are likely to coordinate around a token, so it’s the one group these networks reliably recruit.

It should be quite clear that we need decentralized training to ensure independence for the open weights community, but it’s still an open question of who needs it badly enough to pay for it. It may be a chicken-and-egg problem where once the models are good enough, they’ll see more demand. We’ll need to check back in on this one over time.

The hard problems to solve to make decentralized training work

Just like decentralized inference, there are some hairy technical challenges that need to be solved to make decentralized training work at any meaningful scale.

  1. Bandwidth. This is the big one. Every training step has to combine gradient information across all the machines and for a 70GB model that’s about 140GB, and these models aren’t getting smaller. Inside a datacenter that’s fast. Over consumer uplinks it can run 5+ orders of magnitude slower.

Epoch AI quantified it: over a 60 Mbps connection, naively training a DeepSeek-v3-class model would take about 5,000 years, and sub-ten-minute syncs over normal internet cap you around a 600-million-parameter model.

Every project is working through this one constraint, and almost all the techniques descend from two approaches.

  • Sync less often - that’s DiLoCo, DeepMind’s 2023 method of running many local steps between communications, the canon most of the field builds on

  • Send less per sync - compress, sparsify, and quantize the update until it fits through a consumer uplink. Nous Research’s DeMo is the leading method.

  1. Latency. Even with infinite bandwidth, you still run into the limitations of physics. Fiber across 4,800 km is ~24 milliseconds one way, and a sync has to finish inside a few percent of a batch’s compute time or throughput collapses. Epoch’s follow-up analysis shows you can’t amortize the speed of light past a certain scale.

  2. Heterogeneity and stragglers. Synchronous training runs at the speed of the slowest machine. In a scattered swarm, mismatched GPUs and flaky links are the default condition, not edge cases, so one slow node taxes everyone.

  3. Verification. In a permissionless network, how do you know a node actually ran the training it’s claiming rewards for, instead of submitting garbage?

This is nastier than it sounds. Proof-of-learning is the obvious approach but it’s spoofable: there’s a published attack that forges a valid-looking proof more cheaply than training honestly.

The robust fixes all tend to quietly re-introduce some form trust: run it inside a hardware enclave (now you trust Intel or NVIDIA) or use a trusted coordinator (now you’re not permissionless). Zero-knowledge proofs of training are real but are nowhere near practical at frontier scale. I go into some of these methods in more detail in my Proof-of-Useful-Work article.

  1. Incentive and Byzantine security. Volunteer altruism doesn’t supply enough compute so you need some incentive mechanism, typically crypto tokens. But the more permissionless you make the network - the entire selling point - the larger the attack surface for poisoning, and robust defenses against it can be expensive and slow everything down.

  2. Governance. It’s possible to build something that’s decentralized in the training math and centralized in who actually controls it, but you have to address the governance layer else you can end up with a community in chaos (surely that would never happen in crypto, right?).

The best way to evaluate the legitimacy of the projects working on decentralized training tends to come down to how they solve those 6 problems above alongside the proof points of what they’ve actually trained. So let’s take a look at the teams.

Six projects to know in decentralized training

Six serious projects have been tackling these hard problems and achieving progressively better results with each run. For each project we get into the background, the approach, the traction, and the key questions.

Prime Intellect

Background. Founded in 2024 in San Francisco by Vincent Weisser and Johannes Hagemann. Weisser a serial founder out of the crypto and open-science world (VitaDAO) and Hagemann the ML-systems half of the pair. They came out of the gate strong with OpenDiLoCo in July 2024, an open-source reproduction and scale-up of DeepMind’s DiLoCo method. They have since shipped two major proof points, and a third (less decentralized) one noted further down:

  • INTELLECT-1 - a 10B model pretrained from scratch across 3 continents on 1T tokens, late 2024

  • INTELLECT-2 - a 32B reasoning model RL-trained across a permissionless swarm, May 2025

The approach. Their insight is that reinforcement-learning post-training decentralizes far better than pretraining, because it splits into a huge pile of communication-light “rollouts” (which run fine on consumer GPUs) and a small heavy update step. Their PRIME-RL system separates rollout, verification, and training into independent roles and tolerates workers being up to four steps out of date. Around it they built three core primitives:

  • TOPLOC - cheap probabilistic verification

  • SHARDCAST - efficient weight broadcast

  • PCCL - a fault-tolerant “collectives over the internet” library where nodes can join and leave a live run (they measured 45 Gbit/s within Western Europe, 25 Gbit/s across the Atlantic)

Traction:

  • Funding: they raised a $15M round in 2025 led by Founders Fund alongside prominent angels like @karpathy and @dylan522p, and just as I was about to publish this article, they announced a massive $130M raise making it the most well-funded project in the category.

  • Product: from a product perspective they remain on the forefront of the technical progress towards bringing decentralized training to the 100B+ size category with the INTELLECT-1 (10B) and INTELLECT-2 (32B) runs.

Open Question. Prime Intellect’s newest flagship, INTELLECT-3 (a 106B mixture-of-experts model, November 2025, scoring 90.8 on AIME 2024), was trained centrally - 512 H200s on InfiniBand - not on their own decentralized network. Their stated reason: the async RL they use needs tighter synchronization than internet links allow. So when a top team in decentralized training wanted maximum quality fast, they reached for a datacenter. The key open question is whether they continue to push the boundaries on decentralized training or take a more enterprise-first approach for product development that may deprioritize further decentralized training.

Nous Research / Psyche

Background. Psyche is the decentralized training network of Nous Research, the creator of the incredibly popular Hermes Agent. Nous Research started in 2023 as something closer to a movement than a company - a partly pseudonymous, Discord-native collective of open-source researchers whose Hermes fine-tunes racked up tens of millions of downloads.

The turn from model-tuning collective to training-infrastructure player came with the DisTrO report in August 2024, the bandwidth-crushing optimizer work that made internet-scale training look plausible. Psyche is that research productized.

The approach. Underneath sits DeMo / DisTrO, their own optimizer, which attacks the bandwidth problem from the opposite direction to DiLoCo - instead of syncing less often, it sends far less data per sync (by a claimed three-to-four orders of magnitude) by transmitting only the fastest-moving, most-important slices of the gradient.

Coordination runs through a Solana smart contract. Networking uses a peer-to-peer layer with fuzzy “did this node do real work” similarity checks, because they argue exact bitwise verification is impossible on mismatched hardware.

Traction:

  • Funding: Nous raised $50M Series A in 2025 led by @paradigm which brings its total funding to nearly ~$70m. Note: Delphi Ventures is an investor.

  • Product: Consilience 40B - a 40 billion parameter model pretrained over the internet on ~20 trillion tokens - is their headline run and one of the largest internet-pretraining efforts to date. Nous also has an impressive presence on Hugging Face with extensive contributions to the open source AI community.

Open question. Nous has exploded in popularity lately due to Hermes Agent and many signs point towards their growing influence across enterprise, so the open question is how does the continued decentralized training fit into the priority stack.

Pluralis

Background. Founded in 2024 by Alexander Long, who spent three-plus years as a research scientist at Amazon AI after his PhD and left on a specific conviction: open-source AI loses to the closed labs unless contributors can own a piece of what they help train. He recruited a team of ex-FAANG research scientists and has developed one of the the most interesting economic designs in the field.

The approach. “Protocol Models” is the key distinction with Pluralis. The model is sharded across nodes as pipeline stages where each node holds whole layers and passes only activations between them, never full weights - so no single participant ever holds the complete model. This opens up a new potential economic design where all the nodes that help train the model could potentially benefit in the economic returns from the model if they keep the weights closed (no single node would have them).

A separate paper on Unextractable Protocol Models makes the shards incompatible across time steps so the weights can’t be reassembled even by colluding nodes. That’s what makes their model elegant: you can pool compute and earn a claim on a model that stays open for use but unextractable, so value can flow to contributors without anyone copying it out and walking.

Their “Beyond Top-K” method is another big step progress step towards solving the bandwidth problem. Essentially it is a smarter compression method that compresses the inter-layer communication which lets them achieve 90% compression without degrading performance and roughly double training throughput even on connections as slow as 80 Mbps.

Traction.

  • Funding: Pluralis raised a $7.6M seed co-led by USV and CoinFund along with Topology, Variant, Eden Block, and prominent angels @balajis and @ClementDelangue

  • Product: the peer-reviewed Protocol Models, NeurIPS 2025 is at ~2B parameters and shows a compressed decentralized model over an 80 Mbps link reaching near-identical perplexity to a centralized 100 Gbps baseline at comparable throughput. Their headline completed run is “Node0,” where they trained a 7.5B parameter model over the public internet using volunteer consumer GPUs, connected over ordinary home broadband running as slow as 80–100 Mbps. The run lasted over three weeks and processed 36B tokens of training data with a low loss rate that tracked a centralized baseline - a very solid showing.

Open question. The technical innovations and the ownership economics are beautiful in theory. What I want to see is whether they can truly scale this approach into the 10-100B parameter range.

Templar / Covenant

Background. Templar grew up inside the Bittensor ecosystem as Subnet 3, one of its incentivized-training subnets, and the team behind it now operates as Covenant AI. For a stretch in early 2026 they were Bittensor’s poster child: @chamath used their training run to introduce Bittensor to Jensen Huang on the All-In podcast in March 2026, and Huang’s praise sent the Bittensor $TAO token up 17 percent. Then a few weeks later the team made the controversial decision to leave the Bittensor community.

The approach. Their bandwidth solution is SparseLoCo, which sits in the DiLoCo/LoCo lineage: instead of syncing after every step, each peer runs ~30 local optimizer steps, then shares a compressed set of data making a communication reduction of ~146x versus dense gradient sharing. The result is ~20 minutes of compute against only ~70 seconds of communication per round, at 94.5 percent compute utilization compared to INTELLECT-1’s 8.3-minute rounds at 82 percent on a model seven times smaller. The primitives are compounding fast, and this is a big step forward.

Traction:

  • Funding: no traditional VC fundraises, they have primarily funded operations through Bittensor emissions ($TAO sales) and they are one of two projects (including Gensyn) with a live token $SN3 ($23m mcap / fdv as of time of writing).

  • Product: the Covenant-72B run was completed in March this year and is the largest fully-decentralized pretraining run on record, and the one with the cleanest primary paper in the whole field. The run included 72.7B parameters, ~1.1T tokens, 70+ unique permissionless peers (each on a minimum 8x8 B200 rig - not consumer hardware) over commodity internet links. It scored 67.1 zero-shot on MMLU, edging out centralized baselines like LLaMA-2-70B making it a genuine milestone.

Open question. When Covernant quit Bittensor in April 2026, publicly, calling it “decentralization theatre” and called its own host network fake-decentralized, it made the governance problem concrete, and it’s a warning that hangs over every project running on someone else’s network. So the open question is how they reposition outside of Bittensor and maintain decentralization without the theater.

Macrocosmos

Background. A London-based company founded in 2024 by Will Squires (CEO) and Steffen Cruz (CTO) - both out of the OpenTensor Foundation itself, where Cruz was CTO and a core developer of Bittensor’s Subnet 1 (his PhD is in subatomic physics). The roughly twenty-person shop runs a portfolio of subnets for pretraining (IOTA), reasoning, data, and protein folding - the closest thing in Bittensor to a vertically integrated decentralized lab.

The approach. IOTA attacks the bandwidth problem from a different angle than most of the field. The standard play (Prime Intellect, Nous, Templar) is data parallelism: every machine holds a full copy of the model, trains on its own slice of data, and the network syncs everyone’s updates. IOTA instead splits one model into layers across machines, assembly-line style - the same pipeline-parallel family as Pluralis - so no single machine ever holds or syncs the whole model. What travels between machines is just the intermediate signal passing from one set of layers to the next, and IOTA compresses that up to 128 times before sending it (in the Orion run, a roughly 140MB transfer became 2.2MB).

The known weakness of an assembly line is that one vanished worker stalls everyone behind them, so the design treats churn as the normal condition rather than a failure: its averaging scheme, Butterfly All-Reduce, is built so that even if 35% of the machines drop out mid-run, more than 90 percent of the model’s weights still come out correct, and validators spot-check each machine’s actual outputs to catch freeloaders claiming rewards for work they didn’t do.

Traction.

  • Funding: no traditional VC fundraises, they have primarily funded operations through Bittensor emissions ($TAO sales) and good old bootstrapping through revenues across some of their non-training product offerings.

  • Product: Orion-100B (June 2026): a 100-billion-parameter model trained actually over the open internet across 48 A100s in five datacenters, at ~65 percent of datacenter efficiency (30.8 percent MFU). On paper the largest-parameter internet run yet, although not quite as decentralized as the Covenant-72B run with 70+ permissionless peer nodes).

Open question. Read the asterisk on Orion-100B carefully: it’s a ~two-day demonstration run, stopped for cost - a capability proof, not a shipped model so what we want to look for is a longer-format proof with a shipped model. Also worth calling out that Macrocosmos runs on Bittensor and is subject to the same governance exposure that drove Covenant out - not especially concerning but worth noting.

Gensyn

Background. The oldest company in the field, and the one whose origin story predates the category. Ben Fielding hit the compute wall in 2015 during his PhD work on neural architecture search and came away convinced that aggregating the world’s scattered idle compute was the only way around it. He met Harry Grieve and founded Gensyn that September - years before DiLoCo made the bandwidth problem look solvable.

Approach: Gensyn doesn’t train large models, they are positioned one layer down as the compute-coordination-and-verification protocol that decentralized training would run on top of. I decided to cover them in this article due to their infrastructure and research towards decentralized training, including:

  • Verde - a verification protocol for training on untrusted hardware; its core trick is “Reproducible Operators” (RepOps), math primitives that produce bit-for-bit identical results across different GPUs so a checker can catch a node that computed something wrong. This is the backbone of Gensyn’s trustless-verification.

  • NoLoCo - a low-bandwidth training method that throws out the usual global “all-reduce” synchronization step and instead has nodes average their weights pairwise via gossip, reaching similar convergence at a fraction of the communication cost.

  • CheckFree - fault-recovery method for distributed pipeline training that restores a failed node’s state without traditional checkpointing, cutting the overhead and downtime that node failures normally cause.

  • RL Swarm - a method for multiple models to train collectively over the internet, critiquing and revising each other’s outputs to improve collectively.

Traction:

  • Funding: raised a $43M Series A led by a16z crypto in 2023 with reports of $78M+ raised in total, making it one of the better funded projects in the space.

  • Product: Aside from the research listed above, the primary point of traction is the mainnet that launched in April this year with the intent to provide the trust and coordination (proving anonymous nodes did the work correctly, paying them, tracking who contributed what to a shared model) layer that other decentralized training projects featured in this article could use.

Open question. Gensyn is the only project in this report with a live token ($AI- $35m mcap / $268m FDV as of this report) and their mainnet went live with a prediction market as its first app(named Delphi, no relation) so any real contribution towards decentralized training is speculative today. I would pay attention to what the project remains focused on to see if they make credible contributions to this space - e.g., do other projects that are advancing decentralized training need to use Gensyn for anything?

Where I land

Decentralized training is real, it’s compounding faster than many expected, but it is still nowhere near replacing the datacenter. That said, I remain optimistic, not blindly so, but based on the progress over the past year where we’re seeing legitimate 10B+ models complete training runs and given that the stakes are so high I believe we will see continued investment and focus in this space.

On the technical side, one reason for my optimism is the physics argument has softened this year. Epoch AI - the people with the most credible “this won’t work” numbers - moved from “unlikely this decade” to conceding it’s technically feasible at frontier scale if you stack the tricks.

Covenant-72B was a big model genuinely trained over the open internet, with sync times falling an order of magnitude in a year. The engineering is looking more real with each run.

On the other hand, Prime Intellect, the top funded team, trained its latest flagship 100B run centrally. When the people who are best at decentralized training want their best result, or need to pursue revenue, they may opt for techniques that are less decentralized. That’s why this sector needs ongoing support and investors with a long term horizon.

So here’s how I view it: decentralized training is not a near-term replacement for or competitor to hyperscaler pretraining, but it is a complement, a strategically important one, and a hedge against an oligopoly on frontier capability. This for me makes it an imperative, not a nice-to-have technical innovation.

Two more things I wan to point out:

  • Governance in any decentralized network is a first-class risk - it adds a layer of complexity that can slow progress (show me a DAO that acts with speed and true agency please), so we need to be mindful that it’s not just about making the tech work, it’s also about getting the governance working in a functional way.

  • Tokens can be great for bootstrapping traction but they can also be a distraction, or at worse, a destructive force. The same pattern I flagged on the inference side applies here in full: when the token starts to look like the product and speculation takes over, it can lead to bad outcomes that hurt the underlying fundamentals of the project. As of writing, only 2 of the 6 projects profiled have a native token and I hope the others remain mindful of the perils of launching a token prematurely.

On a final note, I want to callout a theme I mentioned in the inference paper about the inference->training loop. Inference generates traces, traces become training data, training updates open models, and updated models flow back into inference. A team that runs the full loop - Prime Intellect and Nous are the obvious ones - has a compounding position a pure training network never will. The base model is becoming a commodity and the owners of the loops control the moat.

Like most things at the intersection of decentralization tech and AI - it’s early, and that’s exactly when it’s most interesting.

Disclosure: Delphi Ventures is an investor in Nous Research. I personally do not hold investments in any of the projects mentioned in this article. I was not influenced by, or received any compensation from, any project mentioned in this article - the framing and judgments are my own.

If you’re building at the intersection of AI and crypto with a real model underneath, my DMs are open.

Sources & references

The physics

Foundational methods

The runs (primary)

Governance