Power & Sites · United States

Growing Pains: How Distributed AI Training Changes The Network Between Datacenters

Credited to The Next Platform · nextplatform.com · 8h ago

Photograph · The Next Platform

Large-scale AI training has already escaped the confines of a single datacenter. Google said Gemini was trained synchronously across clusters in multiple locations; Microsoft has connected AI data centres in Wisconsin and Georgia into what it describes as one distributed AI supercomputer; AWS has connected AI compute clusters across wide areas to allow Anthropic to build Claude models; Meta has built high-capacity datacenter interconnects to support model training; and CoreWeave and Google Cloud recently announced cross-cloud training with a private interconnect, with Azure likely to follow later in the year.

This growing geographic spread of model training is partly due to the limits of single datacenters or campus clusters, which can be strained as they try to meet the compute and power demands of new models. Cisco estimates that training models today can require clusters with tens of thousands of GPUs. By 2030, the largest individual frontier training runs could draw 4-16GW of power, according to researchers at Epoch AI. Distributing compute allows hyperscalers and datacenter operators to build in locations with more available power or space and fewer planning constraints.

How To Train An LLM

At the core of an LLM is a neural network with billions of numerical parameters, or weights, that are adjusted during training. A batch of training data passes through the model, which makes a prediction. The system measures the error and calculates how the weights should change. The weights are updated and the process starts again. At the scale of modern models, this work is distributed across thousands of GPUs and other accelerators, such as AWS Trainium and Google TPUs.

Different GPUs can process separate batches of data or different parts of the model. But they cannot work entirely independently: at various points they must exchange results and synchronize before the next stage of training can begin.

What networks carry between the accelerators is generally not the text or images used to train the model, but large arrays of numerical data such as gradients and intermediate results. Thousands of accelerators may need to exchange and combine that data at roughly the same time.

"If you look at any training job, it is basically a repetition of compute and synchronization," explains Ramesh Sivakolundu of Cisco's Silicon One architecture team. "You compute the weights, communicate those weights to synchronize every GPU in the cluster, and then restart the computation. That means bursty traffic happens periodically throughout the training process."

Thousands of accelerators can finish a computation phase and start communicating at the same time. This can lead to "incast" problems where many senders converge on the same destination and traffic arrives faster than the next link can handle.

The fundamental issue lies with synchronization. In a synchronous training job, one group of GPUs cannot simply ignore a slower group and continue independently. The network-wide exchange must complete before the participating GPUs can move on to the next stage. A network delay therefore becomes compute delay.

"You don't want the network to be the bottleneck in the training process," says Sivakolundu. "The idea is to keep the GPUs fully occupied and not have the network introduce additional latency into the overall training program."

That does not mean that every instance of network congestion or packet loss brings the entire model training to a complete halt. However, a delayed or dropped flow may require retransmission, prolong a collective communication operation, and leave other accelerators waiting. More serious failures can require training to return to a saved checkpoint.

Different Kind Of Interconnect

"Inside a single datacenter, you can assume you have built a full network connecting the GPU racks with the bandwidth you planned for," says Itamar Gold, director of product management at Cisco. Gold’s point is that with scale-across, coordinated AI workloads may have to traverse a more constrained inter-site fabric than the network within each facility. Instead, he says, traffic destined for the other facility may be funneled through what is effectively a narrower pipe. A synchronized burst can turn that pipe into a temporary choke point.

Traditional DCI connects otherwise independent datacenter environments, and is typically built for redundancy, reach and workload distribution. The traffic that runs over them is often replication traffic or application data, and the flows are asynchronous, meaning one flow does not have to finish in lockstep with the others.

In scale-across AI training, the inter-site network becomes part of a coordinated distributed computation rather than simply moving data between otherwise independent systems. It is therefore essential to have sufficient bandwidth and predictable delivery.

From Rack To Region

Within a rack, bandwidth requirements can be enormous – up to 800 Gb/sec and 1.6 Tb/sec – but the distances are short. High-speed serial lanes combine to make a GPU cluster appear as a single compute unit. When the AI fabric is extended across an entire facility, “scale out” racks will still require connections up to 1.6 Tb/sec. At these data rates, copper is generally confined to short reaches. Connections across rack rows and datacenter floors instead use pluggable optics, with scale-out links extending from hundreds of meters to around 2km.

“Scale-across” extends a coordinated AI fabric across multiple facilities, making inter-site network behavior part of workload performance. Depending on the architecture, those links may span metro or regional distances, and potentially farther depending on the architecture. According to Cisco modelling of large distributed AI deployments, aggregate bandwidth requirements can reach roughly 14x a conventional DCI baseline.

For starters, short-range datacenter optics are not designed to carry 400-800 Gbps signals over hundreds of kilometers. For these distances, operators need coherent optics that use sophisticated modulation and high-performance digital signal processing to keep high-rate optical signals usable over metro and regional fiber spans, compensating for physical impairments that become more pronounced with distance. The number of ports matters too. Cisco estimates that connecting two 100 megawatt AI sites could require 12,000 to 32,000 coherent optical ports at the scale-across layer, compared with roughly 1,000 to 2,000 for conventional DCI between comparable facilities. At 400 Gb/sec or 800 Gb/sec per port, that adds up to multi-petabit aggregate capacity. Moving that much data is also a power and space problem. Coherent pluggables put the coherent transponder function directly into router or switch ports, avoiding a separate bank of standalone DWDM transponders. Cisco says this reduces the additional rack space, power and cooling needed for the inter-site optical layer which is an important consideration when the AI factories themselves are already power-constrained.

The Latency Problem

Even with sufficient bandwidth and suitable optics, distance adds latency. If congestion develops, the network needs time to signal the sender to slow down. Over a local AI fabric, feedback can arrive quickly. However, if the connection is stretched across 100 kilometers of fiber, considerably more data can be transmitted before the sender even learns that there is a problem.

Cisco calculates that on an 800 Gb/sec connection spanning 100 kilometers, around 100 MB of data can already be in transit during that feedback interval. Cisco argues that these longer feedback loops increase the value of networking silicon designed with substantially deeper buffering. The shallow-buffer switching architectures designed for very low latency inside a datacenter have less capacity to absorb traffic while congestion feedback crosses a long-distance link.

"The reason you need deeper buffers as the distances get longer is that you don't get feedback about the transfer of packets for a longer period of time," says Sivakolundu. "If you don't buffer deeply enough, you can end up dropping and retransmitting packets, increasing latency."

Deep buffering absorbs a temporary burst or disruption while traffic management and congestion signaling address the underlying problem. Sustained congestion still needs more capacity or a change in how traffic is distributed.

The same buffer also helps with oversubscription. If the combined capacity feeding into an inter-site connection is greater than the capacity of the link itself, a synchronized burst can arrive faster than the link can drain it. Buffering gives that excess traffic somewhere to wait without getting dropped while congestion controls respond.

Rather than dividing the available buffer into fixed amounts for each port, a shared buffer is a common pool of packet memory that can be allocated wherever congestion appears. That lets a busy link absorb a larger burst without dropping packets.

"The longer you're reaching out, the more you may need to buffer," says Gold. "You also need the flexibility to put as much buffer as possible where it is needed, perhaps on one port or a few ports where you've identified a problematic flow. Our approach is a single, fully shared buffer."

AI training does offer one useful characteristic: much of its collective communication is repetitive and therefore more predictable than DCI traffic. Cisco's approach is to use proactive congestion management to steer or schedule traffic before a predictable burst creates a problem, while retaining deep buffering as a safety net for transient congestion and unpredictable events such as a link failure. The two approaches are complementary: one tries to avoid the queue while the other gives the network somewhere to put the traffic when a queue forms.

Cisco’s Bet On Co-Design

Cisco’s scale-across architecture uses the Silicon One P200, a 51.2 Tb/sec programmable, deep-buffer routing processor included in the Cisco 8223 and Cisco N9000 platforms. Those systems can be paired with Cisco's 400G and 800G coherent pluggable optics for the long-distance links, while open line systems such as Cisco Open Transport 3000 and Cisco NCS 1014 handle the optical transport layer where required.

Coherent pluggables generate the DWDM wavelengths directly from the routing system, while the line system amplifies and carries those wavelengths across the fiber plant. For links requiring multiple parallel fiber pairs, Cisco's Open Transport 3000 uses a multi-rail design that combines the optical components for multiple rails onto one line card. Cisco claims this reduces power per rail by 75 percent and rack space by 80 percent. Where a separate transport system is required, the NCS 1014 can provide 12.8 Tb/sec of capacity from a 1RU line card.

The P200 has a programmable run-to-completion network processor and P4 tooling, which Cisco says allows protocol support, telemetry and other packet-processing features to be developed in software rather than making every change dependent on a new silicon generation.

For scale-across, where operators are still testing different topologies and traffic-management methods, "the same programmable flexibility can let network functions evolve without every new requirement forcing a chip replacement,” says Sivakolundu.

Cisco also includes hardware-based protection for inter-site traffic in that design, its argument being that encryption should not create another processing bottleneck for the training fabric.

A key part of Cisco's pitch is the co-design. A networking ASIC is defined years before the finished router reaches a datacenter. Cisco argues that because its silicon, system and optics teams sit within the same company, requirements for port density, buffering, power, telemetry and the optical interfaces can feed back into the chip design rather than the system team receiving a commercial ASIC and working out what it can build around it.

That does not mean scale-across has settled on a single approved architecture. Cisco itself divides the problem into campus, metro, and regional deployments, and different operators are experimenting with various combinations of networking and training techniques.

Implementations vary significantly by operator, including distance, topology and workload design, while the common problem is maintaining coordinated AI performance across sites. The technology is still at an early stage, however. "We are seeing early deployments now, but it is a process and I think it will take a few years," says Gold.

Designing a network for distributed training will not eliminate the latency of distance, but it can keep that interconnect from becoming the bottleneck. That gives hyperscalers and datacenter operators the flexibility to add compute where power and space are available, while still treating multiple sites as part of the same training infrastructure.

Power & SitesUnited States

Power system plans to meet large-load demand miss near-term solutions: analysts

Credited to Utility Dive · utilitydive.com9h ago

Useful

Major work by regional planners is focused on flexible access to existing transmission. Some say greater emphasis on advanced transmission technologies can quickly create room for large loads.

Power system planners — including utilities, grid operators and regulators — are racing to develop new ways to bring large loads online faster, but they are often overlooking solutions that could increase capacity quickly while long-term, structural remedies are being negotiated and implemented, analysts and stakeholders told Utility Dive.

Shortages of skilled labor and grid equipment are slowing the buildout of new generation to meet rising demand from AI data centers, while market reforms must go through a regulatory process that can be prolonged.

In response to directives from federal regulators and the White House, planners working to address load growth are focused on innovative integrations of large loads and generation to create more transmission availability.

But some experts say there should also be greater focus on interventions like high-capacity conductors and advanced transmission technologies that are more cost effective and can quickly create room for new large loads.

Grid operators under pressure

On June 18, the Federal Energy Regulatory Commission issued “show cause” orders to six system operators directing them to submit plans for speeding the interconnection of large loads. A ll six have requested more time to develop processes for bringing these loads online without passing the costs on to other customers or threatening system reliability.

FERC is not the only official body pushing power system operators to hurry up. The White House is leaning on tech companies and utilities to help the country “win the AI race” while adhering to a voluntary Ratepayer Protection Pledge to make hyperscalers pay for the infrastructure needed to serve them. President Donald Trump has also taken executive actions declaring infrastructure planning an urgent priority.

But this pressure campaign cannot change the underlying market and regulatory structures that determine who builds what where. Many large loads are waiting to interconnect while Congress debates proposed legislation that would affect them.

Meanwhile, local governments and states are going their own way, with some imposing new vetting requirements or incentives, while others have proposed or enacted outright moratoria or pauses on new data center interconnections.

“The elements of Order 1920 showed the way, and large loads add the exclamation point on the need for new planning processes... But there has not been a complete rethink of utility planning yet.”

Director of Research and Policy, Grid Strategies

Given the increasing difficulty of bringing new data centers and new generation online, some in the industry say increasing the utilization of the transmission system can accommodate these loads more quickly.

Transmission investments are increasing, but most projects continue to be driven by reliability and asset renewal needs rather than load growth, according to FERC’s 2025 State of the Market report.

Large loads require a “shift from project-by-project upgrades to proactive and coordinated long-term planning,” according to a July report by the Energy Systems Integration Group and the Brattle Group.

The report, titled “Transmission Planning with Large Loads,” says a new framework is needed and must include load flexibility and allow for scenario-based, least-regrets planning.

It must also allow for the uncertainty in today’s load growth forecasts, said paper co-author Johannes Pfeifenberger, a principal with The Brattle Group.

That means including “fast and cost-effective near-term options that address speed to power,” like advanced carbon conductors with fiber sensing and analytics, “while longer-term transmission solutions are being planned and built,” he added.

Some see the potential for solutions to emerge from FERC Order 1920, which was issued in 2024 and overhauled long-term regional transmission planning. Most entities covered by Order 1920 began the process of filing their plans this year, with more compliance filings due in 2027.

The order could be useful in the era of large-load demand growth because it created a “robust framework” to address uncertainties from traditional generation retiring and new, variable resources coming online, said Zachary Zimmerman, Grid Strategies’ director of research and policy.

“The elements of Order 1920 showed the way, and large loads add the exclamation point on the need for new planning processes,” said Zimmerman. “But there has not been a complete rethink of utility planning yet.”

Southwest Power Pool sets an example

The Southwest Power Pool has emerged as a potential model for holistic grid planning — though its recently-approved Consolidated Planning Process, which merged its interconnection and transmission tracks, was in development before large-loads became the dominant concern in the electric power sector.

Nevertheless, the plan was praised by FERC officials and others. It is the right type of planning framework for today because it “explicitly brings different planning tracks together,” while other regions are only developing elements of coordination, ESIG said in its report.

SPP planners have felt a significantly greater sense of urgency since the surge in large loads began around 2025, said Natasha Henderson, the system operator’s senior director of strategic policy and grid innovation, and Casey Cathey, its vice president of engineering.

In addition to the Consolidated Planning Process, SPP has introduced several other pathways for large loads willing to make concessions for interconnection. In January, FERC approved SPP’s plan for interconnecting High Impact Large Loads, or HILL, and the associated High Impact Large Load Generation Assessment, or HILLGA.

Streamlined HILL 90-day interconnection studies will also apply to associated HILLGA interconnections if the associated new generation is within two substations and specifically support HILLs, Henderson said.

FERC has also approved SPP’s Conditional High Impact Large Load, or CHILL, process for interconnecting loads willing to be curtailed when transmission is inadequate, Henderson said. It is an approach to new loads that shares some similarities with ERCOT’s “connect and manage” process for generators that agree to curtailment to get online, she added.

SPP’s pending Price Adaptive Load Service, or PALS, under consideration at FERC is “true connect and manage,” Henderson said, because it does not require loads to be matched with generation or transmission. Under that plan, loads would face significant curtailment risk, she added.

SPP is also developing what may be the first regional Intermediate Planning Process, or IPP, said Cathey. It will accelerate speed to power with easily deployed upgrades, grid enhancing and advanced transmission technologies, reconductoring and load shedding to maximize reliability, he added.

Other grid operators have indicated they intend to take different approaches, mostly focused on trading load flexibility for speed of interconnection to create availability on existing transmission.

PJM embraces load flexibility

PJM Interconnection may see 70% of future U.S. data center development and, with costs spiking, it is under pressure from the region’s governors and legislators and from FERC to comply with the show cause orders, according to a brief from law firm White and Case.

PJM’s proposed long-term transmission planning reforms, as outlined in PJM’s June 30 response to a Dec. 2025 show cause order directing the grid operator to create new pathways for colocated load, are “a step in the right direction,” said Grid Strategies’ Zimmerman.

PJM proposed an Interim Resource Adequacy Service, or IRAS, that is also similar to the Texas connect and manage rules. Under PJM’s proposal, large loads could interconnect without new capacity but must, when called on, reduce load or switch to on-site resources.

In August, FERC granted PJM an extension until Nov. 16 to comply with regulators’ directives on large loads and colocation.

Batch Zero pushed Texas toward consolidated grid planning

The Electric Reliability Council of Texas, though not FERC-jurisdictional, is also moving ahead on new processes, said Warren Lasher, president of Lasher Energy Consulting and a former ERCOT senior director of system planning.

In June, the Public Utility Commission of Texas approved ERCOT’s “Batch Zero” processing — a system-wide analysis of large loads in the interconnection queue. That system-wide analysis led the system operator to a consolidated process that merged the Batch Zero review with the annual six-year transmission planning look-ahead, Lasher said.

The connect-and-manage interconnection process that ERCOT has been using for decades added a significant amount of new generation, Lasher said. During Batch Zero stakeholder discussions in 2025 and 2026, ERCOT decided to combine connect and manage with batch studies of flexible loads and loads that bring their own generation to expand conditional access to transmission, he said.

In that ongoing process, ERCOT also developed other interconnection tracks: the Withdrawal-Limited Private Use Network, WLPUN, and the Provisional Controllable Load Resource, or PCLR, according to a Batch Zero explainer published by the grid operator.

WLPUN allows large loads to use their own on-site generation to offset the amount of transmission capacity they need, allowing a faster interconnection. PCLR allows large loads faster interconnections if ERCOT can curtail their controllable load during local transmission constraints.

Batch Zero processing is on pause pending an audit of the queue at the direction of Gov. Greg Abbott.

MISO reworks existing plans

The Midcontinent Independent System Operator is also rethinking how it plans infrastructure, but is reworking existing processes rather than creating new approaches from the ground up like SPP, said James Okullo, director of system planning with the Energy Systems Integration Group and a former MISO manager for strategic assessments.

Some of these changes can be seen in shorter study periods, such as the 90-day study approved as part of MISO’s Expedited Resource Addition Study, or ERAS, in July 2025, he said. MISO’s working group has also developed planning initiatives to address large loads like the Expedited Project Review process that will review proposed transmission projects every two months, Okullo said.

New planning scenarios will enable 120-day studies and a hotly debated Zero-injection Generator Interconnection Agreement to streamline approval for large loads with co-located generation, Okullo added.

Advanced transmission technologies a missing piece

Policymakers “now realize the new large load forecasts are real,” but “they are still not enabling the urgently needed short-term solutions,” said Theodore Paradise, chief policy and grid strategy officer for advanced powerline manufacturer CTC Global.

Advanced transmission technologies and high-capacity conductors can be deployed in 36 months or less to cut costs and in some cases double transmission capacity on a line, Paradise added, citing a recent ClearPath report.

Regional planners should be required to put advanced conductors and advanced transmission technologies in the planning process early on, because planning that only includes them as an afterthought misses their system-wide optimization value, Paradise said. At least six states have passed laws since 2024 requiring utilities to consider advanced transmission technologies.

Policymakers “now realize the new large load forecasts are real,” but “they are still not enabling the urgently needed short-term solutions.”

Chief Policy and Grid Strategy Officer, CTC Global

“Using them can expand availability of existing transmission” and “lower customer costs,” Paradise said.

Advanced technologies are not being broadly integrated, despite work validating them by large utilities such as AEP in Texas, PPL in Pennsylvania and Southern California Edison, Okullo said. Existing production cost models and power flow simulations can model the new technologies and asset management practices are evolving for them, Okullo said.

“The urgency driven by large load growth means new transmission must be built and deploying advanced technologies in the process is the more efficient solution,” he added.

Google's €13bn data center buildout in Finland under investigation for 300-hectare forest removal

Credited to Data Center Dynamics · datacenterdynamics.com · 9h ago

Photograph · Data Center Dynamics

Finnish Licensing and Supervision Authority requests reports from local municipalities

Google's data center projects in Finland are facing scrutiny after the company reportedly razed some 300 hectares of forest land prior to securing mandatory environmental impact assessments.

In mid-September, Finnish news service STT reported that the Finish Association for Nature Conservation said that it suspected that illegal logging had been carried out at Google's sites in Muhor and Kajaani.

On September 18, Finland's Licensing and Supervision Agency (LVV) requested reports from the municipality of Muhos, the city of Kajaani, and Tuike Finland Oy - the project manager - on the planned Google data center projects, their permits, and the measures taken in the areas.

According to reports, forestry works began months ago at the data center project sites, despite the environmental impact assessment of such works still being underway.

According to LVV, around 300 hectares of forest have been felled for the Muhos data center project. The agency notes: "Logging in a master plan area requires a landscape work permit in accordance with the Building Act, if the master plan so provides. Muhos has a partial master plan in force. In Kajaani, the partial master plan has been partially announced in force.

"The municipality of Muhos and the city of Kajaani have approved the local plans. In both municipalities, the local plans have been appealed to the administrative court, which is why a building ban is in force in the areas. The building ban in question requires a landscape work permit for landscape-changing measures, such as felling trees and landscape-changing earthworks, starting from the decision to approve the local plan. The municipal building control authority is responsible for monitoring the legality of the measures with regard to the Construction Act."

The LVV concludes that EIA procedures are coordinated by the agency itself, and that: "The issuance of a reasoned conclusion in the EIA procedure is a key document in the processing of the permits required for the projects."

In a statement sent to STT by Google, the company's communications manager, Michiel Sallaets, has denied any illegal forestry activities. Sallaets said that the logging has been done on land owned by Google "in full accordance with the requirements of the Forestry Act." He added that all potential profits from selective logging will be donated to local projects.

Another Google spokesperson, Sondre Ronander, told AFP: "We have conducted extensive nature surveys covering the entire area as part of the zoning and Environmental Impact Assessment processes, to ensure high-value nature zones within the sites have been protected during forestry works."

The Finnish Association for Nature Conservation has called for construction at the site to be halted, as has the Safety and Chemicals Agency, until the environmental impact assessment is completed.

Google detailed its major buildout plans in Finland at the start of September 2026, stating it would be investing €13 billion ($15.13bn) in digital infrastructure in Finland across 2027 and 2028.

The company has had a presence in Finland since it acquired a former paper mill in Hamina in 2009 and transformed it into a data center. The data center is cooled with seawater, and has been upgraded numerous times over the last 15 years. In 2022, the company purchased 50 acres of adjacent land to accommodate further expansion.

In 2024, Google acquired 1,400 hectares of land in Kajaani and Muhos from state-run forest agency Metsähallitus, but did not detail plans for the sites at the time. In February of this year, it added to its Finland land portfolio again with the acquisition of 900 hectares in Vaala for a possible data center project.