Models & Labs · United States

CoreWeave targets AI inference bottlenecks with full-stack optimization

Credited to SiliconANGLE · siliconangle.com · 13h ago

Photograph · SiliconANGLE

AI inference is fast becoming the workload that decides the economics of the AI boom. Training built the first wave of GPU clouds, but serving models faster and cheaper will define the next.

That shift is pushing specialized cloud providers beyond raw GPU capacity into storage, networking and software. One provider is layering managed services for training, post-training and inference atop its infrastructure, according to Urvashi Chowdhary (pictured), vice president of product and AI services at CoreWeave Inc.

“I think if you look at the AI developer’s journey, they’re looking to solve a problem and they want to do it as quickly as they can with the best performance and the cost to scale,” Chowdhary said. “So, we’ve been really focused on building up the layers of our stack, building on top of the reliable infrastructure to build more managed services, whether it’s for training, post-training, inference.”

Chowdhary spoke with theCUBE Research’s Dave Vellante and John Furrier at the Fully Connected event, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed AI inference, managed services across the full stack and CoreWeave RL Rollouts, a new capability designed to accelerate agentic model iteration. (* Disclosure below.)

Optimizing the AI inference stack layer by layer

Inference demand is climbing fast. A survey of CoreWeave customers and prospects by theCUBE Research found one healthcare customer’s inference workload share rose from about 10% in the first year to 40% in the second, with a roughly 50% share expected within 12 months. CoreWeave’s answer is tuning every layer above the hardware, from the vLLM engine to quantized models and custom speculative decoders, Chowdhary explained.

“One thing that we’ve been very intentional about is leveraging open source tools and technologies, contributing back to open systems so customers have flexibility and then also building our services on top of each other,” she said.

Reinforcement learning adds new pressure. When customers train agentic models with rewards and verifiers, inference becomes the bottleneck during rollouts, according to Chowdhary. CoreWeave RL Rollouts, a preview capability built on Nvidia Corp.’s Dynamo framework, loads new checkpoints into a live deployment. In testing, the capability improved model reload latency by 15x compared with a baseline configuration.

“When you’re doing RL rollouts, you’re continually creating new model checkpoints and versions and you want those to roll out into your inference setup so you can scale it independently,” she said. “And we were able to speed that up by 15x, which means your training runs fast and your inference is scaling while you’re continuing to train your model quickly.”

Those capabilities now sit inside CoreWeave Forge, a platform launched at the event that connects serving, observability, post-training and evaluation. Forge is free to start, with paid tiers offering additional capabilities, extending access to AI development tools and services to individual developers, Chowdhary explained.

“We want to create accessibility for these leading technologies,” she said. “Even if you’re an individual developer signing up today on your own, you still get the best performance, you still get the best reliability and you don’t have to compromise.”

Here’s the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of the Fully Connected event:

(* Disclosure: TheCUBE is a paid media partner for the the Fully Connected event. Neither CoreWeave, the sponsor of theCUBE’s event coverage, nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Support our mission to keep content open and free by engaging with theCUBE community. Join theCUBE’s Alumni Trust Network, where technology leaders connect, share intelligence and create opportunities.

Are you an AWS customer? Support SiliconANGLE financially by buying your AWS services from our Marketplace portal page and links: https://siliconangle.com/aws-marketplace/

Founded by tech visionaries John Furrier and Dave Vellante, SiliconANGLE Media has built a dynamic ecosystem of industry-leading digital media brands that reach 15+ million elite tech professionals. Our new proprietary theCUBE AI Video Cloud is breaking ground in audience interaction, leveraging theCUBEai.com neural network to help technology companies make data-driven decisions and stay at the forefront of industry conversations.

White House bars Microsoft from sponsoring green card applications as JD Vance accuses it of fraud

AI stocks crumble on report that OpenAI's annualized revenue is much lower than believed

Gallatin AI raises $50M in funding for its military logistics platform

Elastic's AlertZero puts AI agents to work on security alert overload

Anthropic launches critical infrastructure program and free OSS Scanner for open source

Alphabet spinoff Isomorphic Labs reportedly raising funding at up to $50B valuation

SECURITY - BY DUNCAN RILEY. 11 HOURS AGO

SECURITY - BY DUNCAN RILEY. 12 HOURS AGO

Join our community

CapitalUnited States

OpenAI ARR Snafu Highlights Flawed Metric

Credited to The Information$ · theinformation.com15h ago

Useful

Here’s a news flash: The quasi-revenue numbers for OpenAI that got wide circulation last week were off— way off. Instead of $70 billion in annualized revenue, as we and others reported, OpenAI is generating closer to $50 billion in annualized revenue (sometimes shortened to…

View the article · $The Information

Front Method

Models & Labs · United States

HBM4 Demand Holds Firm As Rubin Keeps 12 Layers

Credited to Business Korea · businesskorea.co.kr · 20h ago

Samsung and SK hynix Are Set to Benefit as HBM4 Mass Production Expands While Counterpoint Says Demand Concerns Are Overstated

Analysis indicates that the 6th-generation High Bandwidth Memory (HBM4) equipped in Nvidia’s next-generation artificial intelligence (AI) accelerator ‘Rubin’ will remain centered on 12-layer configurations. This suggests that concerns over slowing memory demand due to a reduction in HBM stacking layers are exaggerated.

As Samsung Electronics and SK hynix have begun mass production of HBM4, expanding demand for high-value-added memory is expected to support the earnings growth of the two companies.

On Oct. 8, market research firm Counterpoint Research announced the results of its internal review regarding the recently raised rumors of reduced HBM capacity, stating, “The concern about slowing memory demand is a misunderstanding.” The assessment is that confidence in memory demand growth has actually strengthened in October.

Some industry voices claimed that Nvidia requested major HBM suppliers to lower the number of stacked layers from the existing 12 to 8. This prompted forecasts that if HBM capacity decreases, the production of commodity DRAM will increase, which could lead to a drop in memory prices.

However, Counterpoint Research analyzed that most products equipped with Rubin are actually converging toward a 12-layer configuration.

This is because securing the memory bandwidth and capacity necessary for data processing is more important than cost reduction in AI computing. The firm explained that, contrary to concerns, 4-layer products have not been observed in the ASIC market either.

Counterpoint Research viewed that if HBM demand is maintained, the supply shortage of commodity DRAM is also highly likely to continue. HBM production requires more than three times the production capacity of commodity DRAM. Expanding HBM production restricts commodity DRAM supply, creating a structure that supports memory prices.

In its preliminary third-quarter earnings released on Oct. 8, Samsung Electronics announced that it recorded 195 trillion won (about $145 billion) in revenue and 107.4 trillion won in operating profit. These figures represent increases of 126.6% and 782.5%, respectively, compared to the same period last year. It is the first time a domestic company’s quarterly operating profit has exceeded 100 trillion won.

Counterpoint Research identified exchange rates as the reason there was no massive earnings surprise recently seen in semiconductor companies, despite the record-high performance. The firm explained that because Samsung Electronics has a high share of dollar-denominated revenue, a sharp drop in the dollar-won exchange rate imposes a burden of reduced won-converted profits.

Counterpoint Research stated, “The concern about slowing memory demand is a misunderstanding,” and forecast, “There is a possibility that Samsung Electronics’ record-high performance will not end as a one-time event.”