Models & Labs · United States

AWS’ Colleen Aubrey: Stop inventing new AI metrics — it’s your business metrics that matter

Credited to SiliconANGLE · siliconangle.com · 12h ago

There isn’t a business or information technology leader today who doesn’t have artificial intelligence at the top of their priority list. The challenge for most organizations is deciding what to measure and how to determine whether an AI initiative was successful.

To better understand how customers should approach AI metrics, I spoke with Colleen Aubrey from Amazon Web Services Inc,, who has some deep experience in this area.

As senior vice president of Applied AI Solutions at AWS and a member of Amazon’s S-team, she builds on that foundation with lessons from 20-plus years at Amazon, where she built customer-facing products for millions of sellers and advertisers. The playbook she drew from that experience gives enterprises a practical path to move out of AI pilot purgatory, and it starts with throwing out the AI return-on-investment slide.

She argues that understanding AI ROI requires thinking in terms of business metrics. “I worry that people start trying to come up with new metrics to measure the impact of AI, but it’s the same metrics,” she explained. “It’s your business metrics that matter.” AWS has spent 20 years building cloud infrastructure. AI business applications are a much newer opportunity.

When Aubrey joined AWS, she inherited 35 application-layer programs. “I would say Connect for the contact center was probably our most successful example,” she told me in a recent interview ahead of AWS re:Invent Nov. 30-Dec. 4.

She studied what made it work: The team had deep operational experience running Amazon’s customer service function; it chose a sharp wedge (outstanding voice quality on the AWS backbone rather than chasing feature parity; and the team knew its customers. The lessons from that success are now the blueprint for the expanded Amazon Connect family: Connect Customer, Connect Talent, Connect Health and Connect Decisions.

The playbook Aubrey built for the Connect family closely mirrors what IT leaders need to move their own AI programs out of pilot mode. Here are my five takeaways that should matter to IT leaders:

Feature parity is a trap

The first lesson Aubrey drew from her assessment was about where to start. Nobody runs a better supply chain than Amazon, which is exactly why the early version of AWS Supply Chain was such a useful learning experience. When Aubrey got close to the product, she saw a team working to catch up rather than lead. They had been building five modules that “largely were about getting to feature parity” with entrenched enterprise resource planning vendors that had a decades-long head start.

Connect had taken the opposite path, and it’s instructive. “They understood they weren’t going to achieve feature parity,” Aubrey said of the Connect team. Instead, they led with outstanding voice quality on the AWS backbone, started narrow and built from there.

Aubrey applied that same lesson to the supply chain. She rebooted it as Connect Decisions by asking a different question: where is the pain most acute, and where is the current tooling weakest? The answer was planners. “They’re firefighting the whole time,” she said. “They’re overwhelmed by a whole bunch of signals. By the time they get to decide, the moment has passed.”

The same applies to enterprises. AI projects that stall often try to replicate an entire workflow. The ones that succeed identify a painful decision point and make it dramatically better.

The business metrics haven’t changed

Aubrey’s point about measurement deserves unpacking. “Companies are stuck on what’s the ROI of AI, and they’re looking at return on tokens and this sort of stuff,” she said. “The question is, can you move your business metrics by working in a different way?”

Her proof point is Wells Vehicle Electronics, which, according to AWS, cut inventory by 7% in three months while expanding the SKUs they actively monitor from 20 to 2,000. Nobody at Wells is talking about tokens. They’re talking about working capital. If a project can’t be tied to a metric the chief financial officer already tracks, it’s an experiment, not an investment.

AI doesn’t remove bottlenecks; it moves them

Amazon started where most companies start: with software development, making AI tools available across the company. Aubrey admits there’s a bit of chaos at first. “Everyone’s just experimenting with a bunch of things,” Aubrey said.

She sees that as a feature, not a bug, because that’s where the real learning happens. “You go through this learning period of, you know what, our test coverage sucks,” she said. Fix test coverage, and you find “our pipelines suck,” so you fix pipeline health, eliminate manual intervention and deploy faster. The result in her group this year: a twofold improvement in the velocity of pushes to production.

The lesson is that the velocity didn’t come from the coding assistant. It came from fixing everything downstream that the assistant exposed. Enterprises that buy the tool and skip the plumbing will see activity rise while output stays flat.

Own the outcome, not the task

I asked Aubrey whether Amazon has adjusted its goals to reflect what employees can now do with AI. Her answer was an honest “yes and no.” In her user experience team, roles have collapsed: designers are being pushed to ship code to an alpha environment, and Amazon has created a new “UX tech” role.

The “no” is that org design and goal-setting are still catching up. Her message to her team is blunt: “The only choice you can’t make is to own the task. You have to choose to own the outcome.” Every IT leader should borrow that line. Task-based job descriptions will age poorly.

Good AI with bad UX will fail

AWS’s design philosophy for the Connect family is what she calls “humorphism,” the idea that AI should fit the way people already work, not the other way around. Her biggest concern is the lag between internal prototypes and production.

“Shipping good AI with a bad user experience will fail, in my opinion,” she said. Her team has invested heavily in closing that gap, moving beyond traditional command-based interfaces toward AI that picks up cues, remembers decisions, and carries work forward. The goal, she said, is to make AI so usable that people change how they work, not because they’re told to, but because it makes sense.

She was equally candid about maturity. TE Connectivity has scaled Connect Decisions to 2,000 planners, but Connect Talent reached general availability only in September and has early customers in deployment. Aubrey said the team is focused on getting the product right before aggressively scaling, reflecting the same “pick a wedge” discipline she applied across the Connect solutions.

What this means for IT leaders

Final thoughts

What made this conversation valuable was Aubrey’s willingness to share how she thinks about building AI products that work in production, from studying what made Connect successful to applying those lessons across supply chain, hiring, health care, and customer engagement. That candor is a useful model for IT leaders, because most enterprise AI programs are stuck for the same reasons AWS’s applications once were: too much breadth, too little focus and success measured by the wrong yardstick.

The AI era won’t reward companies with the most pilots. It will reward those that pick a sharp problem, fix the plumbing it exposes, redesign work around outcomes and move the metrics they already have.

Zeus Kerravala is a principal analyst at ZK Research, a division of Kerravala Consulting. He wrote this article for SiliconANGLE.

Support our mission to keep content open and free by engaging with theCUBE community. Join theCUBE’s Alumni Trust Network, where technology leaders connect, share intelligence and create opportunities.

Are you an AWS customer? Support SiliconANGLE financially by buying your AWS services from our Marketplace portal page and links: https://siliconangle.com/aws-marketplace/

Founded by tech visionaries John Furrier and Dave Vellante, SiliconANGLE Media has built a dynamic ecosystem of industry-leading digital media brands that reach 15+ million elite tech professionals. Our new proprietary theCUBE AI Video Cloud is breaking ground in audience interaction, leveraging theCUBEai.com neural network to help technology companies make data-driven decisions and stay at the forefront of industry conversations.

Midterm elections will put AI and data centers on trial, but Big Tech may end up winning anyway

The AI boom's missing balance-sheet question: What has actually been verified?

Atomicwork launches certified AI coworker marketplace for enterprise IT and security

Dropbox acquires Tigris Data for $350M, releases new AI productivity features

Autonomous combat drone maker Shield AI buys radar startup Echodyne for $5.3B

Five9 buys contact center intelligence startup Level AI after posting narrow earnings beat

CLOUD - BY MARIA DEUTSCHER. 17 HOURS AGO

EMERGING TECH - BY MIKE WHEATLEY. 18 HOURS AGO

Join our community

Models & LabsUnited States

Palantir, Changshin Partner to Transform Footwear Development and Manufacturing With AI

Credited to TheElec · thelec.net14h ago

Useful

Global footwear manufacturer Changshin INC. plans to adopt artificial intelligence (AI) and data platforms from Palantir Technologies to integrate information and workflows spanning footwear development, sample planning and factory production. The initiative aims to give Changshin's departments comprehensive visibility across the entire process.

The companies recently signed a partnership agreement to deploy Palantir's data operations platform, Foundry, and its Artificial Intelligence Platform (AIP) across Changshin's development and production operations.

According to Palantir, the partnership is one of the outcomes of its recent AgentCamp program, which allows participants to build AI agents and apply them to real-world business tasks. Changshin said its engineers used company data to develop and demonstrate initial workflows during AgentCamp, confirming their potential to improve operational efficiency.

Building on those results, the companies plan to connect decisions made during product development with factory production planning and execution. Changshin aims to consolidate information and feedback across the entire process to identify problems earlier, reduce inefficiencies in handoffs between departments and incorporate production results into future decision-making.

Palantir emphasized that Changshin will retain control over how AI is applied at every stage.

Palantir Foundry is a data operations platform that integrates fragmented enterprise data and connects it to analytics, decision-making and operational execution. One of Palantir's flagship products, it primarily serves private-sector businesses and enables operational teams and developers to work from a shared data model.

Palantir AIP structures enterprise data and business rules into an ontology and securely integrates large language models (LLMs). This enables companies to develop and operate AI agents and applications for practical business use.

Original · TheElec

Front