How NVIDIA Verification Engineers Find Problems Before AI Systems Reach Customers

Sakeena Fiza describes her job as a verification engineer at NVIDIA in terms more akin to a detective story than a world-class engineering lab.

“Verification engineers see the shadows and shine light into the nooks and crannies,” Fiza says. “Every time we get a system, the first thing we think about is, how can we break it?”

And when will it happen?

“I always like to think of it as a mystery to be solved,” she says.

Testing the systems powering the AI era

At NVIDIA, the systems Fiza and her colleagues in the Data Center Systems Engineering Lab research are the engines of the AI era. Her work begins in the lab, before the world knows a product exists, when power is first applied to a new system.

Components are developed one by one, boards are integrated, and firmware and software teams work together as engineers watch for the first signs of life.

One of Fiza’s first and most lasting memories of working at NVIDIA was seeing the NVIDIA Rubin GPU work for the first time at the system level.

“It literally just said ‘NVIDIA Corporation Device,’” she recalls. “And this is the first time in the world that Rubin GPUs have been enumerated at the system level, so everyone is cheering and celebrating.”

Emotional moments are just the beginning. From there, engineers must make systems more resilient—from trays to racks, clusters, production lines and customers’ AI factories.

Finding problems before customers do

Fiza describes validation—the process of making sure a physical device works correctly before mass production—as being the “first customer for the product.” The goal is to make hardware perform at its limits in a variety of real-world conditions before anyone else relies on it.

“The goal is to always find problems before the customer knows there is a problem,” she says.

From robotics to data center systems

Fiza joined NVIDIA after earning her bachelor’s degree from the University of California, Irvine, where she studied computer science and engineering. Her path to hardware grew from a fascination with systems.

Growing up in Dubai, she learned coding through Logo programming. That early experience encouraged future work in system design. She built a Mars rover at her high school’s robotics camp and worked on developing an unmanned aerial vehicle in college.

She started working in data center systems because she had the opportunity to work with entire machines. At NVIDIA, she says, validation sits at the intersection of firmware, hardware, software, mechanical design, thermal behavior, manufacturing and customer experience.

“You can be a mechanical engineer when you want to be,” she says. “You can be an electrical engineer when you want to be. You can be a firmware engineer when you want to be.”

Solving complex hardware failures

The failures Fiza pursues can be immeasurable or microscopic. Rack-scale issues can involve high-speed signals, thermal margins or power integrity. Other causes may involve over-tightened screws or dust levels within a customer’s facility.

“Solutions may be difficult to find,” Fiza says. “We have to follow the clues, ignore the red herrings and know where to look.”

If logs show how something failed, Fiza’s job is to discover why. Verification engineers reproduce the problem, modify conditions, examine firmware, remove mechanical variables, analyze signals and review oscilloscope captures to narrow down possible causes.

A single board can contain tens of thousands of components. A rack could reach 500,000 units. Those parts should not just coexist; they must operate as one system at scale and under stress, amid the complex realities of production and deployment across different AI factory configurations.

“I want people to understand how complex the hardware required to run AI is,” Fiza says.

“Like the Avengers coming together”

For Fiza, the pressure of the work is inseparable from its excitement. According to her, development is “like the Avengers coming together.” Architects, designers, software engineers, firmware engineers and verification engineers gather in a room and work toward a functioning system.

“One of the things I realize when I come to work is that I am never alone,” she says.

Being a verification engineer means practicing disciplined doubt. You believe the system will work, and then try to think of every way it might not.

The job requires the tenacity of a master detective, the diagnostic abilities of a general practitioner and the temperament of someone solving a crossword while confronting the catastrophic failures of others.

Each project brings new puzzles and new failure modes—and, in turn, a chance to make the next system better.

“We are very excited about the products we are currently developing,” Fiza says. “They will change the world.”