First time’s a charm for FPGA verification at Lockheed Martin Space Systems Company www.mentor.com handful. Automation and standardization do come with upfront costs that can be significant, including training existing staff or even hiring new engineers. The Lockheed Martin team says such costs are worth it and that the new approach can even improve the embattled reputation of FPGA design and verification teams. “When we first said that we were going to have success on our first try, a lot of people were laughing, frankly” said David Foggy, Lockheed Martin’s FPGA lead for the GOES-R project. “I think, in general around the industry, it had sort of become accepted that FPGAs would have problems and that the FPGA design A small team of engineers at Lockheed Martin Space Systems Company in Denver has achieved a first-pass success in verifying field-programmable gate arrays (FPGAs) bound for the NOAA/ NASA GOES-R satellite, scheduled for launch in 2015. Keys to the team’s result included a pilot verification methodology built on SystemVerilog and Open Verification Methodology (OVM), and also various support and tools from Mentor Graphics, most notably Questa Verification Management. The methodology is a departure from the group’s longstanding verification practices, which mostly focused on writing directed tests in VHSIC Hardware Description Language (VHDL). Such tests made sure designs did the basics. However, because of the limits to directed VHDL, the design wasn’t fully verified and corrected until the hardware stage. The success of the new approach, the best measure of which is NASA’s request that the Lockheed Martin team present their technique to the agency’s own engineers, perhaps carries a few broader lessons for those interested in verification of all types of FPGAs, but especially those bound for use in outer space. Why verification is different for space-based FPGAs Space-related verification work comes with a slew of considerations, including that the devices be treated like application-specific integrated circuits (ASICs). That is, design and verification teams must get things right the first time given their system will eventually operate at anywhere from a few hundred to several thousand miles above the Earth’s surface. This distance makes it impossible to just upload a new image to patch a faulty FPGA. (And in the case of GOES-R the point is moot, since the FPGAs are of the burn-once variety.) Also, engineers working on unique systems headed to space work hard to limit the number of hands tinkering with the hardware as it makes its way to the launch pad. So the oft-used practice of repeatedly burning a new FPGA, and then seeing what happens in a testing lab is off limits, especially since antifuse radiation- hardened FPGAs cost several thousand dollars each. The alternative to inefficient and expensive lab debug is to create a robust testbench environment. Lockheed Martin created such an environment in SystemVerilog/OVM. Combined with the right sort of expertise, especially a background in object-oriented programming, a SystemVerilog/OVM methodology makes possible reusable, self-checking testbenches capable of constrained random testing of both corner cases and overall robustness. Automation is vastly superior to the manual alternative, which required an engineer to briefly scan a waveform and make a cursory analysis. This is especially true given that a self checking testbench can check many hundreds or thousands of cycles in the time it takes that engineer to manually check a small “When we first said that we were going to have success on our first try, a lot of people were laughing, frankly.” — DAVID FOGGY, LOCKHEED MARTIN CUSTOMER PROFILE Lockheed Martin Space Systems Company designs, develops, tests, manufactures and operates a full spectrum of advanced-technology systems for national security and military, civil government and commercial customers. Design Challenge ■ Take verification beyond writing directed tests in VHDL ■ Achieve first pass success in a burn-once FPGA bound for a 2015 NOAA/NASA satellite Solution ■ Train verification team in SystemVerilog and OVM ■ Build a testbench framework that makes possible constrained random testing of corner cases and overall robustness ■ Use Mentor Graphics Questa Verification Management to track code and functional coverage metrics in the test plan
Transcript
First time’s a charm for FPGA verification at Lockheed Martin Space Systems Company
w w w . m e n t o r . c o m
handful. Automation and standardization do come with upfront costs that can be significant, including training existing staff or even hiring new engineers. The Lockheed Martin team says such costs are worth it and that the new approach can even improve the embattled reputation of FPGA design and verification teams.
“When we first said that we were going to have success on our first try, a lot of people were laughing, frankly” said David Foggy, Lockheed Martin’s FPGA lead for the GOES-R project. “I think, in general around the industry, it had sort of become accepted that FPGAs would have problems and that the FPGA design
A small team of engineers at Lockheed Martin Space Systems Company in Denver has achieved a first-pass success in verifying field-programmable gate arrays (FPGAs) bound for the NOAA/NASA GOES-R satellite, scheduled for launch in 2015. Keys to the team’s result included a pilot verification methodology built on SystemVerilog and Open Verification Methodology (OVM), and also various support and tools from Mentor Graphics, most notably Questa Verification Management.
The methodology is a departure from the group’s longstanding verification practices, which mostly focused on writing directed tests in VHSIC Hardware Description Language (VHDL). Such tests made sure designs did the basics. However, because of the limits to directed VHDL, the design wasn’t fully verified and corrected until the hardware stage. The success of the new approach, the best measure of which is NASA’s request that the Lockheed Martin team present their technique to the agency’s own engineers, perhaps carries a few broader lessons for those interested in verification of all types of FPGAs, but especially those bound for use in outer space.
Why verification is different for space-based FPGAs
Space-related verification work comes with a slew of considerations, including that the devices be treated like application-specific integrated circuits (ASICs). That is, design and verification teams must get things right the first time given their system will eventually operate at anywhere from a few hundred to several thousand miles above the Earth’s surface. This distance makes it impossible to just upload a new image to patch a faulty FPGA. (And in the case of GOES-R
the point is moot, since the FPGAs are of the burn-once variety.) Also, engineers working on unique systems headed to space work hard to limit the number of hands tinkering with the hardware as it makes its way to the launch pad. So the oft-used practice of repeatedly burning a new FPGA, and then seeing what happens in a testing lab is off limits, especially since antifuse radiation-hardened FPGAs cost several thousand dollars each.
The alternative to inefficient and expensive lab debug is to create a robust testbench environment. Lockheed Martin created such an environment in SystemVerilog/OVM. Combined with the right sort of expertise, especially a
background in object-oriented programming, a SystemVerilog/OVM methodology makes possible reusable, self-checking testbenches capable of constrained random testing of both corner cases and overall robustness. Automation is vastly superior to the manual alternative, which required an engineer to briefly scan a waveform and make a cursory analysis. This is especially true given that a self checking testbench can check many hundreds or thousands of cycles in the time it takes that engineer to manually check a small
“When we first said that we were going to have success on our first try,
a lot of people were laughing, frankly.”
— DaviD Foggy,
lockheeD Martin
CuStoMer ProFiLe
Lockheed Martin Space Systems Company designs, develops, tests, manufactures and operates a full spectrum of advanced-technology systems for national security and military, civil government and commercial customers.
Design Challenge
■ Take verification beyond writing directed tests in VHDL
■ Achieve first pass success in a burn-once FPGA bound for a 2015 NOAA/NASA satellite
Solution
■ Train verification team in SystemVerilog and OVM
■ Build a testbench framework that makes possible constrained random testing of corner cases and overall robustness
■ Use Mentor Graphics Questa Verification Management to track code and functional coverage metrics in the test plan
w w w . m e n t o r . c o m
and verification teams will be the last people off of most projects. We’ve gone a long way to dispel that myth.”
Spreading the word about oVM at Lockheed Martin
Foggy, who Lockheed Martin colleague Michael Donnelly described as a management-designated “prophet” of SystemVerilog/OVM verification, is now parachuted into Lockheed Martin projects in need of a verification overhaul. The most recent example: Foggy and Donnelly spent several months in late fall 2011 working at Lockheed Martin’s Sunnyvale, Calif. campus on another U.S. government project.
The ASIC in question had three major functional blocks. Consistent with what Foggy described as “fast and loose,” the verification engineers assigned to each block were using different, uncoordinated verification methodologies. Foggy and Donnelly decided to throw out these methodologies, and with three months remaining before the device was sent to the foundry, to use the framework they honed for GOES-R.
“One of the things I’m most proud of is that we’ve gotten things to the point where we can just unzip a file, run a rename script and have a skeleton of an environment up in a matter of minutes,” said Donnelly, a senior FPGA design
engineer. “It takes all the tedium of stitching everything together out of the equation. We can dive right into the fun stuff — writing sequences, agents and so on.”
Keeping coders on the straight and narrow
The framework itself is as much or more about basic testbench architecture and workflow as about the specifics of SystemVerilog or OVM. For starters, Foggy and his team spent months developing a set of template classes. These classes force his team — for the design and verification of the GOES-R Command and Data Handling (C&DH) subsystem FPGAs that includes about 13 engineers -- to conform to the methodology created with help from Mentor Graphics OVM experts.
Next, Foggy sent everyone on his team to in-person SystemVerilog and OVM training offered by Mentor Graphics. Mentor’s OVM experts also provided on-site consulting and Foggy’s team took ample advantage of the online educational modules offered through Mentor Graphics Verification Academy. All such training was useful given the changes compelled by SystemVerilog
and OVM, both of which are based on object-oriented principles. Among the ironies of the shift in verification technologies: software expertise is suddenly as or even more important than a hardware background.
“I’ve asked management to only hire electrical or computer engineers with a strong hardware software [OOP] background,” said Foggy. “We’ve really shifted the paradigm in the company.”
Metrics, new and otherwise
Another shift, and one that wasn’t always easy to get used to, had to do with precisely how verification success was measured and communicated. One metric in particular generated consternation: bugs identified during simulation. The consternation came from the fact that the new technique turned up more problems during simulation runs — a good outcome given that catching bugs earlier in the development cycle drives down the overall cost of the project. However, since most engineers loathe bugs no matter where they turn up, Foggy and his team occasionally had to reassure their colleagues that the increasing bug count was actually a welcome development and a sign that
Lockheed Martin’s FPGA is bound for the Geostationary Operational Environmental Satellite-R Series (GOES-R),the next generation of geostationary weather satellites, scheduled to launch in 2015.
“[T]here hasn’t been a single functional error that has come up in the lab that we just flat out missed in verification.”
For the latest product information, call us or visit:
MGC 06-12 1030640-w
w w w . m e n t o r . c o m
the new technique was working. (A new agile flow implemented by Foggy and Lockheed Martin Fellow Tim Gallagher also helped to catch more early-stage bugs.)
A major measure of success is how well the FPGAs have been functioning in-circuit in the lab.
“We’ve had four or five FPGAs in the testing lab since working on the GOES-R command and data handling project,” said Ellen Prommersberger, a senior FPGA design engineer. “In all the testing that has been performed, there hasn’t been a single functional error that has come up in the lab that we just flat out missed in verification.”
The main reason Prommersberger knew they hadn’t missed anything: she and her colleagues had adopted a comprehensive coverage driven flow as another crucial metric. This flow tracked code coverage and funcational coverage. All the coverage measurements were tied together using Mentor Graphics tools.
Mentor Graphics Questa Verification Management helped the GOES-R verification team track the progress of their verification effort and communicate
to NASA. The tool, which integrates seamlessly into a SystemVerilog/OVM framework, generates easy-to-read HTML reports that give updates on all the various code and functional coverage metrics in a test plan. It also allows a test plan, in Lockheed Martin’s case captured in an Excel spreadsheet, to become an executable part of the HTML output.
Foggy, however, said that the best and simplest measurement can be summarized in just three words: “first pass success.”
Such success is good news for any firm working on FPGAs bound for space-based applications. The watchword for these applications: complexity. On any given satellite, one FPGA today might do what a handful of single-purpose cards did a few years ago.
Foggy and his team often work with the Actel RTAXS family of FPGAs, higher-end parts that sell for upwards of tens of thousands of dollars apiece. These FPGAs are packed full of functions, including arbitrating across multiple inputs, and dealing with the addition of SpaceWire (IEEE 1355) communications and DMA (Direct Memory Access) cores.
“Any time you add an input packet source, you also add one more thing that can go wrong in the course of accepting that input and passing it onto the next part of the chain,” said Donnelly.
However, increasing complexity ultimately pales to the fundamental reason to try a new verification methodology. Given a shaky and uncertain global economy, and increasing competition across all of the electronics industry, the pressure is on to deliver on business basics.
“Sure, the new technique helped us deal with complexity,” said Foggy. “But the real impetus for us didn’t have anything to do with that.
“Mostly we wanted to do everything possible to improve affordability and also both engineering and operational excellence,” he said. “Any FPGA design and verification engineer with similar goals, which I imagine are universal, might be well served to check out SystemVerilog and OVM.”
“We’ve gotten things to the point where we can just
unzip a file, run a rename script and have a skeleton of an environment up in a matter of minutes. We
can dive right into the fun stuff — writing sequences,
agents and so on.”
— Michael Donnelly,
lockheeD Martin
(Left to right) Lockheed Martin engineers David Foggy, Michael Donnelly and Ellen Prommersberger
