Date post: | 03-Dec-2014 |
Category: |
Technology |
Upload: | igalia |
View: | 210 times |
Download: | 1 times |
Testing and HW Virtualization
Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
7th WhiteRabbit Workshop - November 2012
Agenda
About IgaliaVirtual vs RealLinux support and making of FMC TDC boardTesting, Virtual Hardware and WhiteRabbitTechnical details and demo
Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
Who is Igalia?
Free and open software company50 hackers around the globeHacking in different areas
kernelvirtualizationgraphics/renderingcompilersdistros...
www.igalia.com
Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
Virtual vs Real
Using virtual HW is all about having the possibility to testlow level software, such as firmwares, HDL cores orkernel/drivers, without real HW. It is not about replacingreal hardware at all.The idea is analysing every test case on real hardware inorder to know if we can run it on virtual hardware. Thisapproach saves resources and time.
Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
Linux support and making of FMC TDC board
Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
FMC TDC Linux support
Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
FMC TDC Linux support
Time-To-Digital-Converter (TDC) boardUsed for measuring the time of arrival of each incomingpulse in order to save the timestamp into the memory,ready to be read later
CERN/Igalia effort to support FMC-TDC board in Linuxdrivers, user-space library and test cases developed
Integration with ZIO framework, use FMC bus and SPECdrivers
Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
Making of FMC TDC Linux support
We face the typical issues when developing drivers for nonmature hardware
Hardware not availableBuggy firmwares delaying developmentIs it a hardware or software bug? Where is it?Flashing firmware, running tests... are time-intensive tasksHardware specification evolving along the project...
Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
Making of FMC TDC Linux support
In order to minimize the previous issues we explored thevirtual way with the following results ...
No real hardware bottle-neck, parallel developmentNo buggy firmwares issuesGot a virtual baseline in order to test old and newspecifications/firmwares, bug identification really easySaved time in real flashing and testing approachesDeveloped continuous integration and automatic testingtooling for software (kernel, libraries, etc) on top of virtualhardwareGot good-enough virtual models (buses, I/O spaces,memory maps, etc) to cover goals.Further emulation (VHDL, firmware, etc) possible but itwasn’t required for this project
Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
Testing, Virtual Hardware and WhiteRabbit
Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
Testing, Virtual Hardware and WhiteRabbit
Some thoughs ...It is not possible catching all bugs in a complex systemA complex system based on hardware/software requiredifferent, but integrated, testing approachesPragmatic testing is a must. Keep all possible testing in thesoft side. It is more affordable, fast and flexibleIt doesn’t matter where the bug is (software or hardware).If it appears in a critical system then it is a critical bugIn the long, continuous integration and automatic testingis the wayFlexible and robust testing approach is key to get greatreleasesAt the end, final testing should always run in the realsystem
Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
Testing, Virtual Hardware and WhiteRabbit
Potential WhiteRabbits components to testTools and compilers (ARM, LM32)BuilrootBarebox (bootloader)IPLLinux kernel and drivers (versions, configurations, etc)User space applications (ptpd, hal, libs, etc)VHDL and LM32 binaries
Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
Testing, Virtual Hardware and WhiteRabbit
Testing with Virtual Hardware would be possible on thefollowing components
No virtual Hardware requiredTools and compilers (ARM, LM32)Builroot
Virtual Hardware requiredLinux kernel and drivers code (similar FMC TDC approach)LM32 binaries (real-time)VHDL (behaviour)
Not exploredBarebox (bootloader)IPLUser space applications (ptpd, hal, libs, etc)
Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
Technical details and demo
Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
Our work
Virtualized models
Industry PackTEWS TPCI-200.GE IP-OCTAL-232.
FMC TDCSPEC board (only needed bits)FMC TDC board
Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
FMC TDC board
Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
FMC TDC board
It was needed to test error conditions on the driver
Generic stuffThe virtual board in being detected by the driver as agenuine device.Normal mode: no error injection.
DMADMA error: TDC_DMA_STAT_ERR andTDC_DMA_STAT_ABORT.DMA timeout.
Input pulsesFilter events with pulse width lower than 100 ns(requirement).Test different input pulse arrival rate.Test timestamp threshold setup.Test time threshold setup.
Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
What was virtualized?
SPECPCI: (sub)Vendor ID, (sub)Device ID. No MSI support.Memory Map: BAR addresses.
FMC TDC FirmwareMemory map: status and configuration registers, Event’scircular buffer.DMA core: memory map, registers.
Input pulse generationDifferent parameters setup: input pulse arrival rate, pulsewidth, etc.
Error injectionAs it was previously mentioned.
Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
How we did that
QEMU (http://wiki.qemu.org)Open source machine emulator and virtualizer.
Buildbot (http://trac.buildbot.net/)Continuous integration system designed to automate thebuild/test cycle.Python. Web interface to check the logs.
Our own testing suiteBased on Sam’s experience working with PTS in the past.Needed more flexibility: different setups for the same test.Developed in Python, as PTS.
Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
Testing suite design
Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
Testing suite design
Launch QEMU with some setups:Choose memory size, virtual machine image, network, etc.Snapshot mode.Shared folders to interchange data.No screen.The kernel is pass as a parameter to QEMU -> Not usingthe one in the virtual machine.Timeout. If QEMU is still running after a given time,something wrong happened.Forward kernel logs to QEMU’s serial port. We can readdmesg info from host.
Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
The result (I)
Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
The result (II)
Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
The result (III)
Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
The result (IV)
Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
Potential improvements
Improve the virtual model.Do a more realistic model of the device.Add VHDL virtualization.
QEMU configuration: use config files?Buildbot: fix issues.Testing suite
Add new features (hooks, etc).Choose Red Hat’s autotest framework?
Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
Testing and HW Virtualization
Questions & Answers
Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
Testing and HW Virtualization
Thank you!
Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez
Links
http://www.ohwr.org/projects/spec
http://www.ohwr.org/projects/fmc-tdc
http://www.ohwr.org/projects/fmc-tdc-sw
http://blogs.igalia.com/berto/2012/10/03/industrypack-qemu-and-linuxcon/
http://events.linuxfoundation.org/images/stories/pdf/lceu2012_garcia.pdf
Testing and HW Virtualization Javier Muñoz Mellid & Samuel Iglesias Gonsalvez