Date post: | 16-Jul-2015 |
Category: |
Engineering |
Upload: | epics-qt-collaboration |
View: | 139 times |
Download: | 3 times |
Yocto Project Experience: Continuous Integration Mark Hatle Senior Member of Technical Staff Wind River
Edinburgh, Scotland 23 Oct 2013
2 Yocto Project | The Linux Foundation
Agenda
• Our experiences as an OSV, productizing the Yocto Project
• Software Lifecycle • Big-Bang Example • Continuous Integration Example
• Our recommendation
Yocto Project | The Linux Foundation
Productization
What does it take to turn the Yocto Project into a commercial product?
4 Yocto Project | The Linux Foundation
Yocto Project Productization
• What does an OSVs customer’s require? • Up-to-date kernel • Up-to-date toolchain • Up-to-date userspace • One or more specific BSP (hardware support) • Quality improvements • Timely support
5 Yocto Project | The Linux Foundation
Yocto Project Productization
• Up-to-date • When the Yocto Project release is complete, it is generally
considered to be very Up-to-date (nothing older than 6 months)
• Up-to-date in the customer world is roughly nothing older than one generation, or 12-18 months
• Toolchain is at the current community supported version • Kernel is at the generally accepted stable version (LTSI or
otherwise) • …
6 Yocto Project | The Linux Foundation
Yocto Project Productization
• Hardware Support • Customers require the hardware of their choice to be
supported. • Generally new hardware requires newer versions of the
Linux kernel • Semiconductor specific optimizations for toolchains, drivers
and other components are often required.
7 Yocto Project | The Linux Foundation
Yocto Project Productization
• Quality • Anything that is released from an OSV needs to be at or
better than Open Source quality • Requires significant test resources (people and machines)
8 Yocto Project | The Linux Foundation
Yocto Project Productization
• Timely support • When something doesn’t work, the OSV is expected to be
the expert on the problem! • The OSV must understand the system as a whole • The OSV must work with the community to find existing fixes • The OSV must work with the community suggest new fixes
10 Yocto Project | The Linux Foundation
Software Lifecycle
• Yocto Project Lifecycle
• Commercial Product Lifecycle
• Real World Examples
11 Yocto Project | The Linux Foundation
Yocto Project Lifecycle
• 6 month development cycle • 4 – 4 week development milestones • 5th milestone is stabilization • Maintenance releases managed for roughly 1 year
12 Yocto Project | The Linux Foundation
Yocto Project Lifecycle 4 - 4 week development milestones
https://wiki.yoctoproject.org/wiki/Yocto_1.5_Schedule
13 Yocto Project | The Linux Foundation
Commercial Product Lifecycle Examples
Big-Bang Continuous Integration Start with community release Work in parallel with community Add ‘missing’ requirements Influence community work Add new value-add features Add new value-add features QA/Verify OSS QA/Verify new components QA/Verify value-add features
QA/Verify OSS
QA/Verify value-add features Work to resolve bugs internally Work with community to fix bugs Release to customers Release to customers Maintain Release (5-10 years) Maintain Release (5-10 years)
Examples assume approx 12-18 month release cycles
14 Yocto Project | The Linux Foundation
Big-Bang Lifecycle Example
• Big-Bang refers to the work • Starts with a large amount of community software • Need to learn how it works • Learn what required product features need to be
implemented • More of the traditional approach • Follow Open Source…
15 Yocto Project | The Linux Foundation
Big-Bang Lifecycle Example
Oct-Dec Apr-Jun Jan-Mar Jul-Sep Oct-Dec Jan-Mar Apr-Jun
Yocto Project
Commercial
Update Update EOL
U1 U2 U3 U4 U5 U7
• No opportunity to influence design, must follow • Enhancements should be contributed to the next
version, and backported • Bugs found may or may not have been found by
community, may require additional resources to resolve
Customer Adoption
16 Yocto Project | The Linux Foundation
Big-Bang Lifecycle Example
Oct-Dec Apr-Jun Jan-Mar Jul-Sep Oct-Dec Jan-Mar Apr-Jun
Yocto Project
Commercial
Update Update EOL
U1 U2 U3 U4 U5 U6
• At start of commercialization, product components are up to 6 months old
• By the time of release, it’s nearly 12+ months old • Has a “shelf life” of only 6-12 months from release • Decision extend shelf-life or uprev?
Customer Adoption
17 Yocto Project | The Linux Foundation
Big-Bang Lifecycle Example
Apr-Jun Jan-Mar Jul-Sep Oct-Dec Jan-Mar Apr-Jun
Yocto Project
Commercial
Update Update EOL
U1 U2 U3 U4 U5
Customer Adoption
Commercial
Jul-Sep Oct-Dec Jan-Mar
U1 U2 U3 U4 …
• Extend shelf life? • Update kernel, toolchain, BSPs and other critical elements • Backport additional select features • 6 months work, only gains 6-12 months Customer Adoption • No community help
Customer Adoption
18 Yocto Project | The Linux Foundation
Big-Bang Lifecycle Example
Apr-Jun Jan-Mar Jul-Sep Oct-Dec Jan-Mar Apr-Jun
Yocto Project
Commercial
Update Update EOL
U1 U2 U3 U4 U5
Customer Adoption
Commercial
Jul-Sep Oct-Dec Jan-Mar
U1 U2 U3 U4 … • Uprev? • Rebase changes (15-45 days) • Add commercialization time • Finish date has now slipped back a bit more • Still following, not leading…
Customer Adoption
Yocto Project Update Update EOL
19 Yocto Project | The Linux Foundation
Commercial Product Lifecycle Examples
Big-Bang Continuous Integration Start with community release Work in parallel with community Add ‘missing’ requirements Influence community work Add new value-add features Add new value-add features QA/Verify OSS QA/Verify new components QA/Verify value-add features
QA/Verify OSS
QA/Verify value-add features Work to resolve bugs internally Work with community to fix bugs Release to customers Release to customers Maintain Release (5-10 years) Maintain Release (5-10 years)
Examples assume approx 12-18 month release cycles
20 Yocto Project | The Linux Foundation
Continuous Integration Lifecycle Example
• Continuous Integration refers to tracking and contributing to the community development • Work with the community on development • Learn capabilities and feature deficits as development
continues • Ability to influence the community by discussing
requirements and/or providing patches • Ability to monitor OSS quality over a longer period of time • Requires resources to follow the community • Lead with the OSS community
21 Yocto Project | The Linux Foundation
Continuous Integration Lifecycle Example
Oct-Dec Apr-Jun Jan-Mar Jul-Sep Oct-Dec Jan-Mar Apr-Jun
Yocto Project
Commercial
Update Update EOL
U1 U2 U3 U4 U5 U6
• Ability to identify issues and work with the community to resolve them
• Enhancements can be contributed during development
• Bugs can be filed with the community and worked on as a group
Customer Adoption
…
Customer Adoption
22 Yocto Project | The Linux Foundation
Continuous Integration Lifecycle Example
Oct-Dec Apr-Jun Jan-Mar Jul-Sep Oct-Dec Jan-Mar Apr-Jun
Yocto Project
Commercial
Update Update EOL
U1 U2 U3 U4 U5 U6
• During commercialization components are current • By the time of release, it’s only 6-7 months old • Has a “shelf life” of 12-24 months from release • No reason to extend the shelf-life!
Customer Adoption
…
23 Yocto Project | The Linux Foundation
Continuous Integration Lifecycle Example
Apr-Jun Jan-Mar Jul-Sep Oct-Dec Jan-Mar Apr-Jun Jul-Sep Oct-Dec Jan-Mar
Yocto Project
Commercial
Update Update EOL
U1 U2 U3 U4 U5 U6
Customer Adoption
…
Yocto Project Update Update EOL
Commercial U1 U2 U3 U4 U5 U6
Customer Adoption
Yocto Project Update Update EOL
• Uprev! • No rebase required… • Ramp up developers
24 Yocto Project | The Linux Foundation
Real World Examples
• Big-Bang • Took a large product team 6 months to commercialize • Required 6 one month cycles to add enhancements and
backport upstream features • First cycle was devoted to investigation • Required significant developer resources • Release was approx time of next YP release
25 Yocto Project | The Linux Foundation
Real World Examples
• Big-Bang Extended life • Required 6 months development to update kernel, BSPs,
toolchain and other customer essential systems • Required roughly the same development effort as a new
product • Release occurred at appox time of EOL of the base Yocto
Project release
26 Yocto Project | The Linux Foundation
Real World Examples
• Big-Bang Uprev • Requires 45 days (of one engineer) to update the tree 2
Yocto Project releases. This 45 days simply enabled the main development team.
• Projected to required 6 months of development to commercialize and add new features, QA, etc.
• Release was now after the next YP release
27 Yocto Project | The Linux Foundation
Real World Examples • Continuous Integration
• Work done in parallel with the community. • Able to ramp up a small team to full team over the course of
development. • As bugs were found, many filed with the community and
fixed in a timely manner. (Many critical bug fixes were submitted to the community.)
• As missing features were identified, worked with the community to implement functionality.
28 Yocto Project | The Linux Foundation
Real World Examples • Continuous Integration
• Required 1 full time resource to manage integration and tracking of the community.
• Resource was the go-to person for questions about community quality, bug triage, etc.
• Continuous uprev averaged every 1-2 weeks for the first 4 milestones. Bug fixes and QA took most of the two week time. (Longer than we expected.)
• Unexpected change in the 5th milestone caused a single 4 week integration cycle.
• Expect weekly uprevs after product release for point fixes.
29 Yocto Project | The Linux Foundation
Real World Examples • Continuous Integration
• Estimated to be the same amount of “improvement” and commercialization required
• Smaller teams required, as less unexpected ‘reactionary’ work was required
31 Yocto Project | The Linux Foundation
Recommendations
• Semiconductor Mfg – Kernel/BSP support • ‘big-bang’ – address customers wanting a stable approach • ‘continuous integration’ – keep changes timely, and ready to
release for the next stable release • May depend on chip market, release schedules and
customer demand
32 Yocto Project | The Linux Foundation
Recommendations
• OSV • Use continuous integration. Quicker time to market, more
up-to-date software, and longer shelf-life. • Should allow time to sync to semiconductor and customer
requirements.
33 Yocto Project | The Linux Foundation
Recommendations
• ISV / Application Developers • Follow the YP versions your customers need. Most likely to
‘follow’ the stable release, but for large applications the continuous-integration model may make sense.
34 Yocto Project | The Linux Foundation
Recommendations
• Device Developers • Look at what your needs are. • If you don’t need ‘work-in-progress’ features, it’s better to
start with a stable release! • If you expect to be updating the OS over the life of the
product, continuous integration may be useful.