+ All Categories
Home > Technology > Continuous Integration: Improving Quality in Moodle

Continuous Integration: Improving Quality in Moodle

Date post: 19-Jul-2015
Category:
Upload: dan-poltawski
View: 413 times
Download: 2 times
Share this document with a friend
Popular Tags:
20
Continuous Integration: Improving Quality in Moodle Dan Poltawski Integrator Moodle HQ @dan_p the world’s open source learning platform #mootieuk15
Transcript
Page 1: Continuous Integration: Improving Quality in Moodle

Continuous Integration: Improving Quality in Moodle

Dan Poltawski Integrator

Moodle HQ

@dan_p

the world’s open source learning platform

#mootieuk15

Page 2: Continuous Integration: Improving Quality in Moodle

the world’s open source learning platform

Wide topic, today focusing on:

• “Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible.” - Martin Fowler

http://www.martinfowler.com/articles/continuousIntegration.html

Continuous Integration

Page 3: Continuous Integration: Improving Quality in Moodle

the world’s open source learning platform

Integration Process: 2001 - 2010Numerous problems:

• Single developer may be only person aware of a

change

• Inconsistency was rife

• Developers often unaware of the consequences of

some changes

• Testing was not enforced

• Unit tests were limited and frequently broken

New Development

Release

Page 4: Continuous Integration: Improving Quality in Moodle

the world’s open source learning platform

Moved to git and brought about a change in

development process

• Enforced code review

• Enforced ‘manual testing’

• Integration team introduced to move

towards ensuring more consistency

• Introduction of CI systems

Integration Process: Today

Automated Tests

Automated Checks

New Development

Peer Review

Integration Review

Testing

Release

Page 5: Continuous Integration: Improving Quality in Moodle

the world’s open source learning platform

• Needed automated tests to supplement the manual reviews done by the Integration team

• CI Server was setup - Jenkins • Open source/extensible • Self hosted • http://jenkins-ci.org

Post-integration checker

Automated Tests

Automated Checks

New Development

Peer Review

Integration Review

Testing

Release

Page 6: Continuous Integration: Improving Quality in Moodle

the world’s open source learning platform

Post-integration checkerJenkins polls for changes in

integration.gitCode pushed to integration.git

Issue status is set (e.g. failed)

Detects code changes and starts running checked jobs

Jenkins notifies status changes

Integration team Jenkins

Page 7: Continuous Integration: Improving Quality in Moodle

the world’s open source learning platform

Post-Integration Checks

Page 8: Continuous Integration: Improving Quality in Moodle

the world’s open source learning platform

Post-integration checks: whitespace• First job to be created (the whitespace police

started it all!)

• Detect incorrectly spacing added

• Frequent error: • Easy to miss setting up your editor for

Moodle • Not spotted easily by reviewers

Page 9: Continuous Integration: Improving Quality in Moodle

the world’s open source learning platform

Post-integration checks: compare upgrade & install

• Detects DB inconsistencies between install and upgrade code:

1. Runs the installer 2. Run upgrade from a previous

version 3. Compares the database

schema from both 4. Fails in the case of any

differences

• Finds schema inconsistencies previously frequently undetected - (common source of number of 1.9 -> 2.x upgrade bugs)

• Also brings fringe benefit of testing upgrade code (though limited by lack of data)

Page 10: Continuous Integration: Improving Quality in Moodle

the world’s open source learning platform

• Initially support for Moodle 2.0-2.2 with SimpleTest

• Custom CLI generated test data, ran tests and output results as JUnit xml file

• Culturally testing was not embedded • Tests rarely run or added by developers • Had to mock everything - no way to

setup test data

Post-integration checks: Unit testing

Page 11: Continuous Integration: Improving Quality in Moodle

the world’s open source learning platform

• Moodle 2.3 added support for PHPUnit

• Data generators significantly improved unit testing experience

• Helped move cultural change towards automated tests

• Challenges: • Keeping the test time down • ‘Random’ fails

Post-integration checks: Unit testing

Page 12: Continuous Integration: Improving Quality in Moodle

Moodle 2.3 Moodle 2.6 Moodle 2.9

24,273 44,491 58,011

the world’s open source learning platform

Unit test assertions

Page 13: Continuous Integration: Improving Quality in Moodle

the world’s open source learning platform

• Many created a result of mistakes slipping through: • Incorrect upgrade savepoints

• version.php file check

• Check files built correctly (less, shifter, grunt)

• php syntax check (php -l)

• Unresolved merge conflicts

Post-integration checks: Others

Page 14: Continuous Integration: Improving Quality in Moodle

the world’s open source learning platform

• Desirable to check before integration • Can help reduce delays

(timezones) by providing more rapid feedback

• Better suited to more pedantic checks

• Provide supplemental information rather than just pass/fail

Pre-integration checker

Automated Tests

Automated Checks

New Development

Peer Review

Integration Review

Testing

Release

Inline comments must end

in full-stops, exclamation marks, or question

marks.

Page 15: Continuous Integration: Improving Quality in Moodle

the world’s open source learning platform

Pre-integration checker

Merges branch with moodle.git

Run pre check jobs against merged branch

Produce ‘smurffile’ of changes, filtered by lines affected by patch

jshint

phpcs

csslint

phpdoc

php -l

commit

savepoint

thirdpartyReport summary back to tracker

checkstyle.xml

Page 16: Continuous Integration: Improving Quality in Moodle

the world’s open source learning platform

• Not possible to run all our testing scenarios on a per-commit basis due to time to run

• Nightly server does these long-running tasks • Behat (Acceptance tests) • Multi-platform tests

• phpunit: pgsql, mariadb, mysql, mssql, oracle

• behat: firefox, chrome, phantomjs

Nightly checks

Automated Tests

Automated Checks

New Development

Peer Review

Integration Review

Testing

Release

Page 17: Continuous Integration: Improving Quality in Moodle

the world’s open source learning platform

//Teacher creates a discussion @Test public void teacherCreateDiscussion() throws Exception { //Adding a discussion forum.clickForumLink(this.properties.get("nameOfForum")); forum.clickAddNewDiscussionTopicButton(); forumPosts.enterSubjectField(this.properties.get("teacherSubject")); forumPosts.enterMessage(this.properties.get("teacherMessage")); forumPosts.clickPostToForum(); }

Automated UI tests: Selenium Native

• Java-based selenium test suite • Not embedded in development process

• Tests written by test manager, not developers • Maintained in separate repository

Scenarios in 2.3

28

Page 18: Continuous Integration: Improving Quality in Moodle

the world’s open source learning platform

Given I add a "Forum" to section "1" and I fill the form with: | Forum name | Test forum name | | Forum type | Standard forum for general use | | Description | Test forum description | | Subscription mode | Optional subscription | And I add a new discussion to "Test forum name" forum with: | Subject | Test post subject | | Message | Test post message |

Automated UI tests: Behat

• Improving from pure selenium implementation • No java :-) Human-friendly ‘Gherkin’ language • PHP integration code • Tests required to land with features by developers

• Culturally embedded Scenarios in 2.9

412

Page 19: Continuous Integration: Improving Quality in Moodle

the world’s open source learning platform

• Many pre-checker ideas

• Run phpunit/behat/multi-

platform tests in pre-checker

(split by component)

• Intelligent re-checking when

branches get updated

• Backup/restore checks

• Custom jira fields for prechecker

• Automated rejections

• Webservices testing

• Run all post-check’s in pre-checker

• Others as we detect them

Future

Page 20: Continuous Integration: Improving Quality in Moodle

Questions?

@dan_p

the world’s open source learning platform

https://github.com/moodlehq/moodle-local_ci


Recommended