Brussels, 02. - 03.02.2013

The Development Infrastructure

of the

TYPO3 project

Lizenz: CC BY-SA 3.0

(pictures of other authors excluded)

1

Brussels, 02. - 03.02.2013

What to expect from this talk?

Learn to set up a perfect development infrastructure

See the perfect infrastructure of the TYPO3 project

Learn how people in another FLOSS project work

See an infrastructure built from 100% open-source

Find inspiration for your project

Get in touch with me/us for collaboration

✔

✘

✘

✔

??

admin@typo3.org - we’re happy to share!

2

Brussels, 02. - 03.02.2013

Steffen Gebert

• TYPO3 CMS Core Team Member(2010-2012)

• TYPO3 Server Admin Team Member(since 2011)

• PhD Student in Computer Science(Communication Networks, since 2011)

• E-Mail: steffen.gebert@typo3.orgTwitter: @StGebert

3

Brussels, 02. - 03.02.2013

Questions!

• Happy to answeryour questions!

• Come down hereafter the talk!

Foto “Question mark”: Yello-Doghttp://www.sxc.hu/photo/1279316

• Slides availablehttp://slideshare.net/stephenking/

4

comments included!

nice that you found the slides - have fun ☺

Brussels, 02. - 03.02.2013

What’s TYPO3?

• TYPO3 CMS

• Started: 1997, first release: 1999

• More than 5M downloads

• PHP-based, GPLv2

• New products

• TYPO3 Flow: PHP framework

• “TYPO3 Flow and the Joy of Development” by Robert Lemke from yesterday

• TYPO3 Neos

• New CMS based on TYPO3 Flow - alpha state

5

Brussels, 02. - 03.02.2013

Who’s TYPO3?

• TYPO3 is community-driven open-source

• no company behind, TYPO3 Association for financial support

• Core teams steering development (99.x% volunteer work)

• CMS with ~25ppl., Flow/Neos with ~12ppl.

6

Photo: Tobias Liebig

Brussels, 02. - 03.02.2013

Requirements of an

Open Source Project

7

Photo by brokenarts:http://www.sxc.hu/photo/207291

Brussels, 02. - 03.02.2013

Requirements

• Collaboration & Communication

• Source Code & Versioning

• Quality Assurance & Testing

• Documentation & Translation

8

Photo by brokenarts:http://www.sxc.hu/photo/207291

There are couple of temas working on tasks other than development- Design, Usability, Marketing, Screencast, Event teams- Teams and committees of the TYPO3 Association (Board, Expert Advisory Board, Budget Control Committee)

Brussels, 02. - 03.02.2013

Using own infrastructure

• We’ve been there before Github ☺

• Sourceforge.net initially used for CVS

• Got hacked, often unreliable

• Owning our data (esp. issue tracker)

• Own account management

• Provide central place and connected services for the TYPO3 community

• Nevertheless, open towards great, free hosted services for less critical things

• Human resources are a problem, of course

9

Brussels, 02. - 03.02.2013

Communication and Collaboration

10

Image by fangol:http://www.sxc.hu/photo/1269811

Brussels, 02. - 03.02.2013

Communication in the TYPO3 Project

• Mailing lists and news groups used since 2003

• Ten years later (last Monday):

11

forum.typo3.org

Brussels, 02. - 03.02.2013

Forum and Official Communication

• Forum

• Based on FUDforum

• Bi-directional sync with news groups

• Easier access to official channels (we hope..)

• Decisions take place in lists

• Transparency towards more people

• Avoid hidden communication

• Avoid surprises (well, we still fail too often)

12

Image by svilen001:http://www.sxc.hu/photo/1165445

Brussels, 02. - 03.02.2013

Forge Platform

• Based on Redmine

• Project space for 2400+ projects

• Core projects, Community Extensions, Other teams

• Features

• Membership listing

• Issue tracker

• Wiki, Repo Browser, RoadmapNews, Time Tracking

13

Want to contribute?

➪ See who’s responsible

not the best one ☹

forge.typo3.org

It’s often a bit chaotic in a big open source community. Being able to find the persons in charge is important!Forge is the place where to search.

We know that we’re running a very outdated redmine version. We’re sorry and ashamed. We’re working on an upgrade.

Brussels, 02. - 03.02.2013

Source Code and Versioning

14

Source:http://nvie.com/posts/a-successful-git-branching-model/

Brussels, 02. - 03.02.2013

Source Code Versioning

• TYPO3 is using Git since 2010/11

• .. but we use a special flavor of Git

• Unfortunately still lot of extensions restricted to subversion

15

Yes.. it’s true. The high Subversion usage is our (=admin team) fault.. we still haven’t launched official git support for community extensions. But it’s near.

Brussels, 02. - 03.02.2013

Quality Assurance and Testing

16

Photo by bredmakers:http://www.sxc.hu/photo/1280927

Brussels, 02. - 03.02.2013

Motivation for Code Reviews 17

• Bugs do exist

• Ensure code quality through peer-reviews (4 eyes in our case)

• Discussing code improves it

• Reviews before merge

• Learn from each other

• Make development transparent, involves community

• Using Gerrit Code Review as Git server and review tool

Photo by asifthebes:http://www.sxc.hu/photo/912841

every change has to be reviewed by at least two other persons, one of them being team member

Brussels, 02. - 03.02.2013

Gerrit Code Review

review.typo3.org

18

All code-changes for core projects are peer-reviewed by at least two people (one of them team member)(Disclaimer: except trivial changes)

Brussels, 02. - 03.02.2013

Gerrit Code Review 19

Brussels, 02. - 03.02.2013

Gerrit Code Review

• Diff view

20

review.typo3.org

Brussels, 02. - 03.02.2013

More About Gerrit

Interested in more Gerrit details?

“Using Gerrit Code Review in an open-source project”

by myself :-)

Here at 15:40

21

Brussels, 02. - 03.02.2013

Continuous Integration and Code Quality

• ci.typo3.org runs our Jenkins server

• PHP lint (syntax check)

• PHP CodeSniffer (enforcement of Coding Guide Lines)

• PHP Mess detector

• Automatic down-vote in Gerrit in case of violations

• metrics.typo3.org runs Sonar

• Code Metrics

• For Core and Extensions, 5600+ projects

22

Brussels, 02. - 03.02.2013

TravisCI - More Continuous Integration

• TravisCI is

»a hosted continuous integration service for the open source community«

• What it does? It executes tests!

• Fetches your code from Github

• Supported languages: C++, Clojure, Erlang, Go, Groovy, Haskell, Java, JavaScript (with Node.js), Perl, PHP, Python, Ruby, Scala

23

To execute post-merge unit tests, we are replicating our Git repo to Github, from which Travis fetches it.In order to let Travis execute them for every patch set, we are currently building a tool to push and fetch Travis’ results and add them to Gerrit.

Brussels, 02. - 03.02.2013

TravisCI

• Spins up virtual machines (using Vagrant and Chef)

• Runs your unit tests in different an environments

• Example:

24

PHP 5.2 PHP 5.3 PHP 5.4

MySQL

PostgreSQL

✔

✘

✔ ✔

✔ ✔

php:    -­‐  5.2    -­‐  5.3    -­‐  5.4

env:    -­‐  DB=mysql    -­‐  DB=pgsql

Brussels, 02. - 03.02.2013

Documentation and Translation

25

Photo by topfer:http://www.sxc.hu/photo/969875

Photo by ftibor:http://www.sxc.hu/photo/533821

Brussels, 02. - 03.02.2013

Translating TYPO3

• Pootle-based

• Core + Extensions can be translated

• Language packages generated, provided for download

26

translation.typo3.org Photo by ftibor:http://www.sxc.hu/photo/533821

Brussels, 02. - 03.02.2013

Documentation

• Was a raw point for long time

• Used OpenOffice 1.x .sxw files since ages

• Migration to DocBook

• started / planned, but format turned out to be too complicated

• Migration to reStructuredText (reST)

• Started ~2011

• Milestone reached with TYPO3 6.0 (Oct. 2012): docs.typo3.org

• TYPO3 CMS Core documentation converted to reST

27

Photo by topfer:http://www.sxc.hu/photo/969875

- During .sxw times, we used a wiki page to write down pending documentation changes, which sucked

- docbook format is based on xml, pretty heavyweight. Probably because of markdown, people feel better with plain-text?

Brussels, 02. - 03.02.2013

reStructuredText

• Plain-text format, similar to markdown

• Parsed and rendered through sphinx

• Rendering chain at TYPO3

• Managed through TYPO3 Flow app

• Can import old .sxw files

• You can view sources on docs.typo3.org!

• Changes reviewed through Gerrit

created fordocs.python.org

28

lower barrier for adoption

Documentation is parsed and rendered through the tools docutils and sphinx (both created for docs.python.org).http://docutils.sourceforge.net/rst.htmlIntersphinx allows cross-manual referencing

Being able to import old manuals hopefully lowers the barrier to migrate documentation from sxw to reST

Brussels, 02. - 03.02.2013

Solutions at a Glance

• Collaboration & Communication

• Mailing lists / NNTP / Forum using FUDforum, Redmine

• Source Code & Versioning

• Git with Gerrit as server and review tool

• Quality Assurance and Testing

• Gerrit, Jenkins with couple PHP tools, TravisCI

• Documentation & Translation

• reStructuredText, sphinx, Pootle

29

Honestly, we use couple of more tools:

- mailman runs our mailing lists.. what else (ah, we have a sympa server vor TYPO3 Assoc member lists, but I wouldn’t recommend that software)- etherpad-lite for meeting notes and collaborative writing: http://notes.typo3.org- BigBlueButton web conferencing server: http://bigbluebutton.typo3.org- and we have been using Mantis as bug tracker for years. Mantis (http://bugs.typo3.org) is now archived and Redmine (http://forge.typo3.org) acts as tracker for *all* TYPO3 projects.

- yes, we are still running a subversion server for some thousand projects-

Brussels, 02. - 03.02.2013

Questions!

• Happy to answeryour questions!

• @StGebert on Twitter

• steffen.gebert@typo3.org

Foto “Question mark”: Yello-Doghttp://www.sxc.hu/photo/1279316

• Slides availablehttp://slideshare.net/stephenking/

30

comments included!

inspiring people to share

inspiring people to share