Data-Driven Software Development -The quest for the intelligent machine that manages software projects

freiheit.com technologies - Hamburg, September 29, 2015code.talks 2015

© freiheit.com 2

Thanks for coming!WHO AM I?

Me: Stefan Richter

● Dipl.-Ing. + Dipl.-Inf.

● First program: I983

● First language: UCSD Pascal

● First 15 years: Lots of C and 4GL languages

● Last 15+ years: Java (from 1.02), Python, Common Lisp, Clojure, JavaScript

● Current favorite: Go

My company: freiheit.com

● Founder: 1999

● Building large-scale Internet systems since 15+ years

● 90 % developers (degree in computer science, mathematics, physics)

● Self-organizing, self-optimizing, hierarchy-free organization

● Daimler, Tchibo, E&V, METRO, SPIEGEL, Libri, Swiss Life, 1&1/WEB.DE etc.

© freiheit.com GOTO Conference Copenhagen, 2011 3

!

DATA-DRIVEN SOFTWARE DEVELOPMENT

THE QUEST FOR THE INTELLIGENT MACHINE THAT MANAGES SOFTWARE PROJECTS

Since 2003, we have been experimenting with systems that support developers in their decision making.

5© freiheit.com

HISTORY: MAKE PROJECT MANAGEMENT MORE INTELLIGENT

TODAY I WANT TO SHOW YOU

HOW WE HAVE BUILT AN INTELLIGENT AGENT TO HELP US RUN OUR SOFTWARE PROJECTS

© freiheit.com 7

© freiheit.com 8

BUT WE ARE AGILE!WE DON’T NEED PROJECT MANAGEMENT!

WHY IS BUILDING SOFTWARE IN LARGER TEAMS STILL DIFFICULT?

Too many steps, artefacts, roles.Lot’s of communications needed.

11

WATERFALL PROCESS

I think you have seen the short-cut, too ...

12© freiheit.com

AGILE PROCESSES

Product Owner

Value? Code!SoftwareStories

Developer Conclusion: It is not enough to implement the Scrum method. You need to change the whole way how an organization works.

If structures are getting larger and complicated, things become less easy ...

13© freiheit.com

LARGER ORGANIZATION, LARGER TEAMS

Product Owner Developer

Developer

Larger OrganizationSmall Team Larger Team

n x (n-1)5 x 4 = 20

Com

mun

icai

ton Com

municaiton

Manager

Manager

WE NEED TO GET

THE RIGHT INFORMATIONTO THE RIGHT PEOPLEAT THE RIGHT TIME

© freiheit.com 15

Every project looks like this: You have to deliver the highest value to the lowest costs.TIME VS. VALUE

Capacity (highest value to the lowest costs)

Time = Money

Deve

lope

rs

Start Today End

Past

Growth

Future

Shrinks

Given the same quality, faster is always better.

Power = Force x DistanceTime

NOT

Force x Distance x Time

No matter if you measure in points or ideal days!

16© freiheit.com

FASTER IS BETTER: ABOUT MEASURING TIME AND PROGRESS

PROJECT MANAGEMENT

ARE WE ON TRACK?IS THERE ANYTHING SLOWING US DOWN?

“ADVENTURE IS JUST BAD PLANNING.”

- ROALD AMUNDSEN (First to reach the south pole on 14 December 1911)

A PLAN IS AN ASSUMPTION ...

… AND PLANNING IS TO CHECK YOUR ASSUMP-TIONS AND ADAPT THEM EVERY DAY BASED ON YOUR PROGRESS.

ARE WE ON TRACK?- What is the initial plan?- Expected delivery? Feature set, delivery date, customer value?- Re-implementing existing systems or how to sort 5000+ stories into a plan.- Capacity checks, forecasts: We have a dedicated team for that.- Forecasts are difficult no matter if you have 5 or 500 people working.- Is our plan still correct? (80/20 rule of uncertainty)- Having project managers running a software project. We don’t like that.- Doing meetings to keep everybody on the same page?- Having more meetings whenever there is a problem?- Everybody is buying Jira, but does this help? (“A fool with a tool is still a fool.”)

ARE WE WORKING ON THE RIGHT THINGS?

- MoSCoW prioritization method (MUST, SHOULD, COULD, WON’T)- implementation order- transparency without meetings (reporting, using the “Facebook effect”)- Blackboards versus agile project systems- Who is working on what? What is done, not done?- Is everybody working on the planned stories? (Anecdotal evidence)- The feeling of being part of a larger team

ARE THERE ANY PROBLEMS?

- Are we spending an unusual amount of time somewhere?- Are new developers getting up to speed?- External factors? Systematic under/over estimations?- Who needs support? Who is stuck somewhere?- Is “done” really done? Getting feedback from users/stakeholders.- Where are changes needed? Follow-up stories?- What will that do to our plan?

WHAT IF AN INTELLIGENT AGENT COULD DO THIS FOR YOU?

Let’s talk about agent-based systems ...

25© freiheit.com

… or maybe like this.

26© freiheit.com

INTELLIGENT AGENT - DEFINITION:

In artificial intelligence, an intelligent agent (IA) is an autonomous entity which observes through sensors and acts upon an environment using actuators and directs its activity towards achieving goals (i.e. it is "rational", as defined in economics).*

*Source WIKIPEDIA

© freiheit.com 28

Simple reflex, model-based, goal-based, utility-based and general learning agents. INTELLIGENT AGENT

© freiheit.com 29

Siri, Google Now, and Facebook bringing general purpose intelligent agents in place.

© freiheit.com 30

WE ARE COMMUNICATING MORE AND MORE VIA INSTANT MESSAGING SYSTEMS LIKE WHATSAPP, SLACK, WECHAT, FACEBOOK etc.

© freiheit.com 31

32© freiheit.com

SO HERE ISWHAT WE HAVE BUILT ...

© freiheit.com 34

© freiheit.com 35

USING GAME AI TECHNIQUES

● Target date

● Estimation

● Effort

● What is STARTED?

● What is DELIVERED

● What is ACCEPTED?

© freiheit.com 37

Before we can learn from behavior, we have to get the basics right.

Agent

State

Goals

Utility

What is the world right now?

What will it be likeif I do action x?

How happy I will be in such a state?

WorldPercepts EnvironmentSensors

Actuators

⎬ Stories

⎬ Progress

Action

© freiheit.com 38

Goal-oriented action planning is a technique used in building NPCs for Game AI.STATE-MACHINES VS. GOAP

Source: http://gamedevelopment.tutsplus.com/tutorials/goal-oriented-action-planning-for-a-smarter-ai--cms-20793

State machine Loosely coupled goals/action

© freiheit.com 39

To reach a goal, we can select the best next action that reduces high insistence values.GOAL-ORIENTED ACTION PLANNING, INSISTENCE

Goal:Eat=4, Bathroom=3Action:Drink-Soda (Eat-2, Bathroom+3)Action:Visit-Bathroom (Bathroom-4)

Source: Artificial Intelligence for Games

© freiheit.com 40

If we want to consider side-effects of an action we have to calculate the discontentment value.GOAL-ORIENTED ACTION PLANNING, DISCONTENTMENT

Goal:Eat=4, Bathroom=3Action:Drink-Soda (Eat-2, Bathroom+2)- afterwards: Eat=2, Bathroom=5:

Discontentment=29Action:Visit-Bathroom (Bathroom-4)- afterwards: Eat=4, Bathroom=0,

Discontentment=16Source: Artificial Intelligence for Games

HYPOTHESIS

WE CAN MODEL OUR PROJECT WORLD USING THE SAME METHODS.

All aspects of a project are monitored by specialized software bots. These bots are working in swarms.

Slack

Sensors

Actuators

Space Planner/Bots

Slack

API

Agile Planning Tool GOAP AIAgent Framework

Swarm

Swarm

Realtime Databases,

other Sources

© freiheit.com 42

Super Swarm

GOETHE ARCHITECTURE

Sensors● Stories (“Plan”)● Progress (“Done”)● Commits (Code)● Bugs● Calendar● Android/Apple Watch

Actuators● Messengers

(Slack etc.)● IRC, XMPP● Apple/Android Watch● Text-2-Speech

© freiheit.com 43

Bots● Personal Bot● Sprint/Milestone● Project● Anomaly Detection● Pattern Recognition● Neural Networks● Planning● Code Quality

© freiheit.com 44

The actions are using Go text/templates and a homegrown DSL to access the data.DSL EXAMPLES

All Stories from the current milestone

All Stories in the past that are not done

DEMO VIDEOS

© freiheit.com 46

© freiheit.com 47

SIGN-UP HERE FOR OUR CLOSED BETA:http://goethe.instapage.com/

THANK YOUTwitter: @smartrevolutionfreiheit.com technologies

Budapester Str. 45

20359 Hamburg, Germany

kontakt@freiheit.com

T +49 40 890584 0

F +49 40 890584 20

Hackers wanted.

Join us:jobs@freiheit.com