Defending the Bird

Product Security Engineering at Twitter

Alex Smolen (@alsmola)

Justin Collins (@presidentbeef)

YAC, Moscow, 2013

What does it mean to “Defend the Bird”?

500+ million Tweets a day

Hyper-growth

2000+ employees around the world

200+ million daily active users

Twitter as the global town square.

3 floors~700 employees

1 floor~100 employees

5+ floors~2000+ employees

https://twitter.com

https://twitter.com

https://mobile.twitter.com

https://twitter.com

https://ads.twitter.com

https://mobile.twitter.com

https://twitter.com

https://ads.twitter.com

https://dev.twitter.com

https://mobile.twitter.com

https://twitter.com

https://ads.twitter.com

https://dev.twitter.com

https://support.twitter.com

https://mobile.twitter.com

https://twitter.com

https://ads.twitter.com

https://dev.twitter.com

https://support.twitter.com

https://analytics.twitter.com

https://mobile.twitter.com

https://twitter.com

https://ads.twitter.com

https://dev.twitter.com

https://support.twitter.com

https://analytics.twitter.com

https://mobile.twitter.com

https://translate.twitter.com

https://twitter.com

https://ads.twitter.com

https://dev.twitter.com

https://support.twitter.com

https://analytics.twitter.com

https://mobile.twitter.com

https://translate.twitter.com

https://bluefin.com

https://twitter.com

https://ads.twitter.com

https://dev.twitter.com

https://support.twitter.com

https://analytics.twitter.com

https://mobile.twitter.com

https://translate.twitter.com

https://bluefin.com

https://admob.com

https://twitter.com

https://ads.twitter.com

https://dev.twitter.com

https://support.twitter.com

https://analytics.twitter.com

https://mobile.twitter.com

https://translate.twitter.com

https://bluefin.com

https://admob.com

https://posterous.com

https://twitter.com

https://ads.twitter.com

https://dev.twitter.com

https://support.twitter.com

https://analytics.twitter.com

https://mobile.twitter.com

https://translate.twitter.com

https://bluefin.com

https://admob.com

https://posterous.com

https://crashalytics.com

https://twitter.com

https://ads.twitter.com

https://dev.twitter.com

https://support.twitter.com

https://analytics.twitter.com

https://mobile.twitter.com

https://translate.twitter.com

https://bluefin.com

https://admob.com

https://posterous.com

https://crashalytics.com

https://vine.co

https://twitter.com

https://ads.twitter.com

https://dev.twitter.com

https://support.twitter.com

https://analytics.twitter.com

https://mobile.twitter.com

https://translate.twitter.com

https://bluefin.com

https://admob.com

https://posterous.com

https://crashalytics.com

https://vine.co

https://twitter.com

https://ads.twitter.com

https://dev.twitter.com

https://support.twitter.com

https://analytics.twitter.com

https://mobile.twitter.com

https://translate.twitter.com

https://bluefin.com

https://admob.com

https://posterous.com

https://crashalytics.com

https://vine.co

https://tweetdeck.com

https://twitter.com

https://ads.twitter.com

https://dev.twitter.com

https://support.twitter.com

https://analytics.twitter.com

https://mobile.twitter.com

https://translate.twitter.com

https://bluefin.com

https://admob.com

https://posterous.com

https://crashalytics.com

https://vine.co

https://tweetdeck.com

https://trendrr.com

https://twitter.com

https://ads.twitter.com

https://dev.twitter.com

https://support.twitter.com

https://analytics.twitter.com

https://mobile.twitter.com

https://translate.twitter.com

https://bluefin.com

https://admob.com

https://posterous.com

https://crashalytics.com

https://vine.co

https://tweetdeck.com

https://trendrr.com

We are 1 out of 100 engineers.

We can’t do everything.

Automation

Code review

Security features

Automating Security

Avoid tedious tasks

Catch issues early

Notify right people

We need a central location where information is collected and transferred.

Static analysisDynamic analysisInternal metrics

How do we let developers know when they check in bad code?

Brakeman

Static analysis for Rails

Needs infrastructure for integration

Reports to SADB

Coffee Break

Javascript static analysis

Catch DOM-based XSS

Reports to SADB

Phantom Gang

Dynamic HTTP scanning

Specific, not full scan

Reports to SADB

We manually review what slips through the cracks.

Code Reviews

Code goes through a review system

Security is automatically added to sensitive reviews

Security can be manually added to any review

Accountability

Email when there are new reviews

Dashboard of pending reviews

Once a month clean sweep

Teams request security reviews through a self-service form.

Security features

Two-factor authentication

Something we’ve wanted to build for a long time

Designed and implemented by the product security team

How do you build a robust yet simple solution?

SMS-based two-factor

Send a six digit code the user

Requires a temporary password to sign in to other apps and devices

Native two-factor

Client has a private/public keypair

Signs request sent by server over push, which has public key

One-tap sign in

Two-factor challenges

Happy case is easy, sad case is hard

Doesn’t deal with many-to-many account access

People can’t manage their own keys

Twitter was one of the first major

services to require 100% SSL.

HTTP Strict Transport Security

How do you bootstrap?

Tells browser not to use HTTP

Sub-domains, CDNs, mobile

Certificate pinning

Implemented in mobile apps, Chrome

Only one certificate is valid

Also working on TACK

ECDHE

SSL mode with perfect forward secrecy

Ephemeral keys used for conversations

We need to build security in to our custom frameworks.

Security headers

Adds several default security headers

Implements interoperable CSP

https://github.com/twitter/secureheaders

Keybird

Keys delivered securely to production environment

Uses puppet

The bird is big, and we’re small.

We use tools to accomplish more.