Building and Deploying MediaSalsa, a drupal-based DAM as a Service

transcript

Building and Deploying MediasalsaBuilding and Deploying MediasalsaA Drupal Digital Asset Management system as a ServiceA Drupal Digital Asset Management system as a Service

Julien PivottoJulien Pivotto

DrupalCon PragueDrupalCon PragueSeptember 26, 2013September 26, 2013

MediaMosa

• Drupal-based Digital Asset Management systemDrupal-based Digital Asset Management system• Open-sourceOpen-source• Store assetsStore assets• Trancode videosTrancode videos• Create, extract and manage metadata using open standards:Create, extract and manage metadata using open standards:

Dublin Core, Qualified DC, IEEE/LOM, CZPDublin Core, Qualified DC, IEEE/LOM, CZP• Webservice orientedWebservice oriented

Julien Pivotto Building and Deploying Mediasalsa

MediaSalsa

==MediaMosa as a ServiceMediaMosa as a Service

MediaSalsa infrastructure (simplified)

• Backend: Core service (MediaMosa)Backend: Core service (MediaMosa)• Frontends: OptionalFrontends: Optional• Web serversWeb servers• Database serverDatabase server• Solr serverSolr server• Transcoding serversTranscoding servers

for each environmentfor each environment

CultureAutomationMeasurement

Sharing

Inuits

• Inuits is an Open-Source companyInuits is an Open-Source company⇒ We contribute backWe contribute back• ± 40 people in 3 countries± 40 people in 3 countries• One language: EnglishOne language: English

Inuits

What do we do?

• Consulting and Internal projectsConsulting and Internal projects• Development and System administrationDevelopment and System administration

Inuits

Ideal world vs budget and realityIdeal world vs budget and reality⇒ pragmatic approach⇒ pragmatic approach

Distributed team

Communication = hell

How to fix it?

• Daily virtual stand-up over XMPP/HangoutDaily virtual stand-up over XMPP/Hangout• Redmine project managmentRedmine project managment

• TasksTasks• RepositoriesRepositories• Documentation (wiki)Documentation (wiki)

• Internal mailing listsInternal mailing lists• Jenkins notifications by mail and XMPPJenkins notifications by mail and XMPP• Internal trainingInternal training

Dev and Ops

DevDev• Develop new featuresDevelop new features• Write testsWrite tests

OpsOps• Infrastructure (puppet)Infrastructure (puppet)• CD and CI (jenkins)CD and CI (jenkins)

Ops teach dev to think about monitoring and distributed serviceOps teach dev to think about monitoring and distributed serviceDev teach to ops about required libraries and testsDev teach to ops about required libraries and tests

Culture

AutomationMeasurement

Sharing

Puppet

Puppet automates all the things

⇒ mcollective orchestrates all the things

Continuous Delivery vs Continuous DeploymentContinuous Delivery vs Continuous Deployment

• Puppet codePuppet code• Deployed to dev environmentDeployed to dev environment• Same puppet code for each environmentSame puppet code for each environment• User-triggered deployments to UAT and prodUser-triggered deployments to UAT and prod• Feature flags in Puppet code per environmentFeature flags in Puppet code per environment

• Application codeApplication code• Continuous integration in devContinuous integration in dev• User-triggered deployments to UATUser-triggered deployments to UAT• Deployment to prod is a business decisionDeployment to prod is a business decision

Developers test a lot

. . . The tests don’t workThe tests don’t work

. . . It works on my machineIt works on my machine

. . . Wrong platformWrong platform

. . . Wrong PHP versionWrong PHP version⇒ Fixed now, thanks to Jenkins!Fixed now, thanks to Jenkins!

Code is under revision control (git)Code is under revision control (git)• Prefer small commitsPrefer small commits• Local features branchesLocal features branches• PackagesPackages

Using OS packaging system

• Consistency, security, dependenciesConsistency, security, dependencies• Uniquely identify where files are coming fromUniquely identify where files are coming from• Source repo may not be reacheableSource repo may not be reacheable• Little overhead when you automateLittle overhead when you automate• Configuration does not belong to a packageConfiguration does not belong to a package

Pipelines

• A collection of jobsA collection of jobs• Run one after anotherRun one after another• Start on checkout, end on deploymentStart on checkout, end on deployment• From the developers’ side:From the developers’ side:⇒ Git pushGit push⇐ Mail with changes + link to deployMail with changes + link to deploy

Our pipelines

• PuppetizedPuppetized• Puppet code and Application code (backend and frontends)Puppet code and Application code (backend and frontends)• 1 pipeline/project/customer1 pipeline/project/customer

Defining a pipelinesalsajobs::pipeline::frontend {’qwerty-inc’:git_repository => ’ssh://git@redmine/qwerty.git’,dashboard_view => ’mds-frontends’,target_urls => {

’uat’ => ’http://qwerty.uat.mediasalsa.eu’,’prod’ => ’http://qwerty.mediasalsa.eu’,

},require => Salsajobs::Views::Dashboard[’mds-frontends’],

Pipelines steps

• CheckoutCheckout• Syntax: php -lSyntax: php -l• Style: Drupal CoderStyle: Drupal Coder• Package: FPMPackage: FPM• Deploy to dev environment: McoDeploy to dev environment: Mco• Tests in dev environment: drush run-testsTests in dev environment: drush run-tests• Publish package and promote: McoPublish package and promote: Mco

Promotion

• At the end of the pipelineAt the end of the pipeline• Send a simple emailSend a simple email

• A link to the promotion pageA link to the promotion page• The changelogThe changelog

• Promotion page contains one button per environmentPromotion page contains one button per environment• Must promote to UAT before ProductionMust promote to UAT before Production

Tools used with jenkins

• Pulp to manage RPM repositoriesPulp to manage RPM repositories• Mcollective to update packages, run drushMcollective to update packages, run drush

CultureAutomation

MeasurementSharing

Logstash

Collect all the logs

• Drupal logsDrupal logs• Apache logsApache logs• Deployment logsDeployment logs• System logsSystem logs⇒ Logstash, ES, kibana, statsd, graphiteLogstash, ES, kibana, statsd, graphite

Icinga

Monitor everything

• "Basics""Basics"+ vhostsvhosts+ databasesdatabases+ cronjobscronjobs

Graphite + gdash

• CollectdCollectd• Monitor platform usageMonitor platform usage• FFmpeg usageFFmpeg usage• Number of accounts (backend ID’s)Number of accounts (backend ID’s)

FFmpeg

Managed assets

Number of backend ID’s

CultureAutomationMeasurement

Sharing

We have just shared our experience. . .We have just shared our experience. . .

Any question?Any question?

Contact

Julien PivottoJulien Pivottojulien@inuits.eujulien@inuits.eu@roidelapluie@roidelapluie

INUITS bvbaINUITS bvbaDuboisstraat 50Duboisstraat 502060 Antwerp2060 AntwerpBelgiumBelgium+32 473 441 636+32 473 441 636https://inuits.euhttps://inuits.eu

THANK YOU!

WHAT DID YOU THINK?

Locate this session at the DrupalCon Prague website:http://prague2013.drupal.org/schedule

Click the “Take the survey” link

Building and Deploying MediaSalsa, a drupal-based DAM as a Service

Technology