Strategies and Tips for Building Enterprise Drupal Applications - PNWDS 2013

Strategies and Tips for Building Enterprise Drupal Applications

Mack Hardy, Dave Tarc, Damien Norris

October 5th, 2013

Pacific Northwest Drupal Summit

Treat it like a Product

http://affinitybridge.com

● Fixed functionality per release

● Repeatable deployment

● Documentation (process, not just end-user).

● QA and Testing (automated and manual)

Application Components


What do we mean by Enterprise?


● Core to the operations of an organization● Always changing - even post launch● Complex business rules● Reports● Data Exports● Audit Trails● More Secure● More Available

Community KnowledgeKeeper

Community KnowledgeKeeper

11 Instances in production

160+ core & contrib modules enabled on larger installs

40+ custom features enabled on larger installs

27 content types

12 taxonomy vocabularies

10 custom user roles

Release ManagementKeeping the cats herded


Version control and releases

● Git for the profile and custom code● Drush make for fetching

Core+Contrib+Patches● Branch-per-Feature development workflow● Travis-CI and manual testing on stage server● Careful of dev vs. stage/prod differences (e.g.

memory management OSX vs. Linux)


Semantic Versioning

Use Semantic Versioning (http://semver.org)

##.##.##

major #.minor #.point #

Update major # when public API changes

Update minor # on deploys of new functionality (weekly or as needed)

Update point # for hotfixes

Show version to privileged users on every page

● Tag master when a release is ready and deploy it.

● Distribution is profile managed in a make file, allows for a rebuild on deploy

● Features are reverted on deploy

● Git Branch Per Feature - See session Sunday Oct 6th, 11:40 “Git Branch-per-Feature”

Deployment Process


● Behat / Mink / Travis CI for automated testing● Tests run on merge to develop branch● Test results integrated to chat client, so devs

see the failures● Limited test coverage, build a test for each

new function● Smith’s presentation “Behaviour Testing and

Continuous Integration with Drupal” Sunday Oct 6th, 3pm

Automated Testing


BuildingNow, lets make Drupal play nice


Choose a small group of Drupal modules to build around.

Choose wisely, becauseYou will have to commit to them.

You will be getting heavy use out of them, and

pushing their boundaries.

Drupal Base Modules

Features

Ctools

Views

Display Suite

Field Group

Renderable Elements

Context

Omega

Token

Libraries

Entity API

Entity Reference

Organic Groups

Date

Search API

Facet API

Media

Geofield

Building with Features

Features enabled functionality to be grouped together and enabled/disabled.

Features module is almost required in order to build distros.

Features module does have some gaps and will provide some headaches.

Features can cascade (depend on each other), which allows for rich options if done well.

Cascading Features with Field Group

Feature A is required.

Feature B is optional, and requires Feature A.

Feature A implements content type Thing.

In Order for Feature B to safely put fields on Thing (edit form and full view):

- Fields must be placed in field groups

- Fields and field groups exported in Feature B

- Display Suite settings exported in Feature A

Display Suite withRenderable Elements

Display Suite allows for configuration of layouts on content types (as well as other entities)

Renderable Elements allows any type of form to be registered & subsequently configured using Display Suite

With Omega providing responsive design, this allows for a very consistent look across pages, nodes & edit forms.

Customized options across installs

Use taxonomy term reference fields if fields need to have different options across installs.

Taxonomy terms are full entities, fieldable and customizable with Display Suite, and all per vocabulary.

Depending on the feature, adding a term reference to a term in a parent vocabulary can often work a lot better than using a hierarchical vocabulary

Search API & Facet API

Using Solr as a backend to return search results, Search API and Facet API can be used to build extremely customized search pages.

Once search results are returned, it is easy to do other things with them, such as export them in a CSV or KML

Improve the distro by leveraging use of Search API. Building single-use or inflexible reports with views does not do much for the distro.

Pain Points - Role rid’s

Never refer to roles in Features (beyond the exported role itself.

All (?) role-based operations work on role rid, which can be different across sites.

Views access should never be configured by role.

User reference fields should never filter by role.

Use permissions instead.

Pain Points - Default Images

When features exports an image filefield with a default image, the file fid is what is exported.

File fid’s will be different across servers

Updating the file fid on each site will override the feature, and future deploys will wipe it

In order to have default images which will work across installs, a core patch is required.

Pain Points - Update Hooks

Update hooks work great for core and contrib modules that need to pass updates downstream to users

Update hooks are a poor choice for managed distros:

- DB Updates should be done before clearing cache or updating features

Pain Points - Feature Install Hooks

Feature install hooks are problematic in some cases:

- Creating taxonomy terms in a vocabulary exported by the feature will not work as the vocabulary will not exist when the hook is run

There might be better options in Features 2.x

Making updates - Drush scripts

Drush scripts which use EntityFieldQuery and Entity API for loading, modifying, and saving content work great.

Drush scripts can be run multiple times. No need to fiddle around with database values to get an update to run again.

Drush eval calls also very powerful:drush eval ‘my_function($nid);’

Making changes

If you really have to, it may be necessary to change a core component.

For us, OpenLayers was getting more cumbersome to use as the system got bigger.

We decided to update to Leaflet.

Large component changes like this can be time-consuming, messy and a distraction from client desires.

Use the Drupal API

Be wary of adding new modules that offer easy solutions to problems that can be done easily using the core hooks.

These helper modules will always have a cost. Sometimes that cost is worth it, but be wary of adding stuff you really don’t need.

Rules

Rules offers great automatic functionality that non-coders can configure. It’s great for some types of Drupal distros.

If your distro is making extensive use of the core hooks, logic implemented via rules can be more challenging to debug.

Rules is just implementing hooks that could be implemented directly.

Field Permissions

Field Permissions module allows you to assign view and edit permissions for each field.

Hiding fields on node form and node view ends up being pretty easy and clean to do in code.

hook_form_alter() {$form[<FIELD_NAME>][‘#access’] = 0;

hook_node_view() {unset($node->content[<FIELD_NAME>]);

Editable Content

Editablefields module works well in some situations, but is delicate.

Edit module works very well in a variety of cases but still may not solve the problem.

Custom forms built using core Form API put into Display Suite code fields end up working pretty well

Custom Hooks

Create custom hooks within your distro for greater power

module_invoke_all($hook)

We defined custom hooks for:

- Site Settings page

- Site wide Search API index

Features could implement these hooks to include variables on the settings page or fields for search indexing

Keep Things Impeccable

Use consideration when setting up filefield paths (consider filefield paths module)

Adopt a standardized naming system

Adopt a standard module dir layout for include files and drush scripts

Categorize your features

Caching and Performancewe built it, but now they want to go fast too, with scads of data!


Caching and performance

on a single Debian Linux server using:

● PHP under FastCGI● APC: PHP Op-code cache ● Memcached: RAM Caching for Drupal● XSendfile: for private file serving● MySQL tuning with mysqltuner● Varnish 3 front-side cache

Moar? This stack can and should be clustered.


PHP with FastCGI

Default php5 packages for Linux run using the Apache module, mod_php. This is problematic:

● All requests (even ones for CSS, Images, Text files, anything not .php) are handled by an Apache process with PHP loaded in its memory. This is an inefficient use of memory and in our experience slows things down.

● PHP is run as the Apache user (apache2 or www-data). That user is usually different from the user who owns the files.


FastCGI - just like it sounds

CGI: The original, old school web application spec allows a webserver to hand off execution to another program. Environment variables hold values to be passed between the two. Still used!

FastCGI runs a pool of ready-to-go PHP processes and Apache just hands off any requests for a .php to it. It’s the old fashioned way but is really really fast!


FastCGI Other Benefits

● Using SuExec, PHP runs as the correct user! No more permissions problems or chown scripts!

● Easy to compile different versions of PHP and run them side by side on the same server.

For Apache < 2.4, use mod_fcgid for FastCGI.

For Apache 2.4+, try PHP-FPM, an improved implementation.


APC: Op-code Cache

● PHP files are source code and therefore interpreted by default.

So, each time the .php file is loaded for a page view, the server has to compile the source code on the fly and this is repeated over and over.


APC: Op-code Cache

APC stores the compiled version in a cache, greatly speeding up PHP. Instant speed up.

● It can also be used for Drupal caching, or as a session handler, but don’t.

● It can be buggy with certain version combinations. (Debian 6 / Ubuntu 12.04.LTS with PHP 5.3. Try leaving apc.shm_size=32 if you are having problems).

● It was once the official cache of choice, and going to be included in PHP 5.4. Now it looks like Zend OpCache (nee Optimizer) is the new darling.


Memcached : Drupal RAM cache

● Stores Drupal Cache tables in memory● makes sites Extremely fast ● not just for Anonymous users - speeds up

logged in users too because Drupal does lots of caching all over the place.

● Redis also works great for this (and is persistent to disk if reloading cache is expensive)


Memcached : HOWTO (1)

● Install “memcached” server (not “memcachedb” server)

● Get the PHP beta 3.0.8 “memcache” PECL extension (not the memcached PECL extension) … and make sure it shows up in a phpinfo(); output.


Memcached : HOWTO (2)

● Install and enable the memcache Drupal module.

● Add these magic beans to settings.php to actually turn it on:

/* Memcache */

$conf['cache_backends'][] =

'sites/all/modules/memcache/memcache.inc';

$conf['cache_default_class'] = 'MemCacheDrupal';

$conf['cache_class_cache_form'] = 'DrupalDatabaseCache';

$conf['memcache_key_prefix'] =

sha1(serialize($databases['default']['default']));


XSendFile for private files

● Private files must be served using PHP, so that Drupal can determine it’s allowed to serve them up. But for larger files, this is extremely hard on your server.

XSendFile (ported over from LightHTTPd) is a clever way to sidestep this issue using just an Apache module and a HTTP header:

XSendFile - How it works

Instead of actually opening and outputting the file, PHP does everything right up to and including outputting any content.

It outputs a full set of HTTP headers (including a valid Content-length).

It adds a header XSendFile: with the path to the file it wants to serve:

XSendFile: /home/demo/private/private.pdf

XSendFile - How it works (2)

Then, right when it’s time to open and output the file’s contents, PHP just exits abruptly instead, returning no further output after the HTTP headers.

As the (empty) page is returned by Apache, the XSendFile module inspects the HTTP headers and it sees XSendFile: <path> and serves the file’s content with Apache, using the correct content-type of the file.

PHP is now only involved for a split second and you’re back to the performance you expect and love with public files. Have your cake and eat it too! Or at least... just stop cake spraying all over the walls, furniture, and computers every time someone downloads a file.

XSendFile - Howto

● Download, compile, and enable the XSendFile apache module from the source code. You will need Apache development packages installed.

● Place the EXACT path to your private files directory in the Apache VirtualHost file:

# XSendFile provides fast private file xfer using xsendfile Drupal module.

<IfModule mod_xsendfile.c>

XSendFile on

XSendFilePath /home/mysite/private_files

</IfModule>

● Install and enable the Drupal xsendfile module. ● Troubleshoot by looking in your Apache error log. The problem is

always that the XSendFilePath is wrong, and it will print the one it’s trying to use in the error log so you can fix it and restart Apache.

MySQL tuning with MySQLTuner

Default MySQL settings are pretty gutless.MySQL will adjust to its tiny cell, but you can configure it to have way more memory to play with and that will make it go faster.

mysqltuner is an awesome command-line utility for tuning MySQL. It will run and analyze your current database and tell you exactly what settings to increase to improve performance.

Beware though, the more resources you give MySQL, the slower your server will boot up at first. On CKK, the CPU load spikes to 15 or 20 after a boot and stays there for about 5 minutes till things settle down. Mysqltuner will keep urging you to turn it up until you run out of juice, so use moderation when following its advice - don’t give it all the RAM.


Varnish 3 - Front Side HTTP Cache


Finally, Varnish is a HTTP Accelerator / Proxy server that can sit in front of your Apache and serve up entire cached pages from RAM.

● It’s mind-bendingly fast and can handle huge traffic spikes without blinking…. provided the entire site can fit in RAM and all your traffic are Anonymous visitors.

● Varnish takes over answering web requests on port 80, and fetches back-end content (when required) from Apache (either on another port or another IP). e.g. Apache on localhost port 8080, Varnish on public network, port 80.

Varnish 3


● Unfortunately, it cannot cache content for logged in users, but it can still serve them static files like Images, CSS and Javascript. So if you have large numbers of users it is probably still worth it. if you only have a few users, then Apache serving those files isn’t your bottleneck.

● Varnish is configured using a compiled language called VCL. It’s pretty confusing at first. We use the FourKitchens Drupal config for varnish and recommend you start with that.

● The Drupal varnish module allows Drupal to clear the varnish cache and allows for integration with cache expiry modules. (Check the Boost module page for a great list of options). The cache must be purged when content changes so that people see the latest and greatest. (Not seeing changes appear is a leading cause of irate content editors..)


Questions?

Date post:	30-Jun-2015
Category:	Technology
Upload:	mack-hardy
View:	7,201 times
Download:	1 times

Strategies and Tips for Building Enterprise Drupal Applications - PNWDS 2013

Technology