Pieces of the API Puzzle: Leveraging Typed Data in Drupal 8
Matthew Radcliffe mradcliffe
@mattkineme
http://drupalcampatlanta.com/2015-drupalcamp-atlanta/sessions/pieces-api-puzzle-leveraging-typed-data-drupal-8
Composite Data Types• Any data type made up of primitive and other composite
data types i.e. structs, classes.
• Associative arrays
• Loosely-typed arrays to describe all of the things.
• Entity API Metadata Controllers
• Rules
Composite Data Types in Drupal 8
• Typed Data is a meta for composite data types that we need in Drupal that are not otherwise defined.
• Strongly typed interfaces that can be passed to various sub-systems in Drupal such as Serialization.
• Any class itself is composite data type. The Plugin API for instance is a way of describing composite data types.
• For now though… Typed Data.
Use Cases for Typed Data Data Types
• Describe data defined elsewhere i.e. schemas from external systems.
• Education, Media, etc…
• Define special snowflake data types that do not require entity storage.
Use Case: Xero Integration• Xero accounting platform with a Restful API.
• Post invoices from Open Atrium cases. Invoice done. Click send. Back to coding.
• Post bank transactions from Commerce/Ubercart payment transactions. Auto-matching Bank Statements to Bank Transactions is huge cost savings for a small company.
• Do not need to store data twice. It’s already stored off-site.
Escaping Procedural Reality• Make OAuth signed requests via drupal_http_request or
straight up Curl (xero_query).
• Validation via Drupal form validation that does not fit Web Services much at all (xero form helper).
• Model data as associative arrays or strongly-coupled entities (xero_make).
• And sometimes we have to maintain external PHP libraries we do not want to maintain…
116 $filterid = ( count($arguments) > 0 ) ? strip_tags(strval($arguments[0])) : false; 117 if(isset($arguments[1])) $modified_after = ( count($arguments) > 1 ) ? str_replace( 'X','T', date( 'Y-m-dXH:i:s', strtotime($arguments[1])) ) : false; 118 if(isset($arguments[2])) $where = ( count($arguments) > 2 ) ? $arguments[2] : false; 119 if ( is_array($where) && (count($where) > 0) ) { 120 $temp_where = ''; 121 foreach ( $where as $wf => $wv ) { 122 if ( is_bool($wv) ) { 123 $wv = ( $wv ) ? "%3d%3dtrue" : "%3d%3dfalse"; 124 } else if ( is_array($wv) ) { 125 if ( is_bool($wv[1]) ) { 126 $wv = ($wv[1]) ? rawurlencode($wv[0]) . "true" : rawurlencode($wv[0]) . "false" ; 127 } else { 128 $wv = rawurlencode($wv[0]) . "%22{$wv[1]}%22" ; 129 } 130 } else { 131 $wv = "%3d%3d%22$wv%22"; 132 } 133 $temp_where .= "%26%26$wf$wv"; 134 } 135 $where = strip_tags(substr($temp_where, 6)); 136 } else { 137 $where = strip_tags(strval($where)); 138 } 139 $order = ( count($arguments) > 3 ) ? strip_tags(strval($arguments[3])) : false; 140 $acceptHeader = ( !empty( $arguments[4] ) ) ? $arguments[4] : ''; 141 $method = $methods_map[$name]; 142 $xero_url = self::ENDPOINT . $method; 143 if ( $filterid ) { 144 $xero_url .= "/$filterid"; 145 } 146 if ( isset($where) ) { 147 $xero_url .= "?where=$where"; 148 } 149 if ( $order ) { 150 $xero_url .= "&order=$order"; 151 } 152 $req = OAuthRequest::from_consumer_and_token( $this->consumer, $this->token, 'GET',$xero_url); 153 $req->sign_request($this->signature_method , $this->consumer, $this->token); 154 $ch = curl_init();
Drupal 8 and Packagist Provide
• Guzzle 3*
• OAuth 1.0 plugin
• XeroClient via XeroBundle
• The Composer problem.
* At the time I stared through the Looking Glass, Guzzle 3 was the version in core. Guzzle 6 is included in Drupal 8 core at the time of writing this session.
** Core broke everything recently so I have to re-evaluate and figure out if I am permanently forking this library. Welp.
Getting off the Island
• The Dependency Nightmare
• Composer, Autoloaders, Composer Manager, Packaging.
• Libraries and duplicate code
• Some other better solution..?
• GitHub culture
Knowing the Drupal 8 API
• Dependency Injection
• Plugin API
• Field API, Entity API
• Serialization module
• Serializer component
So how do I get there?
• Goal: Make HTTP Requests to integrate with external system
• Need:
• Describe Xero types into something Drupal understands.
• Primitive and Composite data types.
Is Typed Data a Good Fit?
• Why Typed Data?
• Typed Data API gets all of the things, and is injectable into Forms, Route Controllers, Normalizers, etc…
• So that can work well with Core and Contrib modules such as Serialization and Rules.
• Are these composite data types entities?
Xero Types• Accounts
• Invoices
• Payments
• Credit Notes
• Contacts
• Users
• Bank Transactions
• Amount
• Line Items
• and more…
Xero Types• Accounts
• Invoices
• Payments
• Credit Notes
• Contacts
• Users
• Bank Transactions
• Amount
• Line Items
• and more…
Xero Types• Accounts
• Invoices
• Payments
• Credit Notes
• Contacts
• Users
• Bank Transactions
• Amount
• Line Items
• and more…
Xero Types• Accounts
• Invoices
• Payments
• Credit Notes
• Contacts
• Users
• Bank Transactions
• Amount
• Line Items
• and more…
How to Read a Map
• Typed Data API provides most of these, yay!
• There’s one complex data type that is not an entity.
• Map: the new associative array.
• Next challenge: how to create this elusive data type class.
Typed Data API Docs
• When I started, there was no documentation:
• Doc Page: https://www.drupal.org/node/1794140
• API Topic: https://api.drupal.org/api/drupal/core!core.api.php/group/typed_data/8
Data Type Plugin• Data Type is a plugin class. Extends TypedData.
• Plugin? What’s that? All the things.
• Data Type
• Constructor usually inherited. Sets property values according to definition.
• Add useful methods to your class.
1 <?php 2 /** 3 * @file 4 * Provides \Drupal\xero\Plugin\DataType\Payment. 5 */ 6 7 namespace Drupal\xero\Plugin\DataType; 8 9 /** 10 * Xero Payment type. 11 * 12 * @DataType( 13 * id = "xero_payment", 14 * label = @Translation("Xero Payment"), 15 * definition_class = "\Drupal\xero\TypedData\Definition\PaymentDefinition" 16 * ) 17 */ 18 class Payment extends XeroTypeBase { 19 20 static public $guid_name = 'PaymentID'; 21 static public $xero_name = 'Payment'; 22 static public $plural_name = 'Payments'; 23 static public $label = 'PaymentID'; 24 25 }
plugin iddata definition
annotation type
Data Definitions• Complex data type needs definition classes to describe its properties
(or child data types).
• Symfony 2 Constraints
• Label
• Property data type
• When I began, definitions were associative arrays.
• This kind of still exists. The static create method takes an array as a parameter, not the class.
15 class PaymentDefinition extends ComplexDataDefinitionBase { 16 22 public function getPropertyDefinitions() { 23 if (!isset($this->propertyDefinitions)) { 24 $info = &$this->propertyDefinitions; 25 26 $info['Invoice'] = DataDefinition::create(‘xero_invoice')
->setRequired(TRUE)->setLabel('Invoice'); 27 $info['Account'] = DataDefinition::create(‘xero_account')
->setRequired(TRUE)->setLabel('Account'); 29 $info['Date'] = DataDefinition::create(‘string')
->setRequired(TRUE)->setLabel('Date'); 30 $info['Amount'] = DataDefinition::create(‘float')
->setRequired(TRUE)->setLabel('Amount'); 31 $info['CurrencyRate'] = DataDefinition::create(‘string')
->setLabel('Currency rate'); 32 $info['Reference'] = DataDefinition::create(‘string')
->setLabel('Reference'); 33 $info['IsReconciled'] = DataDefinition::create(‘boolean')
->setLabel('Is reconciled?'); 34 } 35 return $this->propertyDefinitions; 36 } 37 }
Data Definition
List Data Definition
• Data definitions can be lists of data types i.e. multiple values or an indexed array equivalent.
• This is important for Xero because the API returns a list of items in addition to other odd types like Tracking Categories, Line Items, and Addresses.
Exercise: Spotify
• Album
• Track[]
• Artist[]
• Type
• Name
https://developer.spotify.com/web-api/object-model/
AlbumDefinitionAlbum
ListDataDefinition Artistartists
DataDefinition stringname
DataDefinition stringtype
DataDefinition stringname
ArtistDefinition
propertyDefinitions
Making Use of All of This• Composite data types defined as Typed Data Data Types
and Data Definitions.
• Normalization service that helps to transform Xero API into our data types and vice versa.
• So now we can use Serializer and Guzzle in a consistent and maintainable way.
• XeroQuery used by a Form to display a list of Accounts.
(De)normalization• Normalizer classes transform data into Typed Data and vice
versa.
• Normalization is obscured from use because it runs as part of serialization.
• Entity normalization is handled already.
• “$context” was a difficult parameter to understand.
• Typed Data Manager use is recursive and confusing.
Defining a Normalizer• Normalization classes must be defined as services in the
service container.
• These are also “tagged” so that they go through a container compiler pass.
services: xero.normalizer: class: Drupal\xero\Normalizer\XeroNormalizer arguments: ['@typed_data_manager'] tags: - { name: normalizer }
xero.services.yml
18 class XeroNormalizer extends ComplexDataNormalizer implements DenormalizerInterface { 19 20 protected $supportedInterfaceOrClass = 'Drupal\xero\TypedData\XeroTypeInterface'; 21 22 public function __construct(TypedDataManager $typed_data_manager) { 23 $this->typedDataManager = $typed_data_manager; 24 } 25 60 public function denormalize($data, $class, $format = NULL, array $context = array()) { 63 if (!isset($context['plugin_id']) || empty($context['plugin_id'])) { 64 throw new UnexpectedValueException('Plugin id parameter must be included in context.'); 65 } 66 67 $name = $class::$xero_name; 68 $plural_name = $class::$plural_name; 69 71 if (count(array_filter(array_keys($data[$plural_name][$name]), 'is_string'))) { 72 $data[$plural_name][$name] = array($data[$plural_name][$name]); 73 } 74 75 $list_definition = $this->typedDataManager
->createListDataDefinition($context['plugin_id']); 76 $items = $this->typedDataManager->create($list_definition, $data[$plural_name][$name]); 77 78 return $items; 79 } 80
DefaultSettingsForm
24 class DefaultSettingsForm extends ConfigFormBase implements ContainerInjectionInterface { 25 64 public function buildForm(array $form, FormStateInterface $form_state) { 65 66 // Get the configuration from ConfigFormBase::config(). 67 $config = self::config('xero.settings'); 68 69 $account_options = array(); 70 71 try { 72 $context = array('plugin_id' => 'xero_account'); 73 $accounts = $this->query 74 ->setType($context['plugin_id']) 75 ->setMethod('get') 76 ->setFormat('xml') 77 ->execute(); 78 79 foreach ($accounts as $account) { 80 // Bank accounts do not have a code, exclude them. 81 if ($account->get('Code')->getValue()) { 82 $account_options[$account->get('Code')->getValue()] = $account->get(‘Name')
->getValue(); 83 } 84 } 85 } 86 catch (RequestException $e) { 87 $this->logger->error('%message: %response', array('%message' => $e->getMessage(), '%response' => $e->getResponse()->getBody(TRUE)));
XeroQuery
102 public function __construct( $client,
Serializer $serializer,
TypedDataManager $typed_data,
LoggerChannelFactoryInterface $logger_factory) {
103 $this->client = $client;
104 $this->serializer = $serializer;
105 $this->typed_data = $typed_data;
106 $this->logger = $logger_factory->get('xero');
107 }
459 public function execute() { 460 try { 461 $this->validate(); 462 465 $this->explodeConditions(); 466 469 $data_class = $this->type_definition['class']; 470 $endpoint = $data_class::$plural_name; 471 $context = array('plugin_id' => $this->type); 472 473 if ($this->data !== NULL) { 474 $this->options['body'] = $this->serializer->serialize($this->data, $this->format, $context); 475 } 476 477 $request = $this->client->{$this->method}($endpoint, $this->options['headers']); 486 487 $response = $request->send(); 488 $data = $this->serializer->deserialize($response->getBody(TRUE), $data_class, $this->format, $context); 489 490 return $data; 491 }
XeroFormBuilder
* // Get an autocomplete for account. * $definition = \Drupal::service('typed_data_manager')->createDefinition('xero_account'); * $form['Account'] = $formBuilder->getElementForDefinition($definition, 'AccountID'); * @endcode */
XeroFormBuilder
/** * The Xero form builder service provides methods to create form elements * from Xero data types defined in Drupal. * * Example usage: * @code * // Get form elements required for a line item. * $formBuilder = \Drupal::service('xero.form_builder'); * $form['lineitems'][] = $formBuilder->getElementFor('xero_line_item'); * * // Get an autocomplete for account. * $definition = \Drupal::service(‘typed_data_manager') * ->createDefinition('xero_account'); * $form['Account'] = $formBuilder * ->getElementForDefinition($definition, 'AccountID'); * @endcode */
public function getElementFor($plugin_id, $property_name = '') { $element = [];
try { $definition = $this->typedDataManager->createDataDefinition($plugin_id);
if (is_subclass_of($definition, '\Drupal\Core\TypedData\ComplexDataDefinitionBase')) { if (empty($property_name)) { // Get all elements for the definition. $element = $this->getElementsForDefinition($definition); } else { // Get the element for the property of the definition. $element = $this->getElementForDefinition($definition, $property_name); } } elseif ($definition instanceof ListDataDefinitionInterface) { $element = $this->getElementsForListDefinition($definition); } else { // Get the element for the definition. $element_type = $this->getElementTypeFromDefinition($definition); $element = [ '#type' => $element_type, '#title' => $definition->getLabel(), '#description' => $definition->getDescription() ? $definition->getDescription() : '', ]; $element += $this->getAdditionalProperties($element_type, $definition); }
protected function getElementTypeFromDefinition(DataDefinitionInterface $definition) { $type = 'textfield';
if ($definition->getDataType() === 'boolean') { $type = 'checkbox'; } elseif ($definition->getConstraint('Choice')) { $type = 'select'; }
return $type; }
protected function getAdditionalProperties($type, DataDefinitionInterface$definition, $parent_type = '') { $properties = [];
if ($type === 'select') { // Add the Constraint options to the select element. $properties['#options'] = $definition->getConstraint('Choice')['choices']; } elseif (array_key_exists('XeroGuidConstraint', $definition->getConstraints())) { // Add an auto-complete path for data types with guids. $properties['#autocomplete_route_name'] = 'xero.autocomplete'; $properties['#autocomplete_route_parameters'] = ['type' => $parent_type]; }
if ($definition->isRequired()) { $element['#required'] = TRUE; }
if ($definition->isReadOnly()) { $element['#disabled'] = TRUE; }
return $properties; } }
Typed Data Summary
• Data Type and Data Definition
• Normalizer service
• Inject Typed Data Manager into forms, controllers, and services.
Typed Data and PHPUnit• Unit tests are awesome and fast, but Typed Data Manager is big
and scary. :(
• Typed Data calls itself so mocking is not straight-forward. :(
• Need to mock the service container. :(
• What is actually necessary for decent code coverage and analysis?
• Composer, GitHub and Drupal are utterly broken as of 8.0.0-rc1 on Travis CI for automated testing.
PHPUnit
• I like seeing pretty coverage and complexity graphs.
PHPUnit
• I like seeing pretty coverage and complexity graphs.
PHPUnit Usefulness
• DataDefinitionInterface::getPropertyDefinitions
• Easy to get coverage for this method as it’s only returning an array of more data definitions.
• Coverage and testing here does not provide much value other than coving in my report in a pretty green color.
15 class PaymentDefinition extends ComplexDataDefinitionBase { 16 22 public function getPropertyDefinitions() { 23 if (!isset($this->propertyDefinitions)) { 24 $info = &$this->propertyDefinitions; 25 26 $info['Invoice'] = DataDefinition::create(‘xero_invoice')
->setRequired(TRUE)->setLabel('Invoice'); 27 $info['Account'] = DataDefinition::create(‘xero_account')
->setRequired(TRUE)->setLabel('Account'); 29 $info['Date'] = DataDefinition::create(‘string')
->setRequired(TRUE)->setLabel('Date'); 30 $info['Amount'] = DataDefinition::create(‘float')
->setRequired(TRUE)->setLabel('Amount'); 31 $info['CurrencyRate'] = DataDefinition::create(‘string')
->setLabel('Currency rate'); 32 $info['Reference'] = DataDefinition::create(‘string')
->setLabel('Reference'); 33 $info['IsReconciled'] = DataDefinition::create(‘boolean')
->setLabel('Is reconciled?'); 34 } 35 return $this->propertyDefinitions; 36 } 37 }
Mocking Typed Data Manager
• The static create method on TypedData classes will call TypedDataManager:
• Mock objects cannot simply return a single known value for complex data types.
• Mock objects must carefully re-construct the order of how create will be called on for every definition in a complex data definition.
98 public function createInstance($data_type, array $configuration = array()) { 99 $data_definition = $configuration['data_definition']; 100 $type_definition = $this->getDefinition($data_type); 101 102 if (!isset($type_definition)) { 103 throw new \InvalidArgumentException("Invalid data type '$data_type' has been given"); 104 } 105 106 // Allow per-data definition overrides of the used classes, i.e. take over 107 // classes specified in the type definition. 108 $class = $data_definition->getClass(); 109 110 if (!isset($class)) { 111 throw new PluginException(sprintf('The plugin (%s) did not specify an instance class.', $data_type)); 112 } 113 return $class::createInstance($data_definition, $configuration['name'], $configuration['parent']); 114 }
30 public function setUp() { 31 // Typed Data Manager setup. 32 $this->typedDataManager = $this->getMockBuilder('\Drupal\Core\TypedData \TypedDataManager') 33 ->disableOriginalConstructor() 34 ->getMock(); 35 36 $this->typedDataManager->expects($this->any()) 37 ->method('getDefinition') 38 ->with(static::XERO_TYPE, TRUE) 39 ->will($this->returnValue(['id' => static::XERO_TYPE, 'definition class' => static::XERO_DEFINITION_CLASS])); 40 $this->typedDataManager->expects($this->any()) 41 ->method('getDefaultConstraints') 42 ->willReturn([]); 43 44 // Validation constraint manager setup. 45 $validation_constraint_manager = $this->getMockBuilder(‘\Drupal\Core \Validation\ConstraintManager') 46 ->disableOriginalConstructor() 47 ->getMock(); 48 $validation_constraint_manager->expects($this->any()) 49 ->method('create') 50 ->willReturn([]); 51 $this->typedDataManager->expects($this->any()) 52 ->method('getValidationConstraintManager') 53 ->willReturn($validation_constraint_manager); 54
51 $this->typedDataManager->expects($this->any()) 52 ->method('getValidationConstraintManager') 53 ->willReturn($validation_constraint_manager); 54 55 // Mock the container. 56 $container = new ContainerBuilder(); 57 $container->set('typed_data_manager', $this->typedDataManager); 58 \Drupal::setContainer($container); 59 60 // Create data definition 61 $definition_class = static::XERO_DEFINITION_CLASS; 62 $this->dataDefinition = $definition_class::create(static::XERO_TYPE); 63 } 64 65 }
51 /** 52 * Test getPhone method. 53 */ 54 public function testGetPhoneNumber() { 55 $string_def = DataDefinition::create('string'); 56 $country = new StringData($string_def); 57 $area = new StringData($string_def); 58 $number = new StringData($string_def); 59 60 $this->typedDataManager->expects($this->any()) 61 ->method('getPropertyInstance') 62 ->with($this->phone, $this->callback(function($subject) { 63 return in_array($subject, array(‘PhoneCountryCode’, 'PhoneAreaCode', 'PhoneNumber')); 64 })) 65 ->will($this->onConsecutiveCalls($country, $area, $number)); 66 67 $this->phone->set('PhoneCountryCode', '01'); 68 $this->phone->set('PhoneAreaCode', '805'); 69 $this->phone->set('PhoneNumber', '255-8542'); 70 71 $this->assertEquals('01-805-255-8542', $this->phone->getPhone()); 72 } 73 }
A poem about mocking the container
What I learn not to do
I resign myself to that fate.
Besides, core does it too.
So please do not hate
this container so small
is not actually in my app at all.
I do not wish to load,
or directly invoke the container,
to get variables into my code.
But to be a better maintainer,
forgive that what I leverage.
All I want is pretty coverage.
-webçick (June 2015)
“You’re crazy!”
Re: Unit Tests, Mocking and Typed Data
Why Go Through All of That?• Previous code was not testable
• Ever written an automated test for an external API?
• In a public, open source repository?
• Loosely-coupled and stricter typing is better.
• Better fit in Drupal 8: Serialization, Rules, Services
• Education (Score, Grade, etc…)
What else..?• The f loor is open to ask, discuss or share:
• Typed Data use cases
• Questions
• Other Drupal 8 APIs that relate
Matthew Radcliffe Kosada
@mattkineme
https://joind.in/talk/view/15819
http://softpixel.com/~mradcliffe/#!/articles/2015/10/dcatl-2015-typeddata
