MarkLogic Production Deployments

Copyright © 2011 Flatirons Solutions Corporation

1

MarkLogic Production Deployments

February 15, 2011

How to make them clean/repeatable/testable


2

Who Am I?

Brad Rix [email protected] Lead for Content Technology Practice

Flatirons Solutions CorporationFlatirons Solutions is a Colorado based system integrator and long-time MarkLogic partner specializing in XML publishing, dynamic content delivery, and digital asset management for both commercial and government clients.


3

Why Do I Care About This Stuff?

Problem: You’ve written some cool XQuery It does great things that will help your business/customer But oops: the more you write code, the harder it gets:

New developers break code they don’t understand Deployments to new environment don’t work and the issues are

hard to resolve More code being written causes more instabilities which causes

longer debug/deploy cycles Which results in you spending less time writing cool XQuery

and more time fixing the problems described above

Answer: Invest some time and engineering to create a

build/test/deploy/continuous-integration infrastructure which will greatly improve the ratio of (time doing fun stuff/time doing un-fun stuff)


4

Agenda

General Goals

Automatic Deployments

Automatic Configuration

Automated Unit Testing

Hudson Automated Builds

Lessons Learned


5

General Deployment Goals

Repeatable Procedures Ability to automate deployment of XQuery

modules Ability to automate building of environments

to replicate exact configuration among dev/test/production environments

Ability to run unit tests against the code & environment

Automate running of unit tests


6

Automated Deployments

XQuery modules stored in database Used ant task to deploy modules Use Docs HTTP server or create a single HTTP

server that has a single module stored that manages deployment

Ant task zips up files and POST them to the HTTP server on the Docs database where the simple XQuery module is deployed.

Setting collections/permissions/etc on the files as they are ingested.


7

Automated Deployments (ant task)

First zip modules into a single zip file

<target name="package“> <mkdir dir="${dist.home}"/> <zip destfile="${dist.home}/${ml.

name}.zip"> <fileset dir="${build.home}/src/${ml.

name}"/> </zip> </target>


8

Automated Deployment (ant task)

Properties set in build.properties configurable per environment

Post zip file to preloaded endpoint in Docs database

<target name=“deploy"> <exec executable="wget" dir="${dist.home}"

failonerror="true" vmlauncher="true"> <arg value="--user=${ml.ingest.user}"/> <arg value="--password=${ml.ingest.password}"/> <arg value="--header=Content-Type:application/zip"/> <arg value="--post-file=${ml.name}.zip"/> <arg

value="${ml.db.docs.host}/admin/loadModulesDB.xqy?modulesdb=${ml.modules.db}&serverroot=/"/>

</exec> </target>


9

Automated Deployment (module)

Modules parse zip file to get file (snippet):

for $zip-input in xdmp:zip-manifest($xquery-modules-zip)//*:part

return

Get Module: xdmp:zip-get($xquery-modules-zip, $filename, <options xmlns="xdmp:zip-get"> </options>)Then load module into database:xdmp:document-insert($uri, $doc,

$permissions)


10

Automated Configuration

Goal is to be able to configure all database configurations including Application servers/application settings/database settings/range indexes/security roles/etc.

Create a configuration file that contains all of the required information.

Allow the configuration to change and able to re-run on new or existing environment

Run the configuration via ant task or xquery module


11

Automated Database Configuration

Configuration file per environment

Partial Configuration of database<database name=“MyDBName”>

<word-positions>true</word-positions><element-value-positions>true</element-value-positions> <element-range-indexes> <index> <scalar-type>string</scalar-type>

<namespace-uri>http://mhhe.com/meta/resolved</namespace-uri>

<local-name>type</local-name> <collation>http://marklogic.com/collation/</collation> <range-value-position>false</range-value-position> </index>

…..


12

Automated Configuration (ant task)

Ant task to call a module to configure the server.

Post the file to an endpoint or have the module load the configuration from a file.

<target name="setupdatabase"> <runxquery

module="admin/configureEnv.xqy?source=filesystem&uri=${ml.home.url}/Docs/config/environment-config.xml&env=${ml.environment}&restart=true"

host="${ml.db.docs.host}" user="${ml.user}" password="${ml.password}"/>

</target>


13

Configuration Module

Configuration module will read the configuration file

Read the XML options and run the appropriate admin services API functions to configure the server/database.

Optionally restart the server depending on parameters that require restart to change


14

Sample Function to Create Indexes

let $database-name := $database-config/@name let $dbid := xdmp:database($database-name) (: remove old ones first :) let $cluster-config := admin:database-delete-range-element-index($cluster-config,$dbid, admin:database-get-range-element-indexes ($cluster-config, $dbid)) let $retv := for $index in $database-config/element-range-indexes/index let $rangespec := admin:database-range-element-index($index/scalar-type,

$index/namespace-uri, $index/local-name, $index/collation, $index/range-value-position ) return try { xdmp:set($cluster-config, admin:database-add-range-element-index($cluster-

config, $dbid, $rangespec) ) } catch ($err) { if ($err/error:code eq "ADMIN-DUPLICATEITEM") then () else error(xs:QName("ERROR"), xs:string($err/error:message)) }

return $cluster-config


15

XQuery Unit Testing

Know that all existing modules work as expected

Changes in modules still work for previous expected behavior

At Flatirons we developed a java XQueryUnit that utilizes XML Unit for differencing XML files

Framework uses ant to run all unit tests Incorporated into entire build process so that

the deploy target can run the tests to ensure that everything works properly in this environment


16

Unit Tests (configuration file)

Call a module and expect known return If XML does not match return, then test will fail

<xqu:test xqu:name="HelloWorldInline"> <xqu:xqy_test> <xqu:xqy_test_file xqu:uri="modules/util/testHelloWorld.xqy" /> </xqu:xqy_test> <xqu:expected_result> <xqu:xml_result> <xqu:xml_result_src> <hello> world </hello> </xqu:xml_result_src> </xqu:xml_result> </xqu:expected_result> </xqu:test>


17

Unit Tests – Testing Restful Endpoint

To test an http restful call into MarkLogic, write an XQuery wrapper that performs the xdmp:http-get (or post).

Return the data back from the module and compare that to expected output in the configuration file.


18

Run Unit Tests

Run ant test

test: [junit] Testsuite:

com.mhhe.xmlunit.UnitTestSuite [junit] Tests run: 5, Failures: 0, Errors: 0,

Time elapsed: 3.078 sec [junit] [junit] Testcase: HelloWorldInline took 0.453

sec [junit] Testcase: HelloWorldFile took 0.016

sec [junit] Testcase: TestCreateProject took 2.109

sec [junit] Testcase: TestFavorites took 0.172 sec [junit] Testcase: TestGetAsset took 0.219 sec


19

Unit Tests (Failed Test)

[junit] [junit] Testcase: HelloWorldInline took 0.313 sec [junit] FAILED [junit] Unexpected error received during test

'HelloWorldInline': junit.framework.AssertionFailedError: org.custommo

nkey.xmlunit.Diff [junit] [different] Expected text value 'world' but was

' [junit] failed_test [junit] ' - comparing <hello

...>world</hello> at /hello[1]/text()[1] to <hello ...> [junit] failed_test [junit] </hello> at /hello[1]/text()[1]


20

Automated Build Testing (Hudson)

Continuous build testing with Hudson (http://hudson-ci.org/)

Hudson is a self contained web application (just install and run). Includes Jetty web server as part of deployment.

Run builds on a schedule (hourly) as each developer changes code.

Build will run full configuration/ code deploy/ unit tests at each interval

If any failure then e-mail user who checked in code (along with buildmeister)

Quick demo of Hudson now….

http://hudson-ci.org/

http://hudson-ci.org/


21

Lessons Learned

Tightly type the input and output of all modules This will detect when function is called with empty or

wrong parameters quickly.

Disable Function Mapping (will attempt to map calls to other functions) declare option xdmp:mapping "false";

Unit Test as much as possible

Date post:	07-Feb-2016
Category:	Documents
Upload:	kevork
View:	62 times
Download:	0 times

MarkLogic Production Deployments

Documents