Breaking Out of the Box:Breaking Out of the Box: Creating Customized Metasearch Creating Customized Metasearch

Services Using an XML APIServices Using an XML API

Roy Tennant, California Digital Library

A Cautionary TaleWherein a NitWit finds the courage to face his Disastrous and Idiotic Mistakes while his respected colleagues Justifiably Point and Laugh

What is Breaking Out of the Box?

• Using an XML-based Application Program Interface (API) to an application instead of the native interface

• Requires building an interface layer to:– Accept user input, package it up, and send it to the API– Receive the XML response from the API and process it,

perhaps spawning additional requests to the API– Perform (optionally) other tasks not performed by the

application; e.g., querying another application and merging the response into the user interface

• Or: You do the work, the villain does not

What is Breaking Out of the Box?

• Using an XML-based Application Program Interface (API) to an application instead of the native interface

• Requires building an interface layer to:– Accept user input, package it up, and send it to the API– Receive the XML response from the API and process it,

perhaps spawning additional requests to the API– Perform (optionally) other tasks not performed by the

application; e.g., querying another application and merging the response into the user interface

• Or: You do the work, the vendor does not

Why Break Out of the Box?

• Much greater interface flexibility• Interface customizations remain despite system upgrades • Increased ability to integrate with other systems• Ability to add new functions and services not supported by

the vendor:– Spell checking– Recommendation services

• Upon which it finally occurs to the NitWit that Arrogance has its Price

A Visit to the Scene of the

Crime: Part the 1st• Started shopping for metasearch software in

late 2002• Purchased MetaLib from ExLibris in Spring

2004• Trained in Fall 2004, then began studying how

to make it easy for campuses to customize the interface

• Interlude: A Discovery is Made

Wherein the NitWit discovers

the full magnitude of his

Error• To customize one screen, change all of the

following file fragments:quick-1, quick-2-end, quick-3-body, quick-details, quick-1-body, quick-2-end.orig, quick-3-head, quick-details-info, quick-1-head, quick-2-head, quick-3-js, quick-full, quick-1-tail, quick-2-tail, quick-3-no-results, quick-help, quick-1-toolbar, quick-2-toolbar, quick-3-tail, quick-help-main, quick-1-toolbar-pd, quick-2-wait, quick-3-toolbar, quick-js, quick-2, quick-3, quick-8-body.spa, quick-main-help

• Then, watch other screens break later in the flow• Rinse and Repeat

Why this was truly Tragic

• Our vision is of many search portals:– Tailored to specific audiences (earth science faculty and

grad students) and/or purposes (a few good things)– Branded locally (we serve ten campuses)

• Integrating many types of content or services not typical of library metasearch applications

• The native interface presented difficult, systemic barriers to customization

• The native interface was rife with browser-specific Javascript and substandard HTML

Undergrad portal with suggestions of subject databases

A Visit to the Scene of the

Crime : Part the 2nd• After struggling with interface changes,

decided to go with API access in late 2004

• Interlude: the NitWit discovers there is no API worthy of the name

In which the NitWit overcomes despair

by Blind, Ill-Considered action• The MetaLib API was primitive, poorly documented, buggy, and lacked most of the functionality of the native interface

• We marshalled the community:– Established an X-Server listserv– Gathered input on upgrades, wrote document

summarizing our priorities

• Worked with ExLibris to improve the API• Began integrating MetaLib with the

Common Framework

The Attentive Audience Asks:

The What?

Common Framework: Principles

• Clear separation of applications from underlying services

• Consistent exposure of services thru SOAP and Java client API

• Separation of services from data storage and other resources

• Easy integration of 3rd-party solutions• Platform independence

The CDL Common Framework

• Programmed in Java• Based on Web Services (e.g., SOAP/REST

interfaces)• Services often also have Java and/or Web-based

client applications• Uses METS & PREMIS (for metadata) and ARKs

(for persistent links), among other standards or draft standards (e.g., OAIS reference model)

• Integrates the Extensible Text Framework (XTF) XML searching and publishing platform — see xtf.sourceforge.net

Common Framework Capabilities

• Presently:– Ingesting, managing, and preserving digital

content – Serving XML to the Web– Searching XML metadata and content

• In Development:– Web crawling – OAI harvesting (based on the UofM harvester)– Metasearching (using the MetaLib X-Server)

A Visit to the Scene of the

Crime : Part the 3rd

• Installed first major upgrade to the API in August 2005• Still coding (no Alpha yet!)• Next major upgrade to the API expected in late Spring 2005

• Wherein the NitWit discovers that implementation schedules are to be broken … once … twice … thrice

• Interlude: An attempt by the NitWit to justify his Dubious Decisions and StaggeringMistakes

A panoply of Lame Excuses

• Most metasearch software is not designed for multiple, tailored deployments

• Product APIs are often incomplete or nonexistent• We would rather delay deployment than give up on our model of

tailored portals• It wasn’t my fault

• Meanwhile, the abject failure of this NitWit is made all the more clear by the work of David Walker at CSU San Marcos, who Rocks!

The Agony is Nearly Over

• We conclude our Unfortunate Tale with advice for those who may be unable to avoid such Tragedy as befell this most Deserving Wretch

The Sad Morals of Our Tale

• Proprietary solutions sucketh; proprietary solutions without an API sucketh rocks

• “Excellent” is the sworn enemy of “easy”

• Neither early adopter nor latecomer be (Pain is best experienced by others)

• The first task of a Project Manager is to select upon whom to pin the Blame