Native XML Support inDB2 Universal Database
Matthias Nicola, Bert van der LindenIBM Silicon Valley Lab
Presented by Mo LiuPresented by Mo Liu , Frate, Joseph Frate, Joseph and and John RussoJohn Russo
Some material in the talk is adapted from Some material in the talk is adapted from the slides of this paper’s conference talk.the slides of this paper’s conference talk.
Agenda
What is DB2 9 (Viper)? Native XML in the forthcoming version of
DB2 Native XML Storage XML Schema Support XML indexes Querying XML data in DB2 Summery
What is DB2 9 (Viper)? IBM DB2 9 is the next-generation hybrid
data server with optimized management of both XML and relational data.
IBM extended DB2 to include:• New storage techniques for efficient
management of hierarchical structures inherent in XML documents.
• New indexing technology
New query language support (for XQuery), a new graphical query builder (for XQuery), and new query optimization techniques
New support for validating XML data based on user-supplied schemas
New administrative capabilities, including extensions to key database utilities
Integration with popular application programming interfaces (APIs)
XML Databases XML-enabled Databases The core data model is not XML (but e.g. relational) Mapping between XML data model and DB’s data model is required, or XML is stored as text E.g.: DB2 XML Extender v8
Native XML Databases Use the hierarchical XML data model to store and process XML internally No mapping, no storage as text Storage format = processing format E.g.: Forthcoming version of DB2
XML in Relational Databases – Today's Challenge
Today’s Challenge: XML must be force fit into relational data model – 2 choices1. Shredding or decomposing − Mapping from XML to relational often too complex − Loses hierarchical dependencies
− Loses digital signature− Often requires dozens or hundreds of tables− Difficult to change original XML document
2. Large Object (BLOB, CLOB, Varchar) It allows for fast insert and retrieval of full documents but it needs XML
parsing at query execution time.− SLOW performance− Search performance is slow (must parse at search time)− Retrieval of sub-documents is expensive− Update inside the document is slow− Indexing is inefficient (based on relative position)− Difficult to join with relational− Costs get worse as document size increases
DB2 Hybrid XML Engine - Overview
Integration of XML & Relational Capabilities in DB2 Native XML data type
(not Varchar, not CLOB, not object- relational)
XML Capabilities in all DB2 components Applications combine XML & relational
data
Integrating XML and Relational in DB2
DB2 Hybrid XML Engine - Interfaces Data Definition create table dept(deptID int, deptdoc xml); Insert insert into dept(deptID, deptdoc) values (?,?) Index create index xmlindex1 on dept(deptdoc) generate key using xmlpattern ‘/dept/name’ as varchar(30); Retrieve select deptdoc from dept where deptID = ? SQL based Query select deptID, xmlquery('$d/dept/name' passing deptdoc as “d") from dept where deptID <> “PR27”; XQuery based Query for $book in db2-fn:xmlcolumn('BOOKS')/book for $entry in db2-fn:xmlcolumn('REVIEWS')/entry where $book/title = $entry/title return <review> {$entry/review/text()} </review>;
Native XML Storage
Efficient Document Tree Storage
Information for Every Node Tag name, encoded as unique StringID A nodeID Node kind (e.g. element, attribute, etc.) Namespace / Namespace prefix Type annotation Pointer to parent Array of child pointers Hints to the kind & name of child nodes (for early-out navigation) For text/attribute nodes: the data itself
XML Node Storage Layout
XML Storage: “Regions Index”
XML Indexes in DB2
Need index support to manage millions of XML documents
Path-specific value indexes on XML columns to index frequently used elements and attributes
XML-aware full-text indexing
XML Value Indexes Table DEPT has two fields: “id” and “dept_doc” Field “dept_doc” is an XML document:
<dept>
<employee id=901>
<name>John Doe</name>
<phone>408 555 1212</phone>
<office>344</office>
</employee>
</dept>
CREATE INDEX idx1 ON DEPT(deptdoc) GENERATE KEY USING XMLPATTERN ‘/dept/employee/name’ AS SQL VARCHAR(35)
Creates XML value index on employee name for all documents
XML Value Indexes (continued)
“xmlpattern” identifies the XML nodes to be indexed
Subset of XPath language Wildcards, namespaces allowed XPath predicates such as /a/b[c=5] not supported
“AS SQL” necessary to define data type, since DB2 does not require single XML schema for all documents in a table (so DB2 may not know data type to use for index)
XML Value Indexes: Data Types
Allowed data types for indexes: VARCHAR(n) VARCHAR HASHED, DOUBLE DATE TIMESTAMP
DB2 index manager enhanced to handle special XML types (e.g., +0, -0, +INF, -INF, NaN)
XML Value Indexes (continued)
Node does not cast to the index typeNo error is raisedNo index entry created for that node
Single document (e.g., XML field from single record) may contain 0, 1, or multiple index entriesDifferent than relational index
XML Value Indexes: unique indexes
Unique indexes enforced within a document, and across all documents
Example of unique index on employee id:
CREATE UNIQUE INDEX idx2 ON DEPT(deptdoc) GENERATE KEY USING XMLPATTERN‘/dept/employee/@id’ AS SQL DOUBLE
XML Value Indexes: multiple elements or attributes Can create indexes on multiple elements
or attributes Example: create index on all text nodes:
CREATE INDEX idx3 ON DEPT(deptdoc) GENERATE KEY USING XMLPATTERN ‘//text()’ AS SQL VARCHAR(hashed)
Example: create index on all attributesCREATE INDEX idx4 ON DEPT(deptdoc) GENERATE KEY USING XMLPATTERN ‘//@*’ AS SQL DOUBLE
XML Value Indexes: namespaces
Can index in a particular namespace XMLPATTERN can contain namespace
declarations and prefixes Example:
CREATE INDEX idx5 ON DEPT(deptdoc) GENERATE KEY USING XMLPATTERN ‘DECLARE NAMESPACE m=http://www.me.com/;/m:dept/m:employee/ m:name’ AS SQL VARCHAR(45)
XML Value Indexes: internal
For each XML document, each unique path mapped to an integer PathID (like StringID for tags)
Each index entry includes: PathID to identify path of indexed node Value of the node cast to the index type RowID
Identify rows containing the matching documents NodeID
Identify matching nodes and regions within the documents
XML Value Indexes: atomic vs. non-atomic
Atomic Node: if it is an attribute, or if it is a text node, or if it is an element that has no child elements and
exactly one text node child Indexes typically defined for atomic nodes Possible to define index on non-atomic
nodes, e.g. index on ‘/dept/employee’
XML Value Indexes: atomic vs. non-atomic
‘/dept/employee’ non-atomic since it has child elements
Single index entry for all of “employee” element, on all text nodes under “employee” (concatenation)
Can be useful for mixed content in text-oriented XML, e.g.: <title>The benefits of <bold>XML</bold></title>
XML Full Text Indexes
Allows full-text search of XML columns Can be fully indexed or partially indexed Example of full index:
CREATE INDEX myIndex FOR TEXT ON DEPT(deptdoc) FORMAT XML CONNECT TO PERSONNELDB
Example query: SELECT deptdoc FROM dept WHERE
CONTAINS(deptdoc,’SECTIONS(“/dept/comment”) “Brazil” ‘) =1
Internal index structure
System RX: One Part Relational, One Part XML
Kevin Beyer, Roberta J Cochrane,
Vanja Josifovski, Jim Kleewein, George Lapis,
Guy Lohman, Bob Lyle, Fatma Özcan,
Hamid Pirahesh, Normen Seemann,
Tuong Truong, Bert Van der Linden, Brian Vickery,
Chun Zhang
Internal index structure
XML index implemented with two B+ treesPath indexValue Index
Internal index structure: Path Index
Path Index maps reverse path (revPath) to a generated path identifier (pathId)
A “reverse path” is a list of node labels from leaf to root Compressed into vector of label identifiers
Analogy to COLUMNS catalog from relational database
Used for efficient processing of descendent queries Example: “//name” query
Internal index structure: Value Index Value Index used to represent nodes Cconsists of the following key:
PathIdvaluenodeIdrid
Internal index structure: Value Index “value” is representation of the node’s data
value when cast to the index’s data type “rid” identifies the row in the table (used for
locking) “nodeId” identifies a node within the
uses a Dewey node identifiercan provide quick access to a node in the XML
store “pathId” to retrieve specific path queries
Internal index structure: Tradeoffs of Value Index key fields Order of keys is a tradeoff pathId first allows quick retrieval of specific
queries e.g., index on //name might match many paths query on /book/author/name still has consecutive
index entries but, query like //name=‘Maggie’ will need to examine
every location in the index per matching path
XML Schema Support
Optional XML Schema validation Insert, Update, Query Limited support for DTDs an external entities Type annotation produced by validation
persisted with document (query execution) Conforms to XML Query standard, XML
Schema standard, XML standard
XML Schema Support
Register XML Schemas and DTDs in DB DB then stores type-annotated documents
on disk, compiles execution plans with references to the XML Schemas
Schemas stored in DB itself, for performanceXML Schema Repository (XSR)
XML Schema Support: XSR
XSR consists of several new database catalog tables:Original XML schema documents for XML
schemaBinary representation of the schema for fast
reference
XML Schema Support: Registration
Example:REGISTER XMLSCHEMA
http://my.dept.com FROM dept.xsd AS departments.deptschema complete
Schema URI is http://my.dept.com File with schema document is “dept.xsd” Schema identifier in DB is “deptschema” Belongs to relational DB schema “departments”
XML Schema Support: Validation
“XMLVALIDATE” function to validate documents in SQL statements
Schema for validation is specified explicitly, orcan be deduced from the schemaLocation hints
in the instance documents Referenced by Schema URI or by identifier
XML Schema Support: Validation
Example (explicit by URI):INSERT INTO DEPT(detpdoc)
VALUES xmlvalidate(?according to xmlschema uri ‘http://my.dept.com’)
Example (explicit by ID):INSERT INTO DEPT(deptdoc)
VALUES xmlvalidate(? according to xmlschema id departments.deptschema)
XML Schema Support: Validation
Example (implcit) DB2 tries to deduce schema from input
document
INSERT INTO dept(deptdoc) VALUES xmlvalidate(?)
Try to find it in repository
XML Schema Support: First repository design principle
Repository will notrequire users to modify a schema before it is being
registeredrequire users to modify XML documents before they
are inserted and validated Once document is validated in DB,it will never
require updates to remain validConsidered infeasible to bulk-update all existing
documents to become valid
XML Schema Support: Second repository design principle
Enable schema evolution Sequence of changes in an XML schema over
its lifetime New or evolving business needs How to accomplish schema evolution is much-
debatedno standardsbusiness demands require it; so constrain problem
XML Schema Support: Second repository design principle
Flexibility of schema repository “paramount importance”
DB2’s schema repository does not require namespace or the schema URI of each registered schema to be unique (user does not have control)
Database-specific Schema identifier must be unique (user does have control)
XML Schema Support: Second repository design principle
Built-in support for one very simple type of schema evolution
If new schema is backwards-cmpatible with old schema, then old schema can be replaced with new schema in the schema repository
DB2 verifies all possible elements and attributes in old schema have same named types in the new schema
Querying XML Data in DB2
Options Supported XQuery/XPath as a stand-alone language SQL embedded in XQuery XQuery/XPath embedded in SQL/XML Plain SQL for full-document retrieval
DB2 treats SQL and XQuery as primary query languages. Both will operate independently on their data models Can also be integrated
Sample Tables
create table ship (shipNo varchar(5) primary key not null,capacity decimal(7,2),class int,purchDate date,maintenance xml
)
create table captain (captID varchar(5) primary key not null,lname varchar(20),fname varchar(20),DOB date,contact xml
)
Notice the xml datatype
Sample XML Data Ship.maintenance<mrecord>
<log><mntid>2353</mntid><shipno>39</shipno><vendorid>2345</vendorid><captid>9875</captid><maintdate>01/10/2007</maintdate><service>Removed rust on hull </service><resolution>complete</resolution><cost>13450.96</cost><nextservice>01/10/2008</nextservice>
</log><log>
<mntid>1254</mntid><shipno>39</shipno><vendorid>1253</vendorid><captid>9234</captid><maintdate>09/20/2005</maintdate><service>Replace rudder</service><resolution>complete</resolution><cost>34532.21</cost><nextservice>NA</nextservice>
</log></mrecord>
Sample XML DataCaptain.contactinfo<contactinfo>
<Address><street>234 Rolling Lane</street><city>Rockport</city><state>MA</state><zipcode>01210</zipcode>
</Address><phone>
<work>9783412321</work><home>9722342134</home><cell>9782452343</cell><satellite>2023051243</satellite>
</phone><email>[email protected]</email>
</contactinfo>
Standalone XQuery in DB2
for $s in db2-fn:xmlcolumn(‘ship.maintenance’)
let $ml:= $s//log
where $ml/cost = > 10000
order by $ml/shipno
return <MaintenanceLog>
{$ml/shipno,$ml}
</MaintenanceLog>
Db2-fn:xmlcolumn returns sequence of all documents in the XML column
SQL Embedded in XQuery
for $m in db2-fn:sqlquery(‘select maintenance from ship where class = 1’)
let $ml := $m//log
order by $ml/shipno
return
<maintenanceLog>
{$ml}
</mantenanceLog>
This will return the documents for all class one ships.
Select Statement using XML Column
Select shipno,class,maintenance
from ship
where class = 1
This will produce the maintenance document for each ship that is class 1.
We can also create views this way
SQL/XML Queries
Restricting results using XML element values select captid,lname,fname from captain
where xmlexists(‘$c/contactinfo/Address[state=“MA”]’
passing captain.contact as “c”
• This will return the captid, lname and fname of all captains who live in Massachusetts
SQL/XML Queries
Projecting XML element values Two functions: XMLQuery and XMLTable
XMLQuery retrieves value for 1 element XMLTable retrieves value for multiple elements
XMLQuery example:select xmlquery(‘$c/contactinfo/email’
passing contact as “c”)
from captain
where state = ‘MA’
This will return email addresses for all captains in Massachusetts
SQL/XML QueriesXMLQuery (Continued)
We could also look for only first email for each captain by changing the first line:
select xmlquery(‘$c/contactinfo/email[1]’ …
Similarly, we could use xmlexists to qualify:select xmlquery(‘$c/contactinfo/email’
passing contact as “c”)from captain
where state = ‘MA’and xmlexists(‘$c/contactinfo/email’passing contact as “c”)
SQL/XML QueriesXMLTable XMLTable retrieves XML elements Elements are mapped into result set
columns Maps XML data as relational data
SQL/XML QueriesXMLTable Exampleselect s.shipNo,sm.mid,sm.vid,sm.md,sm.cost
from ship s,xmltable(‘$c/mrecord/log’ passing s.maintenance as “c”columns varchar(4) mid path ‘mntid’,
varchar(4) vid path ‘vendorid’, date md path ‘maintdate’, decimal(7,2) cost path ‘cost’) as sm
This will produce a list of maintenance logs for all ships
Joining XML and Relational Data
select c.captid,c.lname,c.fname
from captain, ship
where xmlexists(‘$s/mrec/log[captid=$c]’
passing ship.maintenance as “s”, captain.captid as “c”)
If the captain was the captain of any ship when it underwent maintenance, he or she will be listed
Using FLWR Expressions in SQL/XMLselect captid,
xmlquery(‘for $c in $cn/contactinfo
let $x := $c//city
return $x’ passing contact as “cn”)
from captain
where class = 1
Returns captid as well as city information
XMLElement
XML Element allows you to publish relational data as XML
select xmlelement(name “captain”,xmlelement(name “captid”, captid),xmlelement(name “lname”,lname),xmlelement(name “fname”,fname),xmlelement(name “class”,class))
from captainwhere class <= 2
XMLElementOutput from previous command
<captain>
<captid>3563</captid>
<lname>Smith</lname>
<fname>John</fname>
<class>2</class>
</captain>
…
Aggregating and Grouping Data
select xmlelement(name “captainlist”,
xmlagg(xmlelement(name “captain”,
xmlforest(cid as “captid”,lname as “lname”,fname as “fname”,class as “class”))
order by cid))
from captain
group by class
This query produces three captainlist elements each with a number of captains.
Updating and Deleting XML Data
UpdatesUse XMLParse command. You must specify
the entire XML column to update. If you specify only 1 element to update, the rest of the data will be lost.
DeletionSame as standard SQLCan also use xmlexists to use XML as qualifier
Query Execution Plans
•Separate parsers for SQL and XQuery statements
•Integrated query compiler for both languages
•QGMX is an internal query graph model
•Query execution plans contain special operators for navigation (XSCAN), XML index access (XISCAN) and joins over XML indexes (XANDOR)
Source: [2]
Query Run-time Evaluation
3 major components added for processing queries over XML:XML NavigationXML Index RuntimeXQuery Function Library
Summary
Problems with CLOB and Shredded XML storage
Native XML support in DB2 offers:Hierarchical and parsed representationPath-specific XML indexingNew XML join and query methods Integration of SQL and XQuery
References
[1] Nicola, M. and van der Linden, B. 2005. Native XML support in DB2 universal database. In Proceedings of the 31st international Conference on Very Large Data Bases (Trondheim, Norway, August 30 - September 02, 2005). Very Large Data Bases. VLDB Endowment, 1164-1174.
[2] Beyer, K., Cochrane, R. J., Josifovski, V., Kleewein, J., Lapis, G., Lohman, G., Lyle, B., Özcan, F., Pirahesh, H., Seemann, N., Truong, T., Van der Linden, B., Vickery, B., and Zhang, C. 2005. System RX: one part relational, one part XML. In Proceedings of the 2005 ACM SIGMOD international Conference on Management of Data (Baltimore, Maryland, June 14 - 16, 2005). SIGMOD '05. ACM Press, New York, NY, 347-358.
[3] http://www-128.ibm.com/developerworks/db2/library/techarticle/dm-0603saracco2/