Post on 05-Jan-2016
description
transcript
Apache TikaEnd-to-End
An introduction to Apache Tika,
and integrating it to your application
Nick Burch
Software EngineerAlfresco
Apache Tikahttp://tika.apache.org/
• Project which started in 2006• Grew out of the Lucene community, now widely
used• Provides detection of files – eg this binary blob is
really a word file, that one is UTF-8 plain text• Plain text, HTML and XHTML versions of a wide
range of different file formats• Consistent Metadata from different files• Tika hides the complexity of the different formats
and their libraries, instead presents a simple, powerful API
• Easy to use and extend
What's new?
• Lots of new parsers – text, office formats, publishing formats, images, audio, CAD, fonts etc
• Long standing parsers improved – better HTML from word for example
• Embedded resources and containers• Use expanding – used by many SOLR
users, Alfresco, lots of people crunching masses of data on Hadoop
Supported Formats Page 1• Audio – WAV, RIFF, MIDI• DWG (CAD)• Epub• RSS and ATOM Feeds• True Type Fonts• HTML• Images – JPEG, GIF, PNG, TIFF, Bitmap
(including EXIF where found)• iWork (Keynote, Pages etc)• RFC822 mbox Mail
Supported Formats Page 2• Microsoft Outlook .msg Email• Microsoft Office (Binary) – Word,
PowerPoint, Excel, Visio, Publisher, Works• Microsoft Office (OOXML) – Word,
PowerPoint, Excel• MP3 (id3 v1 and v2)• CDF (Scientific Data)• Open Document Format (Open Office)• Old-style Open Office (.sxw etc)• PDF
Supported Formats Page 3
• Zip and Tar archives• RDF• Plain Text• FLV Video• XML• Java class files
And I probably forgot one...!
Metadata
• Tika provides consistent metadata across the range of parsers
• No need to know if it's “Last Author”, “Last Editor” or “Previous Author” in a file format, they all come back with the same metadata key
• Keys and values are strings, but strongly typed metadata entries provide converters to dates, ints etc
Text Content• Tika generates HTML-like SAX events
as it parses• Uses Java SAX API• Events can be captured or transformed• Body Content Handler used for plain text• HTML and XHTML available• Can customise with your own handler,
with XSLT or with E4X from JavaScript• eg HTML Table → CSV
Calling Tika
// Get a content detector, and an auto-selecting ParserTikaConfig config = TikaConfig.getDefaultConfig();ContainerAwareDetector detector = new ContainerAwareDetector(config.getMimeRepository() );Parser parser = new AutoDetectParser(detector);
// We’ll only want the plain text contentsContentHandler handler = new BodyContentHandler();
// Tell the parser what we haveMetadata metadata = new Metadata();metadata.set(Metadata.RESOURCE_NAME_KEY, filename);
// Have it processedparser.parse(input, handler, metadata, new ParseContext());
// Plain text only content handlerContentHandler handler = new BodyContentHandler();String text = handler.toString();
// XHTML content handlerSAXTransformerFactory factory = SAXTransformerFactory.newInstance();TransformerHandler handler = factory.newTransformerHandler();handler.getTransformer().setOutputProperty(OutputKeys.METHOD, "xml");handler.getTransformer().setOutputProperty(OutputKeys.INDENT, "yes");StringWriter sw = new StringWriter();handler.setResult(new StreamResult(sw));String text = sw.toString();
Tika Parsers
Parser Interface• Two key methods – what mime types are
supported, and do the parsing
public interface Parser {Set<MediaType>
getSupportedTypes(ParseContext context);void parse(InputStream stream, ContentHandler
handler, Metadata metadata, ParseContext context) throws IOException, SAXException, TikaException;
}
public class HelloWorldParser implements Parser { public Set<MediaType> getSupportedTypes(ParseContext context) { Set<MediaType> types = new HashSet<MediaType>(); types.add(MediaType.parse("hello/world")); return types; } public void parse(InputStream stream, ContentHandler handler, Metadata metadata, ParseContext context) throws SAXException { XHTMLContentHandler xhtml = new XHTMLContentHandler(handler, metadata); xhtml.startDocument(); xhtml.startElement("h1"); xhtml.characters("Hello, World!"); xhtml.endElement("h1"); xhtml.endDocument(); metadata.set("hello","world"); metadata.set("title","Hello World!"); }}
Demo: Tika-App
Demo: Geo-Tagged Imagesin Alfresco Share via Tika
Any Questions?