+ All Categories
Home > Data & Analytics > Fusing Structured and Unstructured Data for Geospatial Insights in Lumify

Fusing Structured and Unstructured Data for Geospatial Insights in Lumify

Date post: 26-Jan-2015
Category:
Upload: charlie-greenbacker
View: 109 times
Download: 1 times
Share this document with a friend
Description:
Lumify is an open source big data integration, analytics, and visualization platform designed to help users discover connections and explore relationships in their data. It can ingest anything from spreadsheets and text documents, to images and video, representing this diverse data as a collection of entities, properties, and relationships between entities. Everything is stored in a scalable and secure graph database to enable advanced social network analysis and complex graph traversals. Built on proven open source technologies for big data like Hadoop, Storm, and Accumulo, Lumify supports a variety of mission-critical use cases centered around the emerging concepts of activity-based intelligence (ABI), object-based production (OBP), and human geography (HG). Its intuitive web-based user interface provides a suite of analytic options with multiple views on the data, including 2D and 3D graphs, full-text faceted search, histograms with aggregate statistics, and an interactive geographic map exploration feature. This talk will demonstrate how Lumify can be used to fuse structured and unstructured data from multiple sources into a unified knowledge base, and then analyze that knowledge to uncover hidden connections and actionable insights buried within the data's geospatial context.
Popular Tags:
22
Fusing Structured and Unstructured Data for Geospatial Insights in Charlie Greenbacker Susan Feng Altamira Technologies Corporation
Transcript
Page 1: Fusing Structured and Unstructured Data for Geospatial Insights in Lumify

Fusing Structured and Unstructured Data for Geospatial Insights in

Charlie Greenbacker Susan Feng Altamira Technologies Corporation

Page 2: Fusing Structured and Unstructured Data for Geospatial Insights in Lumify

is an open source big data analysis and visualization platform built by Altamira engineers

Page 3: Fusing Structured and Unstructured Data for Geospatial Insights in Lumify

Key Lumify Concepts

structure for organizing information (i.e., your data model) Ontology

any “thing” you want to represent (e.g., person, place, event) Entities

a link between two entities (e.g., leader-of, works-for, sibling-of) Relationships

data about an entity (e.g., first name, last name, date of birth) Properties

collection of entities and the relationships between them Graph

Page 4: Fusing Structured and Unstructured Data for Geospatial Insights in Lumify

What you can do with

Page 5: Fusing Structured and Unstructured Data for Geospatial Insights in Lumify

trafficking

RESULTS

Document 94

FILTER BY ENTITY PROPERTIES

GEO LOCATION REMOVE

Latitude 23.22

Longitude -106.42

Radius 1000

DATE REMOVE

is between 2014-01-01

2014-03-01

ADD FILTER

Video 27 Image 39 Event 21

Raid 21

Drug Lord 25

Person 60

Politician 35

Lumify provides full-text search over everything in your graph. Use custom filters built from properties defined in your ontology to refine your search.

Search

Page 6: Fusing Structured and Unstructured Data for Geospatial Insights in Lumify

Joaquin Guzman Loera

Display related entities, find paths to another entity, and establish new relationships to other entities all from a right-click menu or drag and drop action.

Link Analysis

Connect…

Find Path…

Search Related

Remove Remove from workspace

^

^

Add Related… Items

Raw

^R

Documents

Images

Videos

People

Contact Information

Organizations

Events

Locations

Page 7: Fusing Structured and Unstructured Data for Geospatial Insights in Lumify

Lumify provides many different ways to resolve new entities, establish relationships, and assign properties from the details view, map, or graph.

Knowledge Building

Zarka de Mexico Joaquin Guzman Loera

617-589-9821

Joaquin Guzman…

works at

owns

founded

advises

Page 8: Fusing Structured and Unstructured Data for Geospatial Insights in Lumify

The graph leverages drag-and-drop and context menus to put common actions at your fingertips. Use auto layout options to tame large graphs.

Graph Visualization

2014-02-10 +52 1 825 5536872 +52 1 877 1211498

303-301-5881

303-904-7511

Mazatlan

Mexico City

2014-02-22 2014-02-22

Joaquin Guzman… Zarka de Mexico

Emma Coronel Patraca

Ismael Garcia

Javier Felix

Page 9: Fusing Structured and Unstructured Data for Geospatial Insights in Lumify

Lumify ingests unstructured text documents, images, video, and audio files, then uses a variety of tools to extract & enrich the data for discoverability, analysis, and visualization.

Multimedia Analysis

Drug Lord “El Chapo” Captured in Mexico

PUBLISHED DATE

SOURCE

Audit

2014/02/22 Wikipedia

Add Property

Although Guzman had long hidden successfully in remote areas of the Sierra Madre mountains, the arrested members of his security team told the military he had begun venturing out to Culiacan and the beach town of Mazatlan. A week prior to his capture, Guzman and Zambada were reported to have attended a family reunion in Sinaloa. The Mexican military followed the bodyguards tips to Guzman’s ex-wife’s house, but they had trouble ramming the steel-reinforced front door, which allowed Guzman to escape through a system of secret tunnels that connected six houses, eventually moving south to Mazatlan. He planned to stay a few days in Mazatlan to see his twin baby daughters before retreating to the mountains. On 22 February 2014, at around 6:40 a.m., Mexican authorities arrested Guzman at a hotel in a beach front area in Mazatlan, Sinaloa, following an operation by the Mexican Navy, with joint intelligence from the DEA and

Page 10: Fusing Structured and Unstructured Data for Geospatial Insights in Lumify

Geo-tagged data can be aggregated and viewed using any mapping system with support for OpenLayers, including ESRI and Google Maps.

Geospatial Analysis

Page 11: Fusing Structured and Unstructured Data for Geospatial Insights in Lumify

Geospatial data in

Page 12: Fusing Structured and Unstructured Data for Geospatial Insights in Lumify

Sources of Geospatial Data in Lumify

geotags & coords in database records, metadata, etc. Structured Data

location fields & addresses in spreadsheets, etc. Semi-structured Data

place names mentioned in text documents Unstructured Data

Page 13: Fusing Structured and Unstructured Data for Geospatial Insights in Lumify

CLAVIN: an open source geoparser

geotagging & parsing of unstructured text Turns Text into Maps

resolves place names to gazetteer records Geospatial Entity Resolution

solves the “Springfield problem” Disambiguation

now handles multipart location fields (e.g., [Reston|VA|US]) Versatile

created by Berico Technologies www.clavin.io

Page 14: Fusing Structured and Unstructured Data for Geospatial Insights in Lumify

How does CLAVIN work?

(i.e., machine learning + natural language processing)

Page 15: Fusing Structured and Unstructured Data for Geospatial Insights in Lumify

demo

Page 16: Fusing Structured and Unstructured Data for Geospatial Insights in Lumify

Who can

help?

Page 17: Fusing Structured and Unstructured Data for Geospatial Insights in Lumify

Lumify helps analysts fuse structured and unstructured data from myriad sources into actionable intelligence.

Intelligence Analyst

Page 18: Fusing Structured and Unstructured Data for Geospatial Insights in Lumify

Law enforcement personnel can use Lumify to explore criminal networks, uncover hidden connections, and develop leads.

Police Investigator

Page 19: Fusing Structured and Unstructured Data for Geospatial Insights in Lumify

Lumify analyzes financial data and transaction records to help detect fraud and identify possible insider threats.

Financial Analyst

photo  credit:  “Numbers  And  Finance”  by  Ken  Teegardin  (h<ps://flic.kr/p/9rn9Yh)  CC-­‐BY-­‐SA  2.0  

Page 20: Fusing Structured and Unstructured Data for Geospatial Insights in Lumify

Scientists, law firms, news organizations, and others can track their research in Lumify to unearth latent knowledge and discover critical new insights.

Research Staff

photo  credit:  “A  researcher  at  The  NaJonal  Archives  in  Kew”  by  the  UK  NaJonal  Archives  (h<p://bit.ly/1n9dhR8)  CC-­‐BY  3.0  

Page 21: Fusing Structured and Unstructured Data for Geospatial Insights in Lumify

Built on Scalable Open Source Tech

Hadoop  CDH  4  

Accumulo  

ElasJcSearch  

tesseract  CLAVIN   CMU  Sphinx  OpenNLP   OpenCV   ffmpeg  

Apache  Storm  

Secure  Graph  

custom  code  

Page 22: Fusing Structured and Unstructured Data for Geospatial Insights in Lumify

Questions?

www.lumify.io

try.lumify.io

@lumifyio


Recommended