Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
APEX and SQL/JSON in Action Twitter Analysis for everyone
Carsten Czarski Consulting Member of Technical Staff Oracle Application Express Oracle Deutschland B.V. & Co KG
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
About: Carsten Czarski
• With Oracle Germany since 2001 – based in Munich, Germany
• Member of the APEX development team since March 2016
• Blogs, Web, Twitter – German APEX community newsletter
blogs.oracle.com/apexcommunity_deutsch
– SQL and PL/SQL sql-plsql-de.blogspot.com
– Twitter @cczarski
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
About: Twitter Analysis in Oracle Database 12c Oracle12c SQL/JSON Showcase
• Understand data format
• Understand data structure
• Understand Oracle12c SQL/JSON
• Load Tweet data into the database
• Build queries and visualize results
3
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Twitter data (from the Twitter API)
4
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
JSON: JavaScript Object Notation
• Very simple format
• Much simpler than XML
• Based on Javascript
• Increasing importance on the server node.js
• Broad support by web APIs – Google, Twitter, Facebook
{ "PONumber" : 1600, "Reference" : "ABULL-20140421“, "Requestor" : "Alexis Bull“, "User" : "ABULL“, "CostCenter" : "A50“, "ShippingInstructions" : { "name" : "Alexis Bull“, "Address" : { ... }, "Phone" : [ ... ] }, "Special Instructions" : null, "AllowPartialShipment" : true, "LineItems" : [{ "ItemNumber" : 1, "Part" : { "Description" : "One Magic Christmas“, "UnitPrice" : 19.95, "UPCCode" : 13131092899 }, "Quantity" : 9 }, {…}
] }
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Understanding the twitter data structure
• Tweet represented by a JSON document
– 1 Tweet = 140 Characters 1 JSON = ~4900 Characters on average
• Many redundancies – Each JSON document contains full user details
– Retweet contains complete 'retweeted tweet'
– Twitter entities (Hashtags, Mentions) are being prepared as arrays
• Storage alternatives – Store JSON as is
– Parse JSON and store data in normalized, relational schema
6
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Load Twitter data into the database
• Multiple approaches available: Twitter API, commercial offerings
• Simple approach with the Twitter API
7
http://sql-plsql-de.blogspot.de/2015/09/twitter-analyse-mit-nichts-als.html
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Twitter API and the Oracle database: An approach
• Download "twitter4j" – Java Client library for the Twitter API
• Load Twitter4j into the Oracle Database (we use the Oracle JVM)
• Create Java Classes to access Twitter and map these to PL/SQL functions
• Prepare tables, views and PL/SQL packages
8
http://sql-plsql-de.blogspot.de/2015/09/twitter-analyse-mit-nichts-als.html
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
PL/SQL packages and functions for the Twitter API
9
SQL> desc tweet_pkg FUNCTION GET_GRANTS_SCRIPT RETURNS VARCHAR2 FUNCTION GET_TWEETS_JSON RETURNS TWEET_JSON_CT Argument Name Typ In/Out Defaultwert? ------------------------------ ----------------------- ------ -------- P_SEARCH VARCHAR2 IN P_ID NUMBER IN PROCEDURE STORE_TWEETS Argument Name Typ In/Out Defaultwert? ------------------------------ ----------------------- ------ -------- P_CATID NUMBER IN PROCEDURE STORE_TWEETS SQL> select * from table(tweet_pkg.get_tweets_json('@ogh_nl #apexworld', -1)) where rownum <= 10; COLUMN_VALUE -------------------------------------------------------------------------------- {"retweeted":false,"in_reply_to_screen_name":null,"possibly_sensitive":false,"tr {"retweeted":false,"in_reply_to_screen_name":null,"possibly_sensitive":false,"tr {"retweeted":false,"in_reply_to_screen_name":null,"possibly_sensitive":false,"tr :
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
The result: Our Twitter data is now in the database
10
SQL> select tweet from apextweets where rownum <= 3; TWEET -------------------------------------------------------------------------------- {"retweeted_status":{"contributors":null,"text":"#countdown naar #Apexworld @OGh _nl #iadvise_live @Yvke1983 geeft met #Robeco een client case over #twitterboots trap #formsmigratie #orclapex","geo":null,"retweeted":false,"in_reply_to_screen_ name":null,"truncated":false,"lang":"nl","entities":{"symbols":[],"urls":[],"has htags":[{"text":"countdown","indices":[0 {"retweeted_status":{"contributors":null,"text":"#countdown naar #Apexworld @OGh _nl #iadvise_live @Yvke1983 geeft met #Robeco een client case over #twitterboots trap #formsmigratie #orclapex","geo":null,"retweeted":false,"in_reply_to_screen_ name":null,"truncated":false,"lang":"nl","entities":{"symbols":[],"urls":[],"has htags":[{"text":"countdown","indices":[0 {"contributors":null,"text":"#countdown naar #Apexworld @OGh_nl #iadvise_live @Y vke1983 geeft met #Robeco een client case over #twitterbootstrap #formsmigratie #orclapex","geo":null,"retweeted":false,"in_reply_to_screen_name":null,"truncate d":false,"lang":"nl","entities":{"symbols":[],"urls":[],"hashtags":[{"text":"cou ntdown","indices":[0,10]},{"text":"Apexw
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
The Oracle Concept: JSON and SQL Duality
JSON
API / REST
access JSON storage in the
database
SQL Queries / Analytics
Oracle Database 12c
SQL
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Native JSON Support in the Oracle Database
Store and manage JSON in tables • Store JSON as Text in tables (VARCHAR2, CLOB, BLOB) Schemaless storage – full flexibility
• No separate data type – All existing applications and tools can work with JSON data
API Access to JSON documents • REST Services (ORDS, SODA)
• Java API
Direct SQL queries on JSON documents Reporting, analysis and more.
'JSON and SQL duality'
12
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
SQL/JSON functions
• Query JSON data in tables with SQL/JSON functions – External tables do also work
– JSON data can even reside in a Hadoop Cluster or NoSQL DB Big Data SQL
• SQL/JSON functions available – JSON_VALUE extracts one scalar value
– JSON_QUERY extracts a JSON fragment
– JSON_EXISTS returns TRUE when given JSON path exists
– JSON_TABLE projects JSON data as a relational result
– JSON_TEXTCONTAINS performs full-text searches on JSON data
13
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Example: Simple attribute extraction with JSON_VALUE
14
select distinct json_value( tweet, '$.user.screen_name' RETURNING VARCHAR2(35) ) from apextweets; JSON_VALUE ------------------------------------------------------------------ AdKosigan Enkitec berkleyp delrez_stephane :
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
More sophisticated analysis – on JSON data
15
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
More sophisticated analysis – on JSON data
16
SQL/JSON Part projects JSON attributes as a
relational result.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
More sophisticated analysis – on JSON data
17
"Classic" SQL aggegation
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
More analytics: Tweets by Day
• JSON Attribute: $.created_at
19
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
More analytics: Hashtag Combinations
20
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
More analytics: Hashtag Combinations
21
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
More Informationen
• Oracle Documentation – XML DB Developers Guide
– Chapter 39: JSON in Oracle Database
• Blogs, Web 2.0, Twitter – blogs.oracle.com/jsondb (English)
– json-rest-oracledb.blogspot.com (German)
– sql-plsql-de.blogspot.com (German and English)
22
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
APEX Connect 26th – 28th April 2016 in Berlin
Get your early-bird ticket now!
APEX.DOAG.ORG #APEXCONN16
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
[email protected] http://tinyurl.com/apexcommunity http://sql-plsql-de.blogspot.com http://plsqlexecoscomm.sourceforge.net http://plsqlmailclient.sourceforge.net Twitter: @cczarski