Date post: | 10-Jan-2017 |
Category: |
Data & Analytics |
Upload: | smart-infrastructure-facility |
View: | 354 times |
Download: | 3 times |
From Big Data to Smart dataJie (Jack) Yang | April 2016
—What is Big Data?
—Challenge of Big Data processing
—Smart Learning framework
—Applications
—Conclusions
Outline
—No single standard definition
—5-V information assets that require innovative techniques, algorithms, and analytics that enable decision making, and process automation
Big Data definition
1 – Scale (Volume)12+ TBs
of tweet data every day
25+ TBs oflog data every
day
? TB
s of
data
eve
ry d
ay
2+ billion people on the Web by
end 2011
30 billion RFID tags today
(1.3B in 2005)4.6 billion
camera phones
world wide
100s of millions of
GPS enabled devices sold
annually
76 million smart meters in 2009…
200M by 2014
The ability to manage, analyse, summarise, visualise, and discover knowledge from the collected data in a timely and scalable manner
2 – Speed (Velocity)
Social media and networks(millions of active users)
Mobile devices(tracking objects all the time)
Infrastructure sensors and/or instruments
(measuring all kinds of data)
Various formats, types and structures:
— Text
— Numerical
— Multi-dim arrays
— Images, audio, video, sequences
— Time series
— Graph (network)
— Streaming data
— etc
3 – Complexity (Varity)
4 – Uncertainty (Veracity)
5 – Benefit (Value)
Value ($, time, performance)
Beer & Diaper (Woolworths in Illawarra)“A number of convenience store clerks noticed that men often bought beer at the same time they bought diapers. The store mined its receipts and proved the clerks' observations correct. So, the store began stocking diapers next to the beer coolers, and sales skyrocketed”
A simple example
Hardware
—Choose machines
—System failure
Challenge of Big Data processing
16 Cores, 32G RAM, $AUD6000+
Software
—Different data sources
—Really slow
—Memory issue (out of memory, for 52 million records)
Challenge of Big Data processing
Smart Learning Framework
Data harvesting
data partners
Data mining
Data storage
Data streaming
Data visualisation
Hardware
—Money wise
—Tolerance to hardware failure
Smart Learning Framework
16 Cores, 32G RAM, $AUD6000+ 4 Cores, 8G RAM, $AUD 600+
Main features
— Collection across different platforms and formats
• APIs
• Web crawling
— 1 master and 6 workers
• distributing–working–waiting–reactivating process
— Data volume (per day)
• 20K+ records user activities
• 25K+ records from social platforms
• 200K+ tweets around AU and EU
Data harvesting
Main features
— save data into different formats
• Pure TXT / CSV
• (NO)SQL
— Query across all
— Fast respond
Data storage
SELECT * FROM (SELECT * FROM /web/logs/CSV) t0JOIN ( SELECT country, count(*) FROM mysql.web.users GROUP BY country) t1JOIN (SELECT timestamp FROM s3.root.clicks.json WHERE user_id = 'jdoe‘) t2
Main features
— Preprocessing (filtering, cleansing, feature extraction)
— Event simulation
— Saving to DBs
— Running ML jobs on the fly
• Receiver throughput = 3kb /sec
• Consumer throughput = 2kb /sec
• Consumer latency = 0.23 sec
Data streaming
Main features (35 online training jobs per day)
— Supervised (with a human assisting in classification) / unsupervised machine learning techniques, to assist with classification, clustering and prediction;
— Geospatial analysis: K-pop cluster in geographical regions;
— Network analysis to understand social connections between consumers and producers;
— Other analysis including:
• More sophisticated number crunching of comments, such as time series analysis to examine trends;
• Natural language processing techniques to assist with sentiment analysis.
Data mining
Student behaviour analysis (OLPC, until Feb 2016):
— 153+ schools
— 20K+ active laptops
— 4.2M+ activity records
Application 1
1.2M 2.6M 4.2M0
100020003000
Most popular Apps (per school) App usage (per school)
1.2M 2.6M 4.2M0
1000
2000
Car parking
Application 2
Car parking
— Every 2 minutes
— 604800 records (May to Oct 2015)
— Temporal and spatial features
Application 2
Application 2Average classification accuracy (%) as a function of the size of the selected samples.
Average computational time (second)
Social media analysis
— 70K+ films
— 228K+ users (2M + friendships)
— 1M+ reviews
— 13 features
Application 3
— User profile vs film preference
— User profile vs topics
Application 3
— Network analysis
— Opinion leadership
Application 3
4K nodes + 7K edges 76 nodes + 253 edges
Jie Yang; Jun Ma, A structure optimization algorithm of neural networks for large-scale data sets, Fuzz-IEEE,2014;
Jie Yang; Jun Ma, A Sparsity-Based Training Algorithm for Least Squares SVM, IEEE SSCI, 2014;
Jie Yang, Jun Ma, A big-data processing framework for uncertainties in Transportation data, Fuzz-IEEE, 2015
Jie Yang, Jun Ma, and Sarah K. Howard, A Structure Optimization Algorithm of Neural Networks for Pattern Learning from Educational Data, Springer Studies in Computational Intelligence ANN Modelling, 2015
Jie Yang; Jun Ma, A hybrid gene expression programming algorithm based on orthogonal design, International Journal of Computational Intelligence Systems, 2015
Jie Yang, Brian Yecies, Mining Chinese Social Media UGC A SmartLearning Framework For Analyzing Douban Movie Reviews, Journal of Big Data, 2016
Jie Yang; Jun Ma, A structure optimization framework for feed-forward neural networks using sparse representation, Knowledge-Based Systems, 2016;
Jie Yang; Jun Ma, Sarah K. Howard, Exploring Technology Integration in Education using Fuzzy Representation and Feature Selection, Fuzz-IEEE, 2016
Brian Yecies, Jie Yang, Matthew Berryman, Kai Soh, Marketing Bait: Using SMART Data to Identify E-guanxi Among China’s ‘Internet Aborigines, Film Marketing in a Global Era, 2015
Brian Yecies, Jie Yang, Matthew Berryman, Aegyung Shim, and Kai Soh, Korean Female Writer-Directors and SMART Analysis of Douban commentary Among China’s Digital Natives, Women Screenwriters: An International Guide, 2015
Brian Yecies, Jie Yang, Matthew Berryman, Aegyung Shim, and Kai Soh, Korean Female Writer–Directors and SMART Analysis of Douban Commentary Among China’s Digital Natives, Participations: International Journal of Audience Research, 2016
Sarah K. Howard, Jun Ma, Jie Yang, Kate Thompson, The use of data mining to explore factors of technology integration in learning and teaching, EARLI 2015
Sarah K. Howard, Ellie Rennie, Jun Ma, Jie Yang, Big Data, Big Theory: Moving Beyond New Empiricism to Generate Powerful Explanations, The New Data “Revolution” in Sociology, 2016
Jun Ma, Jie Yang, Rohan W. Denagamage and Murad Safadi, A Conceptual Model for Clustering Local Government Areas using Complex Fuzzy Sets, Fuzz-IEEE, 2016
Publications
— OLPC (ARC-Linkage)
— NSW-DER
— CAAR
— China-South Korean Foundation
— Healthcare (Pubmed, Seer)
— Tourism business project (UTS)
— MTR
Projects and grants
— Big Data processing:
• Data collection; streaming data; data storage; and Machine learning
• Open source libraries
— Other domains:
• Public transportation
• Business Intelligence
• Health care
Conclusions
Thank you