1. 1StockSwingINSIGHT DATA ENGINEERINGFeni Chawla

2. Webapp for comparing stock price fluctuations2 3. Data pipeline3HBASEDATA COLLECTION& FORMATTINGKAFKAHDFSFlask{ .sh, .py }Canvas.jsBATCH PROCESSINGREAL TIME PROCESSINGHappyBaseProducerConsumerConsumer 4. Data SetsHistoric and Real Time stock information4Stock Information (Yahoo Finance) Corp Information (Stock Exchange)PARSING, FORMATTING,NORMALIZATIONSources AMEX, NYSE,NASDAQ 6000 companiesHistoric data Daily granularity 1983 nowReal Time data 15 min granularity Growing data setV E L O C I T Y V O L U M E V A R I E T YDate, Open, High, Low, Close,Volume, Adj CloseDate, Open, High, Low, Close,Volume, Adj CloseTicker", "Name", Sale", "MarketCap",Address", "IPO", "Sector", "industry","Summary QuoteTicker, Date, High, Low, Open,Close, Volume, Adj CloseTicker, Date, High, Low, Open,Close, Volume, Adj Close 5. Design considerationsInteractive Queries on Very Large Growing DatasetDesign choice Why?Pre-computing fluctuation values for allcompanies to store in HBase table5To enable fast querying from the appGranularity of real time data collectionat 15 minutesBottlenecked by network and efficiencyof random writes to HBaseHbase row key selection:Ticker+Date+TimeMaximizing query efficiency bygrouping results 6. About meFeni Chawla Pipeline6 7. Questions?7 8. Backup8 9. Demo screencast9 10. Hbase Schema Daily result table schema Intra-day result table schema10Ticker+Date High Low % fluctuationTicker+Date+Time High Low % fluctuation 11. Future work Can be extended to display results for whole sector Can be extended to more time ranges- E.g. weekly, monthly Can be extended to more metrics- E.g. stock volatility Can add more data sets to analyze their impact on stock pricefluctuation- E.g. news data11 12. Fluctuation Calculation Fluctuation =12High Price Low PriceLow PriceX 100