MongoDB Days Silicon Valley
Winning the Dreamforce Hackathon with MongoDB
db.mongoDaysSV.insert( {
“init”: “Welcome Message”, “author”: “Greg Deeds”,
“twitter”: “@TEGBIZ”, setInterval: 162000
} );
RAPID PROTOTYPING with MongoDB
You want to change the game?
You have 24 hours...
Two True Tales Of MongoDB
MongoDB & Force // GR • API driven rapid prototyping • HEROKU CONTAINERIZATION • MASS IMPORT DATA SCIENCE
DRONEY EXPRESS // MongoDB your range • Geospatial & Autonomous
MongoDB or RDBMS
Winning with MongoDB
• Developing Rapid Prototypes
• Winning Hackathons with MongoDB
• Drones / MongoDB {Range Management}
MongoDB or
stay in the RDBMS Matrix
Give Up The Old And Win
• Traditional DevOps Practitioner moved to Micro-services
{Microsoft .NET & SQL Server}
{Oracle APEX-PL/SQL & Oracle Server} {Any Linux... MySQL/POSTGRES}
• Teamwork to overworked evolutions
• My SQL Dreams never materialized - {WHATS NEXT!}
• AGILITY TRAIN YOUR MIND AS A LIFESTYLE
The Early Days
MY BACKGROUND
• Started in 1979 {Ti99/Tape Deck} • Wrote first database on Commodore • BBS Author/ISP {One of the young ones} • Multimedia Producer // Software Engineer • Worked for corporations to startups
{ Military / Federal / Education } • The Writing is on the Wall - MongoDB Time!
My Computing History
TI-99/4A {circa 1982}
Monuments to Awesome-ness
Languages & Coding {Berkeley Logo}
Languages & Coding {Learning to Communicate}
Languages & Coding {Standing on the Shoulders of Giants to Communicate}
MongoDB Queries Support Projection!
PROBLEM: The data is available but in various formats, not easy to use and is changing
• Accessing your government should not be data science or outside of SalesForce.com
• PLATFORM HEROKU - {the app processing grid}
Our target user ranges from the socially conscious individual to Publicly Traded Companies and Governments. Our comprehensive tool will allow…
• Grassroots Organizations, Associations • Publicly Traded Companies • Lobbyists, PACs and Governments
Manage information
• Up-to-Date Congressional Legislator Directories
• Congressional / State Legislation Tracking
• Easy Meetings Management
• Bill and Amendments Tracking
• Committees and more
We are most proud of the tool because it allows both individuals and organizations to understand the landscape of our Government, politics, and social activism.
MongoDB & Droney Express
ForceGR / SPEC
• FORCE.COM APP Generator • APEX & PL/SQL Classes • Restful NODEJS HEROKU • Mongo for Import, Reduce and Production
A Global Competition
Data Sets Matter• Variety Of Formats
• YAML // XML // JSON // CSV // TAB
• Normalized only for XML / DTD files.
• Multiple websites crawled/parsed {Using RSYNC || PHP To Bulk Load Objects}
• Aggregate Pipeline
• Build Words / Terms from triples
• Semantic to understand {What you say}
• Latent Semantic to understand {What you mean}
• MongoDB // Provides both when used in pipeline
with other word bag or terminology generators
• The results are we find correlation in unnormalized
data
Data Miner Mongo DB
Over 500k Objects
• Monitor the query in MongoDB reactively {Check to see how the indexes need to improve with a statistics collection of performance and counters for the API}
• Compound indexes need to be built on geojson first then other fields when used
• Consider using elastic search to boost the Hackathon Team to victory if free text is slow
Use Conditional Indexes {add Agility_new ways working with this data?}
• Find source files over APIs //
{ to maintain high speed with no throttles. Hackathon bandwidth is varied sometimes when you need it. It gets worse with time }
• Use Old School Shell Tools // { NodeJS is not ideal for all challenges }
• PHP • PERL • PyThon • RSYNC
MISSION: Data Capture
• The focus of TEG’s FORCEGR tool is to present information into SalesForce users Desktop
• Then to make decisions about responses to these decisions
• Further guiding the NodeJS learning engine
APEX CLASSES / INTERACTIONS
question =
(my_department, has, what)
knowledge = (
(I, own, my_apartment),
(my_department, has, my_computer),
(my_department, has, my_vote),
(my_department, is_in, Philadelphia)
)
‘$push’ ‘$pop’
‘$unset’ ‘$set’
bye bye SQL...
Sew the Dataset
The Team
• .NET/Oracle APEX/PLSQL is heavy Spiral DEVOPS
• The Browser Wars- We Won {HTML5/Javascript Rocks}
• SUBVERSION TO GIT DEVOPS TO NEW CHATOPS {Your Dated If You Can’t GIT With It…}
• Shifts in web computer science
• Multitouch surface the GPU and JavaScript are the next big shift
Recognizing Shift
• Practice writing specifications for NOSQL
• Use New tools to trap ideas
• CHECK YOUR ROADMAP? { are you on course & time? }
• Gain momentum by solving problems quicker
• GIT more done { Practice like you play the game }
• Organization to Reproduce Results {use automated ‘Agents’ when possible & makes sense} {containerize your app with docker or Heroku}
Next Generation DevOps
question = (my_department, has, what)knowledge = ( (I, own, my_apartment), (my_department, has, my_computer), (my_department, has, my_vote), (my_department, is_in, Philadelphia) )for each statement in knowledge { if ((statement.subject == question.subject or question.subject == what) { and (statement.predicate == question.predicate or question.predicate == what) and (statement.object == question.object or question.object == what)) call FoundAnswer(statement) }}
Output: Answer: my_department has my_computer Answer: my_department has my_vote
Pseudocode for Question- Answering Logic in NodeJS
• Ask yourself on each project freshly {What else is aggregation good for?}
• HINT: You can use the $out pipeline operator and it will create or replace the collection it spits out, after the results are ready....
Aggregation Tips { Use the Pipeline, Luke! }
if (! Links.find().count()) { // init data var nodeNames = []; var data = ['Harry,Sally,1.2', 'Rubio,Mario,1.3', 'Obama,Barrack,0.2', 'Thompson,Fred,0.5', 'Peter,Johan,0.7'];
_.each(data, function (val) { var comp = val.split(',');
Links.insert({ source: comp[0], target: comp[1], compute: comp[2] }); nodeNames.push(comp[0]); nodeNames.push(comp[1]);
});
// remove duplicates nodeNames = _.uniq(nodeNames); _.each(nodeNames, function (name) { Things.insert({ _id: name }); }); }
Finesse by using new tools
VICTORY It never lasts long enough....
MongoDB & Droney Express
• HTML5 & WebGL ODM Tools
• MongoDB Powering Geoserver Raster
• Realtime 3-D Model Scenes In A Browser
Hibernate ORM is Dead {long live MongoDB ODM}
Hibernate ORM VS MongoDB ODM • Hibernate Maps Java Objects To RDBMS And Is
Proven Not Scalable
• RDBMS load times for a single object graph (Hibernate/SQL Server) from 3 to 4 seconds
• Relational model is not suited for complex object graphs.
• Even with massive parallel processing is not economical
Hibernate Search • Cannot Beat Elastic-search & MongoDB In
Response Speed
Business Opportunity
• ORM Hibernate JBOSS - Object/Relational Model {storage has reached the end of life period…}
• MongoDB is the best object store on the market today • Hibernate is complex when the world needs BSON • MongoDB is Built For Replication • Hibernate Replacement Is An Opportunity.
Market Need {MongoDB Sensor ODM}
Project Requirements • Open Source/No Internet/Limited WiFI
• Runs on variety of low cost hardware
• Auto-deploys Via Docker On IoT Appliance
• Plugin To Existing GeoServer/ESRI/GRASS
• Replace Hibernate With MongoDB
Droney Express {Range management System}
• Interchange GeoJSON set of Shapes & Waypoints
• RasberryPi Flight Computer running 2d Queries
• Objects Linked To External GIS Engine (GRASS/ESRI)
• AutoPilot Configuration In Flight/Over-The-Air
• MongoDB Monitors Situational Awareness Of Missions
• GIS Query Results Stream As BSON Objects
What is a Mission?
Geographic Forensics {sensor data store/search}
The work of Droney Express
• Planned Geofence Scanning
• 2D/3D indexed sensor data
• GIS & MongoDB Is Powerful
• Integrated GIS Remote Monitoring
• Connect Using DDP To Sync Collections Adhoc
Sensor Range Management
Async Data Sync -> Use Oplog For Synchronization
Serve GIS markup with pretty maps using Geoserver...
MongoDB is perfect for Sensors // sorry Oracle
Flight Plan For Best Routes
GIS Engines Accelerated By MonogDB Can Handle
more ecological range management sensors faster coordinated ground and drone assets
GEOJSON / MONGODB 2d DRAINAGE MODELING
GIS ENGINE & PLUGIN Droney Express Flight Planner
Least Cost Power & Route Calculator Export Tools For 3DR Ardupilot.
Replica Sets Are Ideal For Network Segregation
HQ is Always Primary - Drones Always Secondary
Force Constant State of Disaster Recovery - Dedup
Drones Are Small Autonomous Collection Vehicles
Eventual Consistency In Loosely Coupled Networks
MongoDB Makes Life Easy
Secondaries DeDup Secondaries To Update HQ GIS Backends For Mission Planning & Sensor Datasets
Droney Heartbeat modeled from mongoDB University
Airborne MongoDB Prevent Election Of New Primary
Droney Flight Computer Always Have Priority Of 0
They Never Get Elected To Primary
[Priority, Optime, Connections]
Sync Using Oplog To HQ After Weeks In Field
Network Environment Spotty For MongoDB Servers
Statement Replication ...Getting The Data To HQ
Droney Acquires Forensic Sensor Data - ‘A Data Silo’
Facilitates Greater Independence
Replicate Based Oplog For Forensic Time-coding
Member Replica Sync Stored Records Using Oplog
Use Oplog To Journal AutoPilot For Flight Review
MongoDB Secondaries Apply Unique ID Vs Primary
Keeps Mobile Workers From Overwriting Datasets
Cheap IoT Devices MongoDB 32bit Flight Computers & Range Sensors With Embedded MongoDB
In Flight Computers Are Low Power/LINUX Threaded
Sensors Streaming Data To/From MongoDB
Autopilot Flight Recorder Is MongoDB/NodeJS
IoT SD Cards - Ideal For Circular Buffers In Flight
Circular Buffers Makes Room For New Documents
Overwrite The Oldest Documents In A Collection
Final Thought.. NO CAMPING … KEEP CLIMBING!
CONSIDER THE FUTURE OF YOUR TIME STUDY NEXT GENERATION DEVOPS SAY “WHY CANT WE FIRST” SHARPEN UP REGULARLY
TEST YOUR SKILLS OR TEACH A SKILL
Greg Deeds Technology Exploration Group
[email protected] www.TEGbiz.com
@TEGbiz @DRONEYEXPRESS @3dVRviewers