35
Billing on top of MongoDB Ofer Cohen, S.D.O.C. Ltd. MUG IL, February 2014
| Date post: | 20-Aug-2015 |
| Category: |
Technology |
| Upload: | ofer-cohen |
| View: | 121,958 times |
| Download: | 0 times |
Billing on top of MongoDBOfer Cohen, S.D.O.C. Ltd.
MUG IL, February 2014
Agenda1. Who am I2. How it all began3. Small Billing4. Main Billing5. Infrastructure6. Transactions7. Maintainence8. Q&A
Agenda1. Who am I2. How it all began3. Small Billing4. Main Billing5. Infrastructure6. Transactions7. Maintainence8. Q&A
Who am I● Open Source Evangelist● Board member● Started from DevOp● Shifted to Biz● Co-Founder at S.D.O.C. Ltd.
○ open source solutions for enterprises
● Mongo user since 2012
Agenda1. Who am I2. How it all began3. Small Billing4. Main Billing5. Infrastructure6. Transactions7. Maintainence8. Q&A
How it all began● Partnering a startup Golan Telecom (GT)
● Telecom CDR first use - Anti-Fraud
○ CDR = Call/Charge detail record
● CDR Types: MOC, MTC & Data
○ 3 Different data structure
○ MOC/MTC duration=0 means SMS
SQL Implementation $base_query = "SELECT imsi FROM moc WHERE callEventStartTimeStamp >=" . $start
. " UNION SELECT imsi FROM mtc WHERE callEventStartTimeStamp >=" . $start
$base_query = "SELECT imsi FROM (" . $base_query . ") AS qry ";
if (isset($args['imsi']))
$base_query .= "WHERE imsi = '" . $this->_connection->real_escape_string($args['imsi']) . "'";
$base_query .= "GROUP BY imsi ";
$mtc_join_query = "SELECT 'mtc' AS type, imsi, SUM(callEventDuration) AS duration, SUM(CEILING(callEventDuration/60)*60) AS duration_round "
. ", SUM(chargeAmount) charge "
. ", SUM(IF(SUBSTRING(callingNumber, 1, 3)='972', callEventDuration, IF(CHAR_LENGTH(callingNumber)<=10, callEventDuration, 0))) AS israel_duration "
. ", SUM(IF(SUBSTRING(callingNumber, 1, 3)='972', CEILING(callEventDuration/60)*60, IF(CHAR_LENGTH(callingNumber)<=10, CEILING(callEventDuration/60)*60, 0))) AS israel_duration_round "
. ", SUM(IF(SUBSTRING(callingNumber, 1, 3)!='972', IF(CHAR_LENGTH(callingNumber)>10, callEventDuration, 0), 0)) AS non_israel_duration "
. ", SUM(IF(SUBSTRING(callingNumber, 1, 3)!='972', IF(CHAR_LENGTH(callingNumber)>10, CEILING(callEventDuration/60)*60, 0), 0)) AS non_israel_duration_round "
. ", SUM(IF(callEventDuration = 0, 1, 0)) AS sms_count "
. "FROM mtc "
. "WHERE callEventStartTimeStamp >=" . $start . " "
. "GROUP BY type, imsi";
Implementation cont. $moc_join_query = "SELECT 'moc' AS type, imsi, SUM(callEventDuration) AS duration, SUM(CEILING(callEventDuration/60)*60) AS duration_round "
. ", SUM(chargeAmount) charge "
. ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)='972', callEventDuration, IF(CHAR_LENGTH(connectedNumber)<=10, callEventDuration, 0))) AS israel_duration "
. ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)='972', CEILING(callEventDuration/60)*60, IF(CHAR_LENGTH(connectedNumber)<=10, CEILING(callEventDuration/60)*60, 0))) AS israel_duration_round "
. ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)!='972', IF(CHAR_LENGTH(connectedNumber)>10, callEventDuration, 0), 0)) AS non_israel_duration "
. ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)!='972', IF(CHAR_LENGTH(connectedNumber)>10, CEILING(callEventDuration/60)*60, 0), 0)) AS non_israel_duration_round "
. ", SUM(IF(callEventDuration = 0, 1, 0)) AS sms_count "
. "FROM moc "
. "WHERE callEventStartTimeStamp >=" . $start . " "
. "GROUP BY type, imsi";
// $gprs_join_query = $this->build_query($args, "gprs");
It still continues $group_query = "SELECT base.imsi, moc.duration AS moc_duration, moc.charge AS moc_charge, "
. "mtc.duration AS mtc_duration, mtc.charge AS mtc_charge, "
. "mtc.duration_round AS mtc_duration_round, moc.duration_round AS moc_duration_round, "
. "moc.israel_duration AS moc_israel_duration, moc.non_israel_duration AS moc_non_israel_duration, "
. "moc.israel_duration_round AS moc_israel_duration_round, moc.non_israel_duration_round AS moc_non_israel_duration_round, "
. "mtc.israel_duration AS mtc_israel_duration, mtc.non_israel_duration AS mtc_non_israel_duration, "
. "mtc.israel_duration_round AS mtc_israel_duration_round, mtc.non_israel_duration_round AS mtc_non_israel_duration_round, "
. "mtc.sms_count AS mtc_sms_count, moc.sms_count AS moc_sms_count "
. "FROM "
. "( " . $base_query . " ) AS base "
. " LEFT JOIN (" . $mtc_join_query . " ) AS mtc ON base.imsi = mtc.imsi "
. " LEFT JOIN (" . $moc_join_query . " ) AS moc ON base.imsi = moc.imsi " ;
if (isset($args['limit'])) {
$limit = (int) $args['limit'];
} else {
$limit = 100000;
}
How it works with Mongo?$base_match = array(
'$match' => array('source' => 'nrtrde','unified_record_time' => array('$gte' => new MongoDate($charge_time)),
));$where = array(
'$match' => array('record_type' => 'MOC','connectedNumber' => array('$regex' => '^972'),'event_stamp' => array('$exists' => false),'deposit_stamp' => array('$exists' => false),'callEventDurationRound' => array('$gt' => 0), // not sms
),);$group = array(
'$group' => array("_id" => '$imsi',"moc_israel" => array('$sum' => '$callEventDurationRound'),'lines_stamps' => array('$addToSet' => '$stamp'),
),);$project = array(
'$project' => array('imsi' => '$_id','_id' => 0,'moc_israel' => 1,'lines_stamps' => 1,
),);
How it works with Mongo?$having = array(
'$match' => array(
'moc_israel' => array('$gte' => Billrun_Factory::config()->getConfigValue('nrtrde.thresholds.moc.israel'))
),
);
$moc_israel = $lines->aggregate($base_match, $where, $group, $project, $having);//sms out to all numbers
$where['$match']['record_type'] = 'MOC';
$where['$match']['callEventDurationRound'] = 0;
$group['$group']['sms_out'] = $group['$group']['mtc_all'];
unset($group['$group']['mtc_all']);
unset($having['$match']['mtc_all']);
$group['$group']['sms_out'] = array('$sum' => 1);
$having['$match']['sms_out'] = array('$gte' => Billrun_Factory::config()->getConfigValue('nrtrde.thresholds.smsout'));
$project['$project']['sms_out'] = 1;
unset($project['$project']['mtc_all']);
$sms_out = $lines->aggregate($base_match, $where, $group, $project, $having);
Agenda1. Who am I2. How it all began3. Small Billing4. Main Billing5. Infrastructure6. Transactions7. Maintainence8. Q&A
Next - Small (ILDs) Billing● Billing only for ILDs (A.K.A. MABAL)
○ Received 6 types of CDRs *almost* the same
○ Each MABAL do what is under his 7th letter
● We decide from day 1 to go with Mongo
Small Billing - TCO● 2 weeks of dev
● 2 days of deployment and first run
● 10,000 invoices
● Low maintenance=>Avg of 2 hours per month
● Base architecture for the main billing
Agenda1. Who am I2. How it all began3. Small Billing4. Main Billing5. Infrastructure6. Transactions7. Maintainence8. Q&A
Main Billing storage requirements● 10 types of CDRs
○ Some binary hierarchical - Nokia Siemens specification
● 500,000,000 CDRs per month● Requirement to save each CDR in DB
storage.● Fields can be added on the fly
Main Billing storage size
● 500,000,000/28 days/24 hr/60 min/60 sec
○ 232 CDRs per second
Main Billing storage size
● 500,000,000/5 days/24 hr/60 min/60 sec
○ 1152 CDRs per second
Main Billing storage size● Today GT have 400,000 subscribers
● Might be 800,000 in the near future
● Data usage (CDRs) increasing (40% of the storage)
● Zero downtime
● 2 data centers redundancy
Agenda1. Who am I2. How it all began3. Small Billing4. Main Billing5. Infrastructure6. Transactions7. Maintainence8. Q&A
Infrastructure
Thanks to Eran Uzan
Sharding● Shard key
○ Requirement for good balancing
● Updates are really fast with shard key
○ No collection lock yet
○ But lock only part of the DB (1 shard)
The size [GB]> db.lines.stats(1024*1024*1024){
"sharded" : true,"ns" : "billing.lines",
"count" : 1436122538,"numExtents" : 771,"size" : 1275,
"storageSize" : 1302,"totalIndexSize" : 355,"indexSizes" : {
"_id_" : 36,"aid_1" : 36,"aid_1_urt_1" : 48,"sid_1" : 36,"stamp_1" : 126,"type_1" : 30,"urt_1" : 30
},"avgObjSize" : 8.878072492167796e-7, //954.7864099064851 Bytes"nindexes" : 7,"nchunks" : 21516,
The size [GB] - each shard"rs0" : {
"ns" : "billing.lines",
"count" : 238827456,"size" : 212,
"avgObjSize" : 8.876701345426549e-7,
"storageSize" : 217,
"numExtents" : 130,
"nindexes" : 7,
"lastExtentSize" : 1,
"paddingFactor" : 1.0000000000335132,
"systemFlags" : 1,
"userFlags" : 0,
"totalIndexSize" : 59,"indexSizes" : {
"_id_" : 6,
"stamp_1" : 21,
"urt_1" : 5,
"aid_1" : 6,
"sid_1" : 6,
"type_1" : 5,
"aid_1_urt_1" : 8
},
"ok" : 1
},
"rs1" : {
"ns" : "billing.lines",
"count" : 239694511,"size" : 213,
"avgObjSize" : 8.886311126248528e-7,
"storageSize" : 217,
"numExtents" : 130,
"nindexes" : 7,
"lastExtentSize" : 1,
"paddingFactor" : 1.0000000000337341,
"systemFlags" : 1,
"userFlags" : 0,
"totalIndexSize" : 59,"indexSizes" : {
"_id_" : 6,
"stamp_1" : 21,
"urt_1" : 5,
"aid_1" : 6,
"sid_1" : 6,
"type_1" : 5,
"aid_1_urt_1" : 8
},
"ok" : 1
},
Pay attention● With SSD you get x20 more performance
○ 5000 lines/second inserted during peak
● Mongo loves RAM
○ All used indexes must to be in RAM
○ Means, each shard have 64 GB in our env
Agenda1. Who am I2. How it all began3. Small Billing4. Main Billing5. Infrastructure6. Transactions7. Maintainence8. Q&A
● write concern = 1 (by default)● Unique key● findAndModify (A.K.A FAM)● value=oldValue● 2 phase commit on app side● Auto increment (invoice id)● Transaction lock by design
Billing - Transactions
● More in our blog post, written by Shani Dalal
http://www.billrun.net/mongo-acid
Billing - Transactions
Agenda1. Who am I2. How it all began3. Small Billing4. Main Billing5. Infrastructure6. Transactions7. Maintenance, BI & TCO8. Q&A
Billing - Maintenance● MongoDB subscription
○ Leverage your professionalism○ Special for enterprise or start-up○ Low TCO
● Monitoring○ Built in commands for kick-off○ MMS the best tool for this purpose
Billing - Maintenance● Backup - hot or cold
○ With replica it’s pretty easy (without downtime)
○ Can be done with MMS or simple script
Billing - BI● MongoDB to SQL
○ Script convert hierarchy to one dimensional object
○ Script to CSV to MySQL
● Pentaho BI tool
Billing - TCO● 3 months dev
● 1 month without SSD
● 2 months QA
● Solution can be extends easily
○ Without app change
Q & A
MUG IL, February 2014
