MongoDB World 2014 - BillRun, Billing on top of MongoDB

MongoDB World 2014

by S.D.O.C. Ltd.Billing on top of MongoDBOfer Cohen

Who Am I ?

● Open source evangelist

● Former Board member of OpenSourceMatters

(Non-profit organization behind Joomla)

● S.D.O.C. Ltd. Co-Founder

FUD

● FUD: Fear, Uncertainty and Doubt

● Tactic used in sales, marketing, public

relations, politics and propaganda.

Wikipedia

FUD

Preferred name for the presentation

S.D.O.C. Ltd. vision

We increase business success through great

open source technologies and services○ Open & Transparent

○ Don't reinvent the wheel

○ High Quality

○ Lean & Effective

How did we get to billing (history)● Client: Golan Telecom (Israel)

How did we get to billing (history)

● Client: Golan Telecom ○ New & lean player in the market

○ 0=>~1M subscribers in few years

○ Limited resources & loves open source

How did we get to billing (history)● Client: Golan Telecom

○ Start up environment from Day 1

○ Short and aggressive time to market

○ Unlimited plan for ~25$

○ Customer can do almost everything using the website

■ Obviously requires 24/7 uptime

How did we get to billing (history)● Start with Anti-Fraud solution

● 2 Different data structure, 2 separated tables○ Outgoing calls (MOC)

○ incoming calls (MTC)

○ SMS (duration=0 means SMS)

Anti-Fraud in RDBMS How did it look like? Example code... $base_query = "SELECT imsi FROM moc WHERE callEventStartTimeStamp >=" . $start

. " UNION SELECT imsi FROM mtc WHERE callEventStartTimeStamp >=" . $start

$base_query = "SELECT imsi FROM (" . $base_query . ") AS qry ";

if (isset($args['imsi']))

$base_query .= "WHERE imsi = '" . $this->_connection->real_escape_string($args['imsi']) . "'";

$base_query .= "GROUP BY imsi ";

$mtc_join_query = "SELECT 'mtc' AS type, imsi, SUM(callEventDuration) AS duration, SUM(CEILING(callEventDuration/60)*60) AS duration_round "

. ", SUM(chargeAmount) charge "

. ", SUM(IF(SUBSTRING(callingNumber, 1, 3)='972', callEventDuration, IF(CHAR_LENGTH(callingNumber)<=10, callEventDuration, 0))) AS israel_duration

"

. ", SUM(IF(SUBSTRING(callingNumber, 1, 3)='972', CEILING(callEventDuration/60)*60, IF(CHAR_LENGTH(callingNumber)<=10, CEILING

(callEventDuration/60)*60, 0))) AS israel_duration_round "

. ", SUM(IF(SUBSTRING(callingNumber, 1, 3)!='972', IF(CHAR_LENGTH(callingNumber)>10, callEventDuration, 0), 0)) AS non_israel_duration "

. ", SUM(IF(SUBSTRING(callingNumber, 1, 3)!='972', IF(CHAR_LENGTH(callingNumber)>10, CEILING(callEventDuration/60)*60, 0), 0)) AS

non_israel_duration_round "

. ", SUM(IF(callEventDuration = 0, 1, 0)) AS sms_count "

Anti-Fraud in RDBMS How did it look like? Example code... . "FROM mtc "

. "WHERE callEventStartTimeStamp >=" . $start . " "

. "GROUP BY type, imsi";

$moc_join_query = "SELECT 'moc' AS type, imsi, SUM(callEventDuration) AS duration, SUM(CEILING(callEventDuration/60)*60) AS duration_round "

. ", SUM(chargeAmount) charge "

. ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)='972', callEventDuration, IF(CHAR_LENGTH(connectedNumber)<=10, callEventDuration, 0))) AS

israel_duration "

. ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)='972', CEILING(callEventDuration/60)*60, IF(CHAR_LENGTH(connectedNumber)<=10, CEILING

(callEventDuration/60)*60, 0))) AS israel_duration_round "

. ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)!='972', IF(CHAR_LENGTH(connectedNumber)>10, callEventDuration, 0), 0)) AS non_israel_duration "

. ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)!='972', IF(CHAR_LENGTH(connectedNumber)>10, CEILING(callEventDuration/60)*60, 0), 0)) AS

non_israel_duration_round "

. ", SUM(IF(callEventDuration = 0, 1, 0)) AS sms_count "

. "FROM moc "

. "WHERE callEventStartTimeStamp >=" . $start . " "

. "GROUP BY type, imsi";

Anti-Fraud in RDBMS How did it look like? Example code... $group_query = "SELECT base.imsi, moc.duration AS moc_duration, moc.charge AS moc_charge, "

. "mtc.duration AS mtc_duration, mtc.charge AS mtc_charge, "

. "mtc.duration_round AS mtc_duration_round, moc.duration_round AS moc_duration_round, "

. "moc.israel_duration AS moc_israel_duration, moc.non_israel_duration AS moc_non_israel_duration, "

. "moc.israel_duration_round AS moc_israel_duration_round, moc.non_israel_duration_round AS

moc_non_israel_duration_round, "

. "mtc.israel_duration AS mtc_israel_duration, mtc.non_israel_duration AS mtc_non_israel_duration, "

. "mtc.israel_duration_round AS mtc_israel_duration_round, mtc.non_israel_duration_round AS

mtc_non_israel_duration_round, "

. "mtc.sms_count AS mtc_sms_count, moc.sms_count AS moc_sms_count "

. "FROM "

. "( " . $base_query . " ) AS base "

. " LEFT JOIN (" . $mtc_join_query . " ) AS mtc ON base.imsi = mtc.imsi "

. " LEFT JOIN (" . $moc_join_query . " ) AS moc ON base.imsi = moc.imsi " ;

Anti-Fraud in RDBMS How did it feel…?

After moving to Mongo

● One main collection for 2 types (MOC & MTC)

● Aggregation framework is much more simple

After moving to Mongo$base_match = array(

'$match' => array('source' => 'nrtrde','unified_record_time' => array('$gte' => new MongoDate($charge_time)),

));$where = array(

'$match' => array('record_type' => 'MOC','connectedNumber' => array('$regex' => '^972'),'event_stamp' => array('$exists' => false),'deposit_stamp' => array('$exists' => false),'callEventDurationRound' => array('$gt' => 0), // not sms

),);$group = array(

'$group' => array("_id" => '$imsi',"moc_israel" => array('$sum' => '$callEventDurationRound'),'lines_stamps' => array('$addToSet' => '$stamp'),

),);$project = array(

'$project' => array('imsi' => '$_id','_id' => 0,'moc_israel' => 1,'lines_stamps' => 1,

),);

After moving to Mongo$having = array(

'$match' => array(

'moc_israel' => array('$gte' => Billrun_Factory::config()->getConfigValue('nrtrde.thresholds.moc.israel'))

),

);

$moc_israel = $lines->aggregate($base_match, $where, $group, $project, $having);//sms out to all numbers

$where['$match']['record_type'] = 'MOC';

$where['$match']['callEventDurationRound'] = 0;

$group['$group']['sms_out'] = $group['$group']['mtc_all'];

unset($group['$group']['mtc_all']);

unset($having['$match']['mtc_all']);

$group['$group']['sms_out'] = array('$sum' => 1);

$having['$match']['sms_out'] = array('$gte' => Billrun_Factory::config()->getConfigValue('nrtrde.thresholds.smsout'));

$project['$project']['sms_out'] = 1;

unset($project['$project']['mtc_all']);

$sms_out = $lines->aggregate($base_match, $where, $group, $project, $having);

What is billing?

● Group of processes of communications service providers

● responsible to collect consumption data● calculate charging and billing information● produce bills to customers● process their payments and manage debt

collectionWikipedia

What is billing?

● This is how we see it

● The KISS way

Telecom today - database challenges

● Telecom situation world wide today:

○ Unlimited packages, extend 3g usage, with high

competition

○ High-volume - 4g, LTE

○ Different Events - different data structure

5 *main* data-structures (CDR) from different sources

● NSN - Calls

● SMSC - SMS

● MMSC - MMS

● GGSN - Data

● TAP3 - International usage

Billing and MongoDB - Pros

NSN Record> db.lines.findOne({"usaget" : "call", aid:XXXXXXX})

{

"_id" : ObjectId("52bafd818f7ac3943a8b96dc"),

"stamp" : "87cea5dec484c8f6a19e44e77a2e15b4",

"record_type" : "01",

"record_number" : "13857000",

"record_status" : 0,

"exchange_id" : "000006270000",

"call_reference" : "700227d9a3",

"urt" : ISODate("2013-12-25T15:09:15Z"),

"charging_start_time" : "20131225170915",

"charging_end_time" : "20131225171517",

"call_reference_time" : "20131225170914",

"duration" : 362,

"calling_number" : "972546918666",

"called_number" : "26366667",

"call_type" : "3",

"chrg_type" : "0",

"called_imsi" : "000030000000000",

"imsi" : "000089000101076",

"in_circuit_group_name" : "",

"in_circuit_group" : "",

"out_circuit_group_name" : "NBZQZA8",

"out_circuit_group" : "0503",

"tariff_class" : "000000",

"called_number_ton" : "6",

"org_dur" : 362,

"type" : "nsn",

"source" : "binary",

"file" : "CF0322.DAT",

"log_stamp" : "0a3ffdb7c9ccc175e38be2fcc00f8c28",

"process_time" : "2013-12-25 17:35:22",

"usaget" : "call","usagev" : 362,

"arate" : DBRef("rates", ObjectId("521e07fcd88db0e73f0001c9")),

"aid" : XXXXXXX,

"sid" : YYYYY,

"plan" : "LARGE",

"aprice" : 0,

}

SMS Record> db.lines.findOne({"usaget" : "sms", aid:XXXXXXX})

{

"_id" : ObjectId("52e52ff1d88db071648b4ad7"),

"stamp" : "9328f3aaa114aaba910910053a11b3e8",

"record_type" : "1",

"calling_number" : "000972546918666",

"calling_imsi" : "000089200000000",

"calling_msc" : "000972000000000",

"billable" : "000000000000000",

"called_number" : "000972547655380",

"called_imsi" : "425089109386379",

"called_msc" : "000972586279101",

"message_submition_time" : "140125192517",

"time_offest" : "02",

"message_delivery_time" : "140125192519",

"time_offest1" : "02",

"cause_of_terminition" : "100",

"call_reference" : "5137864939035049",

"message_length" : "050",

"concatenated" : "1",

"concatenated_from" : "09",

"source" : "separator_field_lines",

"type" : "smsc",

"log_stamp" : "a5950686e364d1400c13dd1857c3340e",

"file" : "140125192403_5735golan.cdr",

"process_time" : "2014-01-26 17:53:21",

"urt" : ISODate("2014-01-25T17:25:17Z"),

"usaget" : "sms","usagev" : 1,

"arate" : DBRef("rates", ObjectId("521e07fcd88db0e73f0001db")),

"aid" : XXXXXXX,

"sid" : YYYYY,

"plan" : "LARGE",

"aprice" : 0,

"usagesb" : 4,

"billrun" : "201402"

}

Data Record> db.lines.findOne({"usaget" : "data", aid:XXXXXXX})

{

"_id" : ObjectId("539076678f7ac34a1d8b9367"),

"stamp" : "8a9f891ec85c5294c974a34653356055",

"imsi" : "400009209100000",

"served_imsi" : "400009209100000",

"ggsn_address" : "XX.XX.144.18",

"charging_id" : "2814645234",

"sgsn_address" : "XX.XX.145.9",

"served_pdp_address" : "XX.XX.237.95",

"urt" : ISODate("2014-06-05T09:34:47Z"),

"record_opening_time" : "20140605123447",

"ms_timezone" : "+03:00",

"node_id" : "GLTNGPT",

"served_msisdn" : "00002546918666",

"fbc_uplink_volume" : 61298,

"fbc_downlink_volume" : 217304,

"rating_group" : 0,

"type" : "ggsn",

"source" : "binary",

"file" : "GLTNGPT_-_0000056580.20140605_-_1251+0300",

"log_stamp" : "45a227ced1098bc76a44774eae04eb67",

"process_time" : "2014-06-05 16:39:40",

"usaget" : "data","usagev" : 278602,

"arate" : DBRef("rates", ObjectId("521e07fcd88db0e73f000200")),

"apr" : 0.020264916503503202,

"aid" : XXXXXXX,

"sid" : YYYYY,

"plan" : "LARGE",

"aprice" : 0,

"usagesb" : 478682116,

"billrun" : "201406"

}

TAP3 Record (intl roaming)> db.lines.findOne({type:"tap3", aid:9073496}){

"_id" : ObjectId("538d9ac98f7ac3e17d8b4fd6"),"stamp" : "8f6cdc8662307ee2ed951ce640a585b5","basicCallInformation" : {

"GprsChargeableSubscriber" : {"chargeableSubscriber" : {

"simChargeableSubscriber" : {"imsi" : "400009209100000"

}},"pdpAddress" : "XX.XX.227.158"

},"GprsDestination" : {

"AccessPointNameNI" : "internet.golantelecom.net.il"},"CallEventStartTimeStamp" : {

"localTimeStamp" : "20140529205131","TimeOffsetCode" : 0

},"TotalCallEventDuration" : 163

},"LocationInformation" : {

"gprsNetworkLocation" : {"RecEntityCodeList" : {

"RecEntityCode" : ["","\u0000"

]},"LocationArea" : 00001,"CellId" : 0001

},"GeographicalLocation" : {

"ServingNetwork" : "MMMM"}

},"ImeiOrEsn" : false,

"GprsServiceUsed" : {"DataVolumeIncoming" : 195120,"DataVolumeOutgoing" : 48600,"ChargeInformationList" : {

"ChargeInformation" : { "ChargedItem" : "X", "ExchangeRateCode" : 0, "ChargeDetailList" : { "ChargeDetail" : {

"ChargeType" : "00", "Charge" : 001, "ChargeableUnits" : 100000, "ChargedUnits" : 100000, "ChargeDetailTimeStamp" : { "localTimeStamp" : "20140529205131", "TimeOffset" : 0 } }

}}

}},"OperatorSpecInfoList" : {

"OperatorSpecInformation" : ["00000000.0000","00000000.0000","00000000.0000"

]},"record_type" : "e","urt" : ISODate("2014-05-29T18:51:31Z"),"tzoffset" : "+0200","imsi" : "400009209100000","serving_network" : "DEUE2","sdr" : 0.0001,"exchange_rate" : 1.12078,"type" : "tap3","file" : "CDBELHBISRGT02253",

"log_stamp" : "a0ad109c6e795f6c1feeef9ef649d937","process_time" : "2014-06-03 12:50:08",

"usaget" : "data","usagev" : 243720,"arate" : DBRef("rates", ObjectId

("521e07fed88db0e73f000219")),"apr" : 0.46640616,"aid" : 9073496,"sid" : 78288,"plan" : "LARGE","out_plan" : 243720,"aprice" : 0.46640616,"usagesb" : 39139746,"billrun" : "201406"

}

Billing and MongoDB - ProsLoose coupling compared to traditional billing components

Oldw/o MongoDb New

with MongoDb

Billing and MongoDB - Pros● One call can be spread on few CDRs● BillRun merges records only on presentation

layer or export● No DB-level aggregation nor accumulation

○ No need for mediation● Use billing and monitoring CDR in one system

database

Billing and MongoDB - Pros

Sophisticated rating module

● Rating module depends on CDR structure

● Easy to implement recurring rating

Rate module example

MongoDB advantages with Billing

Invoice JSON2PDF process● Use json as metadata for the PDF

● Easy to export

● Fast processing

MongoDB advantages with BillingInvoice metadata

> db.billrun.findOne({billrun_key:"201405", aid:9073496}){ "aid" : NumberLong(9073496), "subs" : [ { "sid" : NumberLong(78288), "subscriber_status" : "open", "current_plan" : DBRef("plans", ObjectId("51bd8dc9eb2f76d2178dd3dd")), "next_plan" : DBRef("plans", ObjectId("51bd8dc9eb2f76d2178dd3dd")), "kosher" : false, "breakdown" : { "in_plan" : { "base" : { "INTERNET_BILL_BY_VOLUME" : { "totals" : { "data" : { "usagev" : NumberLong(1975547725), "cost" : NumberLong(0), "count" : NumberLong(1352) } }, "vat" : 0.18 }, "IL_MOBILE" : { "totals" : { "sms" : { "usagev" : NumberLong(159), "cost" : NumberLong(0), "count" : NumberLong(159) }, "call" : { "usagev" : NumberLong(20788), "cost" : NumberLong(0), "count" : NumberLong(83) } }, "vat" : 0.18 },

"IL_FIX" : { "totals" : { "call" : { "usagev" : NumberLong(1217), "cost" : NumberLong(0), "count" : NumberLong(16) } }, "vat" : 0.18 }, "INTERNAL_VOICE_MAIL_CALL" : { "totals" : { "call" : { "usagev" : NumberLong(60), "cost" : NumberLong(0), "count" : NumberLong(2) } }, "vat" : 0.18 }, "service" : { "cost" : 83.898305085, "vat" : 0.18 } }, "intl" : { "KT_USA_NEW" : { "totals" : { "call" : { "usagev" : NumberLong(149), "cost" : NumberLong(0), "count" : NumberLong(2) } }, "vat" : 0.18 } } },

MongoDB advantages with BillingInvoice metadata

"credit" : { "refund_vatable" : { "CRM-REFUND_PROMOTION_1024-BILLRUN_201405" : -33.898305084746 } } }, "lines" : { "data" : { "counters" : { "20140425" : { "usagev" : NumberLong(11991615), "aprice" : NumberLong(0), "plan_flag" : "in" }, …. /* data by date really long, so let’s cut it from this demonstration */ } } }, "totals" : { "vatable" : 50.000000000254005, "before_vat" : 50.000000000254005, "after_vat" : 59.00000000029973 }, "costs" : { "credit" : { "refund" : { "vatable" : -33.898305084746 } }, "flat" : { "vatable" : 83.898305085 } } }, { "sid" : NumberLong(354961), "subscriber_status" : "open", "current_plan" : DBRef("plans", ObjectId("51bd8dc9eb2f76d2178dd3de")), "next_plan" : DBRef("plans", ObjectId("51bd8dc9eb2f76d2178dd3de")),

"kosher" : false, "breakdown" : { "in_plan" : { "base" : { "service" : { "cost" : 8.466101695, "vat" : 0.18 } } } }, "totals" : { "vatable" : 8.466101695, "before_vat" : 8.466101695, "after_vat" : 9.9900000001 }, "costs" : { "flat" : { "vatable" : 8.466101695 } } } ], "vat" : 0.18, "billrun_key" : "201405", "totals" : { "before_vat" : 58.466101695254004, "after_vat" : 68.99000000039973, "vatable" : 58.466101695254004 }, "_id" : ObjectId("5382fd3cd88db0c31a8b74cc"), "invoice_id" : NumberLong(14738102), "invoice_file" : "201405_009073496_00014738102.xml"}

Infrastructure

BETAFirst Production

Today

Data center 1

Data center 2

Data center 1

Data center 2 Data center 1

Data center 2

Archive

App stack

BillRun application

BillRun core

PHP YAF framework

Zend and more libraries

MongoDB

Web service Cli/Cron services

Mongodloid https://github.com/BillRun/Mongodloid

https://github.com/mongodb/mongo/

https://github.com/BillRun/system

https://github.com/zendframework/https://github.com/twbs/bootstraphttps://github.com/laruence/php-yaf

Pay Attention! What to keep in mind

Transactional (tx) processes● write concern - acknowledged (default in 2.4)● findAndModify (A.K.A FAM) - document tx● value=oldValue● 2 phase commit - app side● Transaction lock by design

Transaction workflow diagram

High performance tricks● With SSD you get x20 more performance● MongoDB loves RAM● Follow MongoDB production notes

○ Readahead low as possible○ THP - transparent hugepages


More tips that happened to all● Shorten property names● MongoDB have database lock

○ Separate database for heavy-write collections


TCO - MongoDB based solution

● 3 Months dev

● 3 Months QA

● Few days of maintenance infra and app

● Easy to add features

● Easy to scale

What can BillRun do

● 5,000 events per second○ Including DB insert and transactional rating

● That means 157,680,000,000 events per year

● Create hundreds of millions invoices

Do not forget!

Thank you, Ofer Cohen

[email protected]@oc666

Billing on Top of

Date post:	25-May-2015
Category:	Technology
Upload:	ofer-cohen
View:	827 times
Download:	3 times

MongoDB World 2014 - BillRun, Billing on top of MongoDB

Technology