Call Data Analysisfor Asterisk & FreeSWITCH

with MongoDB

Arezqui Belaid @areskib<info@star2billing.com>

Problems to solve

- Millions of Call records

- Multiple sources

- Multiple data formats

- Replication

- Fast Analytics

- Multi-Tenant

- Realtime

- Fraud detection

Why MongoDB- NoSQL - Schema-Less

- Capacity / Sharding

- Upserts

- Replication : Increase read capacity

- Async writes : Millions of entries / acceptable losses

- Compared to CouchDB - native drivers

What does it look like? Dashboard

Hourly / Daily / Monthly reporting

Compare call traffic

World Map

Realtime

Under the hood- FreeSWITCH (freeswitch.org)- Asterisk (asterisk.org)

- Django (djangoproject.com)

- Celery (celeryproject.org)

- RabbitMQ (rabbitmq.com)

- Socket.IO (socket.io)

- MongoDB (mongo.org)

- PyMongo (api.mongo.org)

- and more...

Our Data - Call Detail Record (CDR)1) Call info : 2) BSON :

3) Insert Mongo : db.cdr.insert(CDR);

CDR = { ... 'callflow':{ 'caller_profile':{ 'username':'1000', 'destination_number':'5578193435', 'ani':'71737224', 'caller_id_name':'71737224', ... }, ... }, 'variables':{ 'mduration':'12960', 'effective_caller_id_name':'Extension 1000', 'outbound_caller_id_number':'0000000000', 'duration':'3', 'end_stamp':'2012-05-23 15:45:12.856527', 'answer_uepoch':'1327521953952257', 'billmsec':'12960',...

'hangup_cause_q850':'20', 'hangup_cause':'NORMAL_CLEARING', 'sip_received_ip':'192.168.1.21', 'sip_from_host':'127.0.0.1', 'tts_voice':'kal',7', 'accountcode':'1000', 'sip_user_agent':'Blink 0.2.8 (Linux)', 'answerusec':'0', 'caller_id':'71737224', 'call_uuid':'adee0934-a51b-11e1-a18c-00231470a30c', 'answer_stamp':'2012-05-23 15:45:09.856463', 'outbound_caller_id_name':'FreeSWITCH', 'billsec':'66', 'progress_uepoch':'0', 'answermsec':'0', 'sip_via_rport':'60536', 'uduration':'12959984', 'sip_local_sdp_str':'v=0\no=FreeSWITCH 1327491731\n' },...

Pre-Aggregate

Pre-Aggregate - Daily CollectionProduce data easier to manipulate : current_y_m_d = datetime.strptime(str(start_uepoch)[:10], "%Y-%m-%d") CDR_DAILY.update({ 'date_y_m_d': current_y_m_d, 'destination_number': destination_number, 'hangup_cause_id': hangup_cause_id, 'accountcode': accountcode, 'switch_id': switch.id, },{ '$inc': {'calls': 1, 'duration': int(cdr['variables']['duration']) } }, upsert=True)

Output db.CDR_DAILY.find() :{ "_id" : ..., "date_y_m_d" : ISODate("2012-04-30T00:00:00Z"), "accountcode" : "1000", "calls" : 1, "destination_number" : "0045277522", "duration" : 23, "hangup_cause_id" :9, "switch_id" :1 }...

- Faster to query pre-aggregate data- Upsert is your friend / update if exists - insert if not

Map-Reduce - Emit Step- MapReduce is a batch processing of data- Applying to previous pre-aggregate collection (Faster / Less data)

map = mark_safe(u''' function(){ emit( { a_Year: this.date_y_m_d.getFullYear(), b_Month: this.date_y_m_d.getMonth() + 1, c_Day: this.date_y_m_d.getDate(), f_Switch: this.switch_id }, {calldate__count: 1, duration__sum: this.duration} ) }''')

Reduce Step is trivial, it simply sums up and counts :

reduce = mark_safe(u''' function(key,vals) { var ret = { calldate__count : 0, duration__sum: 0, duration__avg: 0 }; for (var i=0; i < vals.length; i++){ ret.calldate__count += parseInt(vals[i].calldate__count); ret.duration__sum += parseInt(vals[i].duration__sum); } return ret; } ''')

Map-Reduce - Reduce Step

Map-ReduceQuery :

out = 'aggregate_cdr_daily'calls_in_day = daily_data.map_reduce(map, reduce, out, query=query_var)

Output db.aggregate_cdr_daily.find() :

{ "_id" : { "a_Year" : 2012, "b_Month" : 5, "c_Day" : 13, "f_Switch" :1 }, "value" : { "calldate__count" : 91, "duration__sum" : 5559, "duration__avg" : 0 } }{ "_id" : { "a_Year" : 2012, "b_Month" : 5, "c_Day" : 14, "f_Switch" :1 }, "value" : { "calldate__count" : 284, "duration__sum" : 13318, "duration__avg" : 0 } }...

Roadmap

- Quality monitoring

- Audio recording

- Add support for other telecoms switches

- Improve - refactor (Beta)

- Testing

- Listen and Learn

WAT else...?

- Website : http://www.cdr-stats.org - Code : github.com/star2billing/cdr-stats - FOSS / Licensed MPLv2 - Get started : Install script Try it, it's easy!!!

Questions ?

Twitter : @areskibEmail : areski@gmail.com