Islamic SIM CardApplication Server v1.40
Aug. 20th 2011
Yiannis HatzopoulosTechnical Architect
The Islamic SIM Card Application Server (aka ISCAS) is a modular and scalable application server bundle, consisting of the following components; the basic components of which are based on open source software.
Linux OS distribution (OpenSuse, Ubuntu etc)
PostgreSQL 8 SQL server or MySQL 5.5 (user selectable)
Up to date Java JDK installation
Apache Tomcat 6 web server
Multiple (1...n) Islamic SIM Card application Boxes (ISCBOX)
Http/SMPP connectivity layer to an MNO's SMSC
Http connectivity to 3rd Party CellID search engine (optional)
The components described can be residing inside one multi-core physical server, or placed inside multiple physical or virtual processing machines in the data-center to maximize the processing throughput, and make use of idle time of other machines to minimize the deployed hardware cost.
The connection between the modules is executed using Internet protocol (IP) socket connections and http; which makes it possible to connect the modules in a fault tolerant manner, and get them operating inside or even outside the data-center in any possible configuration.
The reader should note that this software is based on industry standards, and robust open source tools; which guarantee reliability, good performance and low operation cost, even on low spec server hardware.
The general architecture can be visualized in the following manner
The flow? Time after time, the request flow stems from the mobile phone. The user picks the Islamic SIM applet on his handset, selects something from the applet menu and submits his request.
This selection can generate an SMS message, which is picked by the operator's SMSC and sent to the ISCAS tomcat web server for processing. Apache Tomcat parses the incoming SMS and delegates it to an available ISCBOX under his controlled network. The ISCBOX executes the math that the Islamic SIM mandates and sends back a response to the mobile phone using either SMS or MMS, through the operator's SMSC or MMSC.
MNO billing and auditing is transparently integrated into the mix, based on the SMS flow that is either received or generated.
How to figure out the optimal configuration?
As you can see, the modules that support the ISCAS function are flexible. The administrator can get them working either inside or outside the data-center, in disparate geographic locations, redirect internal flow, since it is all based on Internet protocol; receive automated load balancing -- and in case of faults (power shortages, crashes etc), automatically get the request flow redirected to the healthy part of the installation.
So what are the critical numbers to think when setting up this system?
The one Million user question
Let's assume that we have a population of 1 million users sporting the latest Islamic SIM applet on their phones. This applet delivers either on demand (high priority) requests; e.g. Iwant to know my praying schedule for my current location now! -- I do not want to wait more than 20 secs!! – or M2M daemon-like services working in the background on the SIM (low priority).
The figures shown are experimental ones and actual results depend on deployed hardware.
The ISC admin should expect an average throughput of around 80-100 requests per second per processor used, depending on hardware.
That means that if the server avails 4 processors, the request throughput should be around 300+ requests / sec or around 19,000 requests/min. High spec fast servers yield significantly higher throughput.
Consequently, If we use 2 servers of 4 cores each, the throughput doubles. SMSC capacity must be adjusted accordingly. Now let's get back to our 1 million users scenario and let's see how this throughput translates into business terms.
M2M SIM generated low priority requests really require nothing more than 1 SMS submission from the server to the phone once a week, so they have little impact on the overall system performance.
Let's assume that 30% of our users have adopted our subscription based services, which utilize low priority M2M communication. This generates a steady demand flow of 300.000 SMS messages that have to be delivered to the phones once a week, or 42.857 SMS/day; this translates to just over 29 messages/min. This is low priority stuff, runs transparently 24hrs/day and is evenly spread within the 24hr timeslot.
Let's assume that we have a mid-range server at our disposal, with a fast Hard Disk, 8 Gbytes of RAM, a decent SQL server, 4 processors and a fast connection to a wide capacity SMSC. We should expect around 300+ requests processed per second, or 18000requests/minute to pass through the system.
Of those messages, 29 SMS/minute are allocated to serve our subscriber user-base SMS traffic, which leaves around 17971 messages/minute open to serve on-demand traffic.
What does this translate to?
It all comes down to the statistical behavior of the non-subscription user base. Out of the 700.000 non-subscription users, probably 30% (and a portion of our subscription base...) –or around 210.000 wake up in the morning and decide to give a check on this day's or next
day's praying schedule on their phone. Let's assume that half of them do this within 3 hours between 06.00 – 09.00 in the morning. So this means that 105.000 SMS requests are sent within 180 mins, leading to an average of 583 requests/min hitting the server. Good news! Our average allocated capacity of 17971 messages/min is more than adequate to get the traffic serviced within 30 secs or less of arrival.
What happens when or if statistics are miscalculated and demand outgrows capacity? Still we can cope. All requests are buffered in an SQL server, and nothing is lost on its way to the processing board. The service queues may grow and last longer to process, but service is delivered no matter the load. Nevertheless, we all know that the user does not want to wait! So what can the admin do to improve the situation? The answer is simple! Simply start more processing boxes in other machines; declare them in Tomcat web server -- the load will be evenly distributed, and the processing queues will be made shorter.
So, what is the best recipe on setting up your servers?
Start by getting your figures right on the SMSC channel first, then do the math on the appropriate server resources. Start small, follow your traffic and gradually grow! Since the user population is dynamic, with people adopting the service, some leaving it to use new smartphones; or using more of it, the demand is expected to fluctuate as well. Processing capacity can be added to the system or even removed with in a few minutes.
Is there accurate statistical information or server metrics to consider?
This is a newly introduced service, and the statistical behavior of the user base is hard to predict.
As far as hardware performance is concerned, since the metrics are influenced by the actual hardware specs, and the SMSC channel bandwidth deployed, some stress testing simulation on the deployment machines is the safest way to go; and yes the software benchmarking tools to do so can be made available.
So if we are talking for going from 1mil to 2 mil... 10 mil users, we must simply multiply the figures accordingly; and bring in more processing power to the system, taking into consideration the peak demand that is expected of the service, the available SMS channel throughput and the median response time the users might be content with.
Based on experimental data, one should expect that around 1,000,000 user pool can be serviced with in a median 20 sec waiting period per available server processor. Metrics improve as the hardware speed and resources increase.
Installation: the Linux way
HintThe initial installation of this server can be executed using linux remote desktop, with assistance by the ISCAS support. The following notes are meant for troubleshooting and extending the installation to new machines if needed.
Do a preferably clean installation of an up-to-date linux distribution on your server. Likely candidates are OpenSuse, Ubuntu etc. Disable automated updates to the system. Note that test work on this system has been done using Ubuntu 10.04 LTS.02. If you feel content with this distro, by all means use it.
You will receive by the ISCAS support a zipped folder containing a tree a files. Extract atthe folder of your preference; All the support files you need can be found there. Lets assume you extract this at /home/<username>/ISCDistribution
Also create the following folders
Use your designated package manager to download and install
Latest Java JDK (recommended and tested on Open-JDK)
Latest Apache Tomcat 6 (recommended installation path for tomcat /usr/local/tomcat )
PostgreSQL 8 or MySQL 5.5 (user selectable)
pgAdminIII for PostgreSQL 8 or MySQL Administrator based on used SQL server
Your Apache Tomcat Configuration
User rights for the ISCAS web service are registered in the file tomcat-users.xmlfound in the suggested installation path at /usr/local/tomcat/conf folder. The access model follows the industry standard 'Memory Realm' for Tomcat, using basic user authentication.
Tomcat Login Names and Passwords
At the <tomcat-users> section of tomcat-users.xml define two additional roles
<role rolename="ISCAdmin"/> <role rolename="ISCGuest"/>
Define a login name and password for the Tomcat administrator<user password="adminpassword" roles="admin,manager" username="tomcatadmin"/>
and related passwords and login names (as many as are necessary) for ISCAdmin and ISCGuest roles.
Please note that ISCadmin login names MUST contain the word 'admin' in the username field.
<user username="iscadmin" password="iscadminpassword" roles="ISCAdmin"/> ….<user username="iscguest1" password="iscguestpassword" roles="ISCGuest"/> <user username="iscguest2" password="iscguestpassword" roles="ISCGuest"/> <user username="iscguest3" password="iscguestpassword" roles="ISCGuest"/> …<user username="iscguestn" password="iscguestpassword" roles="ISCGuest"/>
The ISCAS tomcat server has been configured to use http port 8080 on the localhost. If you need external access to the web server – the appropriate setting MUST be introduced to your router to map incoming traffic to the appropriate internal port.
At path /usr/local/tomcat/webapps/ROOT you must extract CONTROL.zip which contains the html files that serve as the web console to the system.
Details on this web application will be presented later on in this document...
To start the Tomcat Web server, open a terminal and enter$ cd /usr/local/tomcat/bin$ sh startup.sh
To end Tomcat$ cd /usr/local/tomcat/bin$ sh shutdown.sh
Next step is to install the Web servlet which powers Tomcat.
Open your browser and point to http://localhost:8080/
Login in as tomcat manager and select for deployment file ISCWeb.war found in /home/<username>/ISCDistribution
Now, when you point your browser to http://localhost:8080/ISCWeb
you should be seeing “ISLAMIC SIM Web Server”
The URL to control the ISCAS server over the web is accessible at
http:// < YourWebserverIP > :8080/Control/ISC.html
or on localhost at
http:// localhost :8080/Control/ISC.html
You should be seeing the ISCAS web console
If all this is OK – the web application segment of the server has been setup OK.
SQL Server Configuration
The SQL server that powers ISCAS can be either PostgreSQL 8 or MySQL 5.5I will not commend on the installation process. Installation is executed quite easily through your linux package manager.
You must setup a password and admin login name for the SQL server using your command line.Details can be found on the web. The machine in which the SQL Server resides can be shared to any other module of the ISCAS, or a separate machine can be used as SQL server if additional processing power is requested.
Using pgAdminIII, which is a GUI control tool for PostgreSQL, you can control your SQL server, check its performance etc.
Similar procedure allows the installation of MySQL and MySQL administrator.
Next, you must create a database schema called ISC_DATABASES and execute the SQL Script contained in the file CreateTablePostGreSQL.txt (or CreateTableMySQL.txt)
These SQL tables serve to store CellID repositories and serve as the SMS incoming request buffer to the system. The systems checks its CellID database to map CellID data into approximate GPS location. The operator's own CellID tower lists will go into this Database, as well as third party data CellID data, like OpenCellID. If the desired SMS throughput is modest (<80 SMS/sec), the system can be configured to use memory buffering, instead of SQL buffering. Contact ISCAS support for details.
The Processing BOX launch
The term BOX in ISCAS terminology is used to described a Socket Server application, which packs all the Math processing functionality that backs the ISCAS system. ISCAS uses 1..n BOXes.
Each BOX is a java socket server application, which waits for socket connections in a specific port.
The default listening port is 4444; port 4445 is reserved for debug tracing; any other valueport value higher than 4445 is acceptable.
To start a new box
open a terminal, go to the folder where ISC_SocketServer.jar is found (…/dist)enter
java -jar "ISC_SocketServer.jar" (for default port 4444)
To listen to a specific port
java -jar "ISC_SocketServer.jar" <portno>
You will see something like this on your terminal
Host ID set at 4444User home @:/home/yiannisMEMORY REQUEST BUFFERRestricted No of Worker Threads...RUNNING THREAD......3rd party Web Cellid Search disabledLocal SQL DB Cellid Search disabledWeb SQL DB Cellid Search disabled------------------------------------------------------------RUNNING THREAD......3rd party Web Cellid Search disabledLocal SQL DB Cellid Search disabledWeb SQL DB Cellid Search disabled------------------------------------------------------------Starting Islamic SIM Card Socket Server. Waiting at IP : port 4444---------------------------------------------------------------- Version 1.39 - April 8 2011
This delivers some basic information on the active processing threads (In this case, I have set it up to use 2/8 which are supported per box), the request buffering options MEM/DB, the CellID search options I requested etc).
HintIf you need to run or more ISCBOXes on the same machine, they have to listen to different ports!
Note that each processing Box uses a maximum of 10 concurrent connections to your SQL server.
How to control Boxes from the Command line?
You can control a box individually using a telnet connection to their IP:porte.g.
telnet localhost 4444
You will see
yiannis@yiannis-desktop:~$ telnet localhost 4444 Trying ::1... Connected to localhost.localdomain. Escape character is '^]'. Connected> help HELP ---> This command help list NEW_PARAMS --> Update SMSC Hosts + Web CellID access URL NEW_LISTS ---> Update MCC / DST / Timezone lists SETUP_DB --> Create SQL server tables LICENSE_INFO --> Get remaining license time CELLID_DB_ADD;filename.txt --->Adds records to cellID lookup table GET_ID ---> Request runID to renew license NEW_LICENSE ---> to apply new license. STATS ---> Get usage summary SHOW_DAILY_LOG ---> Get daily activity summary END_BOX ---> Stops this socket server safely PAUSE_BOX ---> Stops processing new requests RESUME_BOX ---> Resumes processing new requests SMSC_HOSTS ---> Registered SMSC Hosts for sending SMS messages SYSTEM_INFO ---> Get system load information
Each ISCBOX can be controlled by telneting to its IP -port and submitting typed text commands. 'Help' displays the available commands.
Getting the initial license lease on each ISCBox is efficiently done using your telnet connection.
The current license is projected by entering LICENSE_INFO on your telnet terminal.
You will receive the remaining operation time in days.
To renew your license enter GET_ID which yields the Run ID of this BOX.
Send the Byte sequence you will receive to the ISCAS Support, and you will receive a license certificate which will be imported to the system by using the command NEW_LICENCE;<CertificateString>
You can verify if the certificate has cleared, by entering LICENSE_INFO.A license certificate can be used only once for the specific BOX.
There is a very important command which can only be executed by a telnet connection,and that is physically ending the BOX in a safe way.
This is done by typing the command END_BOX.
All other commands, although command line driven, are much more easily executed over the web console as will be explained later on.
Although everything in ISCAS runs fine using just one ISCBOX, it is recommended to launch a minimum of 2 boxes, either residing in one physical server or more. Thus the loadcan be shared evenly; it is simply an easy step to stop a box, end it safely, upgrade its software if needed and restart it again, while the next one is routed all traffic; so the service never has to come down this way and runs 24/7.
If more than one physical servers are available, the same concept applies, since it increases the system's redundancy and throughput. Having 2+ servers in different locations further increases the fault tolerance of the setup (power shortages etc).
Files /Folders and settings
The ISCAS uses 2 folders /home/<username>/ISC and /home/<username>/ISCLog
The first one houses the configuration files that control its operation and the second one the logs that are generated during its operation.
Lets examine the most important setup files...
/home/<username>/ISC/DBParameters.txtwe see its typical content:
DBUSER;postgres;4444 DBPASSWORD;password;4444 DBADMINEMAIL;[email protected]
You can see a series of field attributes and related values:
DBADMINEMAIL is the DB admin's email address. An email is sent automatically in case the DB server declares a fault state.
DBCLASSNAME is the JDBC classname used to connect to the SQL server, in this case org.postgresql.Driver and 4444 is the port of the Box that will use this connection.
DBCONNECTION is the URL-link that the SQL server exposes, in our case jdbc:postgresql://localhost:5432/ISC_DATABASES and 4444 is the Box port that will use this link.DBUSER and DBPASSWORD responds to the authentication credentials that must be presented to the SQL server.
If more boxes share the same SQL server, a new entry must be included in this file markedwith the respective BOX port number. If a port number 0 is used, all BOXes that have access to the specific /home/<username>/ISC folder regardless of port will use the specificattribute value.
/home/<username>/ISC/MCCCountrycodes.txtThis file contains a list of Mobile country codes, the capital city, its GPS coordinate and theDST (daylight saving time) dates that are known to be applicable for the next 2 years. This file will be periodically updated and integrated to the ISCAS system, without stopping the server.
OPERATOR_WEB_CELLID_SEARCH_BASEURL;http://localhost:8080/ISCWebCell/CellReq?CellID= SMSCLINK;0;http://localhost:13013/cgi-bin/sendsms?;yiannis;test;123456 SMSCLINK;512;http://localhost:13014/cgi-bin/sendsms?;babalid;iscdemo;0012345678SMSCLINK;513;http://localhost:13014/cgi-bin/sendsms?;yiannis;test;123456 SMSCLINK;514;http://localhost:13013/cgi-bin/sendsms?;binary;content;+123457899
This file contains the URL for accessing a user defined Web CelliD search server, and the SMSC URL links that the system uses; including locale,username, password and SMSC Mobile service number. Changes to these attributes can be made without stopping the server, using the web console. As similar MMSCLINK will be integrated into the system soon.
SOCKET_SERVER;127.0.0.1;4444 SOCKET_SERVER;127.0.0.1;4446 ALLOWED_IP;127.0.0.1 EMAILNOTIFY;[email protected]
LOG on or off sets the web server log state; log data is written in /home/<username>/ISCLog.
CATALINASTORE is the path to tomcat's base URL for web accessible files.
SOCKET_SERVER;<IP>;<port> declares the listening properties of each ISCAS Box. If new ones are needed, they can be introduced from the web console without stopping the server.
ALLOWED_IP declares incoming SMS commands from accepted IP sources. More than one entries can be included. Changes to this attribute can also be executed using the web console.
EMAILNOTIFY is the email address used by the Server, to send an automated email notification in case a malfunction is detected (eg a Box being rendered unreachable etc). This address can also be changed from the web console.
/home/<username>/ISC/Tzlist.txtContains a net of known GPS coordinates (major world cities) and their time zones. Used to track the Mobile user's current timezone if it is not reported from the mobile network.
/home/<username>/ISCLogIn this folder, 2 files are generated and stored: BoxErrorLog.txt and ISCWeblog.txt.One contains information on incoming SMS messages that failed to be parsed by ISCAS and the other detailed web activity log, which can be turned on/off.
ISCAS avails a web console tool, that the admin can use to administer basic operation properties; and guests can use to receive information on the processing load that have been served and led to billing charges to the end users.
How to start it?
Point your URL to http://<YourWebIP>:8080/Control/ISC.html
You have to login into the system to gain access.
The first selection is “Reports”, which is accessible to both ISCAdmins and ISCGuests roles.
There are two kind of reports available. One is the Usage report summary and the other the Daily user log.
SMS Summary report
This yields something like this....
Right click to Download as .CSV file
GLOBAL SYSTEM REPORT -- Wed Apr 13 09:03:37 EEST 2011
Total count of incoming SMS requests for the whole system, 0
Average SMS throughput/min for the whole system, 0.0
SMS throughput in the last 30 mins, 0.0
--->Total count SMS MECCA_DIR_TEXT, 0
--->Total count SMS MECCA_DIR_GRAPHIC, 0
--->Total count SMS PRAYER_SCHEDULER, 0
--->Total count SMS UPDATE_LOCATION, 0
--->Total count SMS PRAYER_TIMES_AT_DATE, 0
--->Total count SMS CALENDAR_UTILS, 0
Response from BOX0
Wed Apr 13 09:03:37 EEST 2011
BOX Total count of SMS messages processed, 0
BOX Total count of failed SMS messages, 0
BOX Average SMS throughput per minute, 0.0
BOX incoming SMS message count in the last 30 mins, 0
BOX processed SMS message count in the last 30 mins, 0
BOX1 Connection unreachable...
You might wonder: Well, what is all that?
-We get a global SMS report plus one on every running Box; and we will be notified if a Box is unreachable (of course... all the incoming traffic is automatically directed to the healthy Boxes).
The base SMS types serviced are –
Additionally, we receive the average SMS service throughput per minute on each Box; the number of SMS requests that have reached the Box and got successfully serviced, the total count of processed SMS and the number of SMS messages that failed to parse.
All this is critical information for the admin. The throughput averages and the processing queues must be kept within relative consistency to the system's setup specs. If incoming
SMS counts outgrow the processed SMS counts, this means it is time to introduce more processing boxes into the system.
SMS Daily usage Log
This tool allows either ISCadmin or ISCguest to download a detailed log file, with the daily activity log serviced by all BOXes for the last 400 days of operation of the ISCSA. This is intended to be downloaded and processed using Excel, to track down the revenue that was generated by the service on a day-to-day basis.
Monitoring is a tool designed for the ISCadmin of the ISCAS. It allows him to view the processing load of active Boxes, change the SMSC links the system uses, alter allowed IPconnections and check the active license of the system's boxes.
Selecting 'processing state' yields a web page, where we can see if a box is OK, Suspended or Unreachable, its OS kernel, the count of processors that service it, the system load it induces, the number of active connections it services, its memory usage profile, HardDisk availability and Uptime.
Request result: ISC admin email:[email protected]
BOX0 :127.0.0.1 :4444 OK
Number or processors: 1
Current system load: 0,4
No of active connections: 1
Current Free memory available to Virtual Machine (Kb): 13.854
Max memory available for Virtual Machines (Kb): 506.816
Used memory of Virtual Machine (Kb):2.017
Total Disk space (MB): 18.600
Free Disk space (MB):11.214
Processing BOX Uptime in Days: 0,0152
PostgreSQL Server request buffering used
DB admin email: [email protected]
BOX1 :127.0.0.1 :4446 UNREACHABLE
SMSC Connections, simply projects the URLs of active SMSC / MMSC server connections
Request result: Response from BOX0 User =yiannis URL=http://localhost:13013/cgi-bin/sendsms? : Locale=0
User =invalid URL=http://localhost:13014/cgi-bin/sendsms? : Locale=512
User =yiannis URL=http://localhost:13014/cgi-bin/sendsms? : Locale=513
User =binary URL=http://localhost:13013/cgi-bin/sendsms? : Locale=514
BOX1 Connection unreachable...
IP Lists projects the active IP connection filter
Request result: Allowed Incoming IP = *.*.*.*
Active License info projects the license information for each running Box. The license time for each Box is timed in Days. If the time lease expires, the Box falls into a restricted mode operation (slows down considerably).
Request result: Response from BOX0 TIME-LEASE EXPIRED. RUNNING IN RESTRICTED MODE. REQUEST NEW LICENSE!
BOX1 Connection unreachable...
SettingsThis is the web console tab, which allows the admin to add a new processing box connections, pause Box connections, resume or remove them. Removing a Box connection does not physically 'kill' a box; Killing a Box has to be done using a telnet connection to it and entering the command END_BOX.
To add a new processing Box, the Box must first be started as an ordinary java .jar application; a telnet connection must be made to it; a new license certificate must be initially imported as described in a previous section of this document and the BOX IP and port must be declared in the BOX IP and Port field form.
Pausing / resuming / removing a Box connection is used in case a Box software update has to be executed. An old Box must be paused, then safely ended, a new one must be started, and a new connection added.
Incoming SMS command buffering sends all incoming SMS messages to a temporary text file; essentially stopping them being channeled over to processing boxes. The command can be revoked likewise. If all BOX connections die (total network failure etc) the incoming
SMS messages are automatically buffered; the admin is notified by email (SMS to be included in the future).
When the network topology is restored, by selecting 'Unbuffer', the admin will send the buffered content back to the processing Boxes, right along with normal incoming SMS traffic.
Trace Log On and Off turns the trace log feature on and off. The text file generated is placed in /home/<username>/ISCLog folder
This selection tab groups the licensing lease updates on running Boxes. To renew a Box lease, request a Box Run ID, send the server response to the support, and import the license certificate using 'New License key'. A certificate can be used once and only in one Box.
To import new CellIds (or update existing ones) to the DB, form a text file using the following format eg.
1,44.8794617951663,-0.52644743414451,208,10,18122,32951790,0 (all decimal values)
<arbitrary seq number>,
Place this text file in /home/<username>/ISC and select import/update cellIDs. Enter the Box that should receive this data and your filename. If no data is available on location height, use an approximate value (0..100).
The filename can be followed by an MCC + MNC filter, executed on the source cellid data.
will filter and import from cellid.txt those entries, with MCC=208 and MNC=10
Example: cellid.txt;202 with filter and import those entries where MCC=202
Note that the CellID database can be shared by 2+ Boxes as specified by the SQL server configuration file. So the update need not necessarily have to repeated for shared CellIDs databases. It will not cause any malfunction, but would take longer to complete.
This import format is supported by the openCellID DB data, which is freely available at http://www.opencellid.org; so OpenCellID can be readily integrated in your CellID database.
The operator's own CellID data can be co-imported in the same DB.
DST files are list of Daylight Time Saving lists for all countries; and a revised periodically. The updated files go into /home/<username>/ISC; an invocation of the DST update serviceis done and the Boxes get fresh DST data.
The same reasoning applies to updating SMSC URL lists. New versions go in the /home/<username>/ISC folder and the update form leads the data down to the actual Boxes.
New Allowed IP lists go to the designated form. If more than 5 are needed, separate them
using a semi colon eg. 127.0.0.1;188.8.131.52
Admin Notification Email – you can use this form to update the admin contact email, in case a malfunction is detected. An automated warning will be sent, in case a problem is detected. Soon an SMS admin notification service will be integrated to the system.