Administrators can import CDRs to PortaBilling from various external sources by using the xDR Mediation utility. The utility extracts the CDRs from incoming files and arranges the ready for processing CDRs into collections. PortaBilling calculates the charges for these CDRs according to defined rates, and then based on this information, creates transaction records (also called xDRs). The created xDRs are then associated with corresponding accounts and vendors.
Service providers face the necessity to import CDRs into PortaBilling when their telecom partners deliver CDRs to them in one of several file formats or when the legacy equipment used does not support RADIUS and can only save CDRs to files. The ability to import CDRs in such cases helps to ensure accounting and reporting integrity. It eases the limitations on the equipment that can be used and allows the service provider to extend the pool of potential business partners.
xDR Mediation capabilities
- CDRs can be imported to PortaBilling from .csv, fixed-width, TAP3 files, and from ASN1 files generated by Huawei MSoftX3000.
- CDRs can be imported in chronological order based on a time stamp. This is the time when a billing event takes place (e.g., the connect time for a voice call).
- It is possible to import CDRs for any types of services, for instance, voice calls, mobile and broadband data transfer or messaging.
- PortaBilling calculates the charges for imported CDRs.
- Service providers may import xDRs that already contain charged amounts to PortaBilling. In this case, import can be configured to forward these amounts to PortaBilling as they are. Then PortaBilling can apply an extra charge to the corresponding XDRs, if required.
- Using Perl code, an administrator can define tailored data manipulation rules to apply to each type of source data. Created rules may, for example, translate CLD into E.164 format. Complex transformations can be executed as a custom module.
- Several partial CDRs can be merged into a single logical CDR or one CDR can be split into several (i.e. to create separate call records for calling and called parties when calls are made within the same cloud PBX).
- An administrator can check in real time whether CDRs are imported and rated successfully. In case there are issues with any of them, the administrator can adjust the extraction or rating configuration and re-run the process.
xDR mediation process
Let’s have a look at the xDR import process step by step:
- The input file with CDRs is saved to the incoming folder (1). When several files are added to the folder they are automatically placed in an alphanumerical order.
- The queue manager that constantly monitors the incoming folder sees the file, checks that it is not a duplicate and queues it for processing.
- Once queued, the Extraction utility creates an empty CDR collection – a set of CDR records from one or several files – in the interim data storage (referred to as collections’ storage).
- The Extraction utility reads the file content and parses the data records (2). When parsed, each CDR is represented as a set of attribute-value pairs.
- The Extraction utility moves the processed file to the archive location and (optionally) compresses it (3).
- Then the Extraction utility performs the required data manipulation on the parsed CDRs according to the rules defined by the administrator (4).
User-Name=18005551155 Acct-Session-Id=01010101 Calling-Station-Id=18005551155 Called-Station-Id=18005555555 Acct-Session-Time=65 h323-connect-time=02:15:03.000 UTC Mon Dec 23 2014 h323-remote-address=22.214.171.124 PortaOne-Service-Type=Voice aggregation_key=11111
- The Extraction utility checks whether any records have the same aggregation key, and if so, merges these records (5).
- When the file is processed (a delay period ends or the timer fires), the Extraction aggregates the ready-to-import CDRs and adds them to the collection (6).
- The Rating utility monitors (7) the collections’ storage.
- When the collection is updated, the Rating sends CDRs to the billing engine via RADIUS (8).
- PortaBilling authorizes the user, calculates the charges for the CDRs according to defined rates, if required, and then saves the created xDRs to the main database.
- The Rating utility receives information about the import status (success, fail) (9).
- The Rating utility updates the information in the collections’ storage (10).
- An administrator can control the import process via web interface, and in case of mistakes, adjust the extraction or rating settings and re-run the import (11).
xDR mediation modes
How you receive files with CDR records depends on the peculiarities of your business and your agreements with your carrier. For example, your carrier sends you:
- one file per day, or
- one file every 30 minutes, etc.
For this reason, you can import CDRs from one or several files to PortaBilling in different modes:
- sequential mode (the files are imported one by one) – For example, you upload three files to an incoming folder (e.g., A, B and C). The xDR Mediator processes the CDR records from each file and passes the CDRs to the billing engine for rating, file by file.Here, the Extraction utility processes the CDRs from each file and creates a separate CDR collection for each of them (e.g., collection A, collection B, etc.). The Rating utility processes the CDRs from collection A and sends them to the billing server in chronological order. The Rating utility then processes the CDRs from collection B, etc.
- consolidated mode (the data from several files is collected and then imported) – For example, the same files, A, B, and C are in the incoming folder.In this case, the Extraction utility creates a single collection for the CDR records from all the files. It processes the CDRs from each file and waits for 60 minutes by default. At the end of this time, the Extraction utility updates the collection
and places ready-to-import CDRs to be rated. The Rating utility processes the CDRs from the collection and sends them to the billing engine in chronological order.
Both sequential and consolidated modes enable you to import CDRs in chronological order. These modes require the use of one Extraction utility and one Rating utility. The number of simultaneous flows depends on the productivity of your system.
- random mode (several files are simultaneously processed) – For example, some files (e.g., A, B, C and D) are in the folder. The xDR Mediator processes the A, B, and C files at the same time and passes the CDRs to the billing engine for rating.In this case, several Extraction and Rating utilities operate with source files. It is difficult to predict the order of data import since the size of the files and their process time may differ.
Auto-download of xDR files from external systems
The xDR mediator has built-in functionality for communicating with external systems (IMS, third-party gateways, etc.) to receive CDR source files via the FTP/SFTP protocols.
The xDR mediator can operate as either an FTP client or an FTP server.
When it operates as an FTP client, it connects to the external system and downloads CDR source files via FTP/SFTP. To make this happen, the administrator defines the credentials necessary for accessing the external system as well as the path to the folder from which to download the files.
When operating as an FTP server, the procedure is reversed. The administrator configures the entry point to the xDR mediator for the external system. The administrator creates a special user for the file transfer. The external system uses this user’s credentials to access the xDR mediator and upload the CDR source files to the specified folder.
FTP file transfer (inbound or outbound) is the recommended configuration compared to other ones (e.g., via SCP or ssh access) since it increases system security and simplifies xDR mediator support through system updates.
Retroactive billing during xDR import
To calculate charges for a billing session, PortaBilling must consider the delay between the session’s effective time – the time when the session starts/is connected, and the charging time – the time actually charged. For real-time charging this delay is very short (only a couple of seconds). When it comes to xDR import, the delay can be considerable – several hours or even days (e.g., roaming xDRs can arrive several days after actual service usage). In the latter case, the billing periods that these charges and volume discount consumption refer to can already be closed.
Therefore, to ensure correct charging for a closed billing period, PortaBilling retroactively bills during an xDR import and uses the tariff applicable at the time the session started to produce the xDRs. The xDRs are then added to the open billing period.
Volume discount usage is withdrawn from the closed billing period’s counters.
Consider the following example:
Let’s say John Doe has a “100 free roaming minutes” monthly quota with one rollover. Throughout November he makes calls for 90 minutes in roaming and on December 1 his billing period closes. Thus, 10 minutes remain from the November quota.
On December 2nd the roaming vendor sends you CDRs with John’s service usage for November 30, which includes 5 minutes of calls. Upon xDR import, these charges are applied to John’s November billing period and his quota counters for November are adjusted by 5 minutes. When John makes calls on December 3, his December quota now includes 105 minutes (5 minutes rolled over from November).