The xDR Mediation utility allows CDRs from various external sources to be imported into PortaBilling, which bills them as if they were regular calls or messages being sent by a remote gateway.
If source files already contain charged amounts (e.g., calculated by the carrier or some outsource billing agency), simply import them to PortaBilling so that these charges will be included in customer invoices.
This utility is useful for service providers who operate in mobile networks (2.5/3G) and for legacy equipment owners whose gateways do not support RADIUS and can therefore only save xDRs to files.
The xDR Mediation utility consists of three main components:
- CDR Extraction, which parses input data from source files, converts it and then creates CDR collections,
- Opensearch as the interim data storage for CDR collections, and
- CDR Rating, which communicates with the billing engine via RADIUS. It sends CDR collection data for processing and receives notifications about the import status.
Though the xDR Mediation utility is typically used to import CDR records for such common services as voice calls, SMS and data transfer, you can also use it to import and bill customers for IPTV or other quantity-based services (e.g., pizza-delivery orders).
The xDR Mediation utility can process the source data formats such as CSV, fixed-width, TAP3, Generic XML (adjustments to specific structure of XML document may be required), Binary ASN1 files generated by Huawei MSoftX3000, video on demand (VOD).This document provides instructions on how to configure the import of xDRs from the .csv format.
Obtain vendor CDR samples
In this handbook, we assume that you have already obtained a file with the CDR sample from your vendor. The sample shows how CDRs received from the vendor are presented, including storing CLI/CLD in the local format, having a specific column order, providing several records applicable to the same session, etc.
If necessary, the custom data transformation module for xDR Mediator should be implemented to correctly parse and transform the data for further processing.
Usage scenario
Let’s say you are a service provider operating on the CDMA network. You provide SIP on-net calls to your customers via PortaSIP while off-net calls are terminated by your carrier TATA Telecom. TATA Telecom sends you CDRs that you need to import to PortaBilling.
The .csv file from TATA Telecom has the following structure:
To automate the CDR retrieval process, you configure the xDR mediator to automatically download the source CDR files from TATA Telecom via the FTP protocol.
Note that you require the credentials to access TATA telecom system and the path to the directory from where to download CDR files.
Configuration overview
- Setup on the Configuration web server interface
- Add the CDR extraction instance
- Add the CDR rating instance
- Initial configuration of PortaSwitch
- Add the OpenSearch instance
- Create a node
- Configure MNO as a vendor:
- Create a vendor tariff
- Create a vendor
- Define a vendor connection
- Set up the service package:
- Create a customer tariff
- Create a product
- Test the xDR import
1. Setup on the Configuration web server interface
1.1 Add the CDR extraction instance
- Go to the Configurations tab on the Configuration server.
- Clone the existing configuration.
- From the Configuration tree, select Auxiliaries ->CDR Mediation -> CDR Extraction and click the
Instance Create icon.
- Fill in the Instance create form:
- Server – select one of your servers. The one recommended is the web server.
- Service IP – select one of the web server’s IP addresses. You may utilize a private IP address for better security (e.g., to avoid possible brute force attacks). Please note that the CDR Extraction instance cannot have the same IP address as the RADIUS instance.
- Click Save.
- In the CDRSource group specify the following parameters:
- type – select the CDR source file type. Leave .CSV here.
- csv_field_separator – this is the separator to use when splitting .CSV file lines into columns.
- column_list – specify the header names for the .CSV file. Type in the following string: User-Name,Calling-Station-Id,Called-Station-Id,Acct-Session-Time,h323-connect-time,h323-disconnect-time,h323-remote-id,NAS-IP-Address. Please refer to the Mandatory attributes for xDR import section for the list of attributes required for a particular service.
The header order is important. Make sure the names you define directly correspond to the column order in the source .CSV file.
You must use the actual separator here. For example, if your field separator is a “|” (pipe) and not a comma, the value will appear as follows:
User-Name|Calling-Station-Id|Called-Station-Id|Acct-Session-Time|h323-connect-time|h323-disconnect-time|h323-remote-id|NAS-IP-Address
- file_pattern – specify the file name pattern. It may contain the wildcard (e.g., abc*.csv). The xDR mediator will import only those files with names that fit this pattern.
- files_input_directory – specify the path to the folder from where the xDR mediator will extract source files (e.g., /porta_var/upload).
- time_zone – select the time zone in which the xDRs will be imported. However, if a particular time zone is specified in the source file, it will override the selected time zone.
- Define the parameters for automatic CDR download in the CDRSourceFTPClient group:
- enable – set Yes.
- files_directory – specify the path to the directory with the source CDR files on the remote host.
- host – specify the IP address or domain name of your vendor’s host. The xDR mediator will connect to it to download CDR files.
- login, password – specify the credentials obtained from your vendor to access their host.
- protocol – select the protocol for file download: FTP or SFTP.
- In the Global group specify how many parallel processes will convert source CDRs and send CDR collections to interim data storage. Leave the default value here for now.
- In the Output group specify the following:
- cdr_export_delay – specify the time interval to import CDRs from several source files in chronological order (e.g., 43200 seconds meaning 12 hours). This defines how long an extractor waits before adding CDRs to the collection for rating.
The period starts when the first CDR is parsed. When the timer fires, the extractor aggregates the ready-to-import CDRs and adds them to the collection. Then a new period starts. The default value is 0 seconds meaning that processed CDRs from a source file are immediately added to the collection.
The delay period is applied to complete CDRs only. If you configure the Extractor to merge several CDRs into one, the delay period is skipped and the value from the max_wait_incomplete_cdr is considered. - max_wait_incomplete_cdr – this defines how long an extractor waits before adding incomplete CDRs to the collection. The default value is 3600 seconds. This value is applied if you configure the Extractor to merge several CDRs into a single xDR. Please find details about this in the merge several CDRs into a single xDR record section of the How to... handbook.
- min_freq – this defines how often the ready-to-import CDRs must be added to the collection when processing a large source file.
When configured, the value from the delay period overrides this value.
- name_pattern – this specifies the pattern for the collection’s name. If left empty, the system names the collection based on the date and sequential number (e.g., cdr2018-07-22 or cdr20170801_0745_1662100001621_0001).
- separate_collections – this selects how the Extractor creates a new collection:
- per file – a separate collection is created for each file in the input folder.
- per day – a single collection is created for files that appear in the input folder during a current day.
- per export – a separate collection is created for all the files in the input folder.
- cdr_export_delay – specify the time interval to import CDRs from several source files in chronological order (e.g., 43200 seconds meaning 12 hours). This defines how long an extractor waits before adding CDRs to the collection for rating.
You may define data transformation rules that will be applied to the parsed data from the source CDR files. To do this, create your custom Perl module and define its name and the path to it in the DataTransformation group. Please refer to the Data transformation module for xDR import handbook for details.
1.2 Add the CDR rating instance
- From the Configuration tree, select Auxiliaries > CDR Mediation > CDR Rating and click the
Instance Create icon.
- Fill in the Instance create form:
- In the Global group define how many parallel processes will run to process the CDR collections. Leave the default value here.
Mandatory attributes for xDR import
These attributes must be present when importing CDRs for the services listed in the "Service type" column:
Attribute Name |
Type |
Description |
Service type |
User-Name |
String |
This determines whom to charge for the session. |
Voice, Msg, Netaccess, Quantity, IPTV |
PortaOne-Service-Type |
Voice, Msg, Netaccess, Quantity, IPTV |
This is the type of service used within a session, where:
|
Voice, Msg, Netaccess, Quantity, IPTV |
Called-Station-Id |
String |
The destination number or a special destination. |
Voice, Msg |
Event-Timestamp |
Int |
The timestamp for the session. |
Quantity, IPTV |
PortaOne-Used-Resource |
Int |
The number of units transferred. |
Quantity, IPTV |
2. Initial configuration of PortaSwitch
If you have just installed the PortaSwitch software or just dedicated a new billing environment to configure the services described in this handbook, make sure to first perform the initial configuration of PortaSwitch. To do this, use the PortaSwitch initial configuration handbook.
3. Create a node
Now create a node that represents the xDR mediator.
- On the navigation menu, select Infrastructure, then Billing data sources, and click Nodes.
- On the Create node panel, fill in the node details:
- Name – type a short descriptive name for your SIP server (that will be used in the lists).
- Node ID – enter the IP address assigned to the CDR rating instance on the Configuration server web interface.
- IP – this is the IP address assigned to the CDR rating instance on the Configuration server web interface.
- Manufacturer – select PortaOne.
- Type – select Generic.
- Click Save.
4. Configure MNO as a vendor
4.1 Create a vendor tariff
The tariff is a single price list of your calling services or your termination costs. PortaBilling requires that every call be accounted for, both on the revenue side (customers) and on the cost side (vendors). Therefore, the tariff is used to calculate your costs.
Since xDRs are imported into PortaBilling for sessions already completed and the vendor must not participate in call routing, the Routing option must be disabled for the vendor tariff.
- On the navigation menu to the left, select Service catalog and click Tariffs.
- On the Create tariff panel, fill in the tariff details:
- Name – type a short name for the tariff object; this is the name you will see in the select menus (for example, GlobalNet Termination).
- Currency – choose the currency in which the vendor charges you.
The currency for the tariff may be chosen only once, and cannot be changed later.
- Service – choose Voice calls here.
- Applied to – choose Vendor in the Applied to list.
- Routing – disable the Routing option.
- Click Save.
Depending on whether or not you control the vendor costs in PortaBilling by creating vendor xDRs, specify the rates for the following destinations in the tariff:
- Enter the symbolic destination “|” (pipe) and specify a zero price for it if you do not control vendor costs, or
- Upload the rates for geographical destinations according to the tariff sheet provided by the carrier.
Please consult the Rate import section for more details.
4.2 Create a vendor
Vendors are your termination partners. PortaBilling requires that every call be accounted for on both the revenue side (customers) and the cost side (vendors). In the case of calls going via the mobile carrier’s network, no vendor (in the traditional sense) is involved – so you still need to create a vendor that will be used to keep the xDRs for these calls. This vendor and the connections to the vendor are required in order to properly bill for calls.
- In the left upper corner click
to open the navigation menu.
- On the navigation menu, select Infrastructure, then select Vendors.
- On the Create vendor panel, fill in the vendor details:
- Name – type a short name for the vendor object; this will be used on the web interface (for example, TATA Telecom).
- Currency – choose the currency in which this vendor charges you.
- Opening balance – this indicates a starting balance for the vendor; the default is zero.
- Billing period – split period for vendor statistics.
- Click Save.
- Fill in the information about the vendor as described in the Basic residential VoIP service handbook.
4.3 Define connections
PortaBilling requires that every service be accounted for on both the revenue side (customers) and the cost side (vendors). Therefore, a tariff needs to be specified for a vendor in PortaBilling by design to store the corresponding xDRs (extended Detail Records).
The internal vendor connection and tariff will be used to manage information for the services and to track the costs owed to your vendor.
- While on the vendor’s panel, click Connections.
- On the Create connection panel, fill in the connection details depending on the service type:
Voice calls
- Description – type a short description for the connection.
- Service type – select Voice calls.
- Type of connections – select SIP.
- Direction – select To vendor.
- Tariff – select the internal vendor tariff created at step 4.1.
- Internal – leave disabled.
- Active – disable to exclude the connection from routing.
- In the Identify gateway by ID option select the Gateway ID and specify the ID of the vendor’s gateway in the Gateway ID field. Type in CDRIMPORTER.
The other possible options:
- IP – select if you want to identify the gateway by IP address, and set the IP in the Remote IP field, e.g., 1.1.1.1. Note that this IP corresponds to the h323-remote-address=1.1.1.1 attribute in the request.
- Gateway ID/IP – select if you want to identify both by IP address and ID of the vendor’s gateway or switch.
- Capacity – define the capacity value for this connection.
Messaging services
To configure a connection for messaging services, create a service policy for the Messaging services type as described in the SMS services handbook.
- Description – type a short description for the connection.
- Service type – select Messaging service.
- Type of connections – select SMPP.
- Tariff – select the internal vendor tariff created at step 4.1.
- Internal – leave disabled.
- Active – disable to exclude the connection from routing.
- Gateway ID – type SMS. Note that this ID corresponds to the h323-remote-id=SMS attribute in the request.
- Port – type a random port, e.g., 22222. Note that this port corresponds to the NAS-Port-Name=22222 attribute in the request.
- Service policy – select the created service policy.
Internet access services
- Description – type a short description for the connection.
- Service type – select Internet access.
- Tariff – select the internal vendor tariff created at step 4.1
- Node – select the name of the node from step 3.
- Capacity – define the capacity value for this connection
Quantity-base services
- Description – type a short description for the connection.
- Service type – select Quantity based.
- Tariff – select the internal vendor tariff created at step 4.1.
- Node – select the name of the node from step 3.
IPTV
To configure a connection for messaging services, create a service policy for the IPTV type as described in the IPTV services handbook.
- Description – type a short description for the connection.
- Service type – select IPTV.
- Tariff – select the internal vendor tariff created at step 4.1.
- Node – select the name of the node from step 3.
5. Set up the service package
5.1 Create customer tariff
To charge your customers for making calls create the tariffs and define the rates within.
Please refer to the Create customer tariffs and Enter rates sections of the Basic residential VoIP service handbook for guidelines about tariff creation.
To import CDRs for incoming calls, do the following:
- Make sure source CDRs are clearly defined as incoming and outgoing ones.
- Define the PortaBilling_AccessCode attribute in the column_list option for the CDR extraction instance and create an appropriate data transformation rule for it. (e.g., The CDR source file contains the CallType column with an IN value for incoming calls and an OUT value for outgoing calls. Then the data transformation rule will be
PortaBilling_Access_Code=<<ACC if ($data{CallType} eq 'IN') { return 'INCOMING';} elsif ($data{CallType} eq 'OUT') { return 'OUTGOING';} else { return undef; } ACC)
- Restart the CDR extraction service to apply changes. Use the following command to restart the CDR Extractor manually:
sudo systemctl restart cdr-extd@<cdr-extraction_instance_name>
Consider restarting the service during an off-peak time or after all the files in the queue have been processed. - Create a tariff to charge users for incoming calls and include it into the product rating list.
5.2 Create a product
End users will use accounts issued for specific products to access the services you provide. Products are powerful tools that define different ways to bill an account for one or several included services. Product definition is always realized through these steps: product definition, service definition and configuration and the creation of a rating list.
- On the navigation menu, select Service catalog and click Products.
- On the Create product panel, fill in the product details:
- Name – type an internal product name that will be shown on the administrator interface.
- Name visible to end users – type a name of the product that will be shown to end users on their self-care interfaces.
- Product type – select Main product here.
- Currency – choose a currency the product will be priced in.
- Managed by – select Administrator only here, since we are setting up a service without the involvement of resellers.
- Account default ACL – choose an Access Control List (ACL) for accounts with this product assigned. ACLs control which objects end users can access to and which actions they can perform.
- Account role – select Mobile from the list since this product is intended to be used by mobile subscribers.
Included services
Define which service types are included in the product. A service type is a description of the physical service provided to end users.
To add a service type:
- On your product’s panel, click Services.
- On the Services panel, click Add a service.
- In the Select services to add dialog box, select Voice calls and click Add.
Usage charges
The rating list has two functions: it defines the permitted access points (nodes and access numbers) and specifies which tariff must be used for billing each of these points. Now you create the rating entries to handle outgoing calls.
- On your product’s panel, click Charges, then click Usage charges.
- On the Usage charges panel, click Add.
- Fill in the required information:
- Click Save.
Proceed with the customer configuration by performing the Enter SIM cards into the SIM card inventory, Create invoice template, Configure taxation, Create a customer class, Create a customer and Create accounts steps from the MVNO service provisioning handbook.
6. Test the xDR import
6.1 Upload CDR source file
- Log in to the server with the xDR mediator configured (in our example – the web server) using ssh. (For how to configure the ssh access for your user, please refer to the Users section of the PortaSwitch Configuration server web reference guide.)
- Upload the source CDR file to the folder you defined for the xDR mediator configuration by using the following command:
sudo scp user@remote.host: /full_path_to_source_file/CDRs.csv /porta_var/upload/CDRs.csv user@remote.host password:
6.2 Verify the xDR import process
PortaBilling enables you to control the xDR import process and adjust the configuration if any issues occur.
- Go to the xDR mediation panel.
- You can see the most recent xDR collections on the xDR Mediation page. View the following information:
- Status – this is the current status of the xDR collection.
- In Queue – the collection is added to the import queue by the xDR Rating utility.
- Processing – the collection is being processed by PortaBilling.
- Processed – the collection was already processed by PortaBilling.
- xDR collection name – this shows the name of the xDR collection created after parsing source file data.
- Added at – this is the date and time when the collection was added to the import queue.
- Finished at – this is the date and time when the collection was processed.
- Total xDR number, Processed, Rejected – this is the information about xDR record numbers and status within the collection.
- Status – this is the current status of the xDR collection.
- Click on the xDRs collection name to view the list of xDRs within the collection. View the xDRs status and charges calculated for the customer and the vendor. Successfully imported xDRs are shown in the corresponding tab. Click on the other tabs to view the xDRs that were rejected due to a specific error.
- Use filters on the xDRs search within collection panel to search for particular xDRs.
- Click the
Details icon to see detailed information about a specific xDR.
- Click the
Log icon to check log files for a particular xDR record with the Log Viewer.