xDR import configuration

The xDR Mediation utility allows CDRs from various external sources to be imported into PortaBilling, which bills them as if they were regular calls or messages being sent by a remote gateway.

If source files already contain charged amounts (e.g., calculated by the carrier or some outsource billing agency), simply import them to PortaBilling so that these charges will be included in customer invoices.

This utility is useful for service providers who operate in mobile networks (2.5/3G) and for legacy equipment owners whose gateways do not support RADIUS and can therefore only save xDRs to files.

The xDR Mediation utility consists of three main components:

• CDR Extraction, which parses input data from source files, converts it and then creates CDR collections,
• Elasticsearch as the interim data storage for CDR collections, and
• CDR Rating, which communicates with the billing engine via RADIUS. It sends CDR collection data for processing and receives notifications about the import status.

Though the xDR Mediation utility is typically used to import CDR records for such common services as voice calls, SMS and data transfer, you can also use it to import and bill customers for IPTV or other quantity-based services (e.g., pizza-delivery orders).

The xDR Mediation utility can process the following source data formats:

• CSV,
• fixed-width,
• TAP3,
• Generic XML (adjustments to specific structure of XML document may be required),
• Binary ASN1 files generated by Huawei MSoftX3000.

This document provides instructions on how to configure the import of xDRs from the .csv format. For detailed description how to import xDRs in other formats, please refer to the How to… handbook.

Use this checklist to check off the operations you have completed while performing the system setup according to the instructions in this chapter. Please be sure to perform all of the operations in the order designated (all of the boxes should be checked), otherwise, the service will not work.

Prior to configuring the xDR Mediation utility, obtain the file that contains examples of xDRs to be imported to PortaBilling from your carrier. Carriers can present CDRs in their own manner: storing CLI/CLD in the local format, having a different column order, providing several records applicable to the same session, etc.

If necessary, the xDR Mediator module should be implemented to correctly parse and transform the data for further processing.

Let’s say you are a service provider operating on the CDMA network. You provide SIP on-net calls to your customers via PortaSIP while off-net calls are terminated by your carrier TATA Telecom. TATA Telecom sends you CDRs that you need to import to PortaBilling.

The .csv file from TATA Telecom has the following structure:

To automate the CDR retrieval process, you configure the xDR mediator to automatically download the source CDR files from TATA Telecom via the FTP protocol.

Note that you require the credentials to access TATA telecom system and the path to the directory from where to download CDR files.

1. Go to the Configurations tab on the Configuration server.
2. Clone the existing configuration.
3. From the Configuration tree, select Auxiliaries ->CDR Mediation -> CDR Extraction and click the  Instance Create icon.

   

4. Fill in the Instance create form:
   • Server – select one of your servers. The one recommended is the web server.
   • Service IP – select one of the web server’s IP addresses. You may utilize a private IP address for better security (e.g., to avoid possible brute force attacks). Please note that the CDR Extraction instance cannot have the same IP address as the RADIUS instance.
   • Click Save.

     

5. In the CDRSource group specify the following parameters:
   • type – select the CDR source file type. Leave .CSV here.
   • csv_field_separator – this is the separator to use when splitting .CSV file lines into columns.
   • column_list – specify the header names for the .CSV file. Type in the following string: User-Name,Calling-Station-Id,Called-Station-Id,Acct-Session-Time,h323-connect-time,h323-disconnect-time,PortaOne-Service-Type,h323-remote-id,NAS-IP-Address. Please refer to the Mandatory attributes for xDR import section for the list of attributes required for a particular service.
     The header order is important. Make sure the names you define directly correspond to the column order in the source .CSV file.

      

     You must use the actual separator here. For example, if your field separator is a “|” (pipe) and not a comma, the value will appear as follows:

     User-Name|Calling-Station-Id|Called-Station-Id|Acct-Session-Time|h323-connect-time|h323-disconnect-time| PortaOne-Service-Type|h323-remote-id|NAS-IP-Address

   • file_pattern – specify the file name pattern. It may contain the wildcard (e.g., abc*.csv). The xDR mediator will import only those files with names that fit this pattern.
   • files_input_directory – specify the path to the folder from where the xDR mediator will extract source files (e.g., /porta_var/upload).
   • time_zone – select the time zone in which the xDRs will be imported. However, if a particular time zone is specified in the source file, it will override the selected time zone.

     

6. Define the parameters for automatic CDR download in the CDRSourceFTPClient group:
   • enable – set Yes.
   • files_directory – specify the path to the directory with the source CDR files on the remote host.
   • host – specify the IP address or domain name of your vendor’s host. The xDR mediator will connect to it to download CDR files.
   • login, password – specify the credentials obtained from your vendor to access their host.
   • protocol – select the protocol for file download: FTP or SFTP.

     

7. In the Global group specify how many parallel processes will convert source CDRs and send CDR collections to interim data storage. Leave the default value here for now.
8. In the Output group specify the following:
   • cdr_export_delay – specify the time interval to import CDRs from several source files in chronological order (e.g., 43200 seconds meaning 12 hours). This defines how long an extractor waits before adding CDRs to the collection for rating.

     The period starts when the first CDR is parsed. When the timer fires, the extractor aggregates the ready-to-import CDRs and adds them to the collection. Then a new period starts. The default value is 0 seconds meaning that processed CDRs from a source file are immediately added to the collection.

     The delay period is applied to complete CDRs only. If you configure the Extractor to merge several CDRs into one, the delay period is skipped and the value from the max_wait_incomplete_cdr is considered.
   • max_wait_incomplete_cdr – this defines how long an extractor waits before adding incomplete CDRs to the collection. The default value is 3600 seconds. This value is applied if you configure the Extractor to merge several CDRs into a single xDR. Please find details about this in the merge several CDRs into a single xDR record section of the How to… handbook.
   • min_freq – this defines how often the ready-to-import CDRs must be added to the collection when processing a large source file.
     When configured, the value from the delay period overrides this value.
   • name_pattern – this specifies the pattern for the collection’s name. If left empty, the system names the collection based on the date and sequential number (e.g., cdr2018-07-22 or cdr20170801_0745_1662100001621_0001).
   • separate_collections – this selects how the Extractor creates a new collection:
     • per file – a separate collection is created for each file in the input folder.
     • per day – a single collection is created for files that appear in the input folder during a current day.
     • per export – a separate collection is created for all the files in the input folder.

TIP: You may define data transformation rules that will be applied to the parsed data from the source CDR files. To do this, create your custom Perl module and define its name and the path to it in the DataTransformation group. Please refer to the Data transformation module for xDR import handbook for details.

1. From the Configuration tree, select Auxiliaries > CDR Mediation > CDR Rating and click the  Instance Create icon.

   

2. Fill in the Instance create form:
   • Server – select one of your servers. You can use the same server where the CDR-Extraction instance is located.
   • Service IP – select one of the server’s IP addresses. Please note that the CDR Rating instance cannot have the same IP address as the RADIUS instance.
   • Click Save.

     

3. In the Global group define how many parallel processes will run to process the CDR collections. Leave the default value here.

These attributes must be present when importing CDRs for any type of services:

Define these attributes in addition to the global ones when importing CDRs for voice call service:

Define these attributes in addition to the global ones when importing CDRs for messaging service:

Define these attributes in addition to the global ones when importing CDRs for Internet Access service:

Define these attributes in addition to the global ones when importing CDRs for quantity-based service:

Define these attributes when importing CDRs for the IPTV service:

If you have just installed the PortaSwitch software or just dedicated a new billing environment to configure the services described in this handbook, make sure to first perform the initial configuration of PortaSwitch. To do this, use the PortaSwitch initial configuration handbook.

Now create a node that represents the xDR mediator.

1. On the navigation menu, select Infrastructure, then Billing data sources, and click Nodes.
2. On the Create node panel, fill in the node details:
   • Name – type a short descriptive name for your SIP server (that will be used in the lists).
   • Node ID – enter the IP address assigned to the CDR rating instance on the Configuration server web interface.
   • IP – this is the IP address assigned to the CDR rating instance on the Configuration server web interface.
   • Manufacturer – select PortaOne.
   • Type – select Generic.

     

3. Click Save.

The tariff is a single price list of your calling services or your termination costs. PortaBilling requires that every call be accounted for, both on the revenue side (customers) and on the cost side (vendors). Therefore, the tariff is used to calculate your costs.

Since xDRs are imported into PortaBilling for sessions already completed and the vendor must not participate in call routing, the Routing option must be disabled for the vendor tariff.

1. On the navigation menu to the left, select Service catalog and click Tariffs.
2. On the Create tariff panel, fill in the tariff details:
   • Name – type a short name for the tariff object; this is the name you will see in the select menus (for example, GlobalNet Termination).
   • Currency – choose the currency in which the vendor charges you.
     The currency for the tariff may be chosen only once, and cannot be changed later.
   • Service – choose Voice calls here.
   • Applied to – choose Vendor in the Applied to list.
   • Routing – disable the Routing option.

     

3. Click Save.

Rates are the per-destination prices. Please refer to the Call billing parameters section for more information on billing parameters.

Depending on whether or not you control the vendor costs in PortaBilling by creating vendor xDRs, specify the rates for the following destinations in the tariff:

• Enter the symbolic destination “|” (pipe) and specify a zero price for it if you do not control vendor costs, or
• Upload the rates for geographical destinations according to the tariff sheet provided by the carrier.

Please consult the Rate import section for more details.

Vendors are your termination partners. PortaBilling requires that every call be accounted for on both the revenue side (customers) and the cost side (vendors). In the case of calls going via the mobile carrier’s network, no vendor (in the traditional sense) is involved – so you still need to create a vendor that will be used to keep the xDRs for these calls. This vendor and the connections to the vendor are required in order to properly bill for calls.

1. In the left upper corner click  to open the navigation menu.
2. On the navigation menu, select Infrastructure, then select Vendors.
3. On the Create vendor panel, fill in the vendor details:
   • Name – type a short name for the vendor object; this will be used on the web interface (for example, TATA Telecom).
   • Currency – choose the currency in which this vendor charges you.
   • Opening balance – this indicates a starting balance for the vendor; the default is zero.
   • Billing period – split period for vendor statistics.

     

4. Click Save.
5. Fill in the information about the vendor as described in the Basic residential VoIP service handbook.

A connection represents the point from which calls leave or enter your network and/or are directed to or from vendors when charges are incurred.

1. While on the vendor’s panel, click Connections.
2. On the Create connection panel, fill in the connection details:
   • Description – type a descriptive name for this connection. It will be displayed in the list of connections.
   • Service type – select Voice Calls.
   • Type of connections – select SIP.
   • Direction – select To vendor.
   • Tariff – choose the tariff that defines your termination costs for this connection/vendor.
   • Active – use the slider to set this connection as inactive.
   • In the Identify gateway by ID option select the Gateway ID and specify the ID of the vendor’s gateway or switch. Type in CDRIMPORTER.
   • Capacity – define the capacity value for this connection.

     

3. Click Save.

To charge your customers for making calls create the tariffs and define the rates within.

Please refer to the Create customer tariffs and Enter rates sections of the Basic residential VoIP service handbook for guidelines about tariff creation.

 TIP: To import CDRs for incoming calls, do the following:

• Make sure source CDRs are clearly defined as incoming and outgoing ones.
• Define the PortaBilling_AccessCode attribute in the column_list option for the CDR extraction instance and create an appropriate data transformation rule for it. (e.g., The CDR source file contains the CallType column with an IN value for incoming calls and an OUT value for outgoing calls. Then the data transformation rule will be

  PortaBilling_Access_Code=<<ACC
  
    if ($data{CallType} eq 'IN') {
  
      return 'INCOMING';}
  
  elsif ($data{CallType} eq 'OUT') {
  
      return 'OUTGOING';}
  
  else {
  
      return undef;
  
    }
  
  ACC)

• Create a tariff to charge users for incoming calls and include it into the product rating list.

End users will use accounts issued for specific products to access the services you provide. Products are powerful tools that define different ways to bill an account for one or several included services. Product definition is always realized through these steps: product definition, service definition and configuration and the creation of a rating list.

1. On the navigation menu, select Service catalog and click Products.
2. On the Create product panel, fill in the product details:
   • Name – type an internal product name that will be shown on the administrator interface.
   • Name visible to end users – type a name of the product that will be shown to end users on their self-care interfaces.
   • Product type – select Main product here.
   • Currency – choose a currency the product will be priced in.
   • Managed by – select Administrator only here, since we are setting up a service without the involvement of resellers.
   • Account default ACL – choose an Access Control List (ACL) for accounts with this product assigned. ACLs control which objects end users can access to and which actions they can perform.
   • Account role – select Mobile from the list since this product is intended to be used by mobile subscribers.

     

Define which service types are included in the product. A service type is a description of the physical service provided to end users.

To add a service type:

1. On your product’s panel, click Services.
2. On the Services panel, click Add a service.
3. In the Select services to add dialog box, select Voice calls and click Add.

   

The rating list has two functions: it defines the permitted access points (nodes and access numbers) and specifies which tariff must be used for billing each of these points. Now you create the rating entries to handle outgoing calls.

1. On your product’s panel, click Charges, then click Usage charges.
2. On the Usage charges panel, click Add.
3. Fill in the required information:
   • Node – select the node you created for the xDR importer.
   • Access code – leave this field empty.
   • Service – choose Voice calls.
   • Tariff – select the tariff that will be used to calculate charges for outgoing calls.
   • Click Save.

     

4.  Click Save.

Proceed with the customer configuration by performing the Enter SIM cards into the SIM card inventory, Create invoice template, Configure taxation, Create a customer class, Create a customer and Create accounts steps from the MVNO service provisioning handbook.

1. Log in to the server with the xDR mediator configured (in our example – the web server) using ssh. (For how to configure the ssh access for your user, please refer to the Users section of the PortaSwitch Configuration server web reference guide.)
2. Upload the source CDR file to the folder you defined for the xDR mediator configuration by using the following command:

   sudo scp user@remote.host: /full_path_to_source_file/CDRs.csv /porta_var/upload/CDRs.csv
   
   user@remote.host password:

PortaBilling enables you to control the xDR import process and adjust the configuration if any issues occur.

1. Go to the xDR mediation panel.

   

2. You can see the most recent xDR collections on the xDR Mediation page. View the following information:
   • Status – this is the current status of the xDR collection.
     • In Queue – the collection is added to the import queue by the xDR Rating utility.
     • Processing – the collection is being processed by PortaBilling.
     • Processed – the collection was already processed by PortaBilling.
   • xDR collection name – this shows the name of the xDR collection created after parsing source file data.
   • Added at – this is the date and time when the collection was added to the import queue.
   • Finished at – this is the date and time when the collection was processed.
   • Total xDR number, Processed, Rejected – this is the information about xDR record numbers and status within the collection.

     

3. Click on the xDRs collection name to view the list of xDRs within the collection. View the xDRs status and charges calculated for the customer and the vendor. Successfully imported xDRs are shown in the corresponding tab. Click on the other tabs to view the xDRs that were rejected due to a specific error.
4. Use filters on the xDRs search within collection panel to search for particular xDRs.
5. Click the  Details icon to see detailed information about a specific xDR.
6. Click the  Log icon to check log files for a particular xDR record with the Log Viewer.