Troubleshooting tools

Trace session functionality is an efficient tool that allows user sessions to be traced and simplifies troubleshooting for administrators.

The key features of trace session functionality are:

• A single page for session search and display;
• Merged SIP and billing logs to provide an entire overview of the session flow;
• General information about a session in the form of a diagram between the main participants (e.g., SIP phones and PortaSwitch) with the issue description and troubleshooting instructions for the most common issues; and
• The ability to switch to the advanced log view if deeper investigation is required (e.g., to review the call flow among nodes within the PortaSwitch cluster).

For instance, on a call with an administrator, John Doe (12103355787) wants to find out why he cannot call 18660333777. The administrator opens the trace session page and searches the session using the account number and time interval. On the log viewer page, he sees that John Doe is not allowed to call the destination because it is forbidden.

For backward compatibility, an administrator can access billing logs on the old web interface by clicking the Old UI icon. This feature drastically simplifies troubleshooting and reduces the load on your administrative staff. Please refer to the Troubleshooting section of the Unified PortaSwitch handbook collection for more troubleshooting tips.

Administrators can trace the events that have affected a customer’s balance, status, and invoice calculation using the Audit log in PortaBilling. The records have extended descriptions, e.g., “Unallocated payments changed from 10.00 to 70.00 (increased by 60.00). Payments source – from customer payment transaction (xDR id 2204).” This helps administrators find the root of the issue faster when dealing with inquiries from customers.

Let’s say, a customer John Doe receives an invoice for September with an amount of $100 due. John calls the customer service manager to dispute the September invoice. During the call, the customer service manager agrees to reduce the amount due by $30. In two weeks, John contacts the customer service manager again to clarify why he received a notification about an overdue invoice even though he paid in full.

To investigate the issue, the administrator opens Customer > Audit log and filters the records made in October. He finds the record that the invoice was adjusted by $30 and the balance after adjustment was $70. Also, he sees that a $60 payment was made, so the amount due is now $10. No payments were received after that and the invoice status was changed from “Partially paid” to “Overdue”. The administrator decides to check whether the invoice was adjusted correctly. He goes to the CRM system and finds a note about the disputed invoice, confirming that $30 is the adjustment amount that was actually agreed upon with the customer. The administrator calls back the customer to explain that his payment didn’t cover the adjusted invoice amount, so John still needs to pay $10 to cover the invoice in full.

By default, audit logs are stored for 180 days in the Elasticsearch storage. You can change the default storage period on the Configuration server.

1. Open ClusterSuite > Web Cluster > Global environment > Tasks group.
2. In the Log_Max_Age option, define a new period in days. The minimum number of days is 30.
3. Click Save > Verify > Check/Apply.

Benefits

• Administrators can troubleshoot invoice-related issues faster via the PortaBilling web interface.

Logs of periodic tasks such as invoice calculation, taxation, periodic payments, reports are available via the Grafana web interface. Grafana is an open-source platform for data visualization, monitoring, and analysis. With the integration with Grafana, service providers’ engineers can access the information from text log files, potentially located across several servers, via one web interface.

Grafana is a single entry-point for displaying the periodic task logs (taskstack.log/task_queue.log files) that are collected from all servers and stored in Elasticsearch in JSON format.

Currently, the integration with Grafana allows using a simple dashboard to cover the most common troubleshooting requests: filtering logs, e.g., by customer or environment, reviewing them, and downloading logs in .txt format to submit them to support.

Also, service providers can create their own dashboards, e.g., graphs demonstrating the performance of specific tasks.

Grafana must be deployed on premises. To configure Grafana on your PortaSwitch, contact PortaOne support.

Benefits

• Service providers’ engineers can troubleshoot issues related to periodic tasks faster.

The External System Provisioning Framework (ESPF) captures changes in customer/account configuration in PortaBilling and sends provisioning events to an external application (for example, an IPTV platform). After the external application updates the entity’s configuration, it returns the event provisioning status to PortaBilling via the ESPF API or replies with HTTP Status Codes (e.g., “404 Not Found”, or “500 Internal Server Error.”) An administrator can see the provisioning status of events (e.g., all, failed, queued, successful) and check event logs directly in the PortaBilling web interface for these entities:

• Account
• Customer
• Access policy
• Account Add-on Product
• Account commitment
• Connection
• CPE
• CPE profile
• DID number
• Node
• Product
• SIM card

Thus, if the provisioning to an external application fails, an administrator can check the web logs directly on the web interface, instead of accessing logs through a command line interface. This saves time and helps solve the issues faster. After solving the issues, the administrator can re-start the provisioning in one click, right from the web interface.

This is how it works:

Let’s say a service provider needs to provision the information about customers and accounts from PortaBilling to an external application (for example, the IPTV platform). There is an account, John Doe, in PortaBilling. John Doe is assigned a product – Standard TV. The administrator changes John Doe’s product from Standard TV to Premium TV in PortaBilling. These changes are recorded as a provisioning event and sent by the ESPF to the IPTV platform. The IPTV platform receives the provisioning event that contains John Doe’s unique account ID and the information that the product has changed. The IPTV platform then attempts to change John Doe’s product from Standard TV to Premium TV. The product change fails. The external application sends the failed provisioning status to PortaBilling.

The administrator checks the logs on the External provisioning panel on the PortaBilling web interface and sees that provisioning failed because incorrect credentials were used in the API request. The administrator reports the issue to their colleague, who sets the correct credentials on the IPTV platform side. Then the administrator clicks the Re-run button on the PortaBilling web interface to re-provision the event.

As a result, the IPTV platform successfully changes John Doe’s product from Standard TV to Premium TV and sends the successful provisioning status to PortaBilling.

Let’s say a customer service representative (CSR) uses the service provider’s CRM system, which then communicates with PortaBilling via API. The service provider has a new customer, Jerry Price.

The information about this customer needs to be provisioned to PortaBilling and to an external application (for example, the IPTV platform).

To do this, the CSR creates a new customer record for Jerry Price in the CRM system, which provisions this information to PortaBilling. The customer creation is recorded in PortaBilling as a provisioning event and is then sent by the ESPF to the IPTV platform. The IPTV platform fails to create a customer record because it ran out of disc space so it replies with a "500 Internal Server Error" message to PortaBilling.

The CRM system requests the customer provisioning status from PortaBilling via API using the GET/espf/v1/logs method and displays this status on the web interface. The CSR sees that the event provisioning failed and asks their colleague to check the logs on the IPTV platform side. The logs shows that the provisioning failed due to an issue with the disc space. Once the disc space is freed up, the CSR clicks the button on the CRM system’s web interface to re-provision the customer record via the API.

As a result, the customer record for Jerry Price is successfully created in the IPTV platform.

Find the detailed description on how to display the status of a provisioning event for ESPF in the “How to” chapter of the PortaSwitch Configuration server web reference section.

Benefits

• The administrator can control the provisioning of events from PortaBilling web interface to external applications.
• The administrator saves time that might be spent on troubleshooting.

You can emulate calls right from the PortaBilling web interface to find out what privacy/identity headers will be passed to a vendor.

You may need to pass the verified caller identity information to vendors in specific INVITE message headers, depending on the local regulations, the equipment capabilities of the vendor, and so on. Using the call emulation tool, you can find the proper combination of PortaSwitch options on the account, customer, and connection level to comply with specific vendor’s requirements.

It’s possible to try different configurations at the SIP message emulation page and check what privacy/identity headers will be passed on in the outgoing INVITE message to a vendor. Thus, you can find the needed configuration faster and reduce time spent on troubleshooting. Once you find a suitable configuration, you can save the changed options right away.

Currently, you can emulate a call between two accounts in your system or a call from account to vendor. Since the outgoing INVITE messages are not sent for emulated calls, these tests don’t affect the regular telephony service. There’s no charging for such calls, either. Thus, you don’t need to create test entities and can make tests using existing accounts/connections. The test call logs are available along with the normal call logs.

If you need to emulate a call with specific headers that are typically received from an external endpoint, you can either add the specific headers manually one by one or copy and paste the full INVITE from a sample log.

Note that when you paste a custom initial INVITE, the SDP part must be included.

PortaSwitch supports a whole set of options that may impact the privacy/identity headers. They can be configured on the account/customer/connection level and applied via a service policy. You can change all these options for caller/callee right from the SIP message emulation page. After you test the impact of specific options on the outgoing INVITE message, you can save the changed settings for a specific entity.

Benefits

• Faster troubleshooting of identity handling issues via the PortaBilling web interface.
• Shorter time to interconnect with a new vendor.

Let’s consider an example:

GlobalNet, a new vendor of service provider Panda Telecom, requires the caller’s phone number (CLI) in at least one of the following headers: From, P-Asserted-Identity (PAI), or Remote-Party-ID (RPID). Otherwise, the calls are dropped.

Panda Telecom provides their customers with the ability to hide the phone number. Adam, an engineer at Panda Telecom, needs to make sure that in this case the caller’s phone number still will be sent to the vendor. Adam performs the following steps:

1. Opens Toolbox > SIP message emulation.
2. For Caller, selects the account ID, e.g., 16045551260, in the Account dropdown list.
3. Selects a node from the dropdown list.
4. For Callee, clicks Vendor and selects a vendor connection in the dropdown list.
5. Specifies any CLD for the call emulation in the “To” field. The CLD can be specified in the E.164 format or in a custom format according to dialing rules that are set up for the customer/account.
6. Opens the Account config tab and enables the Hide CLI option for the caller’s account.
7. Clicks Test.

Adam can see that now the account’s phone number is not included in the From header. However, the phone number is not present in either header in the outgoing INVITE message. Thus, the vendor will reject such a call.

Adam decides to set up the PAI and RPID headers, so the call will be accepted by the vendor. He opens the Connection config tab and sees that the connection is not marked as trusted (the Caller identity option is set to Do not supply). Adam selects Supply instead and clicks Test. Adam can see that now the account’s phone number is included in the PAI and RPID headers. The resulting INVITE message complies with the vendor’s requirements.

Adam wants to save the updated configuration for the connection right away. On the Connection config tab, he clicks Save configs > Save.

1. If you save the configuration for a service policy, note that the changes will affect all entities this service policy is assigned to, not only the account/connection participating in the emulated call.
2. The configuration is saved on each tab separately. For example, you have changed configuration for the account and the connection. If you open the Account config tab and click Save config, only the changes made for the account are saved.
3. The dynamically matched service policies are not considered in the emulated calls. If you use a dynamically matched service policy, it may apply to calls where specific equipment is used. So, the INVITE message in a real call may differ from the emulated one. To consider the impact of a dynamically matched service policy, you can create a static service policy with the same configuration, and apply it to the tested entity.