Data transformation module for xDR import

Defining data transformation rules and storing them inside a local conf file is workable when the source file requires only trivial modification (e.g., converting cents amounts into dollars or stripping initial 00s from the CLD, etc.).

If you deal with complex data transformation rules that contain multiple subroutines, work with external modules, etc. plus look like mini-programs, writing data transformation rules within a conf file and managing the file itself can be a difficult task.

Therefore, it is recommended that you create custom modules and keep such complex data transformation rules within them. When creating a module, you enjoy benefits such as syntax highlight or auto-indentation. You also have advanced flexibility for working with files similar in nature – so that instead of copying & pasting existing code pieces you simply create a new class that extends or overrides only some data transformation rules for the base class.

The module interface requires the following methods:

• get_fields – returns the list of fields your module calculates;
• splitter – splits a single input record into multiple ones. (If you use Stub class and do not need record splits – then you do not have to do anything, Stub already defines an empty splitter method).

The method returns a reference to an array where odd elements are field names and even ones are field definitions created by the field method from the Porta::CDR_Import::DataTransformer::OO::Stub class based on supplied parameters.

 The parameters for the field method are:

• method – a string with a method name or a coderef. If a coderef is used, then this exact piece of code is invoked. If a method name is provided, then it is invoked as an object method (so potentially a method from the subclass can be invoked).
• required – a boolean. If the required field is evaluated to undef, then the record is considered to be “skipped,” and therefore no further evaluation is done.

The method returns undef if no split is required. Otherwise it must return a reference to an array, where each array element is a reference to a new individual record (hash).

When creating a module it is highly recommended that you use the Mouse Perl extension and have your module extend the Porta::CDR_Import::DataTransformer::OO::Stub class.

package CDR_Convertor::MyModule;

use Mouse;
extends 'Porta::CDR_Import::DataTransformer::OO::Stub';

 sub get_fields {
    my ($self) = @_;
    return [
        'PortaOne-Service-Type' => $self->constant('Voice'),
        'h323-remote-address' => $self->constant('55.66.77.88'),
        'User-Name' => $self->remap_field('customer_ID', 1),
        'h323-conf-id' => $self->remap_field('CallID'),
        'Calling-Station-Id' => $self->field( method => 'cli'),
        'Called-Station-Id' => $self->field( method => \&cld ),
        'h323-connect-time' => $self->field( method => ‘mergeDateTime’,
             required => 1
            ),
        ‘Acct-Session-Time’ => $self->field( method => ‘duration’ )
     ];
}
no Mouse;

1;

When defining fields and declaring rules for data transformation use the following methods.

When possible, use subroutines when composing Perl expressions:

sub { some code; }

A Perl subroutine code receives two parameters: object reference and hash reference (input data).

The fields in the hash are evaluated in the order in which they are provided by the get_fields method. The hash contains all fields previously calculated. It is not recommended that you alter the values there directly, since such assignments may not be returned in the final record or there may be other side effects. The only correct way to modify a field value is to return a new value from its method:

sub cld {
    my ($self, $data) = @_;
    return $data->{B}.$data->{C};
}

A subroutine returns scalar (a new value for the field) or undef (in this case, the field is removed from the final record).

You will almost always have some required field (e.g., PortaOne-Service-Type) that is absent in the input file. You need to define this field and populate it with a constant value. To do this, use the constant method:

 'PortaOne-Service-Type' => $self->constant('Voice'),

The h323-connect-time field is mandatory for importing voice call CDRs, while in the CDR file, the column is named Call Time. You can, of course, rename the column in the file definition – but then that makes troubleshooting difficult, since you need to communicate it to your vendor who sends you your source CDRs – and they still call the field Call Time.

A more elegant solution is to say that h323-connect-time equals Call Time.

'h323-connect-time' => $self->remap_field('Call time'),

 And you can define it as mandatory:

'h323-connect-time' => $self->remap_field('Call time', 1)

Assume the CLD field of the original .csv file contains CLDs. Before forwarding the field values as Called-Station-Id, you want to replace the two leading zeros with 44. You can use the following code for this:

sub get_fields {
    my ($self) = @_;
    return [
‘Called-Station-Id’ => $self->field( method => 'CLD'),
];
}

sub CLD {
    my ($self, $data) = @_;
    my $cld = $data->{CLD};
    $cld =~ s/^0{0,2}/44/;
    return $cld;
}

Suppose that the CLI column in the original .csv file contains calling party numbers. The script checks every cell in the column and if there are empty cells, it returns an unknown value and provides the reason why.

To do this you can use the following code:

sub cli {
    my ($self, $data) = @_;
    my $x = $data->{B};
    if (!defined($x) || $x eq '') {
        $self->log_warn("Empty CLI received");
        $x = 'unknown';
    }   
    return $x;
}

You can define a subroutine without naming it. It will be called the same way as the one already named:

'h323-conf-id' => $self->field(
            method => sub { return rand(); }
           ),

To request a subroutine defined in another package you can simply refer to it. The script will then invoke that object method. Use the following code to define a subroutine reference:

'h323-remote-address' => $self->field(
            method => \&remote         
           )
       ];

If classes inside your module do not undergo changes, you can speed up their execution with the following piece of code:

__PACKAGE__->meta->make_immutable;

Keep your modules in a separate directory and define the module name and the path to it in the Data Transformation group on the Configuration server web interface.

To maintain your specific custom modules through software upgrades, add them to Deposit Files via the Configuration server web interface.