There is no question that revenue assurance revolves around the analysis of data. But what should you do when you want to check something, but you cannot pinpoint any data about the relevant network event? What can you do if there has been no relevant network event to check, or if you want to check somebody else’s network? One solution is to perform a test event on the network and see what happens. I spoke with Mario Margolis and Franco Primavesi of Networks Analytic Corporation (NAC) about the use of independent invasive testing in telecoms, and in particular about the recent NAC study into prepaid calling cards.
Eric: The Hispanic Institute recently asked you to test a series of international prepaid calling cards. The cards you tested are available in the U.S.A. and sold to people wanting to phone relatives in their home countries. Can you tell me about the objectives of the study?
Mario: The Hispanic Institute works in the interests of the Hispanic community in the U.S. They wanted to confirm whether the prepaid calling cards targeted at the Hispanic community were actually delivering the promised value to customers. To do this, they commissioned us to perform a series of test calls using a variety of calling cards.
Eric: What kinds of tests did you perform?
Mario: The different calling cards apply various kinds of charges. To make a call, customers type in the pin code from the calling card and they hear the balance they have remaining. Then they dial the destination number and they hear an announcement saying the total number of minutes their call may last. Our plan was to test the card values by performing a single long duration call to see how long the call would last before it was disconnected. Then we would compare the actual length of the call with the announcement that was made at the start of the call. We wanted to see if the call was actually as long as was promised in the announcement.
Eric: How many calling cards did you test? And what were the results?
Mario: We tested forty-five calling cards in total, and used them to make calls to both Guatemala and Mexico. Out of the forty-five, we were unable to connect a call with seven of the cards…
Eric: You were unable to connect a call? So those seven cards were completely useless? How many times did you try to place a call using those cards?
Franco: We attempted to place between seven and ten calls with each of those seven cards, but failed each time. This was particularly a problem when trying to call Guatemala.
Eric: And what of the other thirty-eight calling cards?
Mario: Only fifteen cards allowed us to achieve our original goal of exhausting the balance with just one call. Calls made with the other cards tended to drop after just a few minutes.
Eric: What did you do if the calls dropped?
Franco: We would make as many repeat calls as necessary to completely run down the balance on the card. Typically it would take seven or eight calls to do that.
Eric: That sounds like it would be very frustrating for real customers.
Franco: Yes, and expensive. Because there are charges applied for making each call, and because call durations are rounded, you get a shorter useful duration if you make several short calls instead of one long call.
Eric: So the real value of the card is greatly reduced if customers have to make several calls.
Mario: That was not the only problem. We found that in some cases the originator dialled the number and was charged, but there was no audio path back from the destination. We test the audio path both ways. The call was no good, but the customer was being charged anyway.
Franco: With some of the calling cards the duration is rounded up to the next three minutes. The customer makes a call, hears nothing and hangs up after a few seconds, but would still be charged for three minutes of use.
Eric: What were the results from the fifteen calling cards where you succeeded in exhausting the balance with a single call?
Franco: Only four calling cards lasted as long as promised. Five lasted somewhere between 80% and 50% of the promised duration. The other six cards were completely exhausted after calls that were half as long, or even less, than the duration announced at the start.
Eric: Do you have any advice for customers on what they should look for to tell the best cards from the worst? Or is it impossible to tell without doing the tests you did?
Mario: Customers need to be aware of the fine print about the charges. The less fine print, the better.
Franco: Customers should also avoid cards where you need to phone a customer services line or check a website to find out the full terms and conditions. Many were unreachable.
Mario: There is no obvious technical explanation, but in our tests the same kind of card may have given the complete duration as announced when used to call Guatemala, but less than the promised duration when used to call Mexico. There is no way to know except by using the card and seeing what happens.
Eric: What about calculating the true price per minute of each card?
Mario: I do not believe customers can really do that. Even a professional cannot determine the price that will be applied with complete certainty. There are many elements to the charges. There are weekly maintenance fees, hang-up fees, surcharges depending on the destination, and the calls are rounded in multiple minute increments.
Eric: I see from your examples that the fees are usually described as a range between a lower and higher number meaning you cannot be sure exactly how to calculate the price…
Franco: And the prices are subject to change without notice!
Eric: How can the calling card companies conduct business like this?
Mario: Many of the immigrants who purchase the cards are afraid about their legal status in the U.S. and have no desire to complain to any government agency. Also, regulations over the carriers vary from country to country.
Eric: Is anyone taking action as a consequence of your study?
Mario: The Hispanic Institute is lobbying for increased regulation to protect consumers. They have also asked us to conduct an expanded test, using more cards to make calls to seven countries in Latin America. Some government agencies, which we cannot name, have contacted us directly, in order to review our findings, our methods and our evidence.
Eric: What kind of evidence did you keep?
Mario: We keep all the data from the calls. For example, in addition to Call Detail Records, we have MP3 recordings of the audio paths from all the test calls that were made. That means there can be no argument about what was said in the announcements or about calls which were charged for but where there was no audio path.
Eric: What about the media? Are they interested in your findings?
Mario: The story has been picked up by a number of websites and local journals [see here and here for examples.] We have also been interviewed by a Washington D.C. television station dedicated to the Spanish-speaking community. Before they can broadcast it, they need to give the calling card companies an opportunity to make their own statements.
Eric: Your final report makes fascinating reading. I hope lots of people take a look at the detailed conclusions on the website of The Hispanic Institute. Are there other studies where you have performed independent testing and published the results publicly?
Mario: We did a study to correlate the audibility of DTMF tones with the line quality as measured on the PAMS scale. This was published by the IEC and presented to the ITU’s Quality of Service Group.
Eric: That sounds interesting. What was the conclusion?
Mario: In summary, recognition was not reliable if the PAMS measure was below 2.5. This sets a cut-off rule where the PAMS measure can be used to determine if the service quality is adequate to support services that rely on automated voice recognition.
Eric: Very useful if your business model is to make money by offering one of those automated information services…
Mario: Imagine trying to find out about a movie from an information service whilst driving to the cinema… you may arrive before you get through the first menu!
Eric: Tell me some of the reasons why telcos engage you to perform independent testing.
Franco: Independent testing enables calls to be performed and measured from the customer’s perspective. We perform tests following a predetermined test script, and keep a precise record of what happened from the customer’s point of view.
Mario: Every aspect of a call is precisely measured. Telecommunications firms use the data to assess a variety of things including the quality of the service given to customers and the accuracy of switch records and billing.
Franco: Calls that don’t complete, can’t be charged and poor call quality will cause customers to hang up rather than persist with a conversation, reducing revenues. If problems persist, customers will inevitably churn.
Mario: There is nothing to prevent one telco testing another and obtaining a competitive analysis. They can compare their own performance to rivals. Or, they pre-qualify a carrier they were considering partnering with. That way they can pick carriers based on the real quality of the service as well as the price, and even set Service Level Agreements based on these end-to-end perceptual metrics.
Eric: Why use an ‘intrusive’ testing approach to check billing?
Mario: To begin with, test calls give you an independent way to determine the actual start and end time of calls, and compare these to what was recorded in the switch CDR. Intrusive techniques enable the business to force something to happen on the network without waiting for customers to use a product. This can be very useful for checking new services.
Eric: We were talking about your study involving prepaid calling cards. Such traffic is typically carried by the so-called “bypass” operators. The motive behind your study was consumer protection. Why might a telco want to test for bypass operators?
Mario: Calling cards are so cheap because bypass operators avoid the termination rates levied by the destination network. Instead of paying the wholesale rate for termination, they only pay the retail rate for a local call. They can do this because they illegally link an international call carried by the bypass operator to a local call from a PBX or simbox in the destination country.
Eric: It is obvious this happens, because the prices boasted by these calling cards are lower than the termination rates to the destination countries. How easy is this to do, on a technical level?
Mario: An international VoIP call is connected to a local retail call, bypassing the need for a legitimate and legal interconnect.
Franco: People can buy the necessary hardware on eBay. All you need is a gateway to connect an interconnect call to a standard PBX. A seemingly honest business is used as a front for this kind of operation. For example, the front company may present itself as a marketing or customer service business, as these may be expected to make lots of calls.
Mario: These businesses generate relatively big bills which they pay on time, so they look like very good customers.
Eric: This makes the evidence for this kind of fraud very different to what you find with most other frauds. Most fraud results in bad debt, but bypass operations are likely to pay all bills on time.
Mario: Exactly. Although they pay their bills, they are still breaking the rules because they are finding a way to avoid the international termination rates that they should be paying.
Franco: There may also be help from people inside the destination operator. An employee may have aided the fraudsters by enabling a direct interconnection which is not being billed for. Another kind of help for fraudsters occurs when the operator’s staff decides to ignore the evidence that local calls are really the final leg of a bypass call. Yet another kind of bypass fraud occurs when stolen prepay SIM cards are taken to another country and put into a simbox. These are then used to make as many calls as possible until the network identifies and blocks the SIMs.
Eric: How does your testing prove what bypass operators are doing?
Mario: Because we originate the calls internationally and have precise details of the test calls being made, it is possible to match the test calls with the CDRs of the local call on the destination network. This is used to identify the address of fraudsters and stop them.
Franco: It is not enough to do the tests for a week or a month. You can cut them off, but the criminals will keep coming back, so it is necessary to keep on testing for them.
Mario: With our equipment, you get hard evidence of bypass operations. The calls are paid for like normal, and as the calls are made and the data recorded using our own equipment, there is no need to obtain anyone’s permission to do the testing.
Eric: Can you summarize the advantages of using external specialists like you to do independent testing on behalf of a telco?
Mario: This kind of testing, monitoring and quality control is not part of the core business of the carrier. By using an independent company, you get dedicated expertise, and a totally objective evaluation. A big benefit that cannot be obtained from in-house testing is that the telco can have its test results benchmarked against the results of other anonymous carriers.
Eric: Thank you for your time.
