In case you were wondering, the words in the title are in the order that I intended. A Google search on the string “extract load transform” returned 1,560 hits. Searching for “extract transform load” generated 88,400 hits, so I am not going to argue about which is the more common sequence for processing data. For the uninitiated, Extract-Transform-Load (ETL) means getting data from the source systems, transforming the data at a way-station then moving it to its final destination, whilst Extract-Load-Transform (ELT) means doing the transform at the end point. Why, then, am I talking about this? The technology to get lots of data from source systems to somewhere people can query it happens to be one of the enabling pillars of modern revenue assurance. If somebody else in the business can do this more efficiently, like the people working on data warehousing in the IT function, then revenue assurance will be able to reap the rewards.
I am not in the business of selling other people’s products, but I recently attended yet another glowing vendor presentation about Netezza, the makers of software to manage data warehouses. This time the presentation was from Carphone Warehouse, the UK multiplay provider. In their presentation, they explained the difference Netezza had made to their revenue assurance. As I have said previously, I am no natural fan of theory that revenue assurance is just a matter of implementing an ‘RAdb’ – a big database used to query lots of data relevant to revenue and cost integrity. There is a lot more to revenue assurance than pouring as much data as you can into a big bucket, stirring it around and then sticking your head in to see what you find. That model is fundamentally reactive. Vendors like to play games with language and pretend that spending gazillions of dollars to reduce processing timelags from last week to yesterday to an hour ago to a minute ago somehow makes you proactive. Reacting rapidly is not the same as being proactive. Reacting rapidly still involves allowing something to happen and then reacting to it. Processing speed just reduces the potential delays in reaction. Processing power gives you potential, not a guarantee of results. No matter how much data you have and how quick you can process it, finding leaks will still take longer than your lifetime if you do not know what to look for, or if there is no data of the type you need to highlight a genuine problem. You can spend all you like on Business Intelligence but the results will be worthless if you cannot apply some human intelligence. That said, the dynamic of how to address revenue assurance using systems has changed a lot in the ten years I have been working in the field. Raw processing power is threatening to turn the principles of how to best exploit data on their head. Models for how to work that once seemed obvious may be set aside. I am not a spokesperson for Netezza. I have not seen enough to tell if others are delivering similar results; it could be that Netezza are just much more focused at targeting the revenue assurance market for sales. If vendors of data warehouse appliances see revenue assurance as a market worth targeting, that is no bad thing. What I do know is that Carphone Warehouse says they have gone through a paradigm shift by engaging Netezza and by changing to ELT instead of ETL. If what they say is true, the long-run implications for the future of revenue assurance may be very significant. It may ultimately mean the end of dedicated revenue assurance tools as we know them.
The days of code monkeys creating reports based on what they know how to do, rather than what the business needs to do, are still with us, but those days are numbered. RA vendors are much slicker at giving customers what they really want these days, and there is much less room for those one-off in-house developers that you would find creating reconciliations that fell apart the moment they left the company. The species of code monkey that tried to pretend they knew RA when talking to technologists and tried to pretend they knew technology when talking to RA people is reaching its evolutionary dead end. However slick the RA-specific vendors get, they now face a serious competitor in the form of leveraging general-purpose data warehouses for RA. For a variety of reasons, dedicated RA tools have always been more popular with RA practitioners than trying to run relevant queries on an enterprise data warehouse. But this may change. If the data in your enterprise warehouse was accurate, robust, detailed, comprehensive, and up to date, why would you want a separate revenue assurance system with separate feeds querying its own version of the data? And if the tools around the warehouse allowed the revenue assurance users to perform queries alongside all the other users of the warehouse, without any impediment or obstructions, what is the advantage in having in a dedicated RA system? I admit I can think of a hundred reasons why life may not be as simple as that, but all I am saying is that if it were that simple, it would be hard to argue for the need for systems dedicated to RA that in essence are there to crunch data that could be crunched elsewhere. This brings me back to the title of this blog. Extract-Transform-Load is intended to handle the processing of data most efficiently, but it can encourage a mindset that sees revenue assurance as making unreasonable demands for the volume, quality and speed of processing data. Pressure to make ETL more efficient can lead to decisions that undermine the goals of BI and of revenue assurance in particular. For example, if extraction is a problem, extract less – but without detail then the power of analysis is lost or at least reduced. If revenue assurance is competing for scarce resources with other parts of the business that find it easier to justify their data processing requirements, then stop trying to support the RA team, or give them their own very own half-baked solution. The latter will probably shut them up – it guarantees a few jobs and if it does not work properly then chances at least some of the blame can be shifted from IT to RA when individuals in the RA team foolishly make the mistake of overestimating their own IT skills. The code monkeys employed by RA functions are not going to change their ways and announce their own limitations, after all. The attention-grabber in the Carphone Warehouse story was the benefits gained by changing to Extract-Load-Transform, and performing transformation on Netezza’s CPUs, whilst cutting out some of the extract and load that had been taking place on their ETL servers. For revenue assurance in Carphone Warehouse this resulted in daily extracts that completed 5.5 hours earlier each day, and a 3.5 hour improvement in availability to the end user.
Getting a report to an RA analyst three and a half hours earlier is unlikely to make a whole lot of difference to the bottom line, but it does mean that Carphone Warehouse is having success with an approach that levers common information for multiple stakeholders, including revenue assurance. If that cuts out the cost of needing a dedicated revenue assurance system, that will be where the real saving is.
