The amount of data captured by businesses has increased significantly as more and more aspects of our lives have moved online. The internet is awash with estimates of the amount of digital data already in existence and predictions of rates of growth, usually expressed in some exotic sounding data unit.
Alongside this incredible growth of data many businesses have been analysing much of this data in order to gain better business insights.
Among the many uses of analytics is finding new or suspicious patterns in the data. For example, fraud management systems check data for unusual data patterns. That is why you will be contacted by your credit card company shortly after making a foreign purchase after travelling abroad for the first time in a while. In a similar vein, telecom companies check for suspicious calling patterns.
In addition to pattern-checking, analytics software is used to help hone marketing initiatives through:
- tracking of conversion and marketing funnels
- A/B feature testing
- analysis of user engagement
- analysis of user retention
- analysis of individual users or cohorts
The data analytics sector has been evolving in response to changing requirements such as the sheer volume of data to be analysed. It’s not just the sheer quantity of data that has been presenting challenges but also the way in which it is held in an ever-increasing number of places.
Barriers to market entry fell considerably when the internet offered software vendors the ability to switch to a service model. Vendors were able to offer services for monthly or annual subscription fees and corporate customers were able to sign up for new analytics services without dealing with the obstacles previously created by their own procurement and IT departments. The end result is that businesses have been finding that more and more of their data is splintered across several different applications or Software-as-a-Service (SaaS) tools. These tools range from customer engagement systems such as Salesforce (which has been the leading SaaS tool since its launch in 1999) to tools for handling general operations, issue tracking, support desks, payments and accounting.
Data from Blissfully, a SaaS management start-up, shows that even relatively small companies find they are soon subscribing to a significant number of SaaS tools. The table below gives the average number of tools for each of the main company sectors in 2020.
|Number of Unique SaaS Apps||288||185||102|
The table shows not only how just how sizable the number of disparate data locations is but that there is also a significant degree of churn. However, the complexity doesn’t end there! As well as having data held in significant numbers of SaaS tools, many businesses have further operational data held in:
- simple data files e.g. XML / JSON
- on-premises databases
- cloud datastores
Historically, software engineering initiatives have been slower to be adopted by the analytics sector. Configuration management is one area where analytics has been something of a “poor cousin” of the industry, as anyone who has recently ran a program with the suffix “_final_final” in the program name will testify! Another poor cousin revolves around the automation of testing.
So how has the analytics sector evolved to meet the challenge of increased data complexity whilst bringing onboard some of the software engineering practises used in the wider software world? The short answer is in several interesting ways. I will give a more comprehensive answer in my next article.