Is data platform tech debt costing your enterprise big money? 

As large organisations migrate their data platforms to the cloud, time pressure on engineering teams can generate technical debt; sub-optimal development solutions that are hastily implemented or incomplete. Buried in code or the detail of platform architecture, tech debt hurts productivity and carries significant financial liabilities which are virtually invisible to business leaders.

According to research by Stripe, typical IT engineers spend approximately 33% of their time dealing with tech debt. If we assume that an engineer working for a big company gets paid $100k annually, that means for a fifty-person engineering team, the total HR bill for tech debt remediation is $1.65 million. That’s more than one and a half million dollars’ worth of non-productive IT work which could otherwise be put toward innovation, customer experience and product development.

Tech debt hurts productivity 

There’s more than one reason to eliminate tech debt from data modernisation projects. Beyond the financial savings, avoiding tech debt also fosters confidence in the data produced by the platform. Having confidence in the data leads to better decision making which in turn enables growth.   

Continuing to accrue tech debt in your data platform can garner ongoing IT productivity problems, including:   

• Slower time-to-market    
• Longer periods of downtime    
• Negative customer experience from service interruptions   
• Time lost fixing problems that are invisible to business leadership   
• Lack of predictability in development sprints   
• Lowered IT team morale  

Where does tech debt come from?   

If your platform architecture isn’t strategically designed to be transparent and scalable, tech debt will almost certainly impact your system.   

One of the biggest factors contributing to tech debt is time, or rather, the lack of it. Tight data project delivery timelines often lead to haste, poor planning and the implementation of tactical solutions that meet immediate requirements, but aren’t sustainable.    

If no provision is made to “pay down” tech debt, short term solutions stay in place and permanently damage the efficiency of the platform.   

Managing tech debt   

Let’s discuss some common data platform tech debt issues and strategies to avoid them.   

Inadequate data migration testing   

Performing reconciliations by only checking if the number of records at source matches that in the target is a common workflow scenario. This approach fails to address issues due to corrupt or null records, which can slip through the cracks.    

In the example table above, we can see that the data is not accurate even though the row counts match.   
   
Recommendation:

Add data quality checks, at least for the key metrics in each table. This will significantly boost the veracity of your data.   

Hard failures of data pipelines   

A common approach to data pipeline implementation is to allow it to fail in the event of data integrity issues such as duplicates, corrupted data, data losses, etc.    

Recommendation:   

A better approach is for your pipeline to accommodate data integrity issues that ensure that at least a partial data set is produced and avoid impacting the end-user.   

In this instance, accommodating data integrity issues means applying necessary rules to:   

• prevent the pipeline from failing, &   
• purge the invalid record from the output and make invalid records visible to relevant teams via slack or email. 

Absence of Data Lineage  

When a data integrity issue occurs, investigating the root cause in the absence of data lineage is like finding structural damage in a building without its blueprint. At best, it will result in several hours of analysis and troubleshooting. You also may end up having to rely on an SME (Subject Matter Expert) who understands how the data gets generated or find yourself digging through code to trace a dataset’s dependency.   

Recommendation:   

Establishing data lineage is critical to empowering the operations and development team with insight into downstream and upstream impacts of changing a data set. This is one of the core components of having a robust data governance framework.    

Inability to recover from a disaster   

What if an engineer in your team inadvertently deleted a critical data set in production? What if the servers running your data platform crashed? What if your data lake or data warehouse were destroyed? Can your platform recover and ensure business continuity?   

It’s vital to know whether your data platform can recover from a disaster. Data is an increasingly valuable asset, so protecting it from catastrophe has never been a greater imperative. Your organisation may have a disaster recovery plan, but whether your data platform can execute that plan is rarely validated. These sorts of crisis scenarios can produce remediation bills in the billions of dollars.   

Recommendation:   

When drawing up the architecture for each component of a data platform, disaster recovery must be one of the key factors contributing to the design. The lack of disaster recovery capability is a common source of tech debt.  

Adding code to orchestrate ETL pipelines   

Engineers often spend significant time on ETL pipeline code to orchestrate the ingestion of new data sets into a platform. This can create tech debt that causes loss of productivity with every pipeline created.   

Recommendation:   

An engineer should only need to develop code for applying business rules to generate data sets. There is a wide variety of ETL (Extract, Transform, Load) tools for orchestrating and implementing pipelines that relieve engineers of the orchestration process, so they can stay focused on solving business problems.    

Time scarcity produces tech debt   

If engineering teams are under pressure, using short-term fixes in place of well-architected strategic solutions, tech debt is inevitable. A scarcity of specialist data skills in an IT team can also contribute to rushed work that doesn’t meet best practice standards.     

Recommendation:

Data platform tech debt produces significant cost overruns if it isn’t addressed proactively.  Working with a specialist data modernisation consultant can help you pay down your existing tech debt, avoid future problems and reduce the burden on your team so they can be more productive.   

Take control of tech debt  

By taking a strategic approach to data platform management, you can regain control of productivity and minimise the tech debt in your system. Relieving the time pressure on your engineers will mean they can work on product development and solutions that drive growth. Long term, it’s best to implement proactive solutions that avoid creating tech debt and foster efficient data platform management. 

Talk to Versent’s data modernisation advisors about eliminating tech debt and saving money on remediation.