Data at Work

The world of data and its many applications. This blog will help you learn how visionary companies are utilising external data to enhance business operations.

How Quadrant is delivering on our Data Quality promise

Quadrant Data Noise Algorithm, KV

As part of our long-term commitment to delivering the best quality data to our customers, we’ve built a new proprietary data noise filtering algorithm at Quadrant.

Not only will this save hours or even days of our clients’ time, it will also save on wasted costs of investigating the causes of bad data where they are not eliminated at source.

Quadrant’s new proprietary algorithm helps ensure our customers won’t experience system “breaks” due to bad data loads. This bad data loads can result in up to half a day of troubleshooting by the in-house engineering team to identify the problem.

 

Introducing Quadrant's Data Noise Algorithm

Image of data engineer working

The new algorithm enables our data engineering team to deliver crystal clean data up to 99 per cent noise-free.

This is an important step Quadrant's journey and vision for data quality, which is our promise to our customers. We’re proud of the team that spent two months on testing and building the data noise algorithm.

For those unfamiliar with the term, “noise” in data sets essentially refers to bad data fields.

Image 2-2

Image of Datasets Table

In location data sets, fields including device ID, latitude, longitude, horizontal accuracy, and IP address are very important. If any of these fields contain noise (invalid data), then the entire row becomes worthless.

As an example (as shown below), we know that Device ID fields should constitute 36 characters.

 

Device ID-1

Image of Device_ID sample

With our new algorithm, we can scan millions of lines of data in real-time to identify where the device ID field has an invalid entry and scrub it from the overall set.

Bringing automation and machine learning intelligence to this process across all data fields is an exciting move forward, reducing reliance on manual checks and slow fixes.

Glolbaldatafeeds

 

Starting with our global datafeeds

As of today, our new data noise filtering algorithm is running on all our global datafeeds.

We believe our emphasis on data quality is what sets us apart from the industry at large. We’re excited to continue building new ways to deliver on that promise – and we hope you are too.

To find out more about all the ways we ensure data quality in all the feeds we offer, get in touch with me here.

Also read here on Quadrant's latest Data Quality Report!


Credits & Acknowledgements for Data Noise Filtering Algorithm

Data Engineering Intern: Aishwarya

Aishwarya Bose, Data Engineering Intern