As part of our long-term commitment to delivering the best quality data to our customers, we’ve built a new proprietary data noise filtering algorithm at Quadrant.
Not only will this save hours or even days of our clients’ time, it will also save on wasted costs of investigating the causes of bad data where they are not eliminated at source.
Quadrant’s new proprietary algorithm helps ensure our customers won’t experience system “breaks” due to bad data loads. This bad data loads can result in up to half a day of troubleshooting by the in-house engineering team to identify the problem.
Introducing Quadrant's Data Noise Algorithm
The new algorithm enables our data engineering team to deliver crystal clean data up to 99 per cent noise-free.
This is an important step Quadrant's journey and vision for data quality, which is our promise to our customers. We’re proud of the team that spent two months on testing and building the data noise algorithm.
For those unfamiliar with the term, “noise” in data sets essentially refers to bad data fields.
Image of Datasets Table
In location data sets, fields including device ID, latitude, longitude, horizontal accuracy, and IP address are very important. If any of these fields contain noise (invalid data), then the entire row becomes worthless.
As an example (as shown below), we know that Device ID fields should constitute 36 characters.
Image of Device_ID sample
With our new algorithm, we can scan millions of lines of data in real-time to identify where the device ID field has an invalid entry and scrub it from the overall set.
Bringing automation and machine learning intelligence to this process across all data fields is an exciting move forward, reducing reliance on manual checks and slow fixes.
Starting with our global datafeeds
As of today, our new data noise filtering algorithm is running on all our global datafeeds.
We believe our emphasis on data quality is what sets us apart from the industry at large. We’re excited to continue building new ways to deliver on that promise – and we hope you are too.
To find out more about all the ways we ensure data quality in all the feeds we offer, get in touch with me here.
Also read here on Quadrant's latest Data Quality Report!
Credits & Acknowledgements for Data Noise Filtering Algorithm
Aishwarya Bose, Data Engineering Intern
