Automated Solution for Procurement Notices
Background
An industry leader in the field of procurement, specializing in centralizing tender data, employing digitalization and automation to modernize the end-to-end tendering process, approached us with a complex problem.
Their objective was to enhance the efficiency, transparency, and accessibility of their procurement offering for governments, businesses, and individuals alike. However, they faced the following challenges:
This would ensure more efficient, transparent, and accessible data delivery.
Impact
Our solution had a significant impact on our client’s operation. By providing a reliable, automated, and normalized data stream, we enabled the client to effectively streamline their data handling process. This not only improved their operational efficiency by 45% but also enhanced the transparency and accessibility of their services.
The historical data handling, in particular, allowed our client to provide a comprehensive, near real-time overview of procurement trends, which is an invaluable tool for governments and businesses. With our solution, they were able to cut costs by up to 30%, thanks to the automation of manual processes and significantly reduced errors in data extraction.
Furthermore, the speed of delivering procurement notices data to their customers improved by 56%, as the automated pipeline efficiently processed and delivered the data on daily basis. Our client was able to offer a more complete and accurate service, ultimately contributing to their goal of modernizing and improving the procurement process for the benefit of all the stakeholders.
Challenges & Solutions
Challenge
Data Accessibility
The data was protected by an advanced anti-bot detection system, which made it difficult to access using standard data extraction methods.
Solution
Advanced Anti-Bot System Bypass
We developed a mechanism to navigate through the advanced anti-bot detection system while adhering to the platform’s data privacy guidelines.
Challenge
Data Inconsistency
The procurement notices had different standard formats, which posed a challenge to extract and structure the data uniformly. This necessitated the development of advanced extraction tool ensuring accuracy of the captured information.
Solution
Dynamic Parsing Mechanism
Given the change in notice structures over time, we created a dynamic parsing algorithm that adapted to different formats. This ensured that all data points of interest were captured accurately and consistently.
Challenge
Data Complexity and Volume Variability
The volume of procurement notices varied daily and had different structure, ranging up to 300 data points. This demanded adaptive extraction method ensuring all relevant information was accurately captured.
Solution
Data Extraction and Cleaning
We built an extraction system capable of handling the variable data volumes, ensuring that every procurement notice, was captured. The system included comprehensive data cleaning and random sample inspection maintaining consistency.
Challenge
Data Format and Delivery
The client preferred the Open Contracting Data Standard (OCDS). Requirements included deliveries in JSON format, uploaded to an AWS S3 bucket daily. This demanded a precise process to ensure integrity while adhering to the preferred format and delivery mechanism.
Solution
Data Normalization and Delivery
We developed a data normalization process to unify the different standard formats into the preferred OCDS. This provided the client with standard data format. We also implemented an automated delivery system to convert and upload the data to the client’s AWS S3 bucket in the preferred format.
Challenge
Historical Data
The client required historical data from 1980 till present, which presented a further obstacle due to the changing structure over such a long time period. Addressing this required a sophisticated approach to harmonize the diverse sets of historical data.
Solution
Historical Data Handling
We implemented a special module to extract, clean, and normalize historical data from 1980 till the present day. This allowed us to deliver complete and uniform historical data to the client.
Key Takeaways
Conclusion
In conclusion, the success of this project is significant. It led to a 30% reduction in costs and 56% improvement in delivery speed. These achievements underscore the importance of proficiently handling complex data extraction, normalization, and delivery tasks, especially when dealing with fluctuating and continuously evolving data structures.
This future-proof solution not only addressed the client’s immediate requirements but also adapted to potential changes, as the field continues to evolve. Through our automation efforts, we delivered an impressive 1.4 million data points daily, a feat that significantly cut manual processes and error rates.
This achievement underscores the importance of custom data solutions in supporting clients to achieve their goals, improve operational efficiency, and enhance service transparency.
By integrating historical data handling, we provided an invaluable tool for understanding procurement trends, thereby contributing to a deeper understanding of data management and promoting continuous advancement within this domain.
Take Action Now
We unlock data’s ability to transform.
Unlock the power of data to drive innovation, optimize operations, and make smarter decisions with Datamam’s comprehensive, integrated solutions.