Public data, locked away
The official tenders website restricts large-scale access and hides its data behind several security layers, which keeps investors, researchers and analysts in the dark.
Public Tenders · Bolivia
Bolivia's public spending, out in the open.
We opened a protected procurement portal and turned two decades of public tenders into a clean, ready-to-analyse dataset for investors, researchers and journalists.
- Coverage
- 2006–2024
- Tenders
10 M- Security layers
- 3
- Status
- Ready to use
Executive summary
The problem, the solution and the result
Scraping, reverse engineering and AI
We combined advanced web scraping, API reverse engineering and a neural network that reads the CAPTCHA, to retrieve structured data programmatically.
A dataset ready to analyse
Ten million historical tenders, from 2006 to 2024, structured for market research, forecasting and academic studies.
Technical approach
Four steps from a closed portal to a clean dataset.
The project automates the extraction of Bolivia's public tender data, overcoming each barrier of the official website with a different technique.
-
01
Identifying the data source and its security
The portal gives no direct access to structured data. It is protected by hidden API endpoints, session-based secret tokens and CAPTCHA verification.
-
02
API reverse engineering
The website generates unique secret tokens and injects them with JavaScript before the page finishes loading, which hides them from ordinary scrapers. Our scraper intercepts the scripts, extracts the tokens and rebuilds the requests to reach the data programmatically.
-
03
CAPTCHA solving with a neural network
Every request for more information requires a CAPTCHA. We trained a Convolutional Neural Network (CNN) to decode the images, and used data augmentation to expand the training set without extensive manual labelling. The model reaches high accuracy on this security layer.
-
04
Data retrieval and pipeline
Because the OCR model is lightweight, we chose multiprocessing over asynchronous requests: one worker per CPU core runs the whole extraction in parallel.
Dataset
Bolivia Public Tenders
Historical, ready-to-use data from 2006 to 2024. Ideal for researchers and businesses: use it for in-depth analysis, market research, forecasting or academic studies, and gain actionable insight into Bolivia.
The data was gathered and is ready for analysis, giving a comprehensive snapshot of the public tenders of Bolivia.
Purpose and impact
Transparency in public spending matters most in a crisis.
This project was born from a national financial crisis. Unlike in most countries, public tenders in Bolivia sit behind security barriers that limit access for investors, journalists and citizens. The next step is a deep analysis of the extracted data, with insights into this issue.
More about this
Want the dataset or a deeper analysis?
To access this dataset or learn more about the project, contact us.
