Disclaimer

This project was developed for the contact centre I work at, in my role as a data engineer, within a constrained corporate IT environment, where access to sources of truth and tooling is heavily restricted. Every effort has been made to gain access to the database that stores this data — and at the time of writing, I’m just over five months into that process.

Overview

The purpose of this project is to automate the collection of call logs made by agents at the contact centre, enabling downstream analytics and gamification.

Previously, the only way to access call data was through a legacy browser-based portal used for training and quality control. This portal allowed users to export a maximum of 1,000 call records at a time — and only for calls made within the past two months, due to a strict data retention policy.

Given that over 1,000 calls are made every day, this limitation made granular analysis virtually impossible. While other portals exist for exporting aggregated reports, they offer little to no control over groupings or filters, leaving users stuck with the same rigid perspectives.

Tools & Workarounds

Due to the limitations of the corporate environment, I had to employ some unconventional but effective solutions to get this project operational:

• Playwright (Python) – Used to automate interaction with the call recording portal’s browser-based UI. This handles authentication, navigation, and data extraction, bypassing the need for official APIs or backend access.
• PostgreSQL (Portable) – Because installing software or provisioning cloud infrastructure isn't permitted, I'm running a portable version of PostgreSQL locally. This gives me a full-featured relational database without relying on IT support or infrastructure.
• Local Automation – The Playwright script runs in an infinite loop on an always-on local machine, fetching new logs automatically whenever they become available.

Workflow (Simple)

At a high level, here’s how the Python script operates:

Startup

1. Open the call recording portal in a headless (invisible) browser and log in
2. Reset all filters to ensure all call logs are visible
3. Sort the table by the 'Created at' datetime column in descending order
4. Click the 'Start Search' button to apply/reset filters

Loop

1. Extract all call logs from the current page into a table
2. Insert all rows into the PostgreSQL database — only rows that don’t already exist
3. Return the number of rows inserted
4. If all rows were inserted, go to the next page and repeat
5. If only some rows were inserted, click 'Start Search' again to refresh the data
6. Repeat until new data stops appearing