Picture This: Founders, a reporting manager, and a team of 40+ people, all relying on daily reports and data sheets manually prepared by running complex SQL queries. Some queries took over a minute to execute, clogging the database and slowing down dashboards. As for the database schema? Let’s just say it had the charm of a treasure map with half the clues missing.
To get something as basic as a list of paid leads, you’d need to:
📊 Generate summary reports.
✏️ Update sheets.
🆕 Create new ones without breaking a sweat.
The process was a bit like trying to bake a cake with ingredients scattered across three different kitchens— one of which didn’t have electricity.
Decoding the database 🔍
My first month was all about playing detective. I dove headfirst into the database schema, understanding table structures, and figuring out how everything connected (or didn’t). With limited resources and a “not-so-great” database setup, I knew optimizing the existing process was going to be critical.
🔄The query overhaul
The first big challenge was the runtime of queries. A minute-long query in a startup feels like an eternity, especially when your dashboards depend on it. Using sub-queries and smarter indexing, I managed to optimize 90% of the queries, slashing runtimes dramatically. No Common Table Expressions (CTEs) were available, which would have made life easier, but hey, we work with what we have.
Summary report to the rescue 💪
Once the queries were lean and mean, I turned my attention to the daily grind of summary reports. Using SQLAlchemy to run SQL queries directly in Python, I pulled the data and transformed it into summary tables with Pandas. This not only saved hours of manual work but also ensured consistency in the reports.
Google Sheets Automation 📊🤖✨
At this point, I noticed a pattern: most team members needed specific sheets regularly. Some were daily MIS reports; others were monthly performance sheets. And then there were those special requests that would pop up like mushrooms after rain.
Using the Google Sheets API, I automated the creation and updating of these sheets. Whether it was:
🔄 Appending new entries.
🗑️ Removing paid ones.
The process was now seamless. Reports that used to take hours were ready within minutes.
Streamlit: The Cherry on Top :)
To make everything user-friendly, I deployed all these ETL operations in Streamlit. With just a click, team members could now generate summary reports, update sheets, or create new ones without breaking a sweat. It was so simple that even the least tech-savvy person could use it effortlessly.
Firing Myself !
By the end of it, I had automated 90% of the daily tasks expected of a Data Analyst. Reports were generated, updated, and delivered faster than ever. The team was happy, operations were efficient, and I…well, I was out of things to do.
And that, ladies and gentlemen, is how I became the Data Analyst who fired himself.
The Takeaway 🍽️📚
This experience taught me two things:
💡 The power of automation is immense. When leveraged properly, it can save time, reduce errors, and improve efficiency.
🛠️ There’s always room to optimize — whether it’s queries, workflows, or even your own role.