My SaaS Server Exploded (& How I Salvaged It) — The Bootstrapped Founder 319


Dear founder,

Earlier this week, I finally had a day to work on a massive refactor of Podscan’s transcription queuing system.

And it... didn't go well.

In case this is too technical for you, consider today’s episode a nerdy deep dive into the heart of where my business builds its value. I’ll explore the safeguards and mental exercises to deal with sizeable setbacks.

And if you’re interested in what a solopreneur’s tech complexity can look like, stick around. I’ll spare no details.

So, what happened?

On Monday this week, for the first time in almost three weeks, I had a full day to myself to just spend on building software. I was at MicroConf two weeks ago, and then the following week was full of calls that I had scheduled so I could catch up on the week that I missed at the conference before. There were just a lot of interruptions happening throughout the last couple of weeks, which meant that I didn’t have any meaningful long block of time to spend on my software business, Podscan.

A quick shoutout to my sponsor, Dome. If you're looking for a great place to deploy your app, I highly recommend checking them out!

Kickstart your business with the hosting platform built for developers, indie hackers, and solopreneurs

Say goodbye to the complexities of execution and hello to a world where launching your tech stack is as simple as clicking a button. With Dome, you're not just choosing a platform; you're stepping onto a launchpad designed for developers, indie hackers, and solopreneurs ready to skyrocket their ideas.

Why choose Dome?

  • Instant Launch: Go from idea to execution faster than you've ever imagined.
  • Zero Hassle Scaling: Your services deploy fully scalable, making growth painless and straightforward.
  • Cloud-First Data Management: Keep your data secure and accessible with a secure cloud infrastructure.
  • Love at First Code: Integration with popular frameworks and databases you already use.
  • Plug-and-Play SaaS Templates: Courtesy of Dome’s Boilercode acquisition, launch your SaaS with product-ready templates, all equipped with essential tools and integrations (auth, email, payments, analytics, customer feedback, etc).

Dome is more than a platform - it's your startup partner, ensuring hosting, scaling, and everything in between is comprehensively covered. Spend time on what matters—innovating, creating, and expanding—while Dome handles the heavy lifting!

So, after those two weeks, I chose a day where I would have a full day of opportunity to work on software. For that, I had to reconfigure my schedule. For the last couple of months, I’ve been very open with my schedule. People could book calls almost every day of my week, mostly because I really wanted the early Podscan users to find the perfect time for them to talk to me. The idea was that I could hop on a call with a potential or paying customer whenever it suited them. I’m in a super early stage, and every little piece of feedback helps. Every early customer interaction with positive feedback is something I can build on later. I wanted to open my schedule to everyone, and I did.

However, this meant that I was constantly interrupted, as people scheduled their conversations with me whenever it best suited them but never me.

Meanwhile, I also had a podcast to record, interviews to organize and research, and guests to invite to the show. I have to manage editing and transcribing, and a lot of extra stuff on certain days adds to the overhead time that I can’t spend on building software.

So, I made the choice that I’d had enough of an open schedule and reduced my calls to two days max.

All my customer service conversations and customer discovery calls now happen on Friday. All my podcast recording calls happen on Thursday. I used to have one interview call a day. Now, I have as many as people want to fit into that day with some time in between. So, I’ll stack my calls on Thursday and Friday, freeing up the rest of the week to write code, do marketing, and sales without constant interruptions.

After setting up this new schedule, I set a day to work fully on my software project. I finally had that day on Monday this week, so I chose the biggest project of all—an “eat the frog” moment.

I chose a completely refactor the heart of my operation.

Podscan ingests thousands of hours of audio data every hour, transcribes it into text, and scans for keywords to send to an API. It’s a lot of data coming in, and it’s not a very flexible system. Whenever my backend servers have capacity, they ask my main server if there’s anything available. My main server then checks the database for an available project and sends it over. That’s how it works.

This has grown over time, and it feels like it can be made more extensible and less convoluted. I thought I needed an internal queue system—a system of many queues that make it easier to move individual podcast episodes through all the stages of analysis.

Podscan works like this: I get an audio file, transcribe it (step one), run inference on it to extract certain information and build a summary (step two), and then scan for keywords and send alerts (step three). I have a few more steps planned, but I was starting with these three. Right now, they all kick off each other. When an audio file is available for transcription, it’s presented on the API. A backend server fetches it, responds with the full transcript, which kicks off the inference step. A server handles inference and sends it back. Then, scanning starts on the main server.

It’s a back-and-forth system, and it works.

But there’s always room for improvement, right? I thought about creating three queues in which candidates live, according to their stage of completeness. Instead of looking in my database, I’d just look into the queue.

Simple enough.

I built this system over the course of a few hours, and locally, it worked well. I spent quite some time on the edge cases: What if there’s an error while the transcript is being created? What if I have to retry it, but there’s another step that needs to be done first?

Fortunately, my work has some structure. I built this sizeable feature on a separate branch using the GitFlow model. I have a development branch for more complicated experiments so I can work on them while fixing bugs in the main branch. After about six hours of diving into the code and testing intensely, I deployed it to my main server. And it worked. There were a couple of bugs, but I could quickly fix them. I tweeted about how well it worked, although I was nervous because it’s the central functional part of Podscan.

If the data ingestion and transcript engine doesn’t work, then nothing works: the API and notifications would break down. It needed to function well, so I let it run and checked my metrics. Gradually, fewer episodes landed in the database. Usually, we handle 2,500 episodes an hour. Now, it dropped to 2,200, then 1,900, and eventually 300. Something was wrong in the system, preventing queue items from moving forward, and I couldn’t see where the problem was.

I couldn’t just play around and delete jobs; they all needed to work. It was a live system after all. So, I rolled back what I’d worked on for six hours to the morning’s version.

I tweeted about that too. Sharing setbacks is part of my approach to building in public. I was frustrated not just by the system failing but also by feeling like I had wasted a day. I went upstairs to talk to Danielle, and as I explained my frustration, I realized something: I may have spent a few hours writing code I won’t use, but I had learned a lot about both my product AND my approach to making improvements. They both had properties I was not aware of before. Yes, the queue system didn’t work, but I learned how complicated my existing system is, and how many interference points exist.

I also learned my feature specification process was incomplete. Adding another abstraction layer wasn’t required. The queuing system wasn’t necessary; my existing state machine was efficient enough if I ensured reliable state transitions. If I ever add more steps, I’ll improve the current state machine instead of replacing it.

Despite the wasted hours, I learned from my attempt. Next time, I won’t dive in with just a few notes in a Notion document. I’ll plan properly and monitor the state machine. This is a feature spec itself: build more observability into the system.

And one more remark: I am so lucky to have systems in place to deal with such a botched deployment.

Rolling back was easy, thanks to Laravel’s Forge and Envoyer tools. They let me revert to a prior deployment quickly because I keep several dozen backups.

Reversibility requires non-destructive database migrations. Instead of modifying tables, I add new ones. This ensures rollback is smooth and won’t impact existing data.

Overall, this was a lesson in reframing a frustrating loss into a learning opportunity to understand my product and my process better.

So next time this happens to you, look at it through the lense of emergent insights: if I hadn’t tried to implement this, i wouldn’t have learned about the existing complexity in my business. No wI know more and have more experience.

Do this a few hundred times and you have all the moat you’ll ever need.


I'll share a few updates about my SaaS on the pod, and I'd love to know what you think about them! Please leave a voice message at podline.fm/arvid 🥰

And if you want to track your brand mentions on podcasts, check out podscan.fm!

Classifieds

I recently launched The Bootstrapper's Bundle, which contains Zero to Sold, The Embedded Entrepreneur, and Find your Following. If you want to start a bootstrapped business and build a validated product and a personal platform while doing it, check out this bundle. It contains all eBooks, audiobooks, video courses and extra materials I ever created. It's just $50, for now.

Thank you for reading this week’s essay edition of The Bootstrapped Founder. Did you enjoy it? If so, please spread the word and ​share this issue on Twitter.

If you want to reach tens of thousands of creators, makers, and dreamers, you can ​apply to sponsor ​an episode of this newsletter. Or just reply to this email!

To make sure you keep getting your weekly dose of Bootstrapped Founder, please add arvid@thebootstrappedfounder.com to your address book or whitelist us.

Did someone forward you this issue of The Bootstrapped Founder? ​You can subscribe to it here!​

Want to change which emails you get from The Bootstrapped Founder or unsubscribe for good? No worries, just click this link: ​change email preferences​ or ​unsubscribe​​.

Our postal address: 113 Cherry St #92768, Seattle, WA 98104-2205

Opt-out of preference-based advertising

Arvid Kahl

Being your own boss isn't easy, but it's worth it. Learn how to build a legacy while being kind and authentic. I want to empower as many entrepreneurs as possible to help themselves (and those they choose to serve).

Read more from Arvid Kahl
Bootstrapped Founder Logo

Podcast, YouTube, Blog Dear founder, There’s this famous saying by Paul Graham from one of his most well-known essays: a command to all founders and startup operators to “do things that don’t scale.” Graham explains that at the beginning of starting a business, there are things you should do as a founder that you know will never work once you have hundreds or thousands of customers. But it’s precisely these things that make it happen. The learning from these hands-on projects allows you to...

Podcast, YouTube, Blog Dear founder, These last weeks have been a bit much. What do they say? “Take a break before the break takes you?” Well, I got quite close to that this week. 🎧 Listen to this on my podcast. I’ve been quite active working on Podscan in many different ways, and it has been overwhelming at times. I found a way to cope with this, and I found a way to still get done what I needed to get done, so I wanted to share the whole journey and the whole emotional state that I was in...

Podcast, YouTube, Blog Dear founder, Recently, I’ve been pondering the difference between being sincere and being serious, especially in the context of entrepreneurship. This reflection was sparked by an article I read about how the most successful people are sincere but not overly serious. It’s a fascinating concept that ties into the idea of playing the infinite game versus the finite game—a theme I’ve explored in my books “Zero to Sold” and “The Embedded Entrepreneur.” Both...