An Embarrassing Tale: Why my server could only handle 10 players

It was the summer of 2017, and I was a bright-eyed, bushy-tailed full-stack developer eager to make my mark on the world. Everywhere I looked, it seemed like multiplayer .io games were the new hotness. Slither.io, agar.io, diep.io — they racked up players by the millions and plastered the front page of every online games portal. I wanted a piece of that action.

"How hard could it be?" I naively thought as I sketched out ideas for my own entry into the .io hall of fame. I settled on a concept: BlobBattle.io. You‘d play as a squishy blob in a virtual petri dish, gobbling up smaller blobs and tactically splitting off bits of yourself to absorb other players. Grow big enough and you could dominate the dish. It was simple, engaging, and endlessly replayable. In other words, the perfect recipe for an io game blockbuster.

I excitedly got to work, fingers flying across the keyboard as I laid out the technical architecture. The frontend would be rendered with the trusty HTML5 canvas API, styled up with some slick CSS3 and animated with silky-smooth JavaScript. jQuery would handle the UI bits. On the backend, Node.js was the obvious choice for its event-driven, non-blocking I/O model — perfect for the real-time nature of the game. Socket.io would handle the WebSocket communication layer between client and server. Gameplay events and stats would be persisted in a MySQL database. Pretty standard stuff for a seasoned full-stack developer like myself.

After a few late nights and gallons of coffee, I had a working prototype. The blob physics felt satisfyingly gooey. Colorful dots made pleasing blipping noises as you absorbed them. It was almost too much fun just playing by myself. Surely the server would be bursting at the seams as soon as I unleashed this bad boy on the world.

I deployed the first version of BlobBattle.io to a medium-sized virtual private server, fired off a few "Show HN"-style posts on the usual indie game forums and subreddits, and waited with bated breath for the player counts to skyrocket.

An hour passed. Then two. I nervously refreshed the real-time Google Analytics dashboard again and again, but the visitor graph remained stubbornly flat. Did my game actually suck? Was I delusional to think I could make the next io game hit? Existential dread began to creep over me.

But then, a miraculous blip. A single user had joined the game! I frantically hopped into the server, materializing as a freshly spawned purple blob. The new player — a greenish blob — warily circled around me. I lunged forward recklessly, hoping to establish dominance, but the cagey green blob deftly dodged my advance. We proceeded to engage in a tense ballet of splits and merges and near-misses, until finally, the green blob emerged victorious, absorbing my shattered blob bits to become the new king of the dish.

"gg", I typed into chat, praying that my opponent wouldn‘t be a sore winner.
"fun game!" they replied a second later. "I like the blob physics."

I practically melted into my chair with relief and joy. A real player enjoyed my game! Maybe I wasn‘t crazy after all. As if the floodgates had opened, a few more visitors trickled in over the next hour. I watched with glee as they careened around the dish, blobs and dots splattering in a rainbow of hues. My dopamine receptors fired non-stop as the player count climbed to 10, then 20. BlobBattle.io was an unqualified success!

But then I noticed something peculiar. The server was struggling to keep up with the action. Blobs zipped around at hyperspeed, then stuttered and froze, only to teleport erratically moments later. The colorful dots flickered wildly. Console errors splattered the server logs like a Jackson Pollock painting. My stomach dropped as I pulled up htop and saw CPU and memory usage spiking into the red.

With only around 25 concurrent players, the Node process was consuming nearly 100% of two 2.8 GHz Intel Xeon cores and leaking memory faster than a sieve. This was not good. Not good at all.

I SSHed into the production server and ran some quick back-of-the-envelope calculations. The game server was processing around 3,000 events per second (EPS), with most of the CPU time spent in the collision detection and broadcasting logic. At this rate, even with a more powerful 8-core/16GB setup, I‘d top out at maybe 100 concurrent players before hitting a bottleneck. Considering most successful .io games boasted thousands of players at peak hours, this was a pitiful showing.

Desperate for answers, I fell down a rabbit hole of Node.js performance articles, V8 garbage collection details, and socket.io benchmarks. I tried every trick in the book: object pooling to reduce memory allocations, binary serialization to minimize socket.io packet overhead, worker threads to parallelize collision checks. But no matter what I did, I couldn‘t seem to push the maximum concurrent players past a few hundred while maintaining a smooth 60 FPS experience. I was still two orders of magnitude away from viability.

After many bleary-eyed nights of profiling and refactoring, I finally discovered the root of the problem. When generating game events, I had naively created a new object for each one and pushed it onto an array to be processed every server tick:

function generateEvent(type, data) {
  eventQueue.push({ type, data });
}

But I never cleaned up these event objects after they were processed and broadcast to clients. They just sat there in memory, endlessly accumulating like dust bunnies under a neglected couch. A classic memory leak.

To make matters worse, the main server loop iterated over the entire eventQueue array every 16 ms to process and send events. As the queue grew in size from the leak, the CPU time spent churning through it ballooned:

function serverLoop() {
  for (const event of eventQueue) {
    processEvent(event);
    broadcastEvent(event);
  }

  // Process other game logic...

  setTimeout(serverLoop, 16);
}

No wonder the server was choking on a mere 25 players and their paltry payloads! In a big match, there could easily be tens of thousands of events generated per second. With no garbage collection to flush them out, they piled up endlessly, dragging the tick rate to a crawl and devouring system memory.

The fix was simple in hindsight — just splice processed events out of the queue to allow the garbage collector to clean up after them:

function serverLoop() {
  for (let i = 0; i < eventQueue.length; i++) {
    const event = eventQueue[i];
    processEvent(event);
    broadcastEvent(event);
    eventQueue.splice(i, 1);
    i--;
  }

  // Process other game logic...

  setTimeout(serverLoop, 16);
}

But the damage to my pride was done. In my haste to ride the .io game wave, I had fallen victim to one of the most embarrassing blunders a programmer can make. And in JavaScript, a language infamous for its slipperiness with memory, no less! I was fortunate the leak manifested so early before I had acquired a substantial player base. Imagine the nightmare of diagnosing it with thousands of angry players breathing down my neck.

Humbled and chastened, I took a step back to reevaluate my architecture. It was clear that Node.js and socket.io alone weren‘t going to cut it for the scale I hoped to achieve. I needed a more robust solution that could efficiently handle the massive stream of events and player actions intrinsic to large-scale multiplayer games.

After much research and experimentation, I settled on a new approach. The frontend would stay largely the same, but on the backend, I‘d run multiple Node.js game servers behind a load balancer, with a Redis server acting as a shared memory store and pub/sub layer. Player events would be published to Redis channels and processed asynchronously by the next available game server. Computationally intensive tasks like leaderboard rankings and analytics would be farmed out to worker dynos. Containerizing the game servers in Docker and orchestrating them with Kubernetes would allow me to quickly scale up and down based on player load.

With this revamped architecture in place, I could finally see the path to BlobBattle.io becoming a true "web-scale" multiplayer game. Load testing showed the new setup could comfortably handle over 10,000 concurrent players. The Redis pub/sub layer provided automatic load balancing and fault tolerance. And most importantly, CPU and memory usage remained stable even under heavy, sustained traffic.

It had been a long, painful journey marked by embarrassing missteps, but in the end, I emerged a better, more enlightened developer. I learned the hard way that architecting performant, scalable multiplayer games is a discipline unto itself, full of traps and pitfalls for the unprepared.

But equipped with this newfound knowledge, I was ready to take another crack at conquering the .io game landscape. Watch out slither and agar — BlobBattle is coming for your crown!