In the last number of years, the cost of HDDs and storage has dropped significantly. While cloud storage services like Google Drive or Dropbox have evolved past generic cloud storage, I wanted to answer the question: How do bigger players do it and what challenges do they face? I figured the best way to answer these would be to design and implement it myself, so I went ahead and did so.

stackdrive logo

The project (Getstackdrive.com)

I first identified the minimum core features I (and a few potential users I interviewed) would want:

  1. Storing any type of file on the cloud
  2. Previewing and retrieving uploaded files
  3. Organize files with folders
  4. Search through files and folders
  5. Be privacy friendly (no mining data from user uploads)

As it turns out, “Design Dropbox/Google Drive” is a common system design interview question, so there are a lot of resources and discussions on how it’s done in production at companies that deal with millions of users. After reading through said resources, I started planning out what pieces of the puzzle I would need and wrote it all out in a doc. At some point, I decided to call it Stackdrive.

Database design

The first step I made was to decide on the high level details on what data would be stored on the database. As a cloud service grows in complexity, one can reasonably expect many different tables to be created. However, a barebones cloud services definitely needs at least a way to store the following:

  • Users
  • Files
  • Folders

First, folders. Since I wanted to support nested subfolders and files, having a parent folder ID intuitively makes sense. This way, we can easily check the contents of a folder (or subfolder) by simply querying based on the parent folder ID. Folders look something like this:

{
    id: 1,
    parent_folder_id: NULL,
    name: "mockups"
}

Next, files. As users upload files, it would take an incredible amount of egress to query only from an S3-like storage service. Thus, I would have to track some metadata that looks something like:

{
    id: "3",
    name: "mockup_macOS.png",
    mime_type: "image/png",
    size: 3000,
    parent_folder_id: "1" 
}

These schemas are relatively simple, but that makes it easily extendable as it grows in complexity.

System design overview

If thousands of files were uploaded and processed concurrently, enormous amounts of data would have to pass through my servers and would take significant bandwidth and compute!

Luckily, this problem is solved with presigned URLs. I can rely on Backblaze (or any S3-like provider’s) API to allocate some space that users can directly upload data to, while processing metadata and storing it in database tables.

Similarly, for downloading/previewing files, we don’t need files to travel to our servers; instead, we can simply return a presigned URL.

Below are some rough diagrams I made of to show the flow actually works: Upload flow Download/preview flow

Tech stack

I didn’t spend too much time deciding on the tech stack. I could always migrate to a different framework/service if I needed to, so I just used what I thought can help me move the fastest and what I was most familiar with:

  • Python/FastAPI
  • Typescript + React
  • PostgreSQL
  • AWS, DigitalOcean, Terraform
  • Backblaze B2

It’s safe to say I made the right choice. Even at peak usage with hundreds of concurrent users, my servers ran relatively smooth and I didn’t experience any severe issues.

Choosing a provider to store unstructured data

AWS’s S3 is the most popular service, and for good reason: they provide relatively cheap storage and high reliability. However, I wanted to minimize costs as much as possible to differentiate Stackdrive so users could potentially store terabytes of data for close to nothing. After doing a bit of research, I found a few suitable providers but I ultimately decided to go with Backblaze B2. It fit my needs since it offered free ingress, a generous amount of free egress, and most importantly was much cheaper than other S3 providers.

Conclusion

Previously, I spent a lot of time reading about technologies and understanding the theory behind things for fun. While theory is definitely important, I think there’s a ton of value getting your hands dirty and implementing from scratch. Building a product or feature end-to-end is a unique experience and I’m really glad I built this out. If you’re an engineer/developer reading this, I would recommend you to work on a fun project like this too. You might be suprised with how much you learn :)

As of writing this, Stackdrive has thousands of users and it’s exciting to know people are using your service when they email you with questions or feature requests. I’m still working on mobile apps, and new features for web should be coming in the future!

If you happen to read the entire post, feel free to share questions, comments, or critiques by emailing me: [email protected] Also, find me on LinkedIn.

I read and respond to all emails very quickly :)