Software development at a big company:
what to figure out before you write code

Engineers who are new to a big company often ask me: “This feature would take two days to write. Why do I need a design doc and a stack of meetings first?”

Honestly, I used to think the same thing. Then I stepped on a few landmines: a feature that shipped and nobody used, a cloud bill bigger than the money the feature saved, getting paged in the middle of the night for a system nobody could understand. That's when it clicked. At a big company, writing the code is usually the easy part. The hard part is working out, before you write it, where things are going to break.

This post organizes the questions I ask when designing a system into three stages:

1. Before you design: figure out what problem you're actually solving
2. While you design: make decisions like an owner
3. In the design review: the questions you'll get asked, and the risks they're really probing for

For every question I'll explain why it gets asked. You can only answer a question well once you understand what it's really trying to prevent. Otherwise you're just memorizing a checklist.

PART 1Before you design

The questions that need answers before you start designing anything.

1. Why are we building this feature?

This is the question people skip most often, and it's the most important one. Engineers tend to hear a requirement and jump straight to architecture: “Which model should we use? Do we need Kafka?” But if you don't know who will use it or what problem it solves, you can easily build something technically beautiful that nobody wants.

I break it into three smaller questions.

Who are the users?

“Support agents” is only the first-layer answer. Dig a little deeper and there are actually three groups:

• Support agents: want to type less, and need the summaries to be accurate
• Support managers: when a complaint comes in, want to understand quickly what happened on that call
• QA staff: need to spot-check call quality

Each group needs different things and different permissions (an agent shouldn't be able to see other agents' calls), and all of that shapes the design later on.

What problem does it solve for them?

On the surface it's “save typing time.” But do the math: 5 calls an hour with 3 minutes saved per call is 15 minutes saved every hour, or 2 hours a day. Across 1,000 agents, that's 2,000 hours of work saved every day.

That number matters. It determines how much the company is willing to spend on this feature, and it will keep coming back when we work out costs.

How did users do this before?

This one gets overlooked the most, and it's the one that saves you most often. Until now, agents typed notes into a free-text field in the CRM, and managers investigating a complaint would pull up the recording and listen to it.

That tells us something: the summary has to end up back in the CRM. Build it as a separate website and agents have to copy and paste between two systems. You've just swapped “typing” for “copy-paste,” and within a month nobody will be using it.

2. How will we build it?

Where does the core run? Endpoint, cloud, or local server

The same feature turns out very differently depending on where its core lives:

Customers regularly read out ID numbers and credit card numbers on support calls, so this is usually not a decision engineers can make alone. Ask security and legal first. First, not after you've built it.

What's the interface? Desktop app, mobile app, or web

Agents keep the CRM open in a browser all day, so the obvious answer is a web plugin inside the CRM.

But that depends on where the recordings come from. If the phone system (PBX) already records calls, the backend just fetches the files, and the frontend is a web plugin that shows the summary. If the phone system doesn't record, you need to capture audio from the softphone on the agent's PC, and a browser can't do that. Now you need a desktop app, and a desktop app means dealing with installation, auto-updates, and locked-down company machines.

A requirement detail that looks tiny ends up deciding the entire frontend.

What's the main tech stack?

The speech recognition and LLM ecosystems are almost entirely Python, so Python makes sense for the workers that process audio. For the API service handling lots of concurrent requests, Go might use fewer resources. And if the frontend is a CRM plugin, you have to follow that CRM platform's development framework. No choice there.

3. How many users? How often? How much data?

You need numbers. “Lots of people use it” is not a scale. “1,000 people, 5 times an hour each, 10MB per request” is.

Let's work it out for our example:

Here's the interesting part: 1.4 requests per second is small enough for a single machine. But 8.8TB of recordings a month adds up to more than 100TB in a year.

The challenge in this system isn't QPS at all. It's bandwidth, storage cost, and processing time. Without running the numbers, it's easy to put effort in the wrong place, like spending two weeks tuning API performance.

“Working hours only” is important information too:

• On nights and weekends you can scale down to the minimum and save money
• But everyone logs in at 9 a.m., so watch out for cold starts
• After monthly bills go out, call volume can multiply, so estimate that peak separately

4. What are the system's constraints?

Every technology has built-in bottlenecks. Better to know them before you design than to hit the wall after launch.

The CAP theorem

A distributed system can only guarantee two of consistency, availability, and partition tolerance. In practice, networks do fail and partitions do happen, so the real choice is: when the network breaks, do you want “the data is always correct” or “the service always responds”?

• Billing and inventory: better to be briefly unavailable than to get the numbers wrong, so choose C
• Like counts and view counts: a few seconds of delay is fine, but the page can't go down, so choose A

Message size and timeout limits

This one is critical for our example, because every recording is 10MB:

Once you know these limits, the design mostly falls out on its own:

• Don't send files through the API. Use a pre-signed URL so the client uploads directly to S3
• Put only the file path on the queue, not the file itself (this is the claim-check pattern)
• Transcription will take longer than 29 seconds, so processing has to be asynchronous

5. What does each user cost?

What's the market price? What's our target? You need this even for an internal tool, because leadership will compare it with just buying an off-the-shelf SaaS product.

Compute: 880 × $0.02 = $17.60 per user per month
Storage: each user adds 8.8GB a month; small at first, but it piles up if nothing is deleted
Cost per user per month ≈ $18 and up, and rising every month

Compared with $30 there seems to be room. But add operations staff, idle machines, and ever-growing storage, and you may end up paying about the same as you would to buy. At that point the design direction changes:

• Could we summarize only calls longer than 2 minutes? (Short calls are usually transfers or wrong numbers)
• Could a smaller model classify calls first, and send only the complex ones to the large model?
• Could recordings be deleted after 30 days, keeping only the text summary?

If this were a product sold to external customers: SaaS companies commonly target 70–80% gross margins, which means the cost of serving each user should stay under roughly 20–30% of the price if it's going to fund the team.

6. How many people and how much time to reach production quality?

When the demo works, you're about halfway done.

Production still needs monitoring and alerting, access control, a security review, load testing, error handling, an on-call runbook, documentation… These often take the other half of the time or more, and they're the things most often left out of the schedule.

When estimating, I split the work into two kinds.

Obvious gaps you'll have to build yourself

For example, support calls often involve heavy accents, regional dialects, or people switching languages mid-sentence, and off-the-shelf speech recognition may not be accurate enough. Work like this, where no ready-made solution exists, is research work: it might take two weeks, or you might still be tuning it three months later.

Pull this out and estimate it separately, and be upfront about the uncertainty. Don't lump it in with regular development work.

Stopgaps that fill the hole now but need revisiting

• v1 handles only the primary language, and low-confidence transcripts get flagged “needs human review”
• v1 doesn't write summaries back to the CRM automatically: an agent clicks “Confirm” first, so the AI can't write bad data
• v1 processes in batches every 5 minutes, and switches to real time later

Compromises are fine, but write them into the doc, file the tickets, and agree on when you'll pay them back. A “temporary” fix nobody wrote down always becomes permanent.

7. What existing systems do we need to fit with?

A big company isn't a blank slate. Whatever you design has to fit into an environment that may have been running for more than a decade. The company's preferred languages, systems, software, and vendors are all part of the design's starting point.

The company's preferred languages

If everyone at the company is a Python or Go expert, think twice before choosing Elixir, even if it seems like a great fit. The reasons are practical:

• Nobody can read it in code review, so review becomes a formality
• If the system breaks while you're on vacation, nobody can debug it
• When it's time to hand it off, nobody wants to take it

Pick languages people already know, and when something goes wrong, someone can quickly review and help.

Existing platforms and data formats

Say the company's data platform runs on Starburst and prefers the Iceberg format. Then historical call summaries should land as Iceberg tables, so the data team can analyze them right away without writing a separate ETL pipeline.

Other common compatibility questions

• Login: the company already has SSO (Azure AD / Okta), so don't build your own username-and-password system
• Monitoring and logging: if the company has standardized on Datadog or ELK, plug into it instead of running your own
• Vendors: if the company has an enterprise discount with one cloud provider, switching may mean months of procurement and legal review
• CRM: to write back to the CRM, check first whether its API has rate limits

PART 2While you design

Make decisions like an owner

The biggest mindset shift when designing a system, I think, is this: imagine the feature is your own company. The cloud bill arrives at your house every month, and when it breaks in the middle of the night, your phone is the one that rings.

Seen that way, you naturally start caring about the following.

1. Scalability: when volume grows, is adding machines enough?

Next year the company acquires another support center, and 1,000 agents become 3,000. In a good design, you go from 5 workers to 15 and you're done. In a bad design, the database can't cope and you need two days of downtime for a data migration.

2. Extensibility: do new features mean a rewrite?

Scalability is about volume; extensibility is about features. Three months after launch, a manager will inevitably ask: “Since we have transcripts anyway, can we add sentiment analysis? Can it categorize complaints automatically?”

If transcription emits a “transcript ready” event when it finishes, a new feature just subscribes to that event. If transcription, summarization, and the CRM write-back are hardwired into one function, every new feature touches the core flow, and every change risks breaking it.

3. Acceptable cost

Keep pulling out that per-user cost while you design. For every component you add (a cache, a queue, a read replica), ask yourself: is what it gives us worth what it costs?

4. Acceptable maintenance burden

Running your own Kafka + Kubernetes + Cassandra sounds impressive, but with a three-person team, just upgrading versions and handling node failures will eat all of the team's time.

Very often, paying a bit more for a managed cloud service so the team can sleep at night is an excellent trade.

5. Give reviewers a choice

A lot of people miss this one: design at least two options and compare their trade-offs.

Why? If you present a single option, reviewers can only say “yes” or “no,” and they'll be wondering: “Did you even consider anything else?” The rest of the meeting turns into everyone pitching the design in their own head, and the discussion goes nowhere.

Put two options side by side and the discussion focuses on which trade-off we actually want:

Finish with which option you recommend, and why. It's also worth one line on why each rejected option lost: when someone asks six months later “why didn't we go with B?”, the doc has the answer.

(The design review examples that follow assume we recommended Option A: security approved sending recordings to the cloud, so we launch quickly on managed services.)

A PoC proves the path works; it isn't a system

This is a very common trap.

A PoC (proof of concept) answers one specific question: “Does this path work?” For example:

Take 50 real support recordings (including dialects, background noise, and angry customers shouting), run speech recognition and summarization on them, and see whether accuracy reaches 85%.

This PoC needs no login, no database, and no nice UI. A script and a results report are enough, and you should have an answer within one or two weeks. Throwing all the code away afterwards is completely fine.

The classic failure goes like this: the PoC slowly grows a UI and a database, it gets demoed to leadership, and leadership says: “Isn't this done? Let's launch next month!” And something with no access control, no monitoring, and no error handling ends up in production.

A prototype comes after the design is decided. It's built on the chosen architecture and given to a small group of real users, and it validates the workflow and the experience: do agents actually read the summaries? Where in the CRM is the summary most convenient?

The order is:

PART 3In the design review

A design review isn't there to make your life hard. It's a group of people who've stepped on these landmines before, helping you find the risks in your system.

And the goal is not zero risk. A zero-risk system doesn't exist, and if it did you couldn't afford it. The real goal is to turn “risks we don't know about” into “risks we know about and choose to accept,” and be clear about what they cost.

For example:

• “A manager's stats report may lag by up to a minute”: acceptable, the cost is small
• “Agent A might be able to see Agent B's calls”: not acceptable, must be fixed

Below are the questions that come up most often in design reviews. For each one I've added what it's really probing for. Think that through first, and your answers will have a much clearer direction.

Q1What problem does the system solve? What features does it provide? How big is the expected scale?

What they're really probingWhether everyone is talking about the same thing.

This is usually the first question. If you can't explain it in a sentence or two, the rest of the discussion loses focus. A good answer has users, value, and numbers:

“Give 1,000 support agents a summary within 1 minute of every call, saving 3 minutes of typing per call. Peak volume is about 5,000 calls an hour, or about 8.8TB of recordings a month.”

It's also worth stating clearly what this version won't do (non-goals), like “no live captions, no sentiment analysis.” Otherwise, halfway through the meeting someone will say: “While we're at it, adding X can't be that hard, right?”

Q2What technical options are there? What are the pros and cons of each?

What they're really probingWhether you actually compared them, or picked an answer first and found reasons later.

This is the “at least two options” comparison table from earlier. Reviewers pay close attention to whether the comparison is fair: if the recommended option lists nothing but strengths and the other lists nothing but weaknesses, people will quickly see it's a straw man.

Being honest about your recommended option's weaknesses actually makes it more convincing.

Q3How are tasks dispatched? Sync or async? Do you need a queue? How do you manage queue length and timeouts?

What they're really probingWhether the system collapses at peak, how long users wait, and whether tasks get lost or run twice.

Transcribing a 10-minute recording might take 30–60 seconds, and as mentioned earlier, API Gateway times out at 29 seconds by default, so synchronous processing won't work. The flow becomes:

Once there's a queue, the follow-up questions always come:

• Queue length: call volume triples at the start of the month and 20,000 tasks pile up. Now what? Set alerts (e.g., “the oldest task has been waiting more than 5 minutes”), and decide ahead of time whether to degrade when the backlog gets too large, for example by transcribing first and filling in summaries later.
• Timeouts: the queue's visibility timeout (how long a task that was picked up but not finished waits before going back on the queue) must be longer than the longest processing time. Set it to 30 seconds when processing takes 60, and a second worker picks up the same task, so the same call ends up with two summaries in the CRM. This really is a very common bug.
• Tasks that keep failing: after 3 failed retries, send them to a dead-letter queue (DLQ) instead of retrying forever and clogging the queue, and alert someone to investigate.

Q4Is the core workload stateless or stateful? How is user state recorded, kept, and restored? Do you need sticky sessions?

What they're really probingWhether machines can be added and removed freely, and whether users lose data when a machine dies.

Stateless means any machine can handle any request: if a machine dies you replace it, and to scale you add machines. Stateful means some data lives only in one machine's memory, and when that machine dies, the data goes with it.

For example: if a worker keeps track of how far it has gotten into a recording in memory, and autoscaling kills that machine while scaling in, the call starts over from scratch. The right approach is to write progress to the database or S3, so the worker itself holds no state.

The trickier part is telling the agent “your summary is ready.” If you push it over a WebSocket, the connection itself is tied to one machine, which makes it inherently stateful. There are two common approaches:

• Sticky sessions: the load balancer uses a cookie to pin each user to the same machine. Simple, but when that machine dies everyone on it disconnects at once, and scaling in means waiting for connections to drain before you can shut a machine down.
• Separate connections from state: when a worker finishes, it publishes a message to pub/sub (e.g., Redis). Every connection node receives it, and whichever node holds that user's connection pushes it out. If any node dies, the client just reconnects to another one.

As for restoring: when an agent reconnects, the frontend looks up the latest state in the database by call_id, rather than relying on what the previous connection remembered.

Q5As users and data grow, how does the system scale? Stateless horizontal scaling, or sharding? What metrics trigger scaling?

What they're really probingWhether growth means “add machines” or “redesign.”

There are two kinds of growth, and they need to be looked at separately.

More users (more requests): stateless horizontal scaling

Going from 1,000 agents to 3,000: as long as the API and workers are stateless, adding machines solves it.

More data (bigger tables): partitioning, hot/cold separation, sharding

A year later the database holds tens of millions of call records, and a manager's query for “every complaint in the past year” keeps getting slower. Adding API machines won't help; the bottleneck is the database. Consider these, in order:

1. Partition by month, so queries only scan the months they need
2. Move cold data older than a year to Iceberg and query it with Starburst (conveniently, the company's existing platform)
3. Consider sharding only when write volume truly can't be handled any other way

The most important thing about sharding is how you choose the shard key. Use the date as the shard key, and all of today's data lands on one shard: that shard is always the hottest while the others sit idle, which is as good as not sharding at all. Support center ID or agent ID spreads the load much more evenly. A bad shard key is nearly impossible to change later, so reviewers usually dig into this in detail.

Metrics that trigger scaling

Many people instinctively use CPU, but in our example that's wrong. Workers spend most of their time waiting on the external speech recognition API (they're I/O bound), so CPU might sit at 20% while 2,000 tasks pile up in the queue. With CPU as the trigger, you'd never scale out.

Better metrics:

• Queue depth ÷ number of workers (the average backlog per worker)
• Wait time of the oldest task
• API p95 latency

Q6Does the system need a cache? What TTL? How do you avoid inconsistent data?

What they're really probingA cache trades consistency for speed. Do you know what you gave up?

First: not every system needs a cache. An internal system for 1,000 people may be perfectly fine hitting the database directly. Every cache layer is one more source of inconsistent data. Measure first, and add one only once you've confirmed you need it.

If you do need one, decide based on the nature of the data:

• A manager's “today's call stats”: computing it scans a lot of data every time, so caching it for 60 seconds is totally fine; nobody cares if it's a minute old
• Call summary content: an agent just edited it, but the screen still shows the old version, so they assume it didn't save, edit it again, or even file a bug

For the latter, clear the cache on write (delete the matching cache key right after updating the database), and treat the TTL as a last line of defense.

There's another common trap: a cache avalanche. If lots of keys are written at the same time with the same TTL (say, a batch load at 9 a.m. with a one-hour TTL), every entry expires at exactly 10:00 and all that traffic lands on the database at once. The fix is to add a little randomness (jitter) to the TTL, e.g., 55–65 minutes, so expirations are spread out.

Q7How is the database designed? CAP trade-offs? Read/write splitting, sharding, backup and restore, Multi-AZ, I/O capacity?

What they're really probingThe database is the hardest part of a system to change later. Get it wrong early and it costs the most.

CAP trade-offs

Different data in the same system can make different choices. A rule like “each call has exactly one official summary” needs strong consistency; the stats report can live with eventual consistency.

Read/write splitting

Managers' big reports are resource-hungry. Run them against a read replica so they don't slow down agents' writes. But replicas lag behind (replication lag): an agent saves a summary, the screen immediately reads it back from the replica, and they think “wait, it didn't save?” So reads of data a user just wrote should go to the primary.

Backup and restore

“We have backups” is not the same as “we can restore.” Reviewers may well ask: have you actually done a restore? How many hours does restoring 2TB take? What does the system do in the meantime?

With backups you've never rehearsed, you usually find out during an incident that a table was never included, or that the restore takes 12 hours.

Multi-AZ

When one data center (an Availability Zone) loses power or network, can the database fail over to another one automatically? The cost roughly doubles, so weigh it against what an hour of downtime would cost you.

I/O capacity

Cloud disks have IOPS limits (AWS gp3, for example, has a 3,000 IOPS baseline). A common story: a nightly batch report maxes out IOPS, online writes slow down with it, and it looks like the database is down, but the CPU is nearly idle. If you're not watching I/O metrics, this takes a long time to track down.

Q8How does data sync work? How do you control concurrent access to data?

What they're really probingWhen two people edit the same record at once, whether someone's changes silently disappear.

Common fixes:

• Optimistic locking: add a version column to each record and update with the condition WHERE id = ? AND version = ?. If someone else has already changed the version, the update fails and the user sees “this record was modified by someone else, please refresh.”
• Split the data: the manager's note doesn't need to live in the same record as the summary. Move it to its own table and the conflict disappears.

Syncing across systems is a separate problem: the summary has to be written back to the CRM, but what if the CRM's API happens to be down?

• First decide what the source of truth is: our database, or the CRM?
• Failed writes need to be retried, and retries must be idempotent. Use call_id as the idempotency key, and even after ten retries the CRM will only have one summary.

Q9How are logs collected? What errors can you analyze? Can you query unusual system events? Are user actions kept in an audit log?

What they're really probingWhen something goes wrong, whether you can find out what happened.

Scenario: an agent reports “my call yesterday afternoon has no summary.” If your logs are just lines of text with no IDs tying them together, you're looking for a needle in millions of log lines.

A few basic principles:

• Structured logs (JSON), so they're easy to search and aggregate
• One trace ID across the whole flow: upload → queue → worker → CRM, so a single ID shows the entire path
• Categorize errors: speech recognition failures, LLM timeouts, and CRM write failures, each counted separately, so you know what to fix first
• Don't log personal data: once ID numbers or phone numbers get into your logs, the logging system itself becomes a source of data leaks

An audit log is a different thing. Call recordings contain customers' personal data, and during a security audit or a breach investigation, someone will definitely ask: “Who listened to which recording, who downloaded which summary, and when?” These logs have to be stored separately from ordinary logs and be tamper-proof, with retention set by company policy or regulation.

Q10What metrics should the system monitor? CPU, memory, disk, network, request volume, cache hit rate, connection pool size, and latency?

What they're really probingWhether you find out something's wrong before your users do.

I split metrics into two layers:

• Resource metrics (CPU, memory, disk, network): tell you whether the machines are healthy
• Service metrics (request volume, error rate, latency, queue depth, cache hit rate, connection pool utilization): tell you whether users are having a good experience

A few things that really happen, and why these metrics matter:

• Connection pools: each worker opens 20 database connections. At peak, autoscaling takes workers from 10 to 50, that's 1,000 connections, and the database's limit is exceeded. Scaling out is what takes the system down. Without monitoring connection pools, you'd have no idea why adding machines broke things.
• Look at p99 latency, not just the average: an 800ms average looks great, but a p99 of 15 seconds means 1 call in every 100 leaves an agent waiting a long time. Averages hide problems.
• Cache hit rate: a drop from 90% to 20% is usually an early warning that the database is about to be overwhelmed.
• Disk: logs fill the disk, the service can't write anything, and it falls over. This one is a classic that almost everyone has hit.

Q11How do authentication and authorization work? How do you stop users from accessing other people's data? What login methods are supported? How are accounts created?

What they're really probingSecurity holes, and the most common, most serious kind.

First, keep the two straight: authentication is “who you are,” and authorization is “what you're allowed to do.”

The most common vulnerability here is called IDOR (insecure direct object reference): an agent opens /calls/12345/summary to view their own call, gets curious and changes the URL to 12346, and sees someone else's call.

Hiding the button in the frontend does nothing. Every API has to check on the backend whether this person is allowed to see this record: for example, it's their own call, or they're a manager and it's a call from someone on their team.

Login: for internal systems, integrate directly with the company's SSO (SAML / OIDC with Azure AD or Okta). The biggest problem with rolling your own usernames and passwords: when an employee leaves, their AD account gets disabled, but their account in your system is still there, and they can still log in.

Account creation: internal systems usually don't allow self sign-up. Accounts sync from AD groups instead (e.g., via SCIM), so permissions follow HR changes automatically.

Q12What does the system cost? How much traffic can it handle? How much can each service unit support?

What they're really probingWhether you have measured numbers for the system's capacity, or just guesses.

A “service unit” is how much a single worker or pod can handle. Measure it with load testing. Once you have that number, you can actually calculate scaling and cost:

Load test: one worker completes 12 ten-minute recordings per minute
Normal peak: 5,000 calls/hour ≈ 83 calls/minute → 7 workers
Headroom (survive 2 failures) → 9 workers
3× traffic at the start of the month: ~250 calls/minute → 21+ workers

Worker count times unit price is your monthly cost; divide by the number of users and you're back to “what does each user cost?”

Also be clear about what happens beyond capacity: does the queue just slow down? Do you rate-limit and reject requests? Or degrade to transcription only?

Q13What's the failover mechanism for each component? How long does recovery take?

What they're really probingEvery component will fail eventually. When it does, what do users see?

Two commonly used metrics:

• RTO (Recovery Time Objective): how quickly service has to be restored
• RPO (Recovery Point Objective): the most data loss you can accept, measured in time

List every component and answer for each one:

That last row is the one people forget most. When an external dependency breaks, you can't fix it at all; all you can do is plan a fallback ahead of time. Also, a broken summary can be regenerated, but a lost recording is gone for good, so their RPO requirements are completely different.

Q14What's the error-handling strategy? Should input be validated or limited? Should output be paginated? Should failures be retried, and how many times, how far apart?

What they're really probingNever trust input, and always assume calls will fail.

Input validation

So input needs limits:

• Maximum file size (e.g., 50MB)
• Maximum recording length (e.g., 2 hours; anything longer goes through a special path)
• File format: check the actual file header, not just the extension

Output pagination

A manager queries “all calls this month,” the API tries to return 100,000 records at once, the API times out, and the browser freezes. List APIs must be paginated; with large datasets, use cursor-based pagination rather than OFFSET (which gets slower the further you page).

Retry strategy

• Only retry errors that might succeed: timeouts, 429 Too Many Requests, and 503 are worth retrying; a 400 bad request won't succeed if you retry it a hundred times
• Exponential backoff + jitter: wait 1 second, 2 seconds, 4 seconds… plus some randomness, capped at, say, 5 attempts
• Avoid retry storms: when an external API slows down and every client retries immediately at the same moment, traffic instantly multiplies and a service that was merely slow gets knocked over completely. Besides jitter, add a circuit breaker (stop calling for a while when the failure rate gets too high)
• Retried operations must be idempotent: as covered earlier, retries must never cause duplicate writes

Q15Which platforms are supported? Do you need responsive design?

What they're really probingEvery extra platform means more development, testing, and maintenance. Are you sure you need it?

• Agents all use company-issued Windows PCs with Chrome: a web app is enough
• Managers want to check reports on their phones: responsive web is enough, no native app needed
• But if the requirement becomes “field staff record audio on their phones at customer sites”: now you're handling offline mode, background uploads, and iOS background-execution limits, which is a completely different level of work

Explicitly listing unsupported platforms matters too, for example “uploading recordings from mobile is not supported,” so it doesn't get reported as a bug after launch.

Q16What are the system's limits? Under what conditions are they hit?

What they're really probingWhether you know where your own system's limits are, and what happens when you hit them.

The “constraints” we asked about before designing were limits of external technology. Here the question is about your own system's limits. For each one, spell out what triggers it and how the system behaves when it does:

Write these down, and a middle-of-the-night incident becomes “known behavior, documented in the design doc,” and support managers know in advance how to respond.

Q17How do you troubleshoot problems? What does the system record about users and system state?

What they're really probingWhether debuggability was designed in from the start, or logging gets bolted on after something breaks.

Let's walk through a real troubleshooting session to see what information you need.

To be able to follow that path, you need to have been recording:

• About users: user ID, role, support center, client version, OS and browser
• About requests: call_id, trace ID, file size, recording length and format, timestamps and results for each stage
• About system state: deployed version, feature flag states, which worker processed it, health of dependencies at the time

Especially version information. Conclusions like “only client version 2.3.1 has the problem” or “it started breaking after the phone system upgrade” are impossible to reach if you never recorded versions.

Q18How long is data retained? Is that acceptable for the system?

What they're really probingKeep data forever and cost grows without limit; keep it too long and legal and breach risk grow with it.

Look back at the earlier numbers: 8.8TB of recordings a month. Kept forever, that's more than 100TB after a year and more than 300TB after three. At a rough $0.023 per GB per month for S3 Standard (actual prices vary by region), recordings alone cost over two thousand dollars a month after the first year, and the bill keeps climbing.

The common approach is tiering data by how often it's accessed:

The retention period is not something engineers decide alone. Work it out with legal and security:

• Too short: may fail audit or industry requirements (some industries require records to be kept for a set number of years)
• Too long: the longer personal data is kept, the bigger the impact of a breach, and it runs against privacy rules such as GDPR's storage-limitation principle, under which personal data shouldn't be kept longer than its purpose requires

One last detail that often gets missed: deletion has to be complete. Delete the recordings but leave the backups, logs, derived transcripts, summaries, and search indexes behind, and you haven't really deleted anything.

WRAP-UPWrapping up

That's a lot of questions, but you don't need a perfect score on every one. The point isn't a thicker design doc. It's to help you do three things:

At a big company, one design decision affects several teams and years of operations. Spending a week or two thinking these questions through saves you months of landmines after launch.

Next time you're stuck on a design doc, try playing the reviewer and asking yourself these questions first. The ones you can answer go in the doc. The ones you can't are exactly where your thinking isn't done yet.