Enhance README.md for Mint.com and Pastebin.com designs
- Updated Mint.com design document to clarify problem investigation, use cases, and constraints, including detailed example dialogues for interviews. - Improved high-level design section with a clearer structure and added a diagram for better visualization. - Revised Pastebin.com design document to enhance clarity on use cases, high-level design, and scaling considerations, including a wrap-up section for summarizing discussions with interviewers.pull/1079/head
parent
579c1161bf
commit
114a621e2d
|
@ -1,36 +1,65 @@
|
||||||
# Design Mint.com
|
# Design personal budget tracking app (Mint.com)
|
||||||
|
|
||||||
*Note: This document links directly to relevant areas found in the [system design topics](https://github.com/ido777/system-design-primer-update#index-of-system-design-topics) to avoid duplication. Refer to the linked content for general talking points, tradeoffs, and alternatives.*
|
*Note: This document links directly to relevant areas found in the [system design topics](https://github.com/ido777/system-design-primer-update#index-of-system-design-topics) to avoid duplication. Refer to the linked content for general talking points, trade-offs, and alternatives.*
|
||||||
|
|
||||||
## Step 1: Outline use cases and constraints
|
## Step 1: Investigate the problem, use cases and constraints and establish design scope
|
||||||
|
|
||||||
> Gather requirements and scope the problem.
|
> Gather main functional requirements and scope the problem.
|
||||||
> Ask questions to clarify use cases and constraints.
|
> Ask questions to clarify use cases and constraints.
|
||||||
> Discuss assumptions.
|
> Discuss assumptions.
|
||||||
|
|
||||||
Without an interviewer to address clarifying questions, we'll define some use cases and constraints.
|
|
||||||
|
Adding clarifying questions is the first step in the process.
|
||||||
|
Remember your goal is to understand the problem and establish the design scope.
|
||||||
|
|
||||||
|
### What questions should you ask to clarify the problem?
|
||||||
|
|
||||||
|
|
||||||
|
Here is an example of the dialog you could have with the interviewer:
|
||||||
|
|
||||||
|
**Example dialog with your interviewer:**
|
||||||
|
|
||||||
|
|
||||||
|
**Interviewer**: Design Mint.com.
|
||||||
|
**Candidate**: Sure—could you remind me what the core value proposition of Mint.com is?
|
||||||
|
**Interviewer**: It aggregates users’ financial accounts, categorizes transactions, and helps them budget.
|
||||||
|
**Candidate**: Got it. How do we get the data from the financial accounts?
|
||||||
|
**Interviewer**: connects to a financial account.
|
||||||
|
**Candidate**: Should we focus on real-time updates or would a daily refresh suffice?
|
||||||
|
**Interviewer**: Daily is fine.
|
||||||
|
**Candidate**: How should categorization work?
|
||||||
|
**Interviewer**: Auto-categorize based on merchant, but allow manual overrides. No re-categorization once set.
|
||||||
|
**Candidate**: Understood. What budget features are needed?
|
||||||
|
**Interviewer**: Recommend budgets by category, allow manual budgets, and notify when users approach or exceed them.
|
||||||
|
**Candidate**: Any non-functional requirements?
|
||||||
|
**Interviewer**: High availability is a must; budget alerts don’t need millisecond latency.
|
||||||
|
**Candidate**: Great. And scale?
|
||||||
|
**Interviewer**: Target ~10 M users, ~30 M linked accounts, ~5 B transactions/mo, with ~500 M reads/mo.
|
||||||
|
|
||||||
### Use cases
|
### Use cases
|
||||||
|
|
||||||
#### We'll scope the problem to handle only the following use cases
|
#### We'll scope the problem to handle only the following use cases
|
||||||
|
|
||||||
* **User** connects to a financial account
|
* **User** connects to one or more financial accounts (bank, credit card, investment).
|
||||||
* **Service** extracts transactions from the account
|
* **Service** extracts transactions from the account
|
||||||
* Updates daily
|
* Daily batch updates for active users (last 30 days).
|
||||||
* Categorizes transactions
|
* Categorizes transactions
|
||||||
|
* Auto-categorization by merchant
|
||||||
* Allows manual category override by the user
|
* Allows manual category override by the user
|
||||||
* No automatic re-categorization
|
* No automatic re-categorization after manual change.
|
||||||
* Analyzes monthly spending, by category
|
* Analyzes monthly spending, by category
|
||||||
* **Service** recommends a budget
|
* **Service** Analyze spending & suggest budgets
|
||||||
* Allows users to manually set a budget
|
* Compute monthly spend per category.
|
||||||
* Sends notifications when approaching or exceeding budget
|
* Recommend a budget per category.
|
||||||
|
* Allow users to set or adjust budgets manually.
|
||||||
|
* Send notifications when spending approaches/exceeds budget.
|
||||||
* **Service** has high availability
|
* **Service** has high availability
|
||||||
|
|
||||||
#### Out of scope
|
#### Out of scope
|
||||||
|
|
||||||
* **Service** performs additional logging and analytics
|
* **Service** performs additional logging and analytics
|
||||||
|
|
||||||
### Constraints and assumptions
|
### Constraints & assumptions
|
||||||
|
|
||||||
#### State assumptions
|
#### State assumptions
|
||||||
|
|
||||||
|
@ -50,19 +79,33 @@ Without an interviewer to address clarifying questions, we'll define some use ca
|
||||||
* 5 billion transactions per month
|
* 5 billion transactions per month
|
||||||
* 500 million read requests per month
|
* 500 million read requests per month
|
||||||
* 10:1 write to read ratio
|
* 10:1 write to read ratio
|
||||||
* Write-heavy, users make transactions daily, but few visit the site daily
|
* Write-heavy, users make transactions daily, but few visit the site daily
|
||||||
|
|
||||||
#### Calculate usage
|
#### Back-of-the-envelope usage calculations
|
||||||
|
|
||||||
**Clarify with your interviewer if you should run back-of-the-envelope usage calculations.**
|
> **Clarify with your interviewer if you should run back-of-the-envelope usage calculations.**
|
||||||
|
> **If** you need to calculate usage, here is calculation example:
|
||||||
|
|
||||||
* Size per transaction:
|
```text
|
||||||
* `user_id` - 8 bytes
|
Size per transaction record:
|
||||||
* `created_at` - 5 bytes
|
user_id 8 bytes
|
||||||
* `seller` - 32 bytes
|
timestamp 5 bytes
|
||||||
* `amount` - 5 bytes
|
merchant_id 32 bytes
|
||||||
* Total: ~50 bytes
|
amount 5 bytes
|
||||||
* 250 GB of new transaction content per month
|
category code 2 bytes
|
||||||
|
--------------------------
|
||||||
|
≈ 52 bytes/transaction
|
||||||
|
|
||||||
|
Monthly data volume:
|
||||||
|
52 bytes × 5 000 000 000 txns ≈ 260 GB/month
|
||||||
|
→ ~9 TB over 3 years
|
||||||
|
|
||||||
|
Request rates:
|
||||||
|
5 000 000 000 transactions / (2.5 million sec/mo) ≈ 2 000 writes/sec
|
||||||
|
500 000 000 reads / (2.5 million sec/mo) ≈ 200 reads/sec
|
||||||
|
```
|
||||||
|
|
||||||
|
* 260 GB of new transaction content per month
|
||||||
* 50 bytes per transaction * 5 billion transactions per month
|
* 50 bytes per transaction * 5 billion transactions per month
|
||||||
* 9 TB of new transaction content in 3 years
|
* 9 TB of new transaction content in 3 years
|
||||||
* Assume most are new transactions instead of updates to existing ones
|
* Assume most are new transactions instead of updates to existing ones
|
||||||
|
@ -76,11 +119,87 @@ Handy conversion guide:
|
||||||
* 40 requests per second = 100 million requests per month
|
* 40 requests per second = 100 million requests per month
|
||||||
* 400 requests per second = 1 billion requests per month
|
* 400 requests per second = 1 billion requests per month
|
||||||
|
|
||||||
## Step 2: Create a high level design
|
|
||||||
|
|
||||||
|
|
||||||
|
## Step 2: Create a high level design & Get buy-in
|
||||||
|
|
||||||
> Outline a high level design with all important components.
|
> Outline a high level design with all important components.
|
||||||
|
|
||||||

|
<!--  -->
|
||||||
|
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
%%{init: { "flowchart": { "htmlLabels": true } }}%%
|
||||||
|
|
||||||
|
flowchart TB
|
||||||
|
%% Client Layer
|
||||||
|
subgraph Client["**Client**"]
|
||||||
|
direction TB
|
||||||
|
WebClient[Web Client]
|
||||||
|
MobileClient[Mobile Client]
|
||||||
|
end
|
||||||
|
|
||||||
|
%% Web Server Layer
|
||||||
|
subgraph WebServer["**Web Server - (Reverse Proxy)**"]
|
||||||
|
direction LR
|
||||||
|
AccountsAPI[Accounts API]
|
||||||
|
Queue[Queue]
|
||||||
|
TransactionExtractionService[Transaction Extraction Service]
|
||||||
|
CategoryService[Category Service]
|
||||||
|
BudgetService[Budget Service]
|
||||||
|
NotificationService[Notification Service]
|
||||||
|
|
||||||
|
AccountsAPI --> Queue
|
||||||
|
Queue --> TransactionExtractionService
|
||||||
|
TransactionExtractionService --> CategoryService
|
||||||
|
TransactionExtractionService --> BudgetService
|
||||||
|
TransactionExtractionService --> NotificationService
|
||||||
|
end
|
||||||
|
|
||||||
|
%% Storage Layer
|
||||||
|
subgraph Storage["**Storage**"]
|
||||||
|
direction LR
|
||||||
|
ObjectStore[(Object Store)]
|
||||||
|
SQLDB[(SQL Database)]
|
||||||
|
end
|
||||||
|
|
||||||
|
%% Data Flow
|
||||||
|
Client --> WebServer
|
||||||
|
WebServer --> Storage
|
||||||
|
AccountsAPI --> SQLDB
|
||||||
|
TransactionExtractionService --> SQLDB
|
||||||
|
TransactionExtractionService --> ObjectStore
|
||||||
|
|
||||||
|
|
||||||
|
%% Styling Nodes
|
||||||
|
|
||||||
|
style WebClient fill:#FFCCCC,stroke:#CC0000,stroke-width:2px,rx:6,ry:6
|
||||||
|
style MobileClient fill:#FFD580,stroke:#AA6600,stroke-width:2px,rx:6,ry:6
|
||||||
|
style AccountsAPI fill:#CCE5FF,stroke:#004085,stroke-width:2px,rx:6,ry:6
|
||||||
|
style TransactionExtractionService fill:#CCE5FF,stroke:#004085,stroke-width:2px,rx:6,ry:6
|
||||||
|
style Queue fill:#D4EDDA,stroke:#155724,stroke-width:2px,rx:6,ry:6
|
||||||
|
style SQLDB fill:#E2E3E5,stroke:#6C757D,stroke-width:2px,rx:6,ry:6
|
||||||
|
style ObjectStore fill:#E2E3E5,stroke:#6C757D,stroke-width:2px,rx:6,ry:6
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
### Get buy-in
|
||||||
|
|
||||||
|
✅ Why This Breakdown?
|
||||||
|
|
||||||
|
Rather than diving into implementation, this diagram tells a story:
|
||||||
|
|
||||||
|
The microservice breakdown is driven by **Separation of Concerns**:
|
||||||
|
|
||||||
|
- **Single-purpose services**: Extraction, Categorization, Budget and Notification each handle only one domain, which simplifies testing, deployment and independent versioning.
|
||||||
|
- **Asynchronous decoupling**: We buffer all raw ingestion through a message queue so that spikes or transient failures don’t block users. (The diagram shows the main queue; internal queues between downstream steps are omitted for clarity.)
|
||||||
|
- **Optimized storage**: Raw transaction dumps live in an object store, while structured metadata resides in SQL—letting us choose the right storage for each access pattern and consistency requirement.
|
||||||
|
- **Scalable**: Services are stateless and can be scaled or replaced independently.
|
||||||
|
|
||||||
|
|
||||||
|
You should ask for a feedback after you present the diagram, and get buy-in and some initial ideas about areas to dive into, based on the feedback.
|
||||||
|
|
||||||
|
|
||||||
## Step 3: Design core components
|
## Step 3: Design core components
|
||||||
|
|
||||||
|
@ -88,7 +207,8 @@ Handy conversion guide:
|
||||||
|
|
||||||
### Use case: User connects to a financial account
|
### Use case: User connects to a financial account
|
||||||
|
|
||||||
We could store info on the 10 million users in a [relational database](https://github.com/ido777/system-design-primer-update#relational-database-management-system-rdbms). We should discuss the [use cases and tradeoffs between choosing SQL or NoSQL](https://github.com/ido777/system-design-primer-update#sql-or-nosql).
|
We could store info on the 10 million users in a [relational database](https://github.com/ido777/system-design-primer-update#relational-database-management-system-rdbms).
|
||||||
|
We should discuss the [use cases and tradeoffs between choosing SQL or NoSQL](https://github.com/ido777/system-design-primer-update#sql-or-nosql).
|
||||||
|
|
||||||
* The **Client** sends a request to the **Web Server**, running as a [reverse proxy](https://github.com/ido777/system-design-primer-update#reverse-proxy-web-server)
|
* The **Client** sends a request to the **Web Server**, running as a [reverse proxy](https://github.com/ido777/system-design-primer-update#reverse-proxy-web-server)
|
||||||
* The **Web Server** forwards the request to the **Accounts API** server
|
* The **Web Server** forwards the request to the **Accounts API** server
|
||||||
|
@ -110,17 +230,30 @@ PRIMARY KEY(id)
|
||||||
FOREIGN KEY(user_id) REFERENCES users(id)
|
FOREIGN KEY(user_id) REFERENCES users(id)
|
||||||
```
|
```
|
||||||
|
|
||||||
We'll create an [index](https://github.com/ido777/system-design-primer-update#use-good-indices) on `id`, `user_id `, and `created_at` to speed up lookups (log-time instead of scanning the entire table) and to keep the data in memory. Reading 1 MB sequentially from memory takes about 250 microseconds, while reading from SSD takes 4x and from disk takes 80x longer.<sup><a href=https://github.com/ido777/system-design-primer-update#latency-numbers-every-programmer-should-know>1</a></sup>
|
We'll create an [index](https://github.com/ido777/system-design-primer-update#use-good-indices) on `id`, `user_id `, `last_update` and `created_at` to speed up lookups. Since indexes are typically implemented with B-trees, index lookup is O(log n) instead of O(n). Frequently accessed indexes (like by `last_update` timestamps) are often cached automatically in RAM by the database’s internal cache and since the indexes are smaller, they are likely to stay in memory. Reading 1 MB sequentially from memory takes about 250 microseconds, while reading from SSD takes 4x and from disk takes 80x longer.<sup><a href=https://github.com/ido777/system-design-primer-update.git#latency-numbers-every-programmer-should-know>1</a></sup>
|
||||||
|
|
||||||
|
|
||||||
|
#### REST API
|
||||||
|
|
||||||
We'll use a public [**REST API**](https://github.com/ido777/system-design-primer-update#representational-state-transfer-rest):
|
We'll use a public [**REST API**](https://github.com/ido777/system-design-primer-update#representational-state-transfer-rest):
|
||||||
|
|
||||||
|
##### Connect to a financial account
|
||||||
```
|
```
|
||||||
$ curl -X POST --data '{ "user_id": "foo", "account_url": "bar", \
|
$ curl -X POST --data '{ "user_id": "foo", "account_url": "bar", \
|
||||||
"account_login": "baz", "account_password": "qux" }' \
|
"account_login": "baz", "account_password": "qux" }' \
|
||||||
https://mint.com/api/v1/account
|
https://mint.com/api/v1/account
|
||||||
```
|
```
|
||||||
|
|
||||||
For internal communications, we could use [Remote Procedure Calls](https://github.com/ido777/system-design-primer-update#remote-procedure-call-rpc).
|
Response:
|
||||||
|
|
||||||
|
```
|
||||||
|
{
|
||||||
|
"user_id": "foo",
|
||||||
|
"account_id": "bar",
|
||||||
|
"action": "connect",
|
||||||
|
"status": "success"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
Next, the service extracts transactions from the account.
|
Next, the service extracts transactions from the account.
|
||||||
|
|
||||||
|
@ -132,17 +265,21 @@ We'll want to extract information from an account in these cases:
|
||||||
* The user manually refreshes the account
|
* The user manually refreshes the account
|
||||||
* Automatically each day for users who have been active in the past 30 days
|
* Automatically each day for users who have been active in the past 30 days
|
||||||
|
|
||||||
Data flow:
|
those are the cases for the transaction extraction service.
|
||||||
|
So each of these cases will `trigger` a `transaction extraction service` request.
|
||||||
|
|
||||||
|
Data flow - user manually refreshes the account:
|
||||||
|
|
||||||
* The **Client** sends a request to the **Web Server**
|
* The **Client** sends a request to the **Web Server**
|
||||||
* The **Web Server** forwards the request to the **Accounts API** server
|
* The **Web Server** forwards the request to the **Accounts API** server
|
||||||
* The **Accounts API** server places a job on a **Queue** such as [Amazon SQS](https://aws.amazon.com/sqs/) or [RabbitMQ](https://www.rabbitmq.com/)
|
* The **Accounts API** server places a job on a **Queue** such as [Amazon SQS](https://aws.amazon.com/sqs/) or [RabbitMQ](https://www.rabbitmq.com/)
|
||||||
* Extracting transactions could take awhile, we'd probably want to do this [asynchronously with a queue](https://github.com/ido777/system-design-primer-update#asynchronism), although this introduces additional complexity
|
* Extracting transactions could take awhile, we'd probably want to do this [asynchronously with a queue](https://github.com/ido777/system-design-primer-update#asynchronism), although this introduces additional complexity
|
||||||
* The **Transaction Extraction Service** does the following:
|
* The **Transaction Extraction Service** does the following:
|
||||||
* Pulls from the **Queue** and extracts transactions for the given account from the financial institution, storing the results as raw log files in the **Object Store**
|
* Pulls from the **Queue** and extracts transactions according to job (e.g. for the given account from the financial institution)
|
||||||
* Uses the **Category Service** to categorize each transaction
|
* Stores the results as raw log or json files in the **Object Store**
|
||||||
* Uses the **Budget Service** to calculate aggregate monthly spending by category
|
* Uses the **Category Service** asynchronously to categorize the transactions
|
||||||
* The **Budget Service** uses the **Notification Service** to let users know if they are nearing or have exceeded their budget
|
* Uses the **Budget Service** asynchronously to calculate aggregate monthly spending by category
|
||||||
|
* The **Budget Service** uses the **Notification Service** asynchronously to let users know if they are nearing or have exceeded their budget
|
||||||
* Updates the **SQL Database** `transactions` table with categorized transactions
|
* Updates the **SQL Database** `transactions` table with categorized transactions
|
||||||
* Updates the **SQL Database** `monthly_spending` table with aggregate monthly spending by category
|
* Updates the **SQL Database** `monthly_spending` table with aggregate monthly spending by category
|
||||||
* Notifies the user the transactions have completed through the **Notification Service**:
|
* Notifies the user the transactions have completed through the **Notification Service**:
|
||||||
|
@ -178,7 +315,7 @@ We'll create an [index](https://github.com/ido777/system-design-primer-update#us
|
||||||
|
|
||||||
#### Category service
|
#### Category service
|
||||||
|
|
||||||
For the **Category Service**, we can seed a seller-to-category dictionary with the most popular sellers. If we estimate 50,000 sellers and estimate each entry to take less than 255 bytes, the dictionary would only take about 12 MB of memory.
|
For the **Category Service**, we can seed a seller-to-category dictionary with the most popular sellers. If we estimate 50,000 sellers and estimate each entry to take less than 255 bytes, the dictionary would only take about 12 MB of memory.
|
||||||
|
|
||||||
**Clarify with your interviewer the expected amount, style, and purpose of the code you should write**.
|
**Clarify with your interviewer the expected amount, style, and purpose of the code you should write**.
|
||||||
|
|
||||||
|
@ -197,23 +334,45 @@ seller_category_map['Target'] = DefaultCategories.SHOPPING
|
||||||
...
|
...
|
||||||
```
|
```
|
||||||
|
|
||||||
For sellers not initially seeded in the map, we could use a crowdsourcing effort by evaluating the manual category overrides our users provide. We could use a heap to quickly lookup the top manual override per seller in O(1) time.
|
For sellers not initially seeded in the map, we could use a crowdsourcing effort by evaluating the manual category overrides our users provide.
|
||||||
|
We could use a heap to quickly lookup the top manual override per seller in O(1) time.
|
||||||
|
|
||||||
|
Here we actually want the **most-popular** category override for a given seller (i.e. the one with the highest user-vote count).
|
||||||
|
However, Python’s heapq only provides a min-heap, so we store counts as negative numbers. Then the “minimum” of those negatives is the largest positive count.
|
||||||
|
Heap queues are not designed to handle multiple threads writing the data at the same time, however since writes are rare and missing one count might be acceptable we can use it for a starter.
|
||||||
|
|
||||||
|
Generally, a heap (priority queue) shines when you need to:
|
||||||
|
|
||||||
|
* **Incrementally insert items** (e.g. new user overrides)
|
||||||
|
* **Quickly retrieve** the current top-priority element (peek or pop)
|
||||||
|
* **Maintain the structure** under continuous updates
|
||||||
|
|
||||||
```python
|
```python
|
||||||
class Categorizer(object):
|
import heapq
|
||||||
|
from collections import defaultdict
|
||||||
|
|
||||||
def __init__(self, seller_category_map, seller_category_crowd_overrides_map):
|
class Categorizer:
|
||||||
|
def __init__(self, seller_category_map):
|
||||||
self.seller_category_map = seller_category_map
|
self.seller_category_map = seller_category_map
|
||||||
self.seller_category_crowd_overrides_map = \
|
# each value is a heap of (–override_count, category)
|
||||||
seller_category_crowd_overrides_map
|
self.overrides: dict[str, list[tuple[int, DefaultCategories]]] = defaultdict(list)
|
||||||
|
|
||||||
|
def add_override(self, seller: str, category: DefaultCategories, count: int):
|
||||||
|
# push negative count so that the largest count comes out first
|
||||||
|
heapq.heappush(self.overrides[seller], ( -count, category ))
|
||||||
|
|
||||||
def categorize(self, transaction):
|
def categorize(self, transaction):
|
||||||
if transaction.seller in self.seller_category_map:
|
seller = transaction.seller
|
||||||
return self.seller_category_map[transaction.seller]
|
if seller in self.seller_category_map:
|
||||||
elif transaction.seller in self.seller_category_crowd_overrides_map:
|
return self.seller_category_map[seller]
|
||||||
self.seller_category_map[transaction.seller] = \
|
|
||||||
self.seller_category_crowd_overrides_map[transaction.seller].peek_min()
|
heap = self.overrides.get(seller)
|
||||||
return self.seller_category_map[transaction.seller]
|
if heap:
|
||||||
|
# peek the “min” of the heap, which is (–max_count, category)
|
||||||
|
_, top_category = heap[0]
|
||||||
|
self.seller_category_map[seller] = top_category
|
||||||
|
return top_category
|
||||||
|
|
||||||
return None
|
return None
|
||||||
```
|
```
|
||||||
|
|
||||||
|
@ -228,9 +387,18 @@ class Transaction(object):
|
||||||
self.amount = amount
|
self.amount = amount
|
||||||
```
|
```
|
||||||
|
|
||||||
|
##### Solving the thread-safety issue / Scaling the categorizer
|
||||||
|
|
||||||
|
* **Lock around the heap** or switch to Python’s built-in PriorityQueue for immediate thread safety.
|
||||||
|
* **Use a lightweight embedded database** (e.g. SQLite in its default serialized mode) and do atomic SQL updates/queries.
|
||||||
|
* **Adopt a simple external store** like Redis for atomic counters with INCR.
|
||||||
|
|
||||||
|
Each approach trades off complexity versus performance and scalability; pick the simplest that meets your concurrency needs today, then evolve as you grow.
|
||||||
|
|
||||||
|
|
||||||
### Use case: Service recommends a budget
|
### Use case: Service recommends a budget
|
||||||
|
|
||||||
To start, we could use a generic budget template that allocates category amounts based on income tiers. Using this approach, we would not have to store the 100 million budget items identified in the constraints, only those that the user overrides. If a user overrides a budget category, which we could store the override in the `TABLE budget_overrides`.
|
To start, we could use a generic budget template that allocates category amounts based on income tiers. Using a **common template**, you avoid materializing all 100 million per-category records up front—each new user simply references the same in-memory or table-driven default rules. Only when a user **actually changes** one of those defaults do you write the data; everyone else continues to implicitly use the template values. When a user overrides a budget category, we could store the override in the `TABLE budget_overrides`.
|
||||||
|
|
||||||
```python
|
```python
|
||||||
class Budget(object):
|
class Budget(object):
|
||||||
|
@ -303,9 +471,9 @@ class SpendingByCategory(MRJob):
|
||||||
Using the categorizer to convert seller to category,
|
Using the categorizer to convert seller to category,
|
||||||
emit key value pairs of the form:
|
emit key value pairs of the form:
|
||||||
|
|
||||||
(user_id, 2016-01, shopping), 25
|
(user_id, 2025-01, shopping), 25
|
||||||
(user_id, 2016-01, shopping), 100
|
(user_id, 2025-01, shopping), 100
|
||||||
(user_id, 2016-01, gas), 50
|
(user_id, 2025-01, gas), 50
|
||||||
"""
|
"""
|
||||||
user_id, timestamp, seller, amount = line.split('\t')
|
user_id, timestamp, seller, amount = line.split('\t')
|
||||||
category = self.categorizer.categorize(seller)
|
category = self.categorizer.categorize(seller)
|
||||||
|
@ -316,14 +484,14 @@ class SpendingByCategory(MRJob):
|
||||||
def reducer(self, key, value):
|
def reducer(self, key, value):
|
||||||
"""Sum values for each key.
|
"""Sum values for each key.
|
||||||
|
|
||||||
(user_id, 2016-01, shopping), 125
|
(user_id, 2025-01, shopping), 125
|
||||||
(user_id, 2016-01, gas), 50
|
(user_id, 2025-01, gas), 50
|
||||||
"""
|
"""
|
||||||
total = sum(values)
|
total = sum(values)
|
||||||
yield key, sum(values)
|
yield key, sum(values)
|
||||||
```
|
```
|
||||||
|
|
||||||
## Step 4: Scale the design
|
### Scale the design
|
||||||
|
|
||||||
> Identify and address bottlenecks, given the constraints.
|
> Identify and address bottlenecks, given the constraints.
|
||||||
|
|
||||||
|
@ -331,12 +499,22 @@ class SpendingByCategory(MRJob):
|
||||||
|
|
||||||
**Important: Do not simply jump right into the final design from the initial design!**
|
**Important: Do not simply jump right into the final design from the initial design!**
|
||||||
|
|
||||||
State you would 1) **Benchmark/Load Test**, 2) **Profile** for bottlenecks 3) address bottlenecks while evaluating alternatives and trade-offs, and 4) repeat. See [Design a system that scales to millions of users on AWS](../scaling_aws/README.md) as a sample on how to iteratively scale the initial design.
|
State you would
|
||||||
|
1) **Benchmark/Load Test**,
|
||||||
|
2) **Profile** for bottlenecks
|
||||||
|
3) address bottlenecks while evaluating alternatives and trade-offs, and
|
||||||
|
4) repeat. See [Design a system that scales to millions of users on AWS](../scaling_aws/README.md) as a sample on how to iteratively scale the initial design.
|
||||||
|
|
||||||
It's important to discuss what bottlenecks you might encounter with the initial design and how you might address each of them. For example, what issues are addressed by adding a **Load Balancer** with multiple **Web Servers**? **CDN**? **Master-Slave Replicas**? What are the alternatives and **Trade-Offs** for each?
|
It's important to discuss what bottlenecks you might encounter with the initial design and how you might address each of them. For example, what issues are addressed by adding a **Load Balancer** with multiple **Web Servers**? **CDN**? **Master-Slave Replicas**? What are the alternatives and **Trade-Offs** for each?
|
||||||
|
|
||||||
We'll introduce some components to complete the design and to address scalability issues. Internal load balancers are not shown to reduce clutter.
|
We'll introduce some components to complete the design and to address scalability issues. Internal load balancers are not shown to reduce clutter.
|
||||||
|
|
||||||
|
## Step 4 Wrap up
|
||||||
|
|
||||||
|
To summarize, we've designed a financial management system that meets the core requirements. We've discussed the high-level design, identified potential bottlenecks, and proposed solutions to address scalability issues. Now it is time to align again with the interviewer expectations.
|
||||||
|
See if she has any feedback or questions, suggest next steps, improvements, error handling, and monitoring if appropriate.
|
||||||
|
|
||||||
|
|
||||||
*To avoid repeating discussions*, refer to the following [system design topics](https://github.com/ido777/system-design-primer-update#index-of-system-design-topics) for main talking points, tradeoffs, and alternatives:
|
*To avoid repeating discussions*, refer to the following [system design topics](https://github.com/ido777/system-design-primer-update#index-of-system-design-topics) for main talking points, tradeoffs, and alternatives:
|
||||||
|
|
||||||
* [DNS](https://github.com/ido777/system-design-primer-update#domain-name-system)
|
* [DNS](https://github.com/ido777/system-design-primer-update#domain-name-system)
|
||||||
|
|
|
@ -20,10 +20,10 @@ Remember your goal is to understand the problem and establish the design scope.
|
||||||
### What questions should you ask to clarify the problem?
|
### What questions should you ask to clarify the problem?
|
||||||
|
|
||||||
|
|
||||||
Here is an example of the dialog you could have with the interviewer:
|
Here is an example of the dialog you could have with the **Interviewer**:
|
||||||
interviewer: Design Pastebin.com.
|
**Interviewer**: Design Pastebin.com.
|
||||||
candidate: Could you please remind me what Pastebin.com does at a high level?
|
**Candidate**: Could you please remind me what Pastebin.com does at a high level?
|
||||||
interviewer: Do you happen to know GitHub Gist? It is similar to Pastebin.com.
|
**Interviewer**: Do you happen to know GitHub Gist? It is similar to Pastebin.com.
|
||||||
|
|
||||||
## 📝 Pastebin.com Overview
|
## 📝 Pastebin.com Overview
|
||||||
|
|
||||||
|
@ -47,17 +47,17 @@ interviewer: Do you happen to know GitHub Gist? It is similar to Pastebin.com.
|
||||||
- Temporary storage of text for collaboration or troubleshooting.
|
- Temporary storage of text for collaboration or troubleshooting.
|
||||||
- Situations where simplicity and speed are paramount.
|
- Situations where simplicity and speed are paramount.
|
||||||
|
|
||||||
candidate: Got it. Since Pastebin can be quite complex, can we focus on just the core features first?
|
**Candidate**: Got it. Since Pastebin can be quite complex, can we focus on just the core features first?
|
||||||
interviewer: Sure—what would you target?
|
**Interviewer**: Sure—what would you target?
|
||||||
candidate: The main requirement is that the user pastes text and immediately receives a shareable link. Correct?
|
**Candidate**: The main requirement is that the user pastes text and immediately receives a shareable link. Correct?
|
||||||
interviewer: Can you elaborate on the link?
|
**Interviewer**: Can you elaborate on the link?
|
||||||
candidate: A randomly generated, unique link.
|
**Candidate**: A randomly generated, unique link.
|
||||||
interviewer: Does it expire?
|
**Interviewer**: Does it expire?
|
||||||
candidate: No.
|
**Candidate**: No.
|
||||||
interviewer: Never?
|
**Interviewer**: Never?
|
||||||
candidate: (_Oops, she doesn’t like that we don’t have expiration._) We can add a timed expiration—user can set the expiration.
|
**Candidate**: (_Oops, she doesn’t like that we don’t have expiration._) We can add a timed expiration—user can set the expiration.
|
||||||
interviewer: Sounds good.
|
**Interviewer**: Sounds good.
|
||||||
candidate: Cool. Let me summarize.
|
**Candidate**: Cool. Let me summarize.
|
||||||
|
|
||||||
Conclusion:
|
Conclusion:
|
||||||
- Use cases
|
- Use cases
|
||||||
|
@ -66,25 +66,25 @@ Conclusion:
|
||||||
• Default setting does not expire
|
• Default setting does not expire
|
||||||
• Can optionally set a timed expiration
|
• Can optionally set a timed expiration
|
||||||
|
|
||||||
candidate: Mobile or desktop client?
|
**Candidate**: Mobile or desktop client?
|
||||||
interviewer: Both.
|
**Interviewer**: Both.
|
||||||
candidate: Is user authentication or account registration required to view or create pastes?
|
**Candidate**: Is user authentication or account registration required to view or create pastes?
|
||||||
interviewer: No registration is needed; it’s anonymous.
|
**Interviewer**: No registration is needed; it’s anonymous.
|
||||||
candidate: Great. Do we need to track usage statistics or analytics for these pastes?
|
**Candidate**: Great. Do we need to track usage statistics or analytics for these pastes?
|
||||||
interviewer: We will record monthly visit stats.
|
**Interviewer**: We will record monthly visit stats.
|
||||||
candidate: Should expired pastes be removed automatically?
|
**Candidate**: Should expired pastes be removed automatically?
|
||||||
interviewer: Yes, the service deletes expired pastes.
|
**Interviewer**: Yes, the service deletes expired pastes.
|
||||||
candidate: What availability SLA do we expect?
|
**Candidate**: What availability SLA do we expect?
|
||||||
interviewer: High availability is a requirement.
|
**Interviewer**: High availability is a requirement.
|
||||||
candidate: For this exercise phase, I would like to suggest that we don't need user accounts, login, or custom shortlinks.
|
**Candidate**: For this exercise phase, I would like to suggest that we don't need user accounts, login, or custom shortlinks.
|
||||||
interviewer: ok, Those are out of scope for now.
|
**Interviewer**: ok, Those are out of scope for now.
|
||||||
candidate: For capacity planning, can you confirm traffic patterns and volumes?
|
**Candidate**: For capacity planning, can you confirm traffic patterns and volumes?
|
||||||
interviewer: Traffic is unevenly distributed; we target 10M users, 10M writes/month, and 100M reads/month.
|
**Interviewer**: Traffic is unevenly distributed; we target 10M users, 10M writes/month, and 100M reads/month.
|
||||||
candidate: Understood. And are pastes text only, with low-latency URL resolution?
|
**Candidate**: Understood. And are pastes text only, with low-latency URL resolution?
|
||||||
interviewer: Correct.
|
**Interviewer**: Correct.
|
||||||
candidate: Finally, any rough numbers on storage and throughput?
|
**Candidate**: Finally, any rough numbers on storage and throughput?
|
||||||
interviewer: I'll leave that to you.
|
**Interviewer**: I'll leave that to you.
|
||||||
candidate: ok. So here is the scope of the problem:
|
**Candidate**: ok. So here is the scope of the problem:
|
||||||
|
|
||||||
### Use cases
|
### Use cases
|
||||||
|
|
||||||
|
@ -154,13 +154,16 @@ Handy conversion guide:
|
||||||
|
|
||||||
> Outline a high level design with all important components.
|
> Outline a high level design with all important components.
|
||||||
|
|
||||||
|
|
||||||
|
<!-- Old image for reference  -->
|
||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
%%{init: { "flowchart": { "htmlLabels": true } }}%%
|
%%{init: { "flowchart": { "htmlLabels": true } }}%%
|
||||||
|
|
||||||
flowchart TB
|
flowchart TB
|
||||||
%% Client Layer
|
%% Client Layer
|
||||||
subgraph Client["**Client**"]
|
subgraph Client["**Client**"]
|
||||||
direction LR
|
direction TB
|
||||||
WebClient[Web Client]
|
WebClient[Web Client]
|
||||||
MobileClient[Mobile Client]
|
MobileClient[Mobile Client]
|
||||||
end
|
end
|
||||||
|
@ -176,15 +179,13 @@ flowchart TB
|
||||||
%% Storage Layer
|
%% Storage Layer
|
||||||
subgraph Storage["**Storage**"]
|
subgraph Storage["**Storage**"]
|
||||||
direction LR
|
direction LR
|
||||||
SQLDB[SQL Database]
|
SQLDB[(SQL Database)]
|
||||||
ObjectStore[Object Store]
|
ObjectStore[(Object Store)]
|
||||||
end
|
end
|
||||||
|
|
||||||
%% Data Flow
|
%% Data Flow
|
||||||
WebClient --> WriteAPI
|
Client --> WebServer
|
||||||
MobileClient --> WriteAPI
|
|
||||||
WebClient --> ReadAPI
|
|
||||||
MobileClient --> ReadAPI
|
|
||||||
|
|
||||||
WriteAPI --> SQLDB
|
WriteAPI --> SQLDB
|
||||||
WriteAPI --> ObjectStore
|
WriteAPI --> ObjectStore
|
||||||
|
@ -458,7 +459,7 @@ for key, count in reduced.items():
|
||||||
|
|
||||||
To delete expired pastes, we could just scan the **SQL Database** for all entries whose expiration timestamp are older than the current timestamp. All expired entries would then be deleted (or marked as expired) from the table.
|
To delete expired pastes, we could just scan the **SQL Database** for all entries whose expiration timestamp are older than the current timestamp. All expired entries would then be deleted (or marked as expired) from the table.
|
||||||
|
|
||||||
## Tradeoffs and Scaling the design
|
### Tradeoffs and Scaling the design
|
||||||
|
|
||||||
> Identify and address bottlenecks, given the constraints.
|
> Identify and address bottlenecks, given the constraints.
|
||||||
|
|
||||||
|
@ -483,6 +484,14 @@ For example, what issues are addressed by adding
|
||||||
|
|
||||||
We'll introduce some components to complete the design and to address scalability issues. Internal load balancers are not shown to reduce clutter.
|
We'll introduce some components to complete the design and to address scalability issues. Internal load balancers are not shown to reduce clutter.
|
||||||
|
|
||||||
|
|
||||||
|
## Step 4 Wrap up
|
||||||
|
|
||||||
|
To summarize, we've designed a text snippet sharer system that meets the core requirements. We've discussed the high-level design, identified potential bottlenecks, and proposed solutions to address scalability issues. Now it is time to align again with the interviewer expectations.
|
||||||
|
See if she has any feedback or questions, suggest next steps, improvements, error handling, and monitoring if appropriate.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
*To avoid repeating discussions*, refer to the following [system design topics](https://github.com/ido777/system-design-primer-update.git#index-of-system-design-topics) for main talking points, tradeoffs, and alternatives:
|
*To avoid repeating discussions*, refer to the following [system design topics](https://github.com/ido777/system-design-primer-update.git#index-of-system-design-topics) for main talking points, tradeoffs, and alternatives:
|
||||||
|
|
||||||
* [DNS](https://github.com/ido777/system-design-primer-update.git#domain-name-system)
|
* [DNS](https://github.com/ido777/system-design-primer-update.git#domain-name-system)
|
||||||
|
|
Loading…
Reference in New Issue