Reorder README.md without changing content
Vinicius
GitHub
parent
4e70c54a5b
commit
df5e8f2be3
708
README.md
708
README.md
|
|
@ -4,181 +4,13 @@
|
|||
|
||||
# The System Design Primer
|
||||
|
||||
<p align="center">
|
||||
<img src="images/jj3A5N8.png">
|
||||
<br/>
|
||||
</p>
|
||||
|
||||
# Intro
|
||||
## Motivation
|
||||
|
||||
> Learn how to design large-scale systems.
|
||||
>
|
||||
> Prep for the system design interview.
|
||||
|
||||
### Learn how to design large-scale systems
|
||||
|
||||
Learning how to design scalable systems will help you become a better engineer.
|
||||
|
||||
System design is a broad topic. There is a **vast amount of resources scattered throughout the web** on system design principles.
|
||||
|
||||
This repo is an **organized collection** of resources to help you learn how to build systems at scale.
|
||||
|
||||
### Learn from the open source community
|
||||
|
||||
This is a continually updated, open source project.
|
||||
|
||||
[Contributions](#contributing) are welcome!
|
||||
|
||||
### Prep for the system design interview
|
||||
|
||||
In addition to coding interviews, system design is a **required component** of the **technical interview process** at many tech companies.
|
||||
|
||||
**Practice common system design interview questions** and **compare** your results with **sample solutions**: discussions, code, and diagrams.
|
||||
|
||||
Additional topics for interview prep:
|
||||
|
||||
* [Study guide](#study-guide)
|
||||
* [How to approach a system design interview question](#how-to-approach-a-system-design-interview-question)
|
||||
* [System design interview questions, **with solutions**](#system-design-interview-questions-with-solutions)
|
||||
* [Object-oriented design interview questions, **with solutions**](#object-oriented-design-interview-questions-with-solutions)
|
||||
* [Additional system design interview questions](#additional-system-design-interview-questions)
|
||||
|
||||
## Anki flashcards
|
||||
|
||||
<p align="center">
|
||||
<img src="images/zdCAkB3.png">
|
||||
<br/>
|
||||
</p>
|
||||
|
||||
The provided [Anki flashcard decks](https://apps.ankiweb.net/) use spaced repetition to help you retain key system design concepts.
|
||||
|
||||
* [System design deck](https://github.com/donnemartin/system-design-primer/tree/master/resources/flash_cards/System%20Design.apkg)
|
||||
* [System design exercises deck](https://github.com/donnemartin/system-design-primer/tree/master/resources/flash_cards/System%20Design%20Exercises.apkg)
|
||||
* [Object oriented design exercises deck](https://github.com/donnemartin/system-design-primer/tree/master/resources/flash_cards/OO%20Design.apkg)
|
||||
|
||||
Great for use while on-the-go.
|
||||
|
||||
### Coding Resource: Interactive Coding Challenges
|
||||
|
||||
Looking for resources to help you prep for the [**Coding Interview**](https://github.com/donnemartin/interactive-coding-challenges)?
|
||||
|
||||
<p align="center">
|
||||
<img src="images/b4YtAEN.png">
|
||||
<br/>
|
||||
</p>
|
||||
|
||||
Check out the sister repo [**Interactive Coding Challenges**](https://github.com/donnemartin/interactive-coding-challenges), which contains an additional Anki deck:
|
||||
|
||||
* [Coding deck](https://github.com/donnemartin/interactive-coding-challenges/tree/master/anki_cards/Coding.apkg)
|
||||
|
||||
## Contributing
|
||||
|
||||
> Learn from the community.
|
||||
|
||||
Feel free to submit pull requests to help:
|
||||
|
||||
* Fix errors
|
||||
* Improve sections
|
||||
* Add new sections
|
||||
* [Translate](https://github.com/donnemartin/system-design-primer/issues/28)
|
||||
|
||||
Content that needs some polishing is placed [under development](#under-development).
|
||||
|
||||
Review the [Contributing Guidelines](CONTRIBUTING.md).
|
||||
|
||||
## Index of system design topics
|
||||
|
||||
> Summaries of various system design topics, including pros and cons. **Everything is a trade-off**.
|
||||
>
|
||||
> Each section contains links to more in-depth resources.
|
||||
|
||||
<p align="center">
|
||||
<img src="images/jrUBAF7.png">
|
||||
<br/>
|
||||
</p>
|
||||
|
||||
* [System design topics: start here](#system-design-topics-start-here)
|
||||
* [Step 1: Review the scalability video lecture](#step-1-review-the-scalability-video-lecture)
|
||||
* [Step 2: Review the scalability article](#step-2-review-the-scalability-article)
|
||||
* [Next steps](#next-steps)
|
||||
* [Performance vs scalability](#performance-vs-scalability)
|
||||
* [Latency vs throughput](#latency-vs-throughput)
|
||||
* [Availability vs consistency](#availability-vs-consistency)
|
||||
* [CAP theorem](#cap-theorem)
|
||||
* [CP - consistency and partition tolerance](#cp---consistency-and-partition-tolerance)
|
||||
* [AP - availability and partition tolerance](#ap---availability-and-partition-tolerance)
|
||||
* [Consistency patterns](#consistency-patterns)
|
||||
* [Weak consistency](#weak-consistency)
|
||||
* [Eventual consistency](#eventual-consistency)
|
||||
* [Strong consistency](#strong-consistency)
|
||||
* [Availability patterns](#availability-patterns)
|
||||
* [Fail-over](#fail-over)
|
||||
* [Replication](#replication)
|
||||
* [Availability in numbers](#availability-in-numbers)
|
||||
* [Domain name system](#domain-name-system)
|
||||
* [Content delivery network](#content-delivery-network)
|
||||
* [Push CDNs](#push-cdns)
|
||||
* [Pull CDNs](#pull-cdns)
|
||||
* [Load balancer](#load-balancer)
|
||||
* [Active-passive](#active-passive)
|
||||
* [Active-active](#active-active)
|
||||
* [Layer 4 load balancing](#layer-4-load-balancing)
|
||||
* [Layer 7 load balancing](#layer-7-load-balancing)
|
||||
* [Horizontal scaling](#horizontal-scaling)
|
||||
* [Reverse proxy (web server)](#reverse-proxy-web-server)
|
||||
* [Load balancer vs reverse proxy](#load-balancer-vs-reverse-proxy)
|
||||
* [Application layer](#application-layer)
|
||||
* [Microservices](#microservices)
|
||||
* [Service discovery](#service-discovery)
|
||||
* [Database](#database)
|
||||
* [Relational database management system (RDBMS)](#relational-database-management-system-rdbms)
|
||||
* [Master-slave replication](#master-slave-replication)
|
||||
* [Master-master replication](#master-master-replication)
|
||||
* [Federation](#federation)
|
||||
* [Sharding](#sharding)
|
||||
* [Denormalization](#denormalization)
|
||||
* [SQL tuning](#sql-tuning)
|
||||
* [NoSQL](#nosql)
|
||||
* [Key-value store](#key-value-store)
|
||||
* [Document store](#document-store)
|
||||
* [Wide column store](#wide-column-store)
|
||||
* [Graph Database](#graph-database)
|
||||
* [SQL or NoSQL](#sql-or-nosql)
|
||||
* [Cache](#cache)
|
||||
* [Client caching](#client-caching)
|
||||
* [CDN caching](#cdn-caching)
|
||||
* [Web server caching](#web-server-caching)
|
||||
* [Database caching](#database-caching)
|
||||
* [Application caching](#application-caching)
|
||||
* [Caching at the database query level](#caching-at-the-database-query-level)
|
||||
* [Caching at the object level](#caching-at-the-object-level)
|
||||
* [When to update the cache](#when-to-update-the-cache)
|
||||
* [Cache-aside](#cache-aside)
|
||||
* [Write-through](#write-through)
|
||||
* [Write-behind (write-back)](#write-behind-write-back)
|
||||
* [Refresh-ahead](#refresh-ahead)
|
||||
* [Asynchronism](#asynchronism)
|
||||
* [Message queues](#message-queues)
|
||||
* [Task queues](#task-queues)
|
||||
* [Back pressure](#back-pressure)
|
||||
* [Communication](#communication)
|
||||
* [Transmission control protocol (TCP)](#transmission-control-protocol-tcp)
|
||||
* [User datagram protocol (UDP)](#user-datagram-protocol-udp)
|
||||
* [Remote procedure call (RPC)](#remote-procedure-call-rpc)
|
||||
* [Representational state transfer (REST)](#representational-state-transfer-rest)
|
||||
* [Security](#security)
|
||||
* [Appendix](#appendix)
|
||||
* [Powers of two table](#powers-of-two-table)
|
||||
* [Latency numbers every programmer should know](#latency-numbers-every-programmer-should-know)
|
||||
* [Additional system design interview questions](#additional-system-design-interview-questions)
|
||||
* [Real world architectures](#real-world-architectures)
|
||||
* [Company architectures](#company-architectures)
|
||||
* [Company engineering blogs](#company-engineering-blogs)
|
||||
* [Under development](#under-development)
|
||||
* [Credits](#credits)
|
||||
* [Contact info](#contact-info)
|
||||
* [License](#license)
|
||||
|
||||
## Study guide
|
||||
|
||||
> Suggested topics to review based on your interview timeline (short, medium, long).
|
||||
|
|
@ -215,6 +47,35 @@ Start broad and go deeper in a few areas. It helps to know a little about vario
|
|||
| Work through [Object-oriented design interview questions with solutions](#object-oriented-design-interview-questions-with-solutions) | Some | Many | Most |
|
||||
| Review [Additional system design interview questions](#additional-system-design-interview-questions) | Some | Many | Most |
|
||||
|
||||
### Learn how to design large-scale systems
|
||||
|
||||
Learning how to design scalable systems will help you become a better engineer.
|
||||
|
||||
System design is a broad topic. There is a **vast amount of resources scattered throughout the web** on system design principles.
|
||||
|
||||
This repo is an **organized collection** of resources to help you learn how to build systems at scale.
|
||||
|
||||
### Learn from the open source community
|
||||
|
||||
This is a continually updated, open source project.
|
||||
|
||||
[Contributions](#contributing) are welcome!
|
||||
|
||||
# Approaching the interview
|
||||
### Prep for the system design interview
|
||||
|
||||
In addition to coding interviews, system design is a **required component** of the **technical interview process** at many tech companies.
|
||||
|
||||
**Practice common system design interview questions** and **compare** your results with **sample solutions**: discussions, code, and diagrams.
|
||||
|
||||
Additional topics for interview prep:
|
||||
|
||||
* [Study guide](#study-guide)
|
||||
* [How to approach a system design interview question](#how-to-approach-a-system-design-interview-question)
|
||||
* [System design interview questions, **with solutions**](#system-design-interview-questions-with-solutions)
|
||||
* [Object-oriented design interview questions, **with solutions**](#object-oriented-design-interview-questions-with-solutions)
|
||||
* [Additional system design interview questions](#additional-system-design-interview-questions)
|
||||
|
||||
## How to approach a system design interview question
|
||||
|
||||
> How to tackle a system design interview question.
|
||||
|
|
@ -284,90 +145,37 @@ Check out the following links to get a better idea of what to expect:
|
|||
* [Intro to Architecture and Systems Design Interviews](https://www.youtube.com/watch?v=ZgdS0EUmn70)
|
||||
* [System design template](https://leetcode.com/discuss/career/229177/My-System-Design-Template)
|
||||
|
||||
## System design interview questions with solutions
|
||||
# Systems Design
|
||||
## Index of system design topics
|
||||
|
||||
> Common system design interview questions with sample discussions, code, and diagrams.
|
||||
> Summaries of various system design topics, including pros and cons. **Everything is a trade-off**.
|
||||
>
|
||||
> Solutions linked to content in the `solutions/` folder.
|
||||
> Each section contains links to more in-depth resources.
|
||||
|
||||
| Question | |
|
||||
|---|---|
|
||||
| Design Pastebin.com (or Bit.ly) | [Solution](solutions/system_design/pastebin/README.md) |
|
||||
| Design the Twitter timeline and search (or Facebook feed and search) | [Solution](solutions/system_design/twitter/README.md) |
|
||||
| Design a web crawler | [Solution](solutions/system_design/web_crawler/README.md) |
|
||||
| Design Mint.com | [Solution](solutions/system_design/mint/README.md) |
|
||||
| Design the data structures for a social network | [Solution](solutions/system_design/social_graph/README.md) |
|
||||
| Design a key-value store for a search engine | [Solution](solutions/system_design/query_cache/README.md) |
|
||||
| Design Amazon's sales ranking by category feature | [Solution](solutions/system_design/sales_rank/README.md) |
|
||||
| Design a system that scales to millions of users on AWS | [Solution](solutions/system_design/scaling_aws/README.md) |
|
||||
| Add a system design question | [Contribute](#contributing) |
|
||||
<p align="center">
|
||||
<img src="images/jrUBAF7.png">
|
||||
<br/>
|
||||
</p>
|
||||
|
||||
### Design Pastebin.com (or Bit.ly)
|
||||
|
||||
[View exercise and solution](solutions/system_design/pastebin/README.md)
|
||||
|
||||
|
||||
|
||||
### Design the Twitter timeline and search (or Facebook feed and search)
|
||||
|
||||
[View exercise and solution](solutions/system_design/twitter/README.md)
|
||||
|
||||
|
||||
|
||||
### Design a web crawler
|
||||
|
||||
[View exercise and solution](solutions/system_design/web_crawler/README.md)
|
||||
|
||||
|
||||
|
||||
### Design Mint.com
|
||||
|
||||
[View exercise and solution](solutions/system_design/mint/README.md)
|
||||
|
||||
|
||||
|
||||
### Design the data structures for a social network
|
||||
|
||||
[View exercise and solution](solutions/system_design/social_graph/README.md)
|
||||
|
||||
|
||||
|
||||
### Design a key-value store for a search engine
|
||||
|
||||
[View exercise and solution](solutions/system_design/query_cache/README.md)
|
||||
|
||||
|
||||
|
||||
### Design Amazon's sales ranking by category feature
|
||||
|
||||
[View exercise and solution](solutions/system_design/sales_rank/README.md)
|
||||
|
||||
|
||||
|
||||
### Design a system that scales to millions of users on AWS
|
||||
|
||||
[View exercise and solution](solutions/system_design/scaling_aws/README.md)
|
||||
|
||||
|
||||
|
||||
## Object-oriented design interview questions with solutions
|
||||
|
||||
> Common object-oriented design interview questions with sample discussions, code, and diagrams.
|
||||
>
|
||||
> Solutions linked to content in the `solutions/` folder.
|
||||
|
||||
>**Note: This section is under development**
|
||||
|
||||
| Question | |
|
||||
|---|---|
|
||||
| Design a hash map | [Solution](solutions/object_oriented_design/hash_table/hash_map.ipynb) |
|
||||
| Design a least recently used cache | [Solution](solutions/object_oriented_design/lru_cache/lru_cache.ipynb) |
|
||||
| Design a call center | [Solution](solutions/object_oriented_design/call_center/call_center.ipynb) |
|
||||
| Design a deck of cards | [Solution](solutions/object_oriented_design/deck_of_cards/deck_of_cards.ipynb) |
|
||||
| Design a parking lot | [Solution](solutions/object_oriented_design/parking_lot/parking_lot.ipynb) |
|
||||
| Design a chat server | [Solution](solutions/object_oriented_design/online_chat/online_chat.ipynb) |
|
||||
| Design a circular array | [Contribute](#contributing) |
|
||||
| Add an object-oriented design question | [Contribute](#contributing) |
|
||||
# Concepts
|
||||
* [System design topics: start here](#system-design-topics-start-here)
|
||||
* [Step 1: Review the scalability video lecture](#step-1-review-the-scalability-video-lecture)
|
||||
* [Step 2: Review the scalability article](#step-2-review-the-scalability-article)
|
||||
* [Next steps](#next-steps)
|
||||
* [Performance vs scalability](#performance-vs-scalability)
|
||||
* [Latency vs throughput](#latency-vs-throughput)
|
||||
* [Availability vs consistency](#availability-vs-consistency)
|
||||
* [CAP theorem](#cap-theorem)
|
||||
* [CP - consistency and partition tolerance](#cp---consistency-and-partition-tolerance)
|
||||
* [AP - availability and partition tolerance](#ap---availability-and-partition-tolerance)
|
||||
* [Consistency patterns](#consistency-patterns)
|
||||
* [Weak consistency](#weak-consistency)
|
||||
* [Eventual consistency](#eventual-consistency)
|
||||
* [Strong consistency](#strong-consistency)
|
||||
* [Availability patterns](#availability-patterns)
|
||||
* [Fail-over](#fail-over)
|
||||
* [Replication](#replication)
|
||||
* [Availability in numbers](#availability-in-numbers)
|
||||
|
||||
## System design topics: start here
|
||||
|
||||
|
|
@ -578,6 +386,67 @@ Availability (Total) = 1 - (1 - Availability (Foo)) * (1 - Availability (Bar))
|
|||
|
||||
If both `Foo` and `Bar` each had 99.9% availability, their total availability in parallel would be 99.9999%.
|
||||
|
||||
# Components
|
||||
|
||||
<p align="center">
|
||||
<img src="images/jj3A5N8.png">
|
||||
<br/>
|
||||
</p>
|
||||
|
||||
|
||||
* [Domain name system](#domain-name-system)
|
||||
* [Content delivery network](#content-delivery-network)
|
||||
* [Push CDNs](#push-cdns)
|
||||
* [Pull CDNs](#pull-cdns)
|
||||
* [Load balancer](#load-balancer)
|
||||
* [Active-passive](#active-passive)
|
||||
* [Active-active](#active-active)
|
||||
* [Layer 4 load balancing](#layer-4-load-balancing)
|
||||
* [Layer 7 load balancing](#layer-7-load-balancing)
|
||||
* [Horizontal scaling](#horizontal-scaling)
|
||||
* [Reverse proxy (web server)](#reverse-proxy-web-server)
|
||||
* [Load balancer vs reverse proxy](#load-balancer-vs-reverse-proxy)
|
||||
* [Application layer](#application-layer)
|
||||
* [Microservices](#microservices)
|
||||
* [Service discovery](#service-discovery)
|
||||
* [Database](#database)
|
||||
* [Relational database management system (RDBMS)](#relational-database-management-system-rdbms)
|
||||
* [Master-slave replication](#master-slave-replication)
|
||||
* [Master-master replication](#master-master-replication)
|
||||
* [Federation](#federation)
|
||||
* [Sharding](#sharding)
|
||||
* [Denormalization](#denormalization)
|
||||
* [SQL tuning](#sql-tuning)
|
||||
* [NoSQL](#nosql)
|
||||
* [Key-value store](#key-value-store)
|
||||
* [Document store](#document-store)
|
||||
* [Wide column store](#wide-column-store)
|
||||
* [Graph Database](#graph-database)
|
||||
* [SQL or NoSQL](#sql-or-nosql)
|
||||
* [Cache](#cache)
|
||||
* [Client caching](#client-caching)
|
||||
* [CDN caching](#cdn-caching)
|
||||
* [Web server caching](#web-server-caching)
|
||||
* [Database caching](#database-caching)
|
||||
* [Application caching](#application-caching)
|
||||
* [Caching at the database query level](#caching-at-the-database-query-level)
|
||||
* [Caching at the object level](#caching-at-the-object-level)
|
||||
* [When to update the cache](#when-to-update-the-cache)
|
||||
* [Cache-aside](#cache-aside)
|
||||
* [Write-through](#write-through)
|
||||
* [Write-behind (write-back)](#write-behind-write-back)
|
||||
* [Refresh-ahead](#refresh-ahead)
|
||||
* [Asynchronism](#asynchronism)
|
||||
* [Message queues](#message-queues)
|
||||
* [Task queues](#task-queues)
|
||||
* [Back pressure](#back-pressure)
|
||||
* [Communication](#communication)
|
||||
* [Transmission control protocol (TCP)](#transmission-control-protocol-tcp)
|
||||
* [User datagram protocol (UDP)](#user-datagram-protocol-udp)
|
||||
* [Remote procedure call (RPC)](#remote-procedure-call-rpc)
|
||||
* [Representational state transfer (REST)](#representational-state-transfer-rest)
|
||||
* [Security](#security)
|
||||
|
||||
## Domain name system
|
||||
|
||||
<p align="center">
|
||||
|
|
@ -1574,6 +1443,251 @@ Security is a broad topic. Unless you have considerable experience, a security
|
|||
* [Security guide for developers](https://github.com/FallibleInc/security-guide-for-developers)
|
||||
* [OWASP top ten](https://www.owasp.org/index.php/OWASP_Top_Ten_Cheat_Sheet)
|
||||
|
||||
# Examples
|
||||
## System design interview questions with solutions
|
||||
|
||||
> Common system design interview questions with sample discussions, code, and diagrams.
|
||||
>
|
||||
> Solutions linked to content in the `solutions/` folder.
|
||||
|
||||
| Question | |
|
||||
|---|---|
|
||||
| Design Pastebin.com (or Bit.ly) | [Solution](solutions/system_design/pastebin/README.md) |
|
||||
| Design the Twitter timeline and search (or Facebook feed and search) | [Solution](solutions/system_design/twitter/README.md) |
|
||||
| Design a web crawler | [Solution](solutions/system_design/web_crawler/README.md) |
|
||||
| Design Mint.com | [Solution](solutions/system_design/mint/README.md) |
|
||||
| Design the data structures for a social network | [Solution](solutions/system_design/social_graph/README.md) |
|
||||
| Design a key-value store for a search engine | [Solution](solutions/system_design/query_cache/README.md) |
|
||||
| Design Amazon's sales ranking by category feature | [Solution](solutions/system_design/sales_rank/README.md) |
|
||||
| Design a system that scales to millions of users on AWS | [Solution](solutions/system_design/scaling_aws/README.md) |
|
||||
| Add a system design question | [Contribute](#contributing) |
|
||||
|
||||
### Design Pastebin.com (or Bit.ly)
|
||||
|
||||
[View exercise and solution](solutions/system_design/pastebin/README.md)
|
||||
|
||||
|
||||
|
||||
### Design the Twitter timeline and search (or Facebook feed and search)
|
||||
|
||||
[View exercise and solution](solutions/system_design/twitter/README.md)
|
||||
|
||||
|
||||
|
||||
### Design a web crawler
|
||||
|
||||
[View exercise and solution](solutions/system_design/web_crawler/README.md)
|
||||
|
||||
|
||||
|
||||
### Design Mint.com
|
||||
|
||||
[View exercise and solution](solutions/system_design/mint/README.md)
|
||||
|
||||
|
||||
|
||||
### Design the data structures for a social network
|
||||
|
||||
[View exercise and solution](solutions/system_design/social_graph/README.md)
|
||||
|
||||
|
||||
|
||||
### Design a key-value store for a search engine
|
||||
|
||||
[View exercise and solution](solutions/system_design/query_cache/README.md)
|
||||
|
||||
|
||||
|
||||
### Design Amazon's sales ranking by category feature
|
||||
|
||||
[View exercise and solution](solutions/system_design/sales_rank/README.md)
|
||||
|
||||
|
||||
|
||||
### Design a system that scales to millions of users on AWS
|
||||
|
||||
[View exercise and solution](solutions/system_design/scaling_aws/README.md)
|
||||
|
||||
|
||||
|
||||
## Object-oriented design interview questions with solutions
|
||||
|
||||
> Common object-oriented design interview questions with sample discussions, code, and diagrams.
|
||||
>
|
||||
> Solutions linked to content in the `solutions/` folder.
|
||||
|
||||
>**Note: This section is under development**
|
||||
|
||||
| Question | |
|
||||
|---|---|
|
||||
| Design a hash map | [Solution](solutions/object_oriented_design/hash_table/hash_map.ipynb) |
|
||||
| Design a least recently used cache | [Solution](solutions/object_oriented_design/lru_cache/lru_cache.ipynb) |
|
||||
| Design a call center | [Solution](solutions/object_oriented_design/call_center/call_center.ipynb) |
|
||||
| Design a deck of cards | [Solution](solutions/object_oriented_design/deck_of_cards/deck_of_cards.ipynb) |
|
||||
| Design a parking lot | [Solution](solutions/object_oriented_design/parking_lot/parking_lot.ipynb) |
|
||||
| Design a chat server | [Solution](solutions/object_oriented_design/online_chat/online_chat.ipynb) |
|
||||
| Design a circular array | [Contribute](#contributing) |
|
||||
| Add an object-oriented design question | [Contribute](#contributing) |
|
||||
|
||||
|
||||
### Real world architectures
|
||||
|
||||
> Articles on how real world systems are designed.
|
||||
|
||||
<p align="center">
|
||||
<img src="images/TcUo2fw.png">
|
||||
<br/>
|
||||
<i><a href=https://www.infoq.com/presentations/Twitter-Timeline-Scalability>Source: Twitter timelines at scale</a></i>
|
||||
</p>
|
||||
|
||||
**Don't focus on nitty gritty details for the following articles, instead:**
|
||||
|
||||
* Identify shared principles, common technologies, and patterns within these articles
|
||||
* Study what problems are solved by each component, where it works, where it doesn't
|
||||
* Review the lessons learned
|
||||
|
||||
|Type | System | Reference(s) |
|
||||
|---|---|---|
|
||||
| Data processing | **MapReduce** - Distributed data processing from Google | [research.google.com](http://static.googleusercontent.com/media/research.google.com/zh-CN/us/archive/mapreduce-osdi04.pdf) |
|
||||
| Data processing | **Spark** - Distributed data processing from Databricks | [slideshare.net](http://www.slideshare.net/AGrishchenko/apache-spark-architecture) |
|
||||
| Data processing | **Storm** - Distributed data processing from Twitter | [slideshare.net](http://www.slideshare.net/previa/storm-16094009) |
|
||||
| | | |
|
||||
| Data store | **Bigtable** - Distributed column-oriented database from Google | [harvard.edu](http://www.read.seas.harvard.edu/~kohler/class/cs239-w08/chang06bigtable.pdf) |
|
||||
| Data store | **HBase** - Open source implementation of Bigtable | [slideshare.net](http://www.slideshare.net/alexbaranau/intro-to-hbase) |
|
||||
| Data store | **Cassandra** - Distributed column-oriented database from Facebook | [slideshare.net](http://www.slideshare.net/planetcassandra/cassandra-introduction-features-30103666)
|
||||
| Data store | **DynamoDB** - Document-oriented database from Amazon | [harvard.edu](http://www.read.seas.harvard.edu/~kohler/class/cs239-w08/decandia07dynamo.pdf) |
|
||||
| Data store | **MongoDB** - Document-oriented database | [slideshare.net](http://www.slideshare.net/mdirolf/introduction-to-mongodb) |
|
||||
| Data store | **Spanner** - Globally-distributed database from Google | [research.google.com](http://research.google.com/archive/spanner-osdi2012.pdf) |
|
||||
| Data store | **Memcached** - Distributed memory caching system | [slideshare.net](http://www.slideshare.net/oemebamo/introduction-to-memcached) |
|
||||
| Data store | **Redis** - Distributed memory caching system with persistence and value types | [slideshare.net](http://www.slideshare.net/dvirsky/introduction-to-redis) |
|
||||
| | | |
|
||||
| File system | **Google File System (GFS)** - Distributed file system | [research.google.com](http://static.googleusercontent.com/media/research.google.com/zh-CN/us/archive/gfs-sosp2003.pdf) |
|
||||
| File system | **Hadoop File System (HDFS)** - Open source implementation of GFS | [apache.org](http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html) |
|
||||
| | | |
|
||||
| Misc | **Chubby** - Lock service for loosely-coupled distributed systems from Google | [research.google.com](http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/archive/chubby-osdi06.pdf) |
|
||||
| Misc | **Dapper** - Distributed systems tracing infrastructure | [research.google.com](http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36356.pdf)
|
||||
| Misc | **Kafka** - Pub/sub message queue from LinkedIn | [slideshare.net](http://www.slideshare.net/mumrah/kafka-talk-tri-hug) |
|
||||
| Misc | **Zookeeper** - Centralized infrastructure and services enabling synchronization | [slideshare.net](http://www.slideshare.net/sauravhaloi/introduction-to-apache-zookeeper) |
|
||||
| | Add an architecture | [Contribute](#contributing) |
|
||||
|
||||
### Company architectures
|
||||
|
||||
| Company | Reference(s) |
|
||||
|---|---|
|
||||
| Amazon | [Amazon architecture](http://highscalability.com/amazon-architecture) |
|
||||
| Cinchcast | [Producing 1,500 hours of audio every day](http://highscalability.com/blog/2012/7/16/cinchcast-architecture-producing-1500-hours-of-audio-every-d.html) |
|
||||
| DataSift | [Realtime datamining At 120,000 tweets per second](http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html) |
|
||||
| Dropbox | [How we've scaled Dropbox](https://www.youtube.com/watch?v=PE4gwstWhmc) |
|
||||
| ESPN | [Operating At 100,000 duh nuh nuhs per second](http://highscalability.com/blog/2013/11/4/espns-architecture-at-scale-operating-at-100000-duh-nuh-nuhs.html) |
|
||||
| Google | [Google architecture](http://highscalability.com/google-architecture) |
|
||||
| Instagram | [14 million users, terabytes of photos](http://highscalability.com/blog/2011/12/6/instagram-architecture-14-million-users-terabytes-of-photos.html)<br/>[What powers Instagram](http://instagram-engineering.tumblr.com/post/13649370142/what-powers-instagram-hundreds-of-instances) |
|
||||
| Justin.tv | [Justin.Tv's live video broadcasting architecture](http://highscalability.com/blog/2010/3/16/justintvs-live-video-broadcasting-architecture.html) |
|
||||
| Facebook | [Scaling memcached at Facebook](https://cs.uwaterloo.ca/~brecht/courses/854-Emerging-2014/readings/key-value/fb-memcached-nsdi-2013.pdf)<br/>[TAO: Facebook’s distributed data store for the social graph](https://cs.uwaterloo.ca/~brecht/courses/854-Emerging-2014/readings/data-store/tao-facebook-distributed-datastore-atc-2013.pdf)<br/>[Facebook’s photo storage](https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf)<br/>[How Facebook Live Streams To 800,000 Simultaneous Viewers](http://highscalability.com/blog/2016/6/27/how-facebook-live-streams-to-800000-simultaneous-viewers.html) |
|
||||
| Flickr | [Flickr architecture](http://highscalability.com/flickr-architecture) |
|
||||
| Mailbox | [From 0 to one million users in 6 weeks](http://highscalability.com/blog/2013/6/18/scaling-mailbox-from-0-to-one-million-users-in-6-weeks-and-1.html) |
|
||||
| Netflix | [A 360 Degree View Of The Entire Netflix Stack](http://highscalability.com/blog/2015/11/9/a-360-degree-view-of-the-entire-netflix-stack.html)<br/>[Netflix: What Happens When You Press Play?](http://highscalability.com/blog/2017/12/11/netflix-what-happens-when-you-press-play.html) |
|
||||
| Pinterest | [From 0 To 10s of billions of page views a month](http://highscalability.com/blog/2013/4/15/scaling-pinterest-from-0-to-10s-of-billions-of-page-views-a.html)<br/>[18 million visitors, 10x growth, 12 employees](http://highscalability.com/blog/2012/5/21/pinterest-architecture-update-18-million-visitors-10x-growth.html) |
|
||||
| Playfish | [50 million monthly users and growing](http://highscalability.com/blog/2010/9/21/playfishs-social-gaming-architecture-50-million-monthly-user.html) |
|
||||
| PlentyOfFish | [PlentyOfFish architecture](http://highscalability.com/plentyoffish-architecture) |
|
||||
| Salesforce | [How they handle 1.3 billion transactions a day](http://highscalability.com/blog/2013/9/23/salesforce-architecture-how-they-handle-13-billion-transacti.html) |
|
||||
| Stack Overflow | [Stack Overflow architecture](http://highscalability.com/blog/2009/8/5/stack-overflow-architecture.html) |
|
||||
| TripAdvisor | [40M visitors, 200M dynamic page views, 30TB data](http://highscalability.com/blog/2011/6/27/tripadvisor-architecture-40m-visitors-200m-dynamic-page-view.html) |
|
||||
| Tumblr | [15 billion page views a month](http://highscalability.com/blog/2012/2/13/tumblr-architecture-15-billion-page-views-a-month-and-harder.html) |
|
||||
| Twitter | [Making Twitter 10000 percent faster](http://highscalability.com/scaling-twitter-making-twitter-10000-percent-faster)<br/>[Storing 250 million tweets a day using MySQL](http://highscalability.com/blog/2011/12/19/how-twitter-stores-250-million-tweets-a-day-using-mysql.html)<br/>[150M active users, 300K QPS, a 22 MB/S firehose](http://highscalability.com/blog/2013/7/8/the-architecture-twitter-uses-to-deal-with-150m-active-users.html)<br/>[Timelines at scale](https://www.infoq.com/presentations/Twitter-Timeline-Scalability)<br/>[Big and small data at Twitter](https://www.youtube.com/watch?v=5cKTP36HVgI)<br/>[Operations at Twitter: scaling beyond 100 million users](https://www.youtube.com/watch?v=z8LU0Cj6BOU)<br/>[How Twitter Handles 3,000 Images Per Second](http://highscalability.com/blog/2016/4/20/how-twitter-handles-3000-images-per-second.html) |
|
||||
| Uber | [How Uber scales their real-time market platform](http://highscalability.com/blog/2015/9/14/how-uber-scales-their-real-time-market-platform.html)<br/>[Lessons Learned From Scaling Uber To 2000 Engineers, 1000 Services, And 8000 Git Repositories](http://highscalability.com/blog/2016/10/12/lessons-learned-from-scaling-uber-to-2000-engineers-1000-ser.html) |
|
||||
| WhatsApp | [The WhatsApp architecture Facebook bought for $19 billion](http://highscalability.com/blog/2014/2/26/the-whatsapp-architecture-facebook-bought-for-19-billion.html) |
|
||||
| YouTube | [YouTube scalability](https://www.youtube.com/watch?v=w5WVu624fY8)<br/>[YouTube architecture](http://highscalability.com/youtube-architecture) |
|
||||
|
||||
# Practice
|
||||
## Anki flashcards
|
||||
|
||||
<p align="center">
|
||||
<img src="images/zdCAkB3.png">
|
||||
<br/>
|
||||
</p>
|
||||
|
||||
The provided [Anki flashcard decks](https://apps.ankiweb.net/) use spaced repetition to help you retain key system design concepts.
|
||||
|
||||
* [System design deck](https://github.com/donnemartin/system-design-primer/tree/master/resources/flash_cards/System%20Design.apkg)
|
||||
* [System design exercises deck](https://github.com/donnemartin/system-design-primer/tree/master/resources/flash_cards/System%20Design%20Exercises.apkg)
|
||||
* [Object oriented design exercises deck](https://github.com/donnemartin/system-design-primer/tree/master/resources/flash_cards/OO%20Design.apkg)
|
||||
|
||||
Great for use while on-the-go.
|
||||
|
||||
### Additional system design interview questions
|
||||
|
||||
> Common system design interview questions, with links to resources on how to solve each.
|
||||
|
||||
| Question | Reference(s) |
|
||||
|---|---|
|
||||
| Design a file sync service like Dropbox | [youtube.com](https://www.youtube.com/watch?v=PE4gwstWhmc) |
|
||||
| Design a search engine like Google | [queue.acm.org](http://queue.acm.org/detail.cfm?id=988407)<br/>[stackexchange.com](http://programmers.stackexchange.com/questions/38324/interview-question-how-would-you-implement-google-search)<br/>[ardendertat.com](http://www.ardendertat.com/2012/01/11/implementing-search-engines/)<br/>[stanford.edu](http://infolab.stanford.edu/~backrub/google.html) |
|
||||
| Design a scalable web crawler like Google | [quora.com](https://www.quora.com/How-can-I-build-a-web-crawler-from-scratch) |
|
||||
| Design Google docs | [code.google.com](https://code.google.com/p/google-mobwrite/)<br/>[neil.fraser.name](https://neil.fraser.name/writing/sync/) |
|
||||
| Design a key-value store like Redis | [slideshare.net](http://www.slideshare.net/dvirsky/introduction-to-redis) |
|
||||
| Design a cache system like Memcached | [slideshare.net](http://www.slideshare.net/oemebamo/introduction-to-memcached) |
|
||||
| Design a recommendation system like Amazon's | [hulu.com](https://web.archive.org/web/20170406065247/http://tech.hulu.com/blog/2011/09/19/recommendation-system.html)<br/>[ijcai13.org](http://ijcai13.org/files/tutorial_slides/td3.pdf) |
|
||||
| Design a tinyurl system like Bitly | [n00tc0d3r.blogspot.com](http://n00tc0d3r.blogspot.com/) |
|
||||
| Design a chat app like WhatsApp | [highscalability.com](http://highscalability.com/blog/2014/2/26/the-whatsapp-architecture-facebook-bought-for-19-billion.html)
|
||||
| Design a picture sharing system like Instagram | [highscalability.com](http://highscalability.com/flickr-architecture)<br/>[highscalability.com](http://highscalability.com/blog/2011/12/6/instagram-architecture-14-million-users-terabytes-of-photos.html) |
|
||||
| Design the Facebook news feed function | [quora.com](http://www.quora.com/What-are-best-practices-for-building-something-like-a-News-Feed)<br/>[quora.com](http://www.quora.com/Activity-Streams/What-are-the-scaling-issues-to-keep-in-mind-while-developing-a-social-network-feed)<br/>[slideshare.net](http://www.slideshare.net/danmckinley/etsy-activity-feeds-architecture) |
|
||||
| Design the Facebook timeline function | [facebook.com](https://www.facebook.com/note.php?note_id=10150468255628920)<br/>[highscalability.com](http://highscalability.com/blog/2012/1/23/facebook-timeline-brought-to-you-by-the-power-of-denormaliza.html) |
|
||||
| Design the Facebook chat function | [erlang-factory.com](http://www.erlang-factory.com/upload/presentations/31/EugeneLetuchy-ErlangatFacebook.pdf)<br/>[facebook.com](https://www.facebook.com/note.php?note_id=14218138919&id=9445547199&index=0) |
|
||||
| Design a graph search function like Facebook's | [facebook.com](https://www.facebook.com/notes/facebook-engineering/under-the-hood-building-out-the-infrastructure-for-graph-search/10151347573598920)<br/>[facebook.com](https://www.facebook.com/notes/facebook-engineering/under-the-hood-indexing-and-ranking-in-graph-search/10151361720763920)<br/>[facebook.com](https://www.facebook.com/notes/facebook-engineering/under-the-hood-the-natural-language-interface-of-graph-search/10151432733048920) |
|
||||
| Design a content delivery network like CloudFlare | [figshare.com](https://figshare.com/articles/Globally_distributed_content_delivery/6605972) |
|
||||
| Design a trending topic system like Twitter's | [michael-noll.com](http://www.michael-noll.com/blog/2013/01/18/implementing-real-time-trending-topics-in-storm/)<br/>[snikolov .wordpress.com](http://snikolov.wordpress.com/2012/11/14/early-detection-of-twitter-trends/) |
|
||||
| Design a random ID generation system | [blog.twitter.com](https://blog.twitter.com/2010/announcing-snowflake)<br/>[github.com](https://github.com/twitter/snowflake/) |
|
||||
| Return the top k requests during a time interval | [cs.ucsb.edu](https://www.cs.ucsb.edu/sites/cs.ucsb.edu/files/docs/reports/2005-23.pdf)<br/>[wpi.edu](http://davis.wpi.edu/xmdv/docs/EDBT11-diyang.pdf) |
|
||||
| Design a system that serves data from multiple data centers | [highscalability.com](http://highscalability.com/blog/2009/8/24/how-google-serves-data-from-multiple-datacenters.html) |
|
||||
| Design an online multiplayer card game | [indieflashblog.com](https://web.archive.org/web/20180929181117/http://www.indieflashblog.com/how-to-create-an-asynchronous-multiplayer-game.html)<br/>[buildnewgames.com](http://buildnewgames.com/real-time-multiplayer/) |
|
||||
| Design a garbage collection system | [stuffwithstuff.com](http://journal.stuffwithstuff.com/2013/12/08/babys-first-garbage-collector/)<br/>[washington.edu](http://courses.cs.washington.edu/courses/csep521/07wi/prj/rick.pdf) |
|
||||
| Design an API rate limiter | [https://stripe.com/blog/](https://stripe.com/blog/rate-limiters) |
|
||||
| Design a Stock Exchange (like NASDAQ or Binance) | [Jane Street](https://youtu.be/b1e4t2k2KJY)<br/>[Golang Implementation](https://around25.com/blog/building-a-trading-engine-for-a-crypto-exchange/)<br/>[Go Implemenation](http://bhomnick.net/building-a-simple-limit-order-in-go/) |
|
||||
| Add a system design question | [Contribute](#contributing) |
|
||||
|
||||
### Coding Resource: Interactive Coding Challenges
|
||||
|
||||
Looking for resources to help you prep for the [**Coding Interview**](https://github.com/donnemartin/interactive-coding-challenges)?
|
||||
|
||||
<p align="center">
|
||||
<img src="images/b4YtAEN.png">
|
||||
<br/>
|
||||
</p>
|
||||
|
||||
Check out the sister repo [**Interactive Coding Challenges**](https://github.com/donnemartin/interactive-coding-challenges), which contains an additional Anki deck:
|
||||
|
||||
* [Coding deck](https://github.com/donnemartin/interactive-coding-challenges/tree/master/anki_cards/Coding.apkg)
|
||||
|
||||
# Additional Resources
|
||||
|
||||
## Contributing
|
||||
|
||||
> Learn from the community.
|
||||
|
||||
Feel free to submit pull requests to help:
|
||||
|
||||
* Fix errors
|
||||
* Improve sections
|
||||
* Add new sections
|
||||
* [Translate](https://github.com/donnemartin/system-design-primer/issues/28)
|
||||
|
||||
Content that needs some polishing is placed [under development](#under-development).
|
||||
|
||||
Review the [Contributing Guidelines](CONTRIBUTING.md).
|
||||
|
||||
|
||||
* [Appendix](#appendix)
|
||||
* [Powers of two table](#powers-of-two-table)
|
||||
* [Latency numbers every programmer should know](#latency-numbers-every-programmer-should-know)
|
||||
* [Additional system design interview questions](#additional-system-design-interview-questions)
|
||||
* [Real world architectures](#real-world-architectures)
|
||||
* [Company architectures](#company-architectures)
|
||||
* [Company engineering blogs](#company-engineering-blogs)
|
||||
* [Under development](#under-development)
|
||||
* [Credits](#credits)
|
||||
* [Contact info](#contact-info)
|
||||
* [License](#license)
|
||||
|
||||
## Appendix
|
||||
|
||||
You'll sometimes be asked to do 'back-of-the-envelope' estimates. For example, you might need to determine how long it will take to generate 100 image thumbnails from disk or how much memory a data structure will take. The **Powers of two table** and **Latency numbers every programmer should know** are handy references.
|
||||
|
|
@ -1645,104 +1759,8 @@ Handy metrics based on numbers above:
|
|||
* [Designs, lessons, and advice from building large distributed systems](http://www.cs.cornell.edu/projects/ladis2009/talks/dean-keynote-ladis2009.pdf)
|
||||
* [Software Engineering Advice from Building Large-Scale Distributed Systems](https://static.googleusercontent.com/media/research.google.com/en//people/jeff/stanford-295-talk.pdf)
|
||||
|
||||
### Additional system design interview questions
|
||||
|
||||
> Common system design interview questions, with links to resources on how to solve each.
|
||||
|
||||
| Question | Reference(s) |
|
||||
|---|---|
|
||||
| Design a file sync service like Dropbox | [youtube.com](https://www.youtube.com/watch?v=PE4gwstWhmc) |
|
||||
| Design a search engine like Google | [queue.acm.org](http://queue.acm.org/detail.cfm?id=988407)<br/>[stackexchange.com](http://programmers.stackexchange.com/questions/38324/interview-question-how-would-you-implement-google-search)<br/>[ardendertat.com](http://www.ardendertat.com/2012/01/11/implementing-search-engines/)<br/>[stanford.edu](http://infolab.stanford.edu/~backrub/google.html) |
|
||||
| Design a scalable web crawler like Google | [quora.com](https://www.quora.com/How-can-I-build-a-web-crawler-from-scratch) |
|
||||
| Design Google docs | [code.google.com](https://code.google.com/p/google-mobwrite/)<br/>[neil.fraser.name](https://neil.fraser.name/writing/sync/) |
|
||||
| Design a key-value store like Redis | [slideshare.net](http://www.slideshare.net/dvirsky/introduction-to-redis) |
|
||||
| Design a cache system like Memcached | [slideshare.net](http://www.slideshare.net/oemebamo/introduction-to-memcached) |
|
||||
| Design a recommendation system like Amazon's | [hulu.com](https://web.archive.org/web/20170406065247/http://tech.hulu.com/blog/2011/09/19/recommendation-system.html)<br/>[ijcai13.org](http://ijcai13.org/files/tutorial_slides/td3.pdf) |
|
||||
| Design a tinyurl system like Bitly | [n00tc0d3r.blogspot.com](http://n00tc0d3r.blogspot.com/) |
|
||||
| Design a chat app like WhatsApp | [highscalability.com](http://highscalability.com/blog/2014/2/26/the-whatsapp-architecture-facebook-bought-for-19-billion.html)
|
||||
| Design a picture sharing system like Instagram | [highscalability.com](http://highscalability.com/flickr-architecture)<br/>[highscalability.com](http://highscalability.com/blog/2011/12/6/instagram-architecture-14-million-users-terabytes-of-photos.html) |
|
||||
| Design the Facebook news feed function | [quora.com](http://www.quora.com/What-are-best-practices-for-building-something-like-a-News-Feed)<br/>[quora.com](http://www.quora.com/Activity-Streams/What-are-the-scaling-issues-to-keep-in-mind-while-developing-a-social-network-feed)<br/>[slideshare.net](http://www.slideshare.net/danmckinley/etsy-activity-feeds-architecture) |
|
||||
| Design the Facebook timeline function | [facebook.com](https://www.facebook.com/note.php?note_id=10150468255628920)<br/>[highscalability.com](http://highscalability.com/blog/2012/1/23/facebook-timeline-brought-to-you-by-the-power-of-denormaliza.html) |
|
||||
| Design the Facebook chat function | [erlang-factory.com](http://www.erlang-factory.com/upload/presentations/31/EugeneLetuchy-ErlangatFacebook.pdf)<br/>[facebook.com](https://www.facebook.com/note.php?note_id=14218138919&id=9445547199&index=0) |
|
||||
| Design a graph search function like Facebook's | [facebook.com](https://www.facebook.com/notes/facebook-engineering/under-the-hood-building-out-the-infrastructure-for-graph-search/10151347573598920)<br/>[facebook.com](https://www.facebook.com/notes/facebook-engineering/under-the-hood-indexing-and-ranking-in-graph-search/10151361720763920)<br/>[facebook.com](https://www.facebook.com/notes/facebook-engineering/under-the-hood-the-natural-language-interface-of-graph-search/10151432733048920) |
|
||||
| Design a content delivery network like CloudFlare | [figshare.com](https://figshare.com/articles/Globally_distributed_content_delivery/6605972) |
|
||||
| Design a trending topic system like Twitter's | [michael-noll.com](http://www.michael-noll.com/blog/2013/01/18/implementing-real-time-trending-topics-in-storm/)<br/>[snikolov .wordpress.com](http://snikolov.wordpress.com/2012/11/14/early-detection-of-twitter-trends/) |
|
||||
| Design a random ID generation system | [blog.twitter.com](https://blog.twitter.com/2010/announcing-snowflake)<br/>[github.com](https://github.com/twitter/snowflake/) |
|
||||
| Return the top k requests during a time interval | [cs.ucsb.edu](https://www.cs.ucsb.edu/sites/cs.ucsb.edu/files/docs/reports/2005-23.pdf)<br/>[wpi.edu](http://davis.wpi.edu/xmdv/docs/EDBT11-diyang.pdf) |
|
||||
| Design a system that serves data from multiple data centers | [highscalability.com](http://highscalability.com/blog/2009/8/24/how-google-serves-data-from-multiple-datacenters.html) |
|
||||
| Design an online multiplayer card game | [indieflashblog.com](https://web.archive.org/web/20180929181117/http://www.indieflashblog.com/how-to-create-an-asynchronous-multiplayer-game.html)<br/>[buildnewgames.com](http://buildnewgames.com/real-time-multiplayer/) |
|
||||
| Design a garbage collection system | [stuffwithstuff.com](http://journal.stuffwithstuff.com/2013/12/08/babys-first-garbage-collector/)<br/>[washington.edu](http://courses.cs.washington.edu/courses/csep521/07wi/prj/rick.pdf) |
|
||||
| Design an API rate limiter | [https://stripe.com/blog/](https://stripe.com/blog/rate-limiters) |
|
||||
| Design a Stock Exchange (like NASDAQ or Binance) | [Jane Street](https://youtu.be/b1e4t2k2KJY)<br/>[Golang Implementation](https://around25.com/blog/building-a-trading-engine-for-a-crypto-exchange/)<br/>[Go Implemenation](http://bhomnick.net/building-a-simple-limit-order-in-go/) |
|
||||
| Add a system design question | [Contribute](#contributing) |
|
||||
|
||||
### Real world architectures
|
||||
|
||||
> Articles on how real world systems are designed.
|
||||
|
||||
<p align="center">
|
||||
<img src="images/TcUo2fw.png">
|
||||
<br/>
|
||||
<i><a href=https://www.infoq.com/presentations/Twitter-Timeline-Scalability>Source: Twitter timelines at scale</a></i>
|
||||
</p>
|
||||
|
||||
**Don't focus on nitty gritty details for the following articles, instead:**
|
||||
|
||||
* Identify shared principles, common technologies, and patterns within these articles
|
||||
* Study what problems are solved by each component, where it works, where it doesn't
|
||||
* Review the lessons learned
|
||||
|
||||
|Type | System | Reference(s) |
|
||||
|---|---|---|
|
||||
| Data processing | **MapReduce** - Distributed data processing from Google | [research.google.com](http://static.googleusercontent.com/media/research.google.com/zh-CN/us/archive/mapreduce-osdi04.pdf) |
|
||||
| Data processing | **Spark** - Distributed data processing from Databricks | [slideshare.net](http://www.slideshare.net/AGrishchenko/apache-spark-architecture) |
|
||||
| Data processing | **Storm** - Distributed data processing from Twitter | [slideshare.net](http://www.slideshare.net/previa/storm-16094009) |
|
||||
| | | |
|
||||
| Data store | **Bigtable** - Distributed column-oriented database from Google | [harvard.edu](http://www.read.seas.harvard.edu/~kohler/class/cs239-w08/chang06bigtable.pdf) |
|
||||
| Data store | **HBase** - Open source implementation of Bigtable | [slideshare.net](http://www.slideshare.net/alexbaranau/intro-to-hbase) |
|
||||
| Data store | **Cassandra** - Distributed column-oriented database from Facebook | [slideshare.net](http://www.slideshare.net/planetcassandra/cassandra-introduction-features-30103666)
|
||||
| Data store | **DynamoDB** - Document-oriented database from Amazon | [harvard.edu](http://www.read.seas.harvard.edu/~kohler/class/cs239-w08/decandia07dynamo.pdf) |
|
||||
| Data store | **MongoDB** - Document-oriented database | [slideshare.net](http://www.slideshare.net/mdirolf/introduction-to-mongodb) |
|
||||
| Data store | **Spanner** - Globally-distributed database from Google | [research.google.com](http://research.google.com/archive/spanner-osdi2012.pdf) |
|
||||
| Data store | **Memcached** - Distributed memory caching system | [slideshare.net](http://www.slideshare.net/oemebamo/introduction-to-memcached) |
|
||||
| Data store | **Redis** - Distributed memory caching system with persistence and value types | [slideshare.net](http://www.slideshare.net/dvirsky/introduction-to-redis) |
|
||||
| | | |
|
||||
| File system | **Google File System (GFS)** - Distributed file system | [research.google.com](http://static.googleusercontent.com/media/research.google.com/zh-CN/us/archive/gfs-sosp2003.pdf) |
|
||||
| File system | **Hadoop File System (HDFS)** - Open source implementation of GFS | [apache.org](http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html) |
|
||||
| | | |
|
||||
| Misc | **Chubby** - Lock service for loosely-coupled distributed systems from Google | [research.google.com](http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/archive/chubby-osdi06.pdf) |
|
||||
| Misc | **Dapper** - Distributed systems tracing infrastructure | [research.google.com](http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36356.pdf)
|
||||
| Misc | **Kafka** - Pub/sub message queue from LinkedIn | [slideshare.net](http://www.slideshare.net/mumrah/kafka-talk-tri-hug) |
|
||||
| Misc | **Zookeeper** - Centralized infrastructure and services enabling synchronization | [slideshare.net](http://www.slideshare.net/sauravhaloi/introduction-to-apache-zookeeper) |
|
||||
| | Add an architecture | [Contribute](#contributing) |
|
||||
|
||||
### Company architectures
|
||||
|
||||
| Company | Reference(s) |
|
||||
|---|---|
|
||||
| Amazon | [Amazon architecture](http://highscalability.com/amazon-architecture) |
|
||||
| Cinchcast | [Producing 1,500 hours of audio every day](http://highscalability.com/blog/2012/7/16/cinchcast-architecture-producing-1500-hours-of-audio-every-d.html) |
|
||||
| DataSift | [Realtime datamining At 120,000 tweets per second](http://highscalability.com/blog/2011/11/29/datasift-architecture-realtime-datamining-at-120000-tweets-p.html) |
|
||||
| Dropbox | [How we've scaled Dropbox](https://www.youtube.com/watch?v=PE4gwstWhmc) |
|
||||
| ESPN | [Operating At 100,000 duh nuh nuhs per second](http://highscalability.com/blog/2013/11/4/espns-architecture-at-scale-operating-at-100000-duh-nuh-nuhs.html) |
|
||||
| Google | [Google architecture](http://highscalability.com/google-architecture) |
|
||||
| Instagram | [14 million users, terabytes of photos](http://highscalability.com/blog/2011/12/6/instagram-architecture-14-million-users-terabytes-of-photos.html)<br/>[What powers Instagram](http://instagram-engineering.tumblr.com/post/13649370142/what-powers-instagram-hundreds-of-instances) |
|
||||
| Justin.tv | [Justin.Tv's live video broadcasting architecture](http://highscalability.com/blog/2010/3/16/justintvs-live-video-broadcasting-architecture.html) |
|
||||
| Facebook | [Scaling memcached at Facebook](https://cs.uwaterloo.ca/~brecht/courses/854-Emerging-2014/readings/key-value/fb-memcached-nsdi-2013.pdf)<br/>[TAO: Facebook’s distributed data store for the social graph](https://cs.uwaterloo.ca/~brecht/courses/854-Emerging-2014/readings/data-store/tao-facebook-distributed-datastore-atc-2013.pdf)<br/>[Facebook’s photo storage](https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf)<br/>[How Facebook Live Streams To 800,000 Simultaneous Viewers](http://highscalability.com/blog/2016/6/27/how-facebook-live-streams-to-800000-simultaneous-viewers.html) |
|
||||
| Flickr | [Flickr architecture](http://highscalability.com/flickr-architecture) |
|
||||
| Mailbox | [From 0 to one million users in 6 weeks](http://highscalability.com/blog/2013/6/18/scaling-mailbox-from-0-to-one-million-users-in-6-weeks-and-1.html) |
|
||||
| Netflix | [A 360 Degree View Of The Entire Netflix Stack](http://highscalability.com/blog/2015/11/9/a-360-degree-view-of-the-entire-netflix-stack.html)<br/>[Netflix: What Happens When You Press Play?](http://highscalability.com/blog/2017/12/11/netflix-what-happens-when-you-press-play.html) |
|
||||
| Pinterest | [From 0 To 10s of billions of page views a month](http://highscalability.com/blog/2013/4/15/scaling-pinterest-from-0-to-10s-of-billions-of-page-views-a.html)<br/>[18 million visitors, 10x growth, 12 employees](http://highscalability.com/blog/2012/5/21/pinterest-architecture-update-18-million-visitors-10x-growth.html) |
|
||||
| Playfish | [50 million monthly users and growing](http://highscalability.com/blog/2010/9/21/playfishs-social-gaming-architecture-50-million-monthly-user.html) |
|
||||
| PlentyOfFish | [PlentyOfFish architecture](http://highscalability.com/plentyoffish-architecture) |
|
||||
| Salesforce | [How they handle 1.3 billion transactions a day](http://highscalability.com/blog/2013/9/23/salesforce-architecture-how-they-handle-13-billion-transacti.html) |
|
||||
| Stack Overflow | [Stack Overflow architecture](http://highscalability.com/blog/2009/8/5/stack-overflow-architecture.html) |
|
||||
| TripAdvisor | [40M visitors, 200M dynamic page views, 30TB data](http://highscalability.com/blog/2011/6/27/tripadvisor-architecture-40m-visitors-200m-dynamic-page-view.html) |
|
||||
| Tumblr | [15 billion page views a month](http://highscalability.com/blog/2012/2/13/tumblr-architecture-15-billion-page-views-a-month-and-harder.html) |
|
||||
| Twitter | [Making Twitter 10000 percent faster](http://highscalability.com/scaling-twitter-making-twitter-10000-percent-faster)<br/>[Storing 250 million tweets a day using MySQL](http://highscalability.com/blog/2011/12/19/how-twitter-stores-250-million-tweets-a-day-using-mysql.html)<br/>[150M active users, 300K QPS, a 22 MB/S firehose](http://highscalability.com/blog/2013/7/8/the-architecture-twitter-uses-to-deal-with-150m-active-users.html)<br/>[Timelines at scale](https://www.infoq.com/presentations/Twitter-Timeline-Scalability)<br/>[Big and small data at Twitter](https://www.youtube.com/watch?v=5cKTP36HVgI)<br/>[Operations at Twitter: scaling beyond 100 million users](https://www.youtube.com/watch?v=z8LU0Cj6BOU)<br/>[How Twitter Handles 3,000 Images Per Second](http://highscalability.com/blog/2016/4/20/how-twitter-handles-3000-images-per-second.html) |
|
||||
| Uber | [How Uber scales their real-time market platform](http://highscalability.com/blog/2015/9/14/how-uber-scales-their-real-time-market-platform.html)<br/>[Lessons Learned From Scaling Uber To 2000 Engineers, 1000 Services, And 8000 Git Repositories](http://highscalability.com/blog/2016/10/12/lessons-learned-from-scaling-uber-to-2000-engineers-1000-ser.html) |
|
||||
| WhatsApp | [The WhatsApp architecture Facebook bought for $19 billion](http://highscalability.com/blog/2014/2/26/the-whatsapp-architecture-facebook-bought-for-19-billion.html) |
|
||||
| YouTube | [YouTube scalability](https://www.youtube.com/watch?v=w5WVu624fY8)<br/>[YouTube architecture](http://highscalability.com/youtube-architecture) |
|
||||
|
||||
### Company engineering blogs
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue