mirrors
/
system-design-primer
mirror of https://github.com/donnemartin/system-design-primer.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

translate: 为社交网络设计数据结构 (#33)

pull/374/head
Jiang Haichao 2017-07-19 15:39:15 +08:00 committed by 根号三
parent 0c04035f90
commit 771a905591
1 changed files with 124 additions and 125 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

249
solutions/system_design/social_graph/README.md
View File

@ -1,66 +1,66 @@
# Design the data structures for a social network # 为社交网络设计数据结构
*Note: This document links directly to relevant areas found in the [system design topics](https://github.com/donnemartin/system-design-primer#index-of-system-design-topics) to avoid duplication. Refer to the linked content for general talking points, tradeoffs, and alternatives.* **注释：为了避免重复，这篇文章的链接直接关联到 [系统设计主题](https://github.com/donnemartin/system-design-primer#index-of-system-design-topics) 的相关章节。为一讨论要点、折中方案和可选方案做参考。**
## Step 1: Outline use cases and constraints ## 第 1 步：用例和约束概要
> Gather requirements and scope the problem. > 收集需求并调查问题。
> Ask questions to clarify use cases and constraints. > 通过提问清晰用例和约束。
> Discuss assumptions. > 讨论假设。
Without an interviewer to address clarifying questions, we'll define some use cases and constraints. 如果没有面试官提出明确的问题，我们将自己定义一些用例和约束条件。
### Use cases ### 用例
#### We'll scope the problem to handle only the following use cases #### 我们就处理以下用例审视这一问题
* **User** searches for someone and sees the shortest path to the searched person * **用户** 寻找某人并显示与被寻人之间的最短路径
* **Service** has high availability * **服务** 高可用
### Constraints and assumptions ### 约束和假设
#### State assumptions #### 状态假设
* Traffic is not evenly distributed * 流量分布不均
* Some searches are more popular than others, while others are only executed once * 某些搜索比别的更热门，同时某些搜索仅执行一次
* Graph data won't fit on a single machine * 图数据不适用单一机器
* Graph edges are unweighted * 图的边没有权重
* 100 million users * 1 千万用户
* 50 friends per user average * 每个用户平均有 50 个朋友
* 1 billion friend searches per month * 每月 10 亿次朋友搜索
Exercise the use of more traditional systems - don't use graph-specific solutions such as [GraphQL](http://graphql.org/) or a graph database like [Neo4j](https://neo4j.com/) 训练使用更传统的系统 - 别用图特有的解决方案例如 [GraphQL](http://graphql.org/) 或图数据库如 [Neo4j](https://neo4j.com/)。
#### Calculate usage #### 计算使用
**Clarify with your interviewer if you should run back-of-the-envelope usage calculations.** **向你的面试官厘清你是否应该做粗略的使用计算**
* 5 billion friend relationships * 50 亿朋友关系
* 100 million users * 50 friends per user average * 1 亿用户 * 平均每人 50 个朋友
* 400 search requests per second * 每秒 400 次搜索请求
Handy conversion guide: 便捷的转换指南：
* 2.5 million seconds per month * 每月 250 万秒
* 1 request per second = 2.5 million requests per month * 每秒 1 个请求 = 每月 250 万次请求
* 40 requests per second = 100 million requests per month * 每秒 40 个请求 = 每月 1 亿次请求
* 400 requests per second = 1 billion requests per month * 每秒 400 个请求 = 每月 10 亿次请求
## Step 2: Create a high level design ## 第 2 步：创建高级设计方案
> Outline a high level design with all important components. > 用所有重要组件概述高水平设计
![Imgur](http://i.imgur.com/wxXyq2J.png) ![Imgur](http://i.imgur.com/wxXyq2J.png)
## Step 3: Design core components ## 第 3 步：设计核心组件
> Dive into details for each core component. > 深入每个核心组件的细节。
### Use case: User searches for someone and sees the shortest path to the searched person ### 用例： 用户搜索某人并查看到被搜人的最短路径
**Clarify with your interviewer how much code you are expected to write**. **和你的面试官说清你期望的代码量**
Without the constraint of millions of users (vertices) and billions of friend relationships (edges), we could solve this unweighted shortest path task with a general BFS approach: 没有百万用户（点）的和十亿朋友关系（边）的限制，我们能够用一般 BFS 方法解决无权重最短路径任务：
``` ```
class Graph(Graph): class Graph(Graph):
@ -99,23 +99,22 @@ class Graph(Graph):
return None return None
``` ```
We won't be able to fit all users on the same machine, we'll need to [shard](https://github.com/donnemartin/system-design-primer#sharding) users across **Person Servers** and access them with a **Lookup Service**. 我们不能在同一台机器上满足所有用户，我们需要通过 **人员服务器** [拆分](https://github.com/donnemartin/system-design-primer#sharding) 用户并且通过 **查询服务** 访问。
* The **Client** sends a request to the **Web Server**, running as a [reverse proxy](https://github.com/donnemartin/system-design-primer#reverse-proxy-web-server) * **客户端** 向 **服务器** 发送请求，**服务器** 作为 [反向代理](https://github.com/donnemartin/system-design-primer#reverse-proxy-web-server)
* The **Web Server** forwards the request to the **Search API** server * **搜索 API** 服务器向 **用户图服务** 转发请求
* The **Search API** server forwards the request to the **User Graph Service** * **用户图服务** 有以下功能：
* The **User Graph Service** does the following: * 使用 **查询服务** 找到当前用户信息存储的 **人员服务器**
* Uses the **Lookup Service** to find the **Person Server** where the current user's info is stored * 找到适当的 **人员服务器** 检索当前用户的 `friend_ids` 列表
* Finds the appropriate **Person Server** to retrieve the current user's list of `friend_ids` * 把当前用户作为 `source` 运行 BFS 搜索算法同时 当前用户的 `friend_ids` 作为每个 `adjacent_node` 的 ids
* Runs a BFS search using the current user as the `source` and the current user's `friend_ids` as the ids for each `adjacent_node` * 给定 id 获取 `adjacent_node`:
* To get the `adjacent_node` from a given id: * **用户图服务** 将 **再次** 和 **查询服务** 通讯，最后判断出和给定 id 相匹配的存储 `adjacent_node` 的 **人员服务器**（有待优化）
* The **User Graph Service** will *again* need to communicate with the **Lookup Service** to determine which **Person Server** stores the`adjacent_node` matching the given id (potential for optimization)
**Clarify with your interviewer how much code you should be writing**. **和你的面试官说清你应该写的代码量**
**Note**: Error handling is excluded below for simplicity. Ask if you should code proper error handing. **注释**：简易版错误处理执行如下。询问你是否需要编写适当的错误处理方法。
**Lookup Service** implementation: **查询服务** 实现：
``` ```
class LookupService(object): class LookupService(object):
@ -130,7 +129,7 @@ class LookupService(object):
return self.lookup[person_id] return self.lookup[person_id]
``` ```
**Person Server** implementation: **人员服务器** 实现：
``` ```
class PersonServer(object): class PersonServer(object):
@ -149,7 +148,7 @@ class PersonServer(object):
return results return results
``` ```
**Person** implementation: **用户** 实现：
``` ```
class Person(object): class Person(object):
@ -160,7 +159,7 @@ class Person(object):
self.friend_ids = friend_ids self.friend_ids = friend_ids
``` ```
**User Graph Service** implementation: **用户图服务** 实现：
``` ```
class UserGraphService(object): class UserGraphService(object):
@ -218,13 +217,13 @@ class UserGraphService(object):
return None return None
``` ```
We'll use a public [**REST API**](https://github.com/donnemartin/system-design-primer#representational-state-transfer-rest): 我们用的是公共的 [**REST API**](https://github.com/donnemartin/system-design-primer#representational-state-transfer-rest)：
``` ```
$ curl https://social.com/api/v1/friend_search?person_id=1234 $ curl https://social.com/api/v1/friend_search?person_id=1234
``` ```
Response: 响应：
``` ```
{ {
@ -244,106 +243,106 @@ Response:
}, },
``` ```
For internal communications, we could use [Remote Procedure Calls](https://github.com/donnemartin/system-design-primer#remote-procedure-call-rpc). 内部通信使用 [远端过程调用](https://github.com/donnemartin/system-design-primer#remote-procedure-call-rpc)。
## Step 4: Scale the design ## 第 4 步：扩展设计
> Identify and address bottlenecks, given the constraints. > 在给定约束条件下，定义和确认瓶颈。
![Imgur](http://i.imgur.com/cdCv5g7.png) ![Imgur](http://i.imgur.com/cdCv5g7.png)
**Important: Do not simply jump right into the final design from the initial design!** **重要：别简化从最初设计到最终设计的过程！**
State you would 1) **Benchmark/Load Test**, 2) **Profile** for bottlenecks 3) address bottlenecks while evaluating alternatives and trade-offs, and 4) repeat. See [Design a system that scales to millions of users on AWS](../scaling_aws/README.md) as a sample on how to iteratively scale the initial design. 你将要做的是：1) **基准/负载 测试**， 2) 瓶颈 **概述**， 3) 当评估可选和折中方案时定位瓶颈，4) 重复。以 [在 AWS 上设计支持百万级到千万级用户的系统](../scaling_aws/README.md) 为参考迭代地扩展最初设计。
It's important to discuss what bottlenecks you might encounter with the initial design and how you might address each of them. For example, what issues are addressed by adding a **Load Balancer** with multiple **Web Servers**? **CDN**? **Master-Slave Replicas**? What are the alternatives and **Trade-Offs** for each? 讨论最初设计可能遇到的瓶颈和处理方法十分重要。例如，什么问题可以通过添加多台 **Web 服务器** 作为 **负载均衡** 解决？**CDN**？**主从副本**？每个问题都有哪些替代和 **折中** 方案？
We'll introduce some components to complete the design and to address scalability issues. Internal load balancers are not shown to reduce clutter. 我们即将介绍一些组件来完成设计和解决扩展性问题。内部负载均衡不显示以减少混乱。
*To avoid repeating discussions*, refer to the following [system design topics](https://github.com/donnemartin/system-design-primer#index-of-system-design-topics) for main talking points, tradeoffs, and alternatives: **避免重复讨论**，以下网址链接到 [系统设计主题](https://github.com/donnemartin/system-design-primer#index-of-system-design-topics) 相关的主流方案、折中方案和替代方案。
* [DNS](https://github.com/donnemartin/system-design-primer#domain-name-system) * [DNS](https://github.com/donnemartin/system-design-primer#domain-name-system)
* [Load balancer](https://github.com/donnemartin/system-design-primer#load-balancer) * [负载均衡](https://github.com/donnemartin/system-design-primer#load-balancer)
* [Horizontal scaling](https://github.com/donnemartin/system-design-primer#horizontal-scaling) * [横向扩展](https://github.com/donnemartin/system-design-primer#horizontal-scaling)
* [Web server (reverse proxy)](https://github.com/donnemartin/system-design-primer#reverse-proxy-web-server) * [Web 服务器（反向代理）](https://github.com/donnemartin/system-design-primer#reverse-proxy-web-server)
* [API server (application layer)](https://github.com/donnemartin/system-design-primer#application-layer) * [API 服务器（应用层）](https://github.com/donnemartin/system-design-primer#application-layer)
* [Cache](https://github.com/donnemartin/system-design-primer#cache) * [缓存](https://github.com/donnemartin/system-design-primer#cache)
* [Consistency patterns](https://github.com/donnemartin/system-design-primer#consistency-patterns) * [一致性模式](https://github.com/donnemartin/system-design-primer#consistency-patterns)
* [Availability patterns](https://github.com/donnemartin/system-design-primer#availability-patterns) * [可用性模式](https://github.com/donnemartin/system-design-primer#availability-patterns)
To address the constraint of 400 *average* read requests per second (higher at peak), person data can be served from a **Memory Cache** such as Redis or Memcached to reduce response times and to reduce traffic to downstream services. This could be especially useful for people who do multiple searches in succession and for people who are well-connected. Reading 1 MB sequentially from memory takes about 250 microseconds, while reading from SSD takes 4x and from disk takes 80x longer.<sup><a href=https://github.com/donnemartin/system-design-primer#latency-numbers-every-programmer-should-know>1</a></sup> 解决 **平均** 每秒 400 次请求的限制（峰值），人员数据可以存在例如 Redis 或 Memcached 这样的 **内存** 中以减少响应次数和下游流量通信服务。这尤其在用户执行多次连续查询和查询哪些广泛连接的人时十分有用。从内存中读取 1MB 数据大约要 250 微秒，从 SSD 中读取同样大小的数据时间要长 4 倍，从硬盘要长 80 倍。<sup><a href=https://github.com/donnemartin/system-design-primer#latency-numbers-every-programmer-should-know>1</a></sup>
Below are further optimizations: 以下是进一步优化方案：
* Store complete or partial BFS traversals to speed up subsequent lookups in the **Memory Cache** * 在 **内存** 中存储完整的或部分的BFS遍历加快后续查找
* Batch compute offline then store complete or partial BFS traversals to speed up subsequent lookups in a **NoSQL Database** * 在 **NoSQL 数据库** 中批量离线计算并存储完整的或部分的BFS遍历加快后续查找
* Reduce machine jumps by batching together friend lookups hosted on the same **Person Server** * 在同一台 **人员服务器** 上托管批处理同一批朋友查找减少机器跳转
* [Shard](https://github.com/donnemartin/system-design-primer#sharding) **Person Servers** by location to further improve this, as friends generally live closer to each other * 通过地理位置 [拆分](https://github.com/donnemartin/system-design-primer#sharding) **人员服务器** 来进一步优化，因为朋友通常住得都比较近
* Do two BFS searches at the same time, one starting from the source, and one from the destination, then merge the two paths * 同时进行两个 BFS 查找，一个从 source 开始，一个从 destination 开始，然后合并两个路径
* Start the BFS search from people with large numbers of friends, as they are more likely to reduce the number of [degrees of separation](https://en.wikipedia.org/wiki/Six_degrees_of_separation) between the current user and the search target * 从有庞大朋友圈的人开始找起，这样更有可能减小当前用户和搜索目标之间的 [离散度数](https://en.wikipedia.org/wiki/Six_degrees_of_separation)
* Set a limit based on time or number of hops before asking the user if they want to continue searching, as searching could take a considerable amount of time in some cases * 在询问用户是否继续查询之前设置基于时间或跳跃数阈值，当在某些案例中搜索耗费时间过长时。
* Use a **Graph Database** such as [Neo4j](https://neo4j.com/) or a graph-specific query language such as [GraphQL](http://graphql.org/) (if there were no constraint preventing the use of **Graph Databases**) * 使用类似 [Neo4j](https://neo4j.com/) 的 **图数据库** 或图特定查询语法，例如 [GraphQL](http://graphql.org/)（如果没有禁止使用 **图数据库** 的限制的话）
## Additional talking points ## 额外的话题
> Additional topics to dive into, depending on the problem scope and time remaining. > 根据问题的范围和剩余时间，还需要深入讨论其他问题。
### SQL scaling patterns ### SQL 扩展模式
* [Read replicas](https://github.com/donnemartin/system-design-primer#master-slave) * [读取副本](https://github.com/donnemartin/system-design-primer#master-slave)
* [Federation](https://github.com/donnemartin/system-design-primer#federation) * [集合](https://github.com/donnemartin/system-design-primer#federation)
* [Sharding](https://github.com/donnemartin/system-design-primer#sharding) * [分区](https://github.com/donnemartin/system-design-primer#sharding)
* [Denormalization](https://github.com/donnemartin/system-design-primer#denormalization) * [反规范化](https://github.com/donnemartin/system-design-primer#denormalization)
* [SQL Tuning](https://github.com/donnemartin/system-design-primer#sql-tuning) * [SQL 调优](https://github.com/donnemartin/system-design-primer#sql-tuning)
#### NoSQL #### NoSQL
* [Key-value store](https://github.com/donnemartin/system-design-primer#key-value-store) * [键值存储](https://github.com/donnemartin/system-design-primer#key-value-store)
* [Document store](https://github.com/donnemartin/system-design-primer#document-store) * [文档存储](https://github.com/donnemartin/system-design-primer#document-store)
* [Wide column store](https://github.com/donnemartin/system-design-primer#wide-column-store) * [宽表存储](https://github.com/donnemartin/system-design-primer#wide-column-store)
* [Graph database](https://github.com/donnemartin/system-design-primer#graph-database) * [图数据库](https://github.com/donnemartin/system-design-primer#graph-database)
* [SQL vs NoSQL](https://github.com/donnemartin/system-design-primer#sql-or-nosql) * [SQL vs NoSQL](https://github.com/donnemartin/system-design-primer#sql-or-nosql)
### Caching ### 缓存
* Where to cache * 缓存到哪里
* [Client caching](https://github.com/donnemartin/system-design-primer#client-caching) * [客户端缓存](https://github.com/donnemartin/system-design-primer#client-caching)
* [CDN caching](https://github.com/donnemartin/system-design-primer#cdn-caching) * [CDN 缓存](https://github.com/donnemartin/system-design-primer#cdn-caching)
* [Web server caching](https://github.com/donnemartin/system-design-primer#web-server-caching) * [Web 服务缓存](https://github.com/donnemartin/system-design-primer#web-server-caching)
* [Database caching](https://github.com/donnemartin/system-design-primer#database-caching) * [数据库缓存](https://github.com/donnemartin/system-design-primer#database-caching)
* [Application caching](https://github.com/donnemartin/system-design-primer#application-caching) * [应用缓存](https://github.com/donnemartin/system-design-primer#application-caching)
* What to cache * 缓存什么
* [Caching at the database query level](https://github.com/donnemartin/system-design-primer#caching-at-the-database-query-level) * [数据库请求层缓存](https://github.com/donnemartin/system-design-primer#caching-at-the-database-query-level)
* [Caching at the object level](https://github.com/donnemartin/system-design-primer#caching-at-the-object-level) * [对象层缓存](https://github.com/donnemartin/system-design-primer#caching-at-the-object-level)
* When to update the cache * 何时更新缓存
* [Cache-aside](https://github.com/donnemartin/system-design-primer#cache-aside) * [预留缓存](https://github.com/donnemartin/system-design-primer#cache-aside)
* [Write-through](https://github.com/donnemartin/system-design-primer#write-through) * [完全写入](https://github.com/donnemartin/system-design-primer#write-through)
* [Write-behind (write-back)](https://github.com/donnemartin/system-design-primer#write-behind-write-back) * [延迟写 (写回)](https://github.com/donnemartin/system-design-primer#write-behind-write-back)
* [Refresh ahead](https://github.com/donnemartin/system-design-primer#refresh-ahead) * [事先更新](https://github.com/donnemartin/system-design-primer#refresh-ahead)
### Asynchronism and microservices ### 异步性和微服务
* [Message queues](https://github.com/donnemartin/system-design-primer#message-queues) * [消息队列](https://github.com/donnemartin/system-design-primer#message-queues)
* [Task queues](https://github.com/donnemartin/system-design-primer#task-queues) * [任务队列](https://github.com/donnemartin/system-design-primer#task-queues)
* [Back pressure](https://github.com/donnemartin/system-design-primer#back-pressure) * [回退压力](https://github.com/donnemartin/system-design-primer#back-pressure)
* [Microservices](https://github.com/donnemartin/system-design-primer#microservices) * [微服务](https://github.com/donnemartin/system-design-primer#microservices)
### Communications ### 沟通
* Discuss tradeoffs: * 关于折中方案的讨论:
* External communication with clients - [HTTP APIs following REST](https://github.com/donnemartin/system-design-primer#representational-state-transfer-rest) * 客户端的外部通讯 - [遵循 REST 的 HTTP APIs](https://github.com/donnemartin/system-design-primer#representational-state-transfer-rest)
* Internal communications - [RPC](https://github.com/donnemartin/system-design-primer#remote-procedure-call-rpc) * 内部通讯 - [RPC](https://github.com/donnemartin/system-design-primer#remote-procedure-call-rpc)
* [Service discovery](https://github.com/donnemartin/system-design-primer#service-discovery) * [服务探索](https://github.com/donnemartin/system-design-primer#service-discovery)
### Security ### 安全性
Refer to the [security section](https://github.com/donnemartin/system-design-primer#security). 参考 [安全章节](https://github.com/donnemartin/system-design-primer#security)
### Latency numbers ### 延迟数字指标
See [Latency numbers every programmer should know](https://github.com/donnemartin/system-design-primer#latency-numbers-every-programmer-should-know). 查阅 [每个程序员必懂的延迟数字](https://github.com/donnemartin/system-design-primer#latency-numbers-every-programmer-should-know)
### Ongoing ### 正在进行
* Continue benchmarking and monitoring your system to address bottlenecks as they come up * 继续基准测试并监控你的系统以解决出现的瓶颈问题
* Scaling is an iterative process * 扩展是一个迭代的过程