Tim Yang Content Manager System LAMP MySQL master slave Memcached PHP CDN Real time Push Pull Push Inbox Outbox inbox Inbox Push Figure User A UpdateAction Inbox Append to 1 s home timeline Inbox Append to 2 s home timeline Inbox Append to 3 s home timeline Followers of User A 1 2 3 Push Pull outbox Inbox Pull User I Get home timeline Outbox statuses sent by A Outbox Statuses sent by B Outbox Statuses sent by C User I s Following List A B C Pull 5 000 578 2 000 IO 5 000 10 000 100 000 Latency DB read timeout timeout 503 error Message Queue MQ products Kestrel by twitter RabbitMQ an Erlang Queue Server Memcacheq Memcacheq Berkeley db Memcached protocol client library stats queue 2 get set 1 Memcacheq 2 Get 3 Set MQ Offl ine work MQ The value of the tweet decreases exponentially with time John Kalucki Twitter Cache Ram is the new the disk Local Cache Memcached Database buffer cache LAMP cache Cache TB QuickLZ LZO gzip SIGSEGV 1 Consistent hash 2 Read through cache Consistent hash Read through cache Read through and Write through Products or projects MySQL memcached UDF Cache money for Ruby on Rails Or wrap a proxy for the db driver in any language Evictions Evections cache Latency evictions cache key Multiget When memcached servers are CPU bound adding more memcached servers doesn t help serve more requests Jeff Rothschild Vice President of Technology at Facebook Cache multiget hole Memcached Max RPS A Memcached Max RPS C Memcached Max RPS B Multiget keys Multiget Multiget Multiget Application Max RPS of application A and B and C Multiget Memcached replication IDC Master slave Master master 2PC 3PC Paxos idc design MySQL master slave Dynamo Cassandra PNUTS API REST API library curl javascript client Sina App Engine Sina App Engine API host app Web 2 0 Q A TimYang Twitter xmpp Email iso1600