CloudServices/Notifications/Push/Scaling

Overview

How can we scale Notifications to 1M users?

An "active user" for the purposes of these estimates is a user with two devices, who gets 1 notification permission request per day, and has 5 active notification subscriptions.

An active notification site may send 0-5 notifications per day, so we'll estimate 2.5 notifications per subscription.

We'll assume that a desktop device reconnects to the socket server twice a day, and that a mobile device reconnects 10 times a day. At any one time, expect 1/2 of the clients to be connected.

There are 6 interesting types of activities that occur on notifications:

• requestRemotePermission - the browser API to get a new push URL
• sync push urls and messages - HTTP API to maintain state between devices, called whenever a client reconnects to the socket server
• get socket server addresses - get a list of socket server addresses, called before connecting to a socket server
• connect to a socket server - create a new long-term websocket connection
• send a notification - third-party sites deliver notifications to users
• receive notifications - notifications delivered to the user over a websocket

Thus, 1M active users corresponds to:

• 1000000 / (24 * 60 * 60) = 12 requestRemotePermission requests/s
• 12 * 1000000 / (86400) = 138 sync requests/s
• 12 * 1000000 / (86400) = 138 socket address requests/s
• 12 * 1000000 / (86400) = 138 new socket connections/s
• 5 * 2.5 * 1000000 / (86400) = 145 new messages received/s
• 1,000,000 open socket connections

Caveat Emptor

These numbers are speculative.

Basic Architecture

• persistence: mysql
• message storage: queuey
• HTTP API: pyramid
• socket server: python tornado or gevent server

Write Activity

• 12 requestRemotePermission + 145 new messages = 157 writes/s to queuey
• 12 requestRemotePermission writes/s to mysql
• .5 writes/s to mysql to update socket server connection numbers
• 145 new messages/s delivered to socket servers

Read Activity

• 138 push URL sync requests + 138 socket address requests = 276 reads/s to mysql
• 138 message sync requests/s to queuey