Archive for December, 2010

You can’t move into a 25,000 square foot building and NOT throw a party. So we’re doing just that:

Mark your calendars and hope to see you there! Invites coming soon!

A few months ago we had a conversation with two telecom industry analysts, Danny Briere and Pat Hurley, about the potential of push notifications on mobile devices and how they compare / contrast with SMS (text messages). Danny Briere wrote up the team’s analysis of IP-based push in today’s Connected Planet. It’s a must-read for anyone weighing the merits of each.

Obviously, we are biased, but we can’t help but agree with their final recommendation: “If Push is not part of your mid-term SMS plan, you should start having some meetings about it soon.”

From an end user’s point of view, they are often not that different, given that they appear on the home screen of their mobile device in a shaded box containing up to 140 characters. On an Android or Blackberry, they are more differentiated but not so much to be truly noticeable for an end user. For the sender, however, the differences between using SMS or push for mobile messaging are significant. For one, consider price: “SMS is up to $1500/month for a shortcode and somewhere in the pennies per message wholesale rate for delivery, while Push notifications through a third-party Push provider are thousandths of cents per message. That’s a several orders of magnitude expense reduction in communications expense alone for ISV/SaaS vendors and other SMS senders.”

But what we really stress to our mobile customers is the merits of push notifications to really drive users into actionable content. By triggering an alert that engages the end user and compels him or her to open the app, push has the upper hand over SMS. A user may view an SMS message, but there’s no way of knowing how — or if — this returns any investment to the business.

Yesterday I visited with tech maven Robert Scoble at his home in Half Moon Bay, a beautiful beach town on the coast near between San Francisco and the Silicon Valley. We had a great discussion about the mobile eco-system and where things are headed in 2011. Scoble is a great guy and extremely knowledgeable about pretty much anything to do with technology, so it was a pleasure to sit down with him for a bit. I showed up, sat down and he started shooting. Loved it. And as often happens when talking about our company, we started off with a conversation about our name:

One of the topics that came up during the conversation was app updates. As Robert points out, updates have been used as a way for app developers to remind users of their app’s existence on the device. The problem with updates is that they have many purposes. One is to fix bugs — this does not instill confidence in the end user, and we don’t recommend it as a strategy for ongoing user engagement. We call this “Push Notifications 1.0.”  Mobile publishers could, and should, employ more sophisticated tools to bring users back into their apps. As we demonstrate in the video with the democrats.org app example, push notifications actually drive users into actionable content. Give users the ability — and a reason — to engage  in a meaningful way in real-time and your app will be much better poised for success.

Editor’s Note: This post was compiled and created by Scott Andreas, who can be reached on twitter: @cscotta.

Last month, we set a goal of improving the throughput of our Android messaging stack. This stack has three primary components, whose interactions are mediated by Beanstalk queues:

1) The frontend API to which app publishers push messages.
2) A worker responsible for preparing these messages for delivery to devices.
3) Our “Helium” cluster – the edge nodes to which Android devices maintain a persistent connection for message delivery.

Previously, we’d discussed the design and implementation of our “Helium” server, a Java NIO-based service designed to maintain active connections to up to 500,000 devices and deliver messages to them in real time. In this post, we move one level up in the stack to look at our message prep tier.

This tier is responsible for six primary tasks: Delivering individual messages, delivering broadcast messages for a particular app to every user, delivering messages to every device registered with a specific “tag” for an app, storing pending notifications for devices not online at the time of a push, registering and activating devices as they come online, and recording billing stats reported by our Helium cluster.

An Ambitious Goal

Our initial goal was to be able to deliver a sustained 10,000 messages per second with at most five servers. After setting this goal, we moved to analyze the performance of our (then-)current “Heworker” implemented in Python using Celery. The data access pattern for this task requires large, sequential reads for key/value data, which we read briskly from Cassandra, with supporting data fetched from a MongoDB collection. After data is received, the task prepares two protobufs, encrypts a message, wraps it in an RPC envelope, and passes it on to a Helium server.

We found that on an EC2 c1.medium High Compute instance, our present broadcast message delivery rate was roughly 312 messages per second, with delivery to tags at 280 messages per second. After realizing that the vast majority of MongoDB queries could be eliminated by grouping broadcast message delivery into buckets of 20,000 at a time and fetching the supporting data from Mongo with a single “$in” query, we were able to roughly double the throughput of this task to around 622 messages/second, at which point it became largely CPU-bound. Slightly disheartened, simple math suggested that our goal would require at least 16 servers, along with the work of parallelizing delivery of broadcast messages across them.

Enter the Octobot

A few months ago, I wrote, documented, and released a JVM-based task queue worker called “Octobot.” Octobot is a task queue worker for applications that must process message queues with extremely high throughput, minimal latency, and high availability – especially those sensitive to the overhead implicit in receiving and processing many lightweight tasks, such as message delivery systems. Octobot can listen on any number of queues backed by AMQP, Beanstalk, or Redis, with any number of workers for each task queue, with each operating at a configurable priority. Tasks can be written in nearly any JVM language, with sample code available for Java, Scala, Clojure, and JRuby. Configuration is handled by a short YML file, and it runs anywhere Java does. Finally, deployment’s a snap – two jars, an init script, the YML config, and optionally, a logging config and you’re done.

Based on the promising results of a demo, I worked to build out a quick spike of a couple tasks for us to test. Due to more pressing issues, the project was put on hold for some time, with work resuming a few weeks later on a tighter schedule toward a full implementation. We implemented the Cassandra and Mongo data models, ported each of the tasks from Python to Java, and implemented a full test suite using ScalaTest. The re-implementation and testing of this tier took one developer about three weeks.

A Pleasant Surprise

The results were a pleasant surprise. The new Octobot-based worker is capable of delivering about 16,500 tag messages per second, and roughly 13,300 broadcast messages per second per Octoworker instance (a meager c1.medium).  Our original goal of 10,000 messages/second across five or fewer machines could now be handled by just one instance at 75% capacity.

Profiling the Embarrassingly Parallel

During development, profiling Octobot and the worker under VisualVM helped us uncover many opportunities for optimization and parallelization of message delivery. To deliver a broadcast message to all devices registered with an application, we need to know which devices are registered with an application (stored in Cassandra), and to which Helium server they’re connected to (stored in Mongo). The new worker minimizes database and IO latency by delivering messages in batches of 20,000 – that is, by fetching all data required for delivery of a batch of messages, then parallelizing the CPU-bound delivery tasks across a thread pool.

The breakdown of a broadcast to 10,000 devices looks about like this:
– 61ms round-trip to Cassandra to fetch 10,000 keys from an index.
– 257ms round-trip to Mongo to fetch 10,000 records from a collection.
– 432ms to construct the device messages in parallel and place them in an internal queue for delivery.

The final task here is CPU-bound. With all data required for message generation in memory, it is highly parallelizable. As such, we’ve applied a poor man’s fork/join pattern (think map/reduce, but inside the application itself) by dividing the workload across a number of worker threads. This strategy enables us to take advantage of all processors in the system. While EC2’s c1.medium instances expose two cores, the application will parallelize equally well across four, eight, twelve, or more cores. As elsewhere but especially here, we’re thankful to be programming in an environment that allows for concurrency as well as parallelism within a single process.

Parallel Message Delivery Implementation

Wrapping Up

Beyond its raw speed, the Octobot architecture brings significant enhancements to our infrastructure in terms of monitoring and introspection. With a quick Munin plugin, we’re provided with graphs of average execution time, successes, failures, and retries per task (click to enlarge the stats at right). Failures result in an error notification e-mail detailing the incoming task, the hostname of the instance that executed it, arguments supplied, and a stacktrace if one is available. This makes for quick debugging and updates.

All told, we’re very pleased by how the project turned out.  If you’re in the process of implementing a distributed system that requires high-throughput background processing, a similar strategy might be worth considering.

Further Reading

Octobot
If you’re interested in evaluating Octobot for your needs, check it out at http://octobot.taco.cat/ The 1.0 release is just about a week away, with additional samples on the way for more JVM languages, better documentation, instrumentation, and monitoring. It’s processed well over 5 million tasks for us in the past week and is well-behaved.

Fork/Join
If the fork/join pattern is new to you and your work is parallelizable, you might enjoy Doug Lea’s “A Java Fork/Join Framework” (PDF).

Urban Airship
And last but not least, if you’re interested in implementing Push Notifications and remote messaging for your Android application, grab an account! You get a million messages free per month, delivered instantly and securely, to your user’s devices.

We cannot emphasize it enough: If your app users are turning off push notifications, you have lost a huge opportunity to engage with them. Engagement is a key element of success in mobile, and you don’t want to miss out. As we discussed in our last post on implementation 101, getting users to opt-in to push notifications in the first place can take some persuasion. Once they do, you need to stick to your end of the bargain and send only messages that contain the compelling content your users have agree to accept, and at an acceptable frequency. A misstep on your end could close the door to future engagement with that user.

When you think about push, don’t think in the same terms as you think about email or SMS. Losing an opted-in push user is not just a  “-1” to your marketing list. It undermines the utility of your app, and by extension, the power of your brand. The consequences for losing an app user are far greater, i.e., your app becomes less effective and may not function as intended. Because your app icon lives in your user’s hand, you don’t want it to serve as a reminder of a poor experience.

So to measure your push notification effectiveness, you need to understand how your users are reacting to your messages. One way to do this is to evaluate how many of your users turn off push notifications.

How to Figure Out if Your Users are Turning Off Push Notifications

What you need to do is compare active device tokens against total device tokens. When a user opts in to receive push notifications from your app upon install, our service collects a device token. An active device token is receiving your messages. An *inactive* device token means either the user deleted the app or turned off push notifications from the app (opted out).

What you can do is compare your active device token (ADT) against your disabled device tokens (DDT). We call this the Push Retention Ratio, and it’s one of the most reliable indicators of your push notification effectiveness. We have some customers with Push Retention Ratios as high as 98% and some as low as 65%. Obviously the higher the ratio, the better engagement you have with your users.

Granted, this is a sort of rudimentary process. We are working on tools to provide more meaningful insight into your app’s Push Retention Ratio but for now the ADT / DDT comparison is most useful. Now, the most likely reason for a disabled device token is that the user uninstalled your app. Why would he or she do this? Because your app is not providing compelling content. You may trace this perception to ineffective push, no push at all, or other for some other reason, but the areas you can control are around push and how well you deploy your mobile messaging.

So go ahead and check your ADT / DDT numbers to get a baseline ratio. Then continue to check it after you send push notifications. As you track this number, pay specific attention to how the rate changes directly after push notifications and how it changes during dormant times. As you fine-tune the best practices for your specific app, you’ll have a better understanding of how your users are responding, even when they aren’t telling you directly. If the ratio is going down you may have a problem. You might be sending messages too frequently, with content that is not compelling or with annoying messages.

Sending Push Notifications to an Inactive Device Could Violate the ToS

Once a user opts out of push notifications, you are not allowed to continue to send them messages. This is against Apple’s Terms of Service. Apple has a service called Feedback Service that outlines this in detail. The feedback service states that Apple APNs monitor developers for their diligence in checking the feedback service and that they refrain from sending push notifications to nonexistent applications on devices.

Blackberry and Android don’t publish such specific details in their ToSs, but we highly recommend you follow the Apple guidelines across platforms.

If you’re using Urban Airship as your push notification provider, you need not worry about violating this clause. Our mobile services provide compliance and automatically removes any inactive device token IDs so neither the user nor the Apple server will receive messages from you, post opt-out.

Please visit our resources on feedback service for a better understanding of how Urban Airship provides compliance with this policy and for further detail on how to view your Push Retention Ratio at any given point.

This post is part of an ongoing series, Push Notification Best Practices. We’ll look at different aspects of push and provide guidance on effective tactics. Up next, we’ll cover Incorporating Push Notifications into the App at the Design Stage.