Notifications appear on the lock screen of a mobile phone, hovering at the top of an open app, as pop-ups (“Toast” notifications) on a desktop device or gaming console, posing a significant distraction to users. In our data, the median user receives five notifications per hour on mobile and desktop devices, which is a distraction every 12 minutes. At the 75th percentile, users receive 17 notifications per hour - one every 3.5 minutes. While users receive many notifications, they engage with very few. Toast notifications in our sample have less than a 5% clickthrough rate, and the most prolific sources of notifications often have the lowest engagement. In this paper we present a large-scale analysis of user engagement with notifications on mobile devices and on the desktop. Our study includes a large sample of billions of Toast and message center notifications sampled over a 2 month period, from hundreds of millions of users. Based on our observations, we show that we can improve the clickthrough rate on Toast notifications from 5% to 35% and on notifications in a message center from 29% to 85%. If we instead rank the notifications in the message center, we improve the precision at rank one from 29% to 49%.