Media has become dominant in all aspects of human lives, from critical applications such as medical, military, and security (e.g. surveillance cameras) to entertainment applications such as social media and media sharing websites. Billions of massive media objects (e.g., videos, photos, documents, etc.) are generated every second with high diversity among them (in terms of sizes and formats). These objects have to be stored and retrieved reliably, with low latency and in a scalable while e cient fashion. Additionally, various types of processing are done on media objects, from simple compressions and format conversion, to more complex machine learning algorithms detecting certain patterns and objects. Existing large-scale storage and processing systems face several challenges when handling media objects. My research focuses on building an uni ed storage and processing environment tailored speci cally for media objects, while maintaining high e ciency and scalability. I have built a scalable, load-balanced, e cient storage system optimized for media objects based on their unique access patterns. Currently, I am working on developing an e cient media processing system and integrating these two systems into one framework.
During the past decade, media has become extremely
popular. Based on [
Handling this massive amount of media poses a number of unique challenges. First, a diverse range of media (photos, videos, documents, etc.) with various sizes (from a few KBs to a few GBs), should be processed e ciently at the same time. Second, there is an ever-growing amount of data that needs to be stored, served, and processed in a linearly scalable fashion. Third, in many applications (e.g., machine learning applications), processing is both data and CPU intensive, causing several di culties in resource scheduling. My research focuses on building an uni ed online environment tailored speci cally for storing and processing these diverse massive media objects, while maintaining e ciency and scalability.
First, we need to store this ever-growing enormous amount
of media e ciently. Existing distributed storage systems,
including le systems [
Additionally, media objects have to be processed in a
timely manner. Various types of processing is conducted on
media objects including: pattern matching (e.g., detecting
pornography), categorization (e.g., tagging photos),
correlation detection (e.g., recognizing a burglary in a network
of surveillance cameras), compression and resizing, format
conversion, and matching media objects (e.g.,
deduplication and checking copy-write). There has been extensive
research on optimizing each of these applications, such as
OpenCV and ShapeLogic [
As the rst step in my research, I have been focusing on designing an e cient storage for media objects. Handling media poses a number of unique challenges. First, due to diversity in media types, media object sizes vary signi cantly from tens of KBs (e.g., pro le pictures) to a few GBs (e.g., videos). The system needs to store both massive media objects and a large number of small media objects e ciently. Second, there is an ever-growing number of media that need to be stored and served. This rapid growth in requests magni es the necessity for a linearly scalable system (with low overhead). Third, the variability in workload and cluster expansions can create unbalanced load, degrading the latency and throughput of the system. This creates a need for loadbalancing. Finally, data has to be stored and retrieved in a fast, durable, and highly available fashion. For example, when a user uploads a media object in social network, all his/her friends from all around the globe should be able to see the media object with very low latency, even if parts of the internal infrastructure fail. To provide these properties, data has to be reliably replicated in multiple datacenters, while maintaining low latency for each request.
Several systems have been designed for handling a large
amount of data, but none of them satisfactorily meet the
requirements and scale media processing needs. There has
been extensive research into distributed le systems [
Many key value stores [
A few systems have been designed speci cally for large
immutable objects including Facebook's Haystack [
Thus, through a collaboration with LinkedIn, we
developed a scalable load-balanced distributed storage system
built speci cally for media objects (described below) 1.
Ambry has been running in LinkedIn's production environment
for the past 2 years, serving up to 10K requests per second
across more than 400 million users. We have published our
work \Ambry: LinkedIn's Scalable Geo-Distributed
Object Store" in SIGMOD 2016 [
Social networks are one of the biggest sources of media objects, with hundreds of millions of users continually uploading and viewing photos, videos, etc. Typically, these media objects are written once, frequently accessed, never modi ed, and rarely deleted. We leveraged this immutable read-heavy access pattern of media objects towards Ambry. Ambry is a scalable distributed storage designed for e ciently handling both massive media objects (GBs) and large number of small media objects (KBs). Ambry utilizes techniques such as decentralized design, asynchronous replication, rebalancing mechanisms, zero-cost failure detection, and OS caching. Using these techniques, Ambry provides high throughput (utilizing up to 88% of the network) and low latency (less than 50 ms latency for 1 MB object), while maintaining load balancing amongst nodes. 3.
As the next step, I am currently working on building a
distributed online processing system, optimized for media
objects. Distributed stream processing systems have been
designed for processing enormous amount of data in a near
real-time fashion [
However, existing systems are not optimized for media and incur a lot of data movement. Many of these system include multiple stages of data copy and/or fetching data over the network. Although data movement may not be a dominant factor for processing small data, this is not true for massive media objects.
One of the main causes of data movement is reading and
writing data from an external storage system (remote state),
as opposed to supporting fault-tolerant locally stored data
(local state). For example, for providing exactly-once
guarantees2, Millwheel [
Currently, I am collaborating with LinkedIn on developing Samza, a scalable distributed stream processing system supporting local state. Samza provides fault-tolerant local state by using local database instances in each node, along with a compacted changelog for failure recovery. Each local database instance stores data on disk, providing TBs of space per machine. Additionally, by batching writes and caching popular data in memory, it reaches performance close to an in memory storage. By using local state, we can implement exactly-once guarantees via storing processed message ids locally (with very low latency), and replaying the changelog if failures happen.
We ran an performance benchmark to evaluate the e ect of using local state, compared to using remote state. We used two workloads
ReadWrite: similar to a word count application
reading the current count of a speci c word and updating.
1The project is open-source and the code can be found at
[
We ran the workload on a 4 node cluster of beefy machines, using multiple stores. Based on our initial results, using local state improves throughput up to 100x compared to using remote state, as shown in Figure 1. Using a local Rocks DB with caching enabled reaches almost the performance of a local in memory store and both cases saturate the network. Even with no caching, Rocks DB reaches almost half the throughput of a in-memory store. However, the remote storage is orders of magnitude (up to 100x) slower.
We also measured the latency for each test and it followed a similar pattern. When using a local (Rocks DB or inmemory) store the latency per input data was a few microseconds in all test (8 to 30 microseconds). However, using a remote store this latency was 3 orders of magnitude higher (4-10 milliseconds). This is mainly because of the added delay for going over the network and data copies, and the overhead of providing consistency in a fault-tolerant distributed store (with multiple replicas).
Moreover, we have developed a host a nity mechanism that tries to reduce the failure recovery time by leveraging the state already stored on disk. This mechanism favors brining up failed containers on machines they were placed before, reusing the state stored on disk. Using host a nity we are able to drop the recovery time 5-10x, reducing the recovery time from several minutes to only a few seconds. Also, using host a nity, the overhead of recovery is almost a constant value irrespective of the size of the state to be rebuilt. This is because only the fraction of the state not ushed to disk has to be rebuilt. Therefore, using this mechanism and local state we are able to provide fast failure recovery, close to a remote state, while not incurring the overhead of querying a remote database on each input message processed.
Although local state signi cantly improves performance (specially for applications sensitive to data movement), it is not su cient for providing an e cient media-processing environment. Due to the large media sizes, the local state may not be enough for storing all the data associated with an application.
For example, assume we have a continuously changing reference set of fraud media, and a fraud detection application that compares recently posted media against a subset of the reference set (based on a similarity metric such as RGB ratio). Using local state, we can partition the reference set based on the similarity metric set across multiple machines; access the set locally; and update the reference set whenever needed. However, if the reference set grows too large (e.g., videos), the local state capacity will not be enough unless by scaling to a bigger cluster with many underutilized machines. I plan to overcome this issue by integrating the distributed storage system and the processing system into one framework. This framework will utilize local state as a cache for storing/retrieving media by o oading data to the storage system, whenever needed.
Additionally, processing even a single massive media object (e.g., a high quality video) in a timely manner can go beyond the capabilities of a single machine. Chunking large media objects into smaller ones and processing chunks in parallel, mitigates this issue. However, chunking introduces many challenges including: rebuilding the large object in a system where data chunks can be processed out of order or in nitely delayed; processing data without losing accuracy; and handling data dependency amongst data chunks. As my future direction, I plan to provide built-in chunking and rebuilding mechanism for large media types, without a ecting accuracy.
In a nutshell, as the future direction, I plan to further optimize Samza to e ciently handle diverse media types and sizes (ranging from a few KBs to a few GBs), and integrate Samza and Ambry into a uni ed scalable media storage and processing environment. 5.