In this paper, a system for the real-time evaluation of video streaming Quality of Experience (QoE) based on the ITU-T P.1203 standard is presented. The proposed approach aims at estimating the Mean Opinion Score (MOS) in live streaming scenarios. It analyzes incoming media segments in real-time, leveraging a sliding window mechanism to assess the quality on a configurable number of segments. The system manages playback interruptions, such as bufering events, by integrating into the assessment pipeline a module to monitor player stalls and quality level switches. Experimental results demonstrate the system's efectiveness in diferent streaming conditions and highlight its ability to capture perceptual quality fluctuations, making it a valuable tool for content providers aiming to optimize user experience in adaptive streaming environments.
In recent years, the rapid growth of multimedia-centric services and applications has significantly grown, afecting the network trafic and becoming a major challenge for video streaming service providers. Assessing the Quality of Experience (QoE) of video content streaming provides multimedia service providers valuable insights on how their networks are performing in delivering the content. As users become more accustomed to high-quality streaming services, their expectations grow accordingly. To deliver services that meet these expectations, providers must develop a shared understanding of the issues afecting users and how these factors influence their perception of service quality.
Unlike Quality of Service (QoS) metrics, which focus on network-level parameters and disregard human perception, QoE captures the subjective user experience by directly assessing the perceived visual and auditory quality of a video sequence during playback. Specifically, real time monitoring of video QoE allows operators to proactively adjust bandwidth allocation and optimize trafic routing to enhance the quality of real-time television broadcasting services over IP networks.
The implementation of the ITU-T P.1203 standard [
Content production and delivery by OTT
OTT
Content
CDN Distribution
CDN Provider CDN Provider CDN Provider
Transport Network
ISP
Access network
DSL DSL FFTH
User 2 User 1 User n
User network
The digital transformation of media provisioning and fruition has ushered in an era where multimedia content delivery dominates global network trafic, with video streaming accounting for more than half of downstream internet trafic worldwide. This paradigm shift has created a complex ecosystem where traditional telecommunications infrastructure must seamlessly support diverse, bandwidthintensive applications while meeting increasingly sophisticated user expectations for uninterrupted, high-definition content delivery. The evolution from broadcast television to live streaming by the socalled Over The Top (OTT) services has fundamentally altered the relationship between content providers and consumers. Unlike traditional broadcasting models where technical limitations were accepted as inherent constraints, modern streaming services operate under implicit service-level agreements where users expect consistent, premium-quality experiences commensurate with their subscription fees. This expectation creates a critical business challenge: technical failures during content delivery are no longer mere inconveniences but represent potential contractual breaches that can trigger financial liabilities and permanent subscriber loss.
In this context, two interrelated concepts, Quality of Service (QoS) and Quality of Experience (QoE), play a pivotal role in defining the success of content delivery. Quality of service refers to the objective and measurable performance of the underlying network and delivery infrastructure, which includes parameters such as latency, jitter, throughput, and packet loss. It represents the technical foundation upon which reliable and eficient multimedia transmission is built. QoE, by contrast, captures the end-user’s subjective perception of service quality, integrating both the technical performance and the user’s expectations, context, and satisfaction. While high QoS is a necessary condition for ensuring smooth streaming, it is not suficient on its own; true competitive advantage lies in optimizing QoE, where the ultimate goal is to deliver an experience that users perceive as seamless, engaging, and worth the subscription.
Figure 1 depicts the context where monitoring the streaming video QoE is relevant. The multimedia content is delivered through a chain of interconnected elements, by the OTT provider, across CDN (Content Delivery Network) distribution, through the transport and access networks, and finally to the user device. Each of such stages may introduce noise and potential degradation of the multimedia content. Although operators have traditionally relied on objective QoS indicators such as latency, jitter, or packet loss to assess network health, these metrics alone often fail to capture the true quality perceived by the end user. This is where QoE monitoring becomes essential.
A widely used metric to quantify the QoE is the Mean Opinion Score (MOS), originally proposed
for subjective assessment of audio quality [
By monitoring MOS alongside QoS metrics, operators can bridge the gap between technical performance and subjective perception, identifying cases where bufering, CDN ineficiencies, or applicationlevel issues degrade the experience despite acceptable network conditions. In this way, MOS monitoring provides a more complete understanding of service quality, ensuring that operational priorities align not only with infrastructure performance but also with user satisfaction and business outcomes.
The rapid growth of video trafic over IP networks, as predicted by Cisco [
The starting point of this work is a recent standard for QoE estimation in HTTP-based streaming, namely
the ITU-T P.1203 model [
The architecture of the ITU-T P.1203 standard is shown in Figure 2. The P.1203 model is composed of three modules devoted to: i) video quality estimation (P.1203.1, ), ii) audio quality assessment (P.1203.2, ), and iii) overall audiovisual quality computation (P.1203.3, ), respectively. The first two modules, and , require bitstreams data and specific metadata in input to produce per-second MOS lists reflecting the perceived quality over time. The last module, , integrates MOS lists with additional playback-related information, such as video player behavior and playout bufer status, to generate a single final MOS score representing the overall quality of the streaming session.
To deal with data availability issues, the modules can operate in four diferent modes. These modes, represented in Figure 3, allow for progressively richer input, ranging from metadata-only configurations to full access to the complete bitstream, thereby supporting a flexible quality estimation framework based on available information.
The P.1203 model has been shown to provide accurate predictions of user QoE even in evaluations carried out on data collected from real-world scenarios, however, it lacks of a real-time implementation.
The ITU-T P.1203 standard has been validated across a wide range of conditions relevant to adaptive streaming. Video compression degradations were tested using H.264/AVC (High Profile) with bitrates ranging from 75 kbit/s to 12.5 Mbit/s, while audio compression degradations were evaluated using AAC-LC (32 − 196 kbit/s). The audio quality module () is also assumed valid for other codecs, such as HE-AACv2, AC3, and MPEG-LII, based on the previous testing in P.1201 for bitrates from 24 − 196 kbit/s.
The validation included video content with varying spatiotemporal complexity, display resolutions up to Full HD (1920×1080), and playback on diferent devices (PC/TV monitors and smartphones, for example, Samsung Galaxy S5). Quality variations due to media adaptation—such as switching between diferent bitrate or resolution layers—were considered, along with frame rates between 8 and 30 fps. Initial loading delays and stalling events were also part of the evaluation. The model has been validated only for inputs of up to 5 min length — that is the duration of the test sequences that have been shown to the test subjects during subjective evaluation.
To enable real-time QoE evaluation, the ITU-T P.1203 standard has been integrated into a local system to emulate streaming by the HLS protocol. In this setup, video and audio segments are progressively delivered to the player and, as they are displayed, simultaneously forwarded to the evaluation system. The latter, implemented as a C application, shown in Figure 4, listens to a TCP port, uses a low-latency protocol based on WebSocket for communication, using raw binary data, and is designed to be multithreaded. Specifically, it employs a timer mechanism with two threads synchronized using a semaphore as a synchronization primitive, to ensure that the QoE evaluation is performed synchronously at ifxed time intervals, while keeping a dedicated thread responsible for accepting incoming segments. It processes each received segment and returns the corresponding MOS to the player. For live and timely assessment, the segments are analyzed by a sliding window composed of segments, and a result is triggered any time segments, i.e. the sliding window step, are received. Given that each segment typically has a duration ranging from 2 to 4 seconds, the resulting MOS is shown to the user approximately every · seconds. Therefore, the total processing time required by the evaluation system must be less than or equal to the interval between the evaluations, that is, · . This condition guarantees that the evaluation process remains synchronized with the media playback timeline and avoids any latency accumulation.
For practical usage of the application, it is possible to provide a JSON-like configuration file to the evaluation system to specify parameters like , and the mode of operation of the standard, allowing flexible assessment based on diferent streaming requirements or experimental setups. Such configuration capabilities also enable the deployment of distributed measurements across multiple instances of the system, making it possible to evaluate the stream under various configurations and network conditions simultaneously.
All experiments have been carried out on a single machine equipped with a multicore 1.2 GHz CPU and 10 MB of L3 cache, hosting both the player and the evaluation system. A full-HD (1920x1080) screen has been used in the tests and the ITU-T P.1203 resolution parameter has been set accordingly. The open source HLS.js media player version 1.6.0 has been used to play the stream, using the player API to enable logging. A fine-tuning of the playback behavior was also performed, as configuration parameters can be provided to hls.js upon the instantiation of the Hls object, to adjust the bufering settings. Although the HLS playlist has been deployed on a separate server, no delays due to routing protocols or network congestion have been introduced, since playback and evaluation are performed on the local machine.
To test the real-time performance of ITU-T P.1203, which is originally designed for sequences of around 5 minutes, longer live streaming segments (e.g. 10 and 15 minutes) are exploited. Stalling events have been also randomly simulated by limiting the network throughput between the player and the server hosting the HLS, by exploiting Google Chrome network throttler feature. A sample stream with a segment duration of = 2 has been used throughout the experiments.
The first set of tests focuses on evaluating live streaming performance across the four defined modes of operation, with particular attention to the handling of playback stalling events. These stalls occurred as a direct consequence of the quality degradation introduced by simulating a 3G network profile, where the available bitrate was limited to approximately 1 Mbps to emulate low user download bandwidth. The playback stalls occurred at some points during the streaming session, directly impacting the user experience. Considering that a stream with = 2 s was used, the first stall appeared at second 70 (segment 35) and lasted about 8 seconds, equivalent to four segments. A second stall happened at second 130 with the same duration of 8 seconds. Later, at second 216, the stream paused again for roughly 6 seconds, corresponding to three segments. Finally, a shorter interruption took place at second 372, lasting 2 seconds, or one segment duration.
The goal is to assess how each mode manages real-time MOS computation under diferent playback conditions. As shown in Figure 5, modes 2 and 3 exhibit higher accuracy in assessing the perceived QoE, especially in presence of playback interruptions and quality switches.
Additional tests have been carried out to evaluate the stream in Mode 3 by varying the window size from 5 to 15, while keeping fixed the step size = 2; results are shown in Figure 6. From our experimental attempts, we can deduce that the sliding window size does not significantly afect the ifnal MOS trend, as long as remains within a reasonable range. However, increasing the window size could produce a slight delay in the responsiveness of the MOS updates since more segments need to be processed before a new score is computed, even though this delay is not perceptible from the user’s perspective.
This work presented the integration of the ITU-T P.1203 standard into a system for real-time QoE evaluation of HTTP Live Streaming content. By simulating realistic conditions, the system enabled detailed analysis of perceived quality. Results showed that Modes 2 and 3 produced more accurate and stable MOS trends, especially during stalling events.
Future work will aim to support evaluation in Modes 0 and 1, even in encrypted or DRM-protected scenarios where bitstream-level access is restricted, as it would be suficient to extract only the segment metadata and construct the input for the model accordingly.
Additionally, integrating the ITU-T P.1204 standard [
Most importantly, the integration should aim to combine both QoE and QoS metrics to ensure a comprehensive assessment of streaming performance from both network and user perspectives, making it more suitable for modern adaptive streaming environments.
During the preparation of this work, the authors used Grammarly to check grammar and spelling. After using this service, the authors reviewed and edited the content as needed and took full responsibility for the published content. model standard for 4K/UHD: ITU-T p.1204.3 — model details, evaluation, analysis and open source implementation, in: 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX), IEEE, 2020.