We present STREAMS, a batch and streaming platform which integrates and manages AI services for rich transcription, translation, voice synthesis and audio description. Its main goal is to provide a unified platform to enhance the processes and products of companies working in multiple sectors. STREAMS has been successfully deployed in real-life batch and multilingual streaming scenarios, ofering a rich array of multilingual services with innovative methods, in particular to enhance portability, responsiveness and output readability.
1https://haiko.es/ 2https://jarkatza.com/ 3https://www.mondragonlingua.com/en 4https://goiena.eus/ 5https://mixer.eus/ 6https://www.noticiasdegipuzkoa.eus/ 7https://www.ondavasca.com/ 8https://www.vicomtech.org/en
The solution can operate in both batch and streaming modes, via independent deployments.
STREAMS consists of the following main components: client requests management, pipelines definition and execution, scalability and error handling. The batch and streaming services are deployed independently, although they are both served within the same solution. • Two web-based graphical user interface (GUI) frontends, one for transcription and translation purposes, and the other for speech synthesis. 2.1. Frontends Both GUIs operate with the same backend in The STREAMS GUIs aim to facilitate the communication batch mode. between the user and the backend within a batch scenario. • A main REST API, which exposes the functional- They were designed from a usability and user experience ities of the backend to the GUIs and/or the user perspective, prioritising simplicity. for direct integration. The principal internal com- The first GUI, shown in Figure 2, provides users with munications between the diferent modules are diferent input options, such as text and audio file, and alperformed through REST APIs as well. lows them to select transcription and translation models • A backend, which orchestrates all the function- to perform the desired tasks. Additionally, it integrates alities of the solution, including the AI services, two main text-boxes to present the transcription and
translation results. It is worth noting that the transcrip- The batch operational mode operates on previously tion and translation results can be downloaded in difer- generated contents in video, audio and texts formats, genent formats (txt, json and srt), to be used for diferent erating transcriptions and translation in diferent output applications. formats. In addition to providing results in text plain
The second GUI for speech synthesis, shown in Fig- (txt), the STREAMS solution also generates outputs in ure 3, includes a main text-box for the input text and a JSON format, which includes confidence scores and time drop-down menu to select the desired voice. stamps at the word level for both transcription and trans
The GUIs were developed using the Angular frame- lation, and SRT subtitling format. work9 and deployed via a Nginx web server10. Figure 2 The streaming operational mode supports the genershows the graphical interface for transcription and trans- ation of multilingual subtitles from a video transmitted lation purposes. over the common RTMP and HLS streaming protocols. The streaming backend was built to manage both the 2.2. REST API input video and audio channels, generate a new subtitle channel in the original or translated languages, and conThe principal REST API connected to the Commander struct a new output stream with the 3 channels of video, serves as the main interface between the GUI and the audio and subtitles perfectly synchronised in time. backend for batch processing. Additionally, it provides Managing the input video and audio channels during an alternative way for the user to directly access all the the subtitle generation process enables the generation features of the solution via http requests, allowing third of perfectly aligned subtitles within a controlled timing party systems to be built on top of STREAMS and thus of the video broadcast, which should not exceed a conextend their functionality. The principal communication ifgurable maximum delay. Depending on the content between all modules of the solution are performed via and speech rates, a maximum delay of between 15 and REST APIs as well. 30 seconds with respect to the original video was suficient in our tests to guarantee good quality and stable 2.3. Backends results over time, without information loss. Based on the experience of the companies in the consortium that The STREAMS solution ofers two types of backends specialise in streaming, these delays were considered perdepending on the operational mode under which contents fectly acceptable in streaming applications designed for are provided as input, supporting batch and streaming web broadcasts. Finally, the output video streams can be processing modes. transmitted over the same RTMP and HLS streaming pro9https://angular.io/ tocols, being thus compatible with the most important 10https://www.nginx.com/ commercial video streaming platforms.
Both batch and streaming backends are composed of engine, the live transcription engine implements an onseveral modules which support the features of the solu- line decoding process, in which the recognition models tion. Whereas the backend for batch processing includes are previously loaded in memory and the features are all the modules, the streaming backend is focused on processed in real time, without having to wait until the generating good quality and well-timed subtitles, thus audio has finished to start decoding. integrating three main modules dedicated to live transcription, machine translation and subtitles generation. 2.3.2. Machine Translation All the main modules are described in turn in the following subsections.
comtech’s proprietary Itzuli Translator engine, a robust
2.3.1. Ofline and Live transcription and scalable text translation system, which can be
deployed under Kubernetes orchestration12 or as a
stanThe Ofline and Live transcription modules are based on dalone platform in a dedicated server. The module
inteVicomtech’s propietary Transkit software library, which grates MarianNMT [
The library is composed of 7 technological modules streaming scenarios, is handled by the MT module, which
connected via configurable pipelines. The modules corre- can return either a fully postprocessed text or arrays of
spond to an audio transcoder which integrates the FFm- translated word with an approximated time distribution.
peg11 tool, an acoustic segmenter based on the Voice The STREAMS MT module also provides support for
Activity Detector module proposed by [
Although all these components can be used for batch ture via an information redistribution algorithm. The processing, 3 modules are mainly used for streaming, translated text is thus distributed into each target subwhich correspond to live transcription, punctuation and title according to the original character distribution in capitalisation, as well as normalisation. Unlike the ofline the corresponding source subtitle, with additional mini11https://fmpeg.org/ mal heuristics to enforce key subtitling constraints in the target language, e.g., enforcing subtitle breaks on final punctuation markers or preventing subtitle lines with single words.
The main purpose of the subtitle generator component is to provide quality subtitles that adhere to standards of readability and usability, featuring appropriate segmentation and persistence, among other criteria. This module can operate in batch or streaming scenarios, returning formatted subtitles from either arrays of words with their timestamps or plain text.
The module includes a portable and eficient segmentation component, to provide quality subtitles that balance grammatical segmentation and adequate persistence. It includes a character-counting method, which ensures that subtitles respect the maximum number of characters per line. The second method for subtitle segmentation is a simple rule-based approach, which uses character counting as well as enforcing segmentation on punctuation marks. These methods are included as fail-safe in scenarios where minimal latency is critical, but are typically discarded in favour of more advanced methods within the platform in most scenarios.
Two additional classes of methods are provided for
quality subtitle generation within STREAMS. The first
one relies on supervised models, based on Conditional
Random Fields (CRF) as proposed by [
in charge of a specific part of the process: pre-processing the input text to normalise numbers and special characters, transforming the input text into a waveform via Text-To-Speech (TTS) technology , and finally, formatting the output into the requested audio format.
The core TTS technology is based on the Tacotron-2
architecture [
Audio-description (AD) is a growing media access service which complements subtitling by providing audio information for key visual elements in media content, a particularly useful feature for blind and/or visually impaired people. This information is commonly generated manually by dubbers in recording studios, using narrative and neutral voices to describe the scenes and their visual elements.
The STREAMS solution incorporates an AD module which enables the automatic generation of this information using the TTS module, given that the synthetic voices were built on content with a narrative and journalistic style. Taking as input a script with the time-marked descriptions of the scenes, the AD module generates a master audio of the same length of the script and/or content, including each audio segment at the corresponding timestamp. The DocParser sub-module developed for the platform (see Figure 1) aids in processing scripts in various formats and extracting information as needed. 2.3.6. Aligner
generating subtitles or time-marked word transcriptions
2.3.4. Speech synthesis from an audio input and its corresponding script.
This module was developed and integrated in response
The speech synthesis module is based on Vicomtech’s to the need expressed by several entities, such as dubbing
proprietary VicomTTS software library for neural speech companies, to generate subtitles from pre-existing script
synthesis, which supports concurrent processing, deploy- and audio. Using the acoustic models and the
forcedment under local or scalable modes with automatic re- recognition mode of the Ofline Transcription module,
quest trafic balancing, as well as dynamic management the Aligner module generates well-timed subtitles by
of decentralised synthesis instances, via REST API access. performing an alignment between the text and the audio,
thus recovering the timestamps at the word level. The
13We use multilingual BERT by default [
We described STREAMS, a batch and streaming platform which integrates and manages AI services for rich transcription, translation, voice synthesis and audio description. STREAMS is meant to enhance the processes and products of companies working in multiple sectors and has been successfully deployed in real-life batch and multilingual streaming scenarios during the project, ofering a rich array of multilingual services with innovative methods, in particular to enhance portability, responsiveness and output readability.
Development Agency, SPRI, under grant agreement ZL2023/00211.
The authors would also like to thank the companies in the consortium that have contributed with their knowledge and experience to the project and the STREAMS solution.