This paper discusses voice-based direct manipulation to extend the conversational interactions with LLMs and, more generally, with intent-driven user interfaces. Inspired by recent work on visual direct manipulation in AIassisted tools, we explore whether similar patterns can enhance Voice User Interfaces (VUIs). Such approaches can improve usability of intent formulation, especially in contexts where voice is the primary or the only interaction modality, with important implications for accessibility and social inclusion.
Recent advancements in Generative AI and its widespread availability are impacting professional and
private lives. This new technology has also contributed to a revolutionary shift in the way humans
interact with technology: a paradigm Nielsen described as “intent-based outcome specification” [
After the ChatGPT launch, back in November 2022, conversational intent-based interactions have
quickly become the standard to interact with Large Language Models (LLMs). In this paradigm, humans
and AI engage in multi-turn conversations, with the user focusing on expressing the desired outcome
in the form of a prompt in natural language, while the model is responsible for capturing the user
intent and converting it into a meaningful result [
Alongside text-based prompting, other techniques that leverage the manipulation of visual elements to streamline prompting and achieve more direct interactions are emerging. This paper discusses these new strategies and introduces possible ways to bring voice-based direct manipulation into intent-driven interfaces. Shifting the focus to the voice modality is fundamental to ensure advancements in AI remain human-centered, preventing social exclusions and encouraging participation while embracing diversity. After illustrating the current panorama of conversational intent-based interaction, especially applied to the interaction with LLMs, the paper discusses how typical patterns for direct manipulation in visual intent-based interfaces can be translated into voice-based interaction mechanisms. The paper also discusses a preliminary prototype informed by a user study that involved Blind and Visually Impaired (BVI) participants.
Although standard prompting, especially text-based prompting, is currently the most common
interaction mechanism in Generative AI systems [
DirectGPT [
DirectGPT, as opposed to conversational interfaces, builds upon the idea that to enable direct manipulation the objects of interest must be continuously accessible inside the interface. For this reason, the long history of messages is replaced by a fixed area at the center of the screen, where the model output is continuously displayed after each modification. Furthermore, the dynamic population of a toolbar of commands makes available the most recent user prompts in the form of buttons, fostering their re-usability. Increased directness is also achieved through physical actions, such as dragging or highlighting, augmenting the expressivity and the flexibility of purely textual prompts. Additionally, error recovery is achieved through undo and redo commands, displayed as buttons located at the top of the interface. The system also highlights the elements afected by the LLM-generated modifications, allowing users to quickly assess the efect of their prompts and immediately revert them if needed.
Content creation is one of the primary applications of Generative AI. As Luera and et al. [
Our research explores how direct manipulation principles, most commonly associated with GUIs, here
can be extended to voice LLMs’ interfaces. The direct-manipulation paradigm was first introduced
to describe interactive systems that apply three main principles [
Overall, the study revealed a range of design opportunities in voice interaction. It also emphasized the need for better ways to manage and organize conversations, making it easier for users to navigate content, revisit important information, and maintain context across sessions. Additionally, the findings highlighted the importance of supporting more efective consumption of generated responses, by enabling smoother navigation through long or complex outputs and direct manipulation of specific output portions. These design insights informed the development of a prototype incorporating the new voice interaction patterns. We are now building on this experience to extend direct manipulation principles to the vocal channel. The discussion presented below is inspired by the previously-discussed examples of canvas UIs and considers some well-known challenges of VUI design from the literature.
Continuous representation of the object of interest. The system could let the user hear the most updated version of the output—the LLM-generated text—aloud. In this way, the efect of user-requested commands could be immediately perceived, without the need for the user to explicitly ask to have the full text read out loud any time a new transformation is applied. However, while this mechanism could be efective when the model’s response is relatively short, for more lengthy outputs, it could decrease usability. In such circumstances, providing a summary of the applied changes might be more appropriate, while still preserving the chance for the user to listen to the full modified text upon request.
Physical actions or labeled button presses instead of complex syntax. While the definition
of visual abstractions acting like shortcuts is relatively easy and efective in GUIs, in voice interfaces
it requires careful consideration. Such an abstraction could be implemented as pre-defined vocal
commands that act as proxies for more complex and longer prompts. A challenge is, however, to inform
the users about the availability of such shortcuts; once aware, they should also be able to recall them.
To address these issues, the system could actively remind of the availability of such commands, asking
at strategic points in the conversation whether the user wants to hear the list of such shortcuts. A less
intrusive alternative would be to provide it only upon request. In both cases, to help the user remember
all the proposed options, no more than three elements should be provided at a time [
Rapid, incremental, reversible operations whose impact on the object of interest is immediately visible. While reversibility can be easily achieved by introducing vocal commands that undo or redo recent modifications, the possibility of performing incremental operations quickly is the most critical part of the interaction. This is because vocal commands can be relatively rapid to issue, but the eficiency with which the user is able to evaluate their efect on the system strongly depends on the way such actions are acknowledged. When the target text can be visually inspected, it is easier for the user to check the diference between the current version of the output and the previous one, especially if the interface provides a dedicated mechanism to inspect and browse the editing history, like the one available in Canvas. In VUIs, instead, the eficiency with which users can evaluate the efect of issued commands depends on the trade-ofs discussed for the first principle.
To assess the potential of the patterns presented in the previous section, we integrated them into a new
paradigm for vocal interaction with LLMs [
At first, the user chooses a message from a chat that will act as a target for editing actions. To support
the refinement process, an editing mode is introduced. This is conceptually analogous to the canvas
area seen in some visual LLM interfaces and can be accessed using a dedicated vocal command, such as
“Enter edit mode”. Being such a space virtually separated from the main chat with the model, it can be
leveraged by the user to iteratively carry out a series of LLM-assisted transformations to the corpus of
the text, while keeping the main chat clean and concise. In fact, in VUIs, the longer the conversation
history, the more frustrating it becomes to fully traverse it [
In edit mode, we can classify user requests to modify text into two categories: • Global editing actions (Figure 4a): they afect the target message in its entirety. Examples include requests to summarize the message or change its tone. Any time a request of this kind is sent to the model, the full modified message is read aloud. In the case of particularly long messages, instead, only a summary of modifications applied is provided. • Localized editing actions (Figure 4b): they afect only a restricted portion of the text, such as a single sentence or word. Examples include rewriting a given sentence using a diferent style or replacing a word with a synonym. To perform this class of actions the user must be able to both traverse the message using a finer-grained and adjustable granularity (e.g. at sentence level), and select the specific portion to be transformed. This behaviour can be achieved through the previously-discussed navigation pattern, implemented as a navigation mode that can be activated using a vocal command like “Enter navigation mode”. Users can optionally specify the desired navigation granularity, otherwise the most appropriate one is selected based on the length of the message. In navigation mode, users can also access helper commands that facilitate message traversal. Landing on a given node automatically triggers a series of actions. At first, the node’s identifier is read aloud followed by its content. Then, the user receives suggestions from the system for possible actions to take. At this stage, any request involving an editing action automatically considers the node currently in focus as the target of the transformation to be applied. Moreover, while traversing the message, any time a request to modify a node is issued, the updated version of its content is read aloud.
Additional complementary strategies to improve the overall interaction include: • Auditory cues: sounds can be used to reinforce and complement verbal feedback about specific actions. E.g., a specific non-verbal sound can be reproduced after the message “You are now in edit mode”, when entering this mode. Such an additional layer of feedback could help increase the users’ awareness of the system’s state and guide them more intuitively through the interaction. • Keyboard shortcuts: they can be introduced as an alternative to commands that already have a voice-based counterpart. For instance, inside navigation mode, users could use the right arrow key to move to the next node instead of pronouncing the word “Next”. The availability of such keyboard shortcuts could streamline the interaction. However, to preserve the hands-free modality, these mechanisms should only represent a form of redundancy. • Help command: to facilitate the exploration of all actions and shortcuts available from a given stage of the interaction, a help command can additionally be introduced.
This paper has illustrated preliminary insights on defining vocal interaction patterns for direct manipulation in chat-based interfaces. Sketching the interaction, as illustrated in the previous section, represents the first step toward validating the feasibility of the proposed patterns. The next steps involve testing the interaction with users. A fast-prototyping approach was envisioned for this purpose, leveraging OpenAI’s custom GPTs as tools for demonstrating the identified interaction patterns. This approach presents several advantages, including access to a pre-built interface with a ready-to-use voice mode available on top of OpenAI’s models. The configuration required to share and run the prototype is thus limited to the definition of a well-formatted system prompt. Despite its limitations, the low resource demand of this solution makes it promising for quickly testing and improving the interaction. However, more sophisticated prototyping techniques will also be explored before approaching the validation step, with the aim of defining a more robust artifact that can lead to a higher quality of the gathered data.
The authors used Writefull for grammar and spelling checking. However, the authors extensively reviewed and edited the text; therefore, they take full responsibility for the publication’s content.