In this paper we describe the Sharp Multi-input Editing Software (SMES), a prototype system which allows players to convert dexterity challenges to planning challenges in existing games. SMES accomplishes this by recording the player input, displaying it in a visual form, allowing the player to edit the recorded playtrace, and replay the edited sequence back again. Our system combines ideas from tool-assisted speedruns (TAS) with interaction design patterns from Casual Creators and Accessible Player Experiences. Because our implementation is based on intercepting input events at the operating system level, no engineering effort is needed to apply the tool to new games. The result is a new mode of play for existing games that further allows players to realize new behaviors in the game. SMES could support making games more inclusive by letting players adapt the challenge type they face. Furthermore we claim that AI systems that operate over playtraces to assist the player is a rich avenue to explore.
Challenge is a crucial component of videogames
Beyond personal preference, however, some challenges
can be much less accessible to certain groups of players.
If one has a motor disability then it is much more
difficult for them to engage with a high-intensity dexterity based
game
When faced with barriers to engagement it is common
for people to modify games through using the different
inCopyright c 2020 for this paper by its authors. Use permitted
under Creative Commons License Attribution 4.0 International (CC
BY 4.0).
terfaces to play the game such as switch controls
Inspired by the spirit of modifying how people engage
with games, in this paper we discuss our work-in-progress
Sharp Multi-input Editing Software (SMES). The SMES
allows converting high-paced dexterity based challenges to
deliberative planning challenges in a narrow set of games
that exist in the wild. SMES was built with difficult
platformers with very short levels in mind such as Celeste
SMES allows the players to save, display, modify and replay their keypress sequences. SMES works through the following four steps: 1. The player attempts the challenge and SMES records their keypresses.
This loop allows the player to change how they engage with dexterity based games. The challenge stops being about the precise timing of button presses, but rather becomes more like a puzzle game where the player incrementally refines the play trace until they clear the level.
When we extract and replay the playtrace of the player,
we treat the sequence of actions as a stateless, fixed
policy. Similar stateless action sequences have been used
successfully for difficult exploration problems
We believe that AI techniques can be effective in assisting players while operating on playtraces. Representing dexterity based games as playtraces, and then recording how player changes allows us to get another type of data as to how a player might improve their playtraces.
In the rest of this paper we will discuss the technical details and the design decisions of SMES, and how the extracted play traces can be a fruitful avenue for AI research.
Videogames provide intrinsically valuable experiences to
millions of individuals
It is important for everyone to be able to engage in this
shared culture. Yet, through a combination of design
decisions of developers and differing abilities of players, a
significant group of people are excluded from engaging with
games
Most assist modes do not modify what type of challenge there is but rather they work by changing the magnitude of the challenge. In this project we wanted to explore whether we could change the type of challenge that the game presented. We use the Taxonomy of Failure (ToF) to categorize the types of challenges (Aytemiz and Smith forthcoming). ToF offers six different categories of failure which map to different types of challenges that exist in a game. For this project the relevant categories are Planning and Execution. Planning refers to the challenge category where the player is deciding on what actions to take. The Execution category on the other hand refers to the challenge of the player to actually take those actions, by pressing the correct buttons in the correct order and timing.
In the context of difficult platformer games, which SMES is targeting, the planning challenge would be deciding what route to take and when to use the relevant abilities such as dash or jump, and the execution challenge would be related to pressing the buttons to activate the relevant abilities at the correct time. Our goal in this project is to replace the challenging execution aspect from playing difficult platformers into a planning one.
Instead of building a new game that has the planning
challenge present as a default, in this project we explore a tool
that targets already existing culturally relevant games and
supports people’s ability to engage with them. While there
have been many successful games that are specifically
designed to target groups with disabilities
In building the SMES we were inspired by the ideas of
transgressive play
it has over 8 million users. This shows that a significant population of players are accustomed to modifying their play experience by reaching outside the game.
The Speedrunning community is another group that inspired this project. Speedrunners are interested in completing a game as fast as humanly possible. Tool Assisted Speedruns (TAS), take this concept a step further and use emulators to complete a game as fast as possible using any and all available tools. While most TAS systems run the game in an emulated environment to have frame by frame control, in order to increase generalizability of our system we simply replay keypress events (which is not guaranteed to reproduce gameplay over long periods of time) instead of attempting to emulate the whole game.
The disability community has also been customizing the
way they engage with games through a combination of
software and hardware
Design Inspirations Before the development phase of the SMES we reached out to different communities on Reddit to inform our design. We had conversations on the disabledgamers, TAS (tool assisted speed run) and pc gaming subreddits. While few in number, the discussions around how such a tool might be used was immensely valuable in prioritizing certain features over others and understanding potential user needs.
Our design was informed by a public discussion of a system prototype on Reddit’s r/disabledgamers community and r/TAS communities
We incorporated casual creator design patterns wherever
possible
Finally SMES also drew inspiration from the
Accessible Player Experiences
Recording Key Presses One decision we had to make early on was what layer of abstraction to have SMES integrate. While extending an emulator would allow us to be much more precise both in recording and replaying our playtraces, it would also limit the games that the SMES would be compatible with to those that could run on the decided emulator. One of our guiding principles while building SMES was to support access to a broader set of culturally relevant games, and we therefore decided not to limit ourselves with any specific emulator
In order to reach a wider range of games SMES uses the Win32 API2 to integrate into the keyboard input layer. After the player starts recording, the SMES intercepts and saves the virtual keyboard calls that happen while the game is focused. When the player stops recording, the keypresses are saved into a text file. Having players seed the starting playtrace with their own action connects with the “No blank slate” pattern of casual creators. While the players can start from scratch to create the full playtrace using SMES, having a starting playtrace to edit makes the iterative process much faster.
To support the quickest edit-and-replay interaction loop, it is the player’s responsibility to include input sequences that would reset the game to a safe state (e.g. accessing an ingame menu to restart the level from a checkpoint). If there is no such obvious sequence to include in the playtrace that will accomplish this goal, the player can always use manual controls (similar to how they recorded the original playtrace) to reset the game before continuing the SMES loop. Representing Key Presses It is common for speed-runs to be shared through a BK23 file. These files are a way to record which combination of keys were pressed down at which frame. Inspired by this, our intermediate representation is in a simple human readable text file. This was decided in part to support the easy sharing of play traces. The easy sharing component is aligned with the “Self promotion” casual creator design pattern that acknowledges the shared and social nature of playing games and generation.
Having a semi-permanent copy of the playtrace that the player can keep editing also aligns with the “Save Early, Save Often” and “Slow it Down” patterns of APX. The player can take as many iterations as they would like to edit the playtrace. This allows the player to engage with the game on their own pace instead of being locked to the speed the game demands. Furthermore, even if they have to close SMES and come back later, the text file will be there for them to pick up where they left of
Because SMES can not know what exact frame each button was pressed, instead of matching keypresses to frames, we record the timing of each press. We record the timing of whenever a key is pressed down and later released. However, instead of recording at what time these actions of key down and key up happen, we record how much time elapses in the middle. This makes keeping track of multiple keypresses and the duration of a specific keypress easier. To be more specific, our representation is a series of two data points, the action type and the modifier. There are three possible actions, key down, key up, or delay, and each action has a modifier associated with it. If the action type is key up or down, then the modifier is the name of the key that is pressed or released. Whereas if the action type is delay, the modifier is how many milliseconds have elapsed since the last entry.
Let us give an example of a playtrace. Imagine a character starts running to the right, then jumps over a gap, and stops running. This can be shown as follows:
In this example, the player holds down “d” for two seconds, while pressing “w” for a single second in the middle. Visualizing and Editing Keypresses After the playtrace is saved into a text file, the SMES GUI loads them and visualizes them. The current GUI is written using the QT framework and is still a work in progress. While designing the visualization we were inspired by digital audio workstation (DAW) software packages. SMES allows the users to manipulate the playtraces by changing when and for how long a key is pressed down. When the desired changes are done the updated playtrace is once again saved into the text document.
This design is aligned with the “Improved Precision” and “Personal Interface” patterns of APX. It is an additional interface that the players can use to engage with dexterity games and by removing the temporal aspect of taking actions the player can be more precise. Depending on motor skills and personal preferences, players may have an easier time with our GUI or with a text editing for which they have built deeper fluency (in which case the GUI forms a passive viewer tool).
Replaying the Playtrace The player can start the replay whenever they want by pressing the replay button. SMES then starts replaying the playtraces that is saved in the intermediate text file. In order to replay the playtrace we use the underlying Win32 API, the same system we used to record the keys. We use the python package winput4 to send Virtual Key Codes as input to the operating system directly.
Due to the game not running in an emulator and the Win32 API not being instantaneous the playtrace that is replayed is not completely deterministic. In our experiments there is an 2% average divergence between the length of the input sequence and its realized replay. Because the errors compound in a playtrace this can result in perceivable differences
in outcomes across different replays of the same playtrace. This is the main bottleneck that limits the usefulness of the current state of the SMES in supporting longer playtraces or styles of play that rely on precise determinism (e.g. luck manipulation strategies used in TAS). These are not critical limitations, however, because we imagine SMES being applied to localized scenes (which can be re-attempted many times) in games that were otherwise completable within the limits of human dexterity.
As mentioned earlier, there is currently no AI system acting
on the saved playtraces. However, other work shows that by
adding a simple level state awareness reusing parts of play
traces can be very useful. For an example from the games
industry, in the fighting game Killer Instinct, the enemy
shadow AI learns how to play by memorizing sequences of
player actions
In the future, we are interested in taking a co-creator
approach to supporting playtraces similar to the
reinforcement learning based mixed-initiative level creator Morai
Maker (Guzdial et al. 2019) or the work on user
empowerment where a reinforcement learning agent is trained to
augment (rather than replace) player input
We imagine that the edits that the user makes to the timeline are not directly applied but are instead taken as training data points to tune a stateful and observation-dependent 5https://forums.ultra-combo.com/t/shadows-ai-retrospectivediscussion/24899 action policy. In this setup, the player is still making deliberate choices about how to act in specific moments, but the local choices they make can have impact on several other moments (e.g. those that arise in the future or as the result of different stochastic elements of the game).
Furthermore, we believe that the datasets describing how a player iteratively refines their playtrace can be interesting for future applications.
In this paper we presented the work-in progress Sharp Multiinput Editing Tool (SMES) which allows the players of difficult platformer games to change the challenge type from a dexterity based execution challenge to a cerebral planning challenge. SMES does this by recording the actions of a player, and then presents it to them in a visual interface where they can make changes and replay the updated playtraces. We also discussed how playtraces can be operated on via AI systems with the goal of assisting and empowering players. We hope that with SMES we can contribute to the ongoing conversation of how we can use AI techniques to assist our player. ble action game. In Computers Helping People with Special Needs, 388–395. Springer Berlin Heidelberg.