=Paper=
{{Paper
|id=Vol-2862/paper2
|storemode=property
|title=SMES: Adapting Dexterity-based Games for Deliberative Play
|pdfUrl=https://ceur-ws.org/Vol-2862/paper2.pdf
|volume=Vol-2862
|authors=Batu Aytemiz,Hunter Lynch,Carl Erez,Adam M. Smith
|dblpUrl=https://dblp.org/rec/conf/aiide/AytemizLES20
}}
==SMES: Adapting Dexterity-based Games for Deliberative Play==
https://ceur-ws.org/Vol-2862/paper2.pdf
SMES: Adapting Dexterity-based Games for Deliberative Play
Batu Aytemiz, Hunter Lynch, Carl Erez, Adam M. Smith
University of California, Santa Cruz
1156 High Street
Santa Cruz, California 95064
{baytemiz, hflynch, cerez, amsmith}@ucsc.edu
Abstract terfaces to play the game such as switch controls (Sethfors
In this paper we describe the Sharp Multi-input Editing Soft- 2018) or Microsoft Universal Controller (Englard 2018), or
ware (SMES), a prototype system which allows players to by changes in software via trainers.
convert dexterity challenges to planning challenges in exist- Inspired by the spirit of modifying how people engage
ing games. SMES accomplishes this by recording the player with games, in this paper we discuss our work-in-progress
input, displaying it in a visual form, allowing the player to Sharp Multi-input Editing Software (SMES). The SMES al-
edit the recorded playtrace, and replay the edited sequence lows converting high-paced dexterity based challenges to
back again. Our system combines ideas from tool-assisted deliberative planning challenges in a narrow set of games
speedruns (TAS) with interaction design patterns from Ca- that exist in the wild. SMES was built with difficult plat-
sual Creators and Accessible Player Experiences. Because formers with very short levels in mind such as Celeste (Matt
our implementation is based on intercepting input events at
the operating system level, no engineering effort is needed
Makes Games 2018) and Super Meat Boy (Team Meat
to apply the tool to new games. The result is a new mode 2010).
of play for existing games that further allows players to real- SMES allows the players to save, display, modify and re-
ize new behaviors in the game. SMES could support making play their keypress sequences. SMES works through the fol-
games more inclusive by letting players adapt the challenge lowing four steps:
type they face. Furthermore we claim that AI systems that
operate over playtraces to assist the player is a rich avenue to 1. The player attempts the challenge and SMES records their
explore. keypresses.
Challenge is a crucial component of videogames (Adams 2. The SMES visualizes the keypresses over time.
2014). From solving puzzles to defeating enemies, from 3. The player makes experimental changes to the keypresses.
scaling cliffs to uncovering mysteries, games often ask their
players to overcome a variety of obstacles. The presence 4. The SMES replays the edited keypresses (and the player
of at least some challenges has shown to be important for returns to step 2).
player engagement and fun (Juul 2009). Yet, not everyone This loop allows the player to change how they engage
wants to engage with the same types of challenges (Hamari with dexterity based games. The challenge stops being about
and Tuunanen 2014). One group of players might find dex- the precise timing of button presses, but rather becomes
terity based challenges more fun whereas others might pre- more like a puzzle game where the player incrementally re-
fer a slower-paced logical puzzle. Furthermore, an individ- fines the play trace until they clear the level.
ual player might want to engage with different types of chal- When we extract and replay the playtrace of the player,
lenges at different times. we treat the sequence of actions as a stateless, fixed pol-
Beyond personal preference, however, some challenges icy. Similar stateless action sequences have been used suc-
can be much less accessible to certain groups of players. cessfully for difficult exploration problems (Ecoffet et al.
If one has a motor disability then it is much more diffi- 2019) (Zhan, Aytemiz, and Smith 2018) and is shown to be
cult for them to engage with a high-intensity dexterity based a promising starting point.
game (Pitaru 2008). It is one thing for a player to choose to We believe that AI techniques can be effective in assisting
not play a certain type, and another for the player to not be players while operating on playtraces. Representing dexter-
able to play the game. ity based games as playtraces, and then recording how player
When faced with barriers to engagement it is common changes allows us to get another type of data as to how a
for people to modify games through using the different in- player might improve their playtraces.
Copyright c 2020 for this paper by its authors. Use permitted un- In the rest of this paper we will discuss the technical de-
der Creative Commons License Attribution 4.0 International (CC tails and the design decisions of SMES, and how the ex-
BY 4.0). tracted play traces can be a fruitful avenue for AI research.
Figure 1: On the left, the approximate path that player took from the bottom left of the map to the top right of the map in the
game Celeste. On the right, the buttons press sequence that the player pressed in order to take this path.
Related Work Planning refers to the challenge category where the player is
deciding on what actions to take. The Execution category on
Videogames provide intrinsically valuable experiences to the other hand refers to the challenge of the player to actu-
millions of individuals (Juul 2011). Maybe more impor- ally take those actions, by pressing the correct buttons in the
tantly, games are an important part of how people connect to correct order and timing.
one another (Organization 2020). While multiplayer games In the context of difficult platformer games, which SMES
offer moment-to-moment connectivity, the connective na- is targeting, the planning challenge would be deciding what
ture of games also extends to single player games through route to take and when to use the relevant abilities such as
creating a set of shared experiences (John 2020) and com- dash or jump, and the execution challenge would be related
munities (Gee and Hayes 2012). to pressing the buttons to activate the relevant abilities at the
It is important for everyone to be able to engage in this correct time. Our goal in this project is to replace the chal-
shared culture. Yet, through a combination of design deci- lenging execution aspect from playing difficult platformers
sions of developers and differing abilities of players, a sig- into a planning one.
nificant group of people are excluded from engaging with Instead of building a new game that has the planning chal-
games (Taylor 2013). To make games more inclusive, in re- lenge present as a default, in this project we explore a tool
cent years there have been a number of games that include that targets already existing culturally relevant games and
assist modes such as Celeste (Matt Makes Games 2018) and supports people’s ability to engage with them. While there
Super Mario Odyssey (Nintendo 2017). These assist modes have been many successful games that are specifically de-
allow the player to change the magnitude of difficulty they signed to target groups with disabilities (Grammenos et al.
encounter by tweaking variables such as game speed, player 2006) we believe that due to the social nature of games it is
and enemy attributes and other relevant factors. The addition important to support players in engaging with pre-existing
of assist mode has been met with positive feedback from the cultural artifacts.
players (Klepek 2019). In building the SMES we were inspired by the ideas of
Most assist modes do not modify what type of challenge transgressive play (Aarseth 2014) and the diversity of ways
there is but rather they work by changing the magnitude of players customize how they engage with games. One ex-
the challenge. In this project we wanted to explore whether ample of this is the WeMod community1 . WeMod allows
we could change the type of challenge that the game pre- players to reconfigure over a thousand single player games
sented. We use the Taxonomy of Failure (ToF) to categorize through a unified interface, letting them customize the game
the types of challenges (Aytemiz and Smith forthcoming). to their liking, often through changing difficulty and pro-
ToF offers six different categories of failure which map to gression variables. As of July 2020 WeMod advertises that
different types of challenges that exist in a game. For this
1
project the relevant categories are Planning and Execution. https://www.wemod.com
it has over 8 million users. This shows that a significant pop- was to support access to a broader set of culturally relevant
ulation of players are accustomed to modifying their play games, and we therefore decided not to limit ourselves with
experience by reaching outside the game. any specific emulator
The Speedrunning community is another group that in- In order to reach a wider range of games SMES uses the
spired this project. Speedrunners are interested in com- Win32 API2 to integrate into the keyboard input layer. After
pleting a game as fast as humanly possible. Tool Assisted the player starts recording, the SMES intercepts and saves
Speedruns (TAS), take this concept a step further and use the virtual keyboard calls that happen while the game is
emulators to complete a game as fast as possible using any focused. When the player stops recording, the keypresses
and all available tools. While most TAS systems run the are saved into a text file. Having players seed the starting
game in an emulated environment to have frame by frame playtrace with their own action connects with the “No blank
control, in order to increase generalizability of our system slate” pattern of casual creators. While the players can start
we simply replay keypress events (which is not guaranteed from scratch to create the full playtrace using SMES, having
to reproduce gameplay over long periods of time) instead of a starting playtrace to edit makes the iterative process much
attempting to emulate the whole game. faster.
The disability community has also been customizing the To support the quickest edit-and-replay interaction loop, it
way they engage with games through a combination of soft- is the player’s responsibility to include input sequences that
ware and hardware (Beeston et al. 2018). The Accessible would reset the game to a safe state (e.g. accessing an in-
Player Experiences (Cairns et al. 2019) design patterns are a game menu to restart the level from a checkpoint). If there
great resource when it comes to making games more inclu- is no such obvious sequence to include in the playtrace that
sive. These 22 patterns target both the ability of the players will accomplish this goal, the player can always use manual
to access the game content, and their ability to adjust the controls (similar to how they recorded the original playtrace)
challenges present in the game. In the later chapter we will to reset the game before continuing the SMES loop.
be discussing how SMES fits these design patterns.
Representing Key Presses It is common for speed-runs
Technical Design of the SMES to be shared through a BK23 file. These files are a way to
Design Inspirations record which combination of keys were pressed down at
which frame. Inspired by this, our intermediate representa-
Before the development phase of the SMES we reached out tion is in a simple human readable text file. This was decided
to different communities on Reddit to inform our design. We in part to support the easy sharing of play traces. The easy
had conversations on the disabledgamers, TAS (tool assisted sharing component is aligned with the “Self promotion” ca-
speed run) and pc gaming subreddits. While few in number, sual creator design pattern that acknowledges the shared and
the discussions around how such a tool might be used was social nature of playing games and generation.
immensely valuable in prioritizing certain features over oth- Having a semi-permanent copy of the playtrace that the
ers and understanding potential user needs. player can keep editing also aligns with the “Save Early,
Our design was informed by a public discussion of a sys- Save Often” and “Slow it Down” patterns of APX. The
tem prototype on Reddit’s r/disabledgamers commu- player can take as many iterations as they would like to
nity and r/TAS communities edit the playtrace. This allows the player to engage with the
We incorporated casual creator design patterns wherever game on their own pace instead of being locked to the speed
possible (Compton and Mateas 2015). Casual creator pat- the game demands. Furthermore, even if they have to close
terns aim to make systems more usable and friendly. While SMES and come back later, the text file will be there for
these design patterns are traditionally used in the context them to pick up where they left of
of generative systems, we found them useful in our design.
Because SMES can not know what exact frame each but-
Like most casual creators, SMES is a tool that prioritizes ap-
ton was pressed, instead of matching keypresses to frames,
proachability and flexibility over control and frame-perfect
we record the timing of each press. We record the timing of
precision.
whenever a key is pressed down and later released. However,
Finally SMES also drew inspiration from the Accessi-
instead of recording at what time these actions of key down
ble Player Experiences (Cairns et al. 2019) (APX) patterns.
and key up happen, we record how much time elapses in
APX patterns have two distinct categories: Access patterns
the middle. This makes keeping track of multiple keypresses
support players accessing the game by allowing them to tune
and the duration of a specific keypress easier. To be more
the experience to meet their unique needs, and challenge pat-
specific, our representation is a series of two data points,
terns on the other hand allow players to modulate the amount
the action type and the modifier. There are three possible
of challenge they want to face.
actions, key down, key up, or delay, and each action has a
Recording Key Presses One decision we had to make modifier associated with it. If the action type is key up or
early on was what layer of abstraction to have SMES in- down, then the modifier is the name of the key that is pressed
tegrate. While extending an emulator would allow us to be or released. Whereas if the action type is delay, the modifier
much more precise both in recording and replaying our play- is how many milliseconds have elapsed since the last entry.
traces, it would also limit the games that the SMES would be
2
compatible with to those that could run on the decided em- https://docs.microsoft.com/en-us/windows/win32/api/
3
ulator. One of our guiding principles while building SMES http://tasvideos.org/Bizhawk/BK2Format.html
Figure 2: The visual path that is described by the given play-
trace.
Let us give an example of a playtrace. Imagine a character
starts running to the right, then jumps over a gap, and stops
running. This can be shown as follows:
key_down , d
delay , 500
key_down , w
delay , 1000
key_up , w
delay , 500
key_up , d
Figure 3: The visual mock-up of the editor. The prototype
GUI is actively in development.
In this example, the player holds down “d” for two sec-
onds, while pressing “w” for a single second in the middle.
Visualizing and Editing Keypresses After the playtrace in outcomes across different replays of the same playtrace.
is saved into a text file, the SMES GUI loads them and visu- This is the main bottleneck that limits the usefulness of the
alizes them. The current GUI is written using the QT frame- current state of the SMES in supporting longer playtraces
work and is still a work in progress. While designing the or styles of play that rely on precise determinism (e.g. luck
visualization we were inspired by digital audio workstation manipulation strategies used in TAS). These are not critical
(DAW) software packages. SMES allows the users to ma- limitations, however, because we imagine SMES being ap-
nipulate the playtraces by changing when and for how long plied to localized scenes (which can be re-attempted many
a key is pressed down. When the desired changes are done times) in games that were otherwise completable within the
the updated playtrace is once again saved into the text docu- limits of human dexterity.
ment.
This design is aligned with the “Improved Precision” and Future work: Manipulating Playtraces with AI
“Personal Interface” patterns of APX. It is an additional in- As mentioned earlier, there is currently no AI system acting
terface that the players can use to engage with dexterity on the saved playtraces. However, other work shows that by
games and by removing the temporal aspect of taking ac- adding a simple level state awareness reusing parts of play
tions the player can be more precise. Depending on motor traces can be very useful. For an example from the games
skills and personal preferences, players may have an easier industry, in the fighting game Killer Instinct, the enemy
time with our GUI or with a text editing for which they have shadow AI learns how to play by memorizing sequences of
built deeper fluency (in which case the GUI forms a passive player actions (Neal 2016). Then the AI agent replays parts
viewer tool). of the player’s action sequences when a correct game state
Replaying the Playtrace The player can start the replay is matched. This method proves effective enough to be de-
whenever they want by pressing the replay button. SMES ployed in a live commercial game, and it has been met with
then starts replaying the playtraces that is saved in the inter- approval from the player base5 .
mediate text file. In order to replay the playtrace we use the In the future, we are interested in taking a co-creator
underlying Win32 API, the same system we used to record approach to supporting playtraces similar to the reinforce-
the keys. We use the python package winput4 to send Virtual ment learning based mixed-initiative level creator Morai
Key Codes as input to the operating system directly. Maker (Guzdial et al. 2019) or the work on user empow-
Due to the game not running in an emulator and the Win32 erment where a reinforcement learning agent is trained to
API not being instantaneous the playtrace that is replayed augment (rather than replace) player input (Du et al. 2020).
is not completely deterministic. In our experiments there is We imagine that the edits that the user makes to the time-
an 2% average divergence between the length of the input line are not directly applied but are instead taken as train-
sequence and its realized replay. Because the errors com- ing data points to tune a stateful and observation-dependent
pound in a playtrace this can result in perceivable differences
5
https://forums.ultra-combo.com/t/shadows-ai-retrospective-
4
https://pypi.org/project/winput/ discussion/24899
action policy. In this setup, the player is still making delib- ble action game. In Computers Helping People with Special
erate choices about how to act in specific moments, but the Needs, 388–395. Springer Berlin Heidelberg.
local choices they make can have impact on several other Guzdial, M.; Liao, N.; Chen, J.; Chen, S.-Y.; Shah, S.; Shah,
moments (e.g. those that arise in the future or as the result V.; Reno, J.; Smith, G.; and Riedl, M. 2019. Friend, collab-
of different stochastic elements of the game). orator, student, manager: How design of an AI-Driven game
Furthermore, we believe that the datasets describing how level editor affects creators.
a player iteratively refines their playtrace can be interesting
Hamari, J., and Tuunanen, J. 2014. Player types: A meta-
for future applications.
synthesis.
John, M. 2020. There are no single player video games.
Conclusion
Juul, J. 2009. Fear of failing? the many meanings of
In this paper we presented the work-in progress Sharp Multi- difficulty in video games. The video game theory reader
input Editing Tool (SMES) which allows the players of dif- 2(01):2009.
ficult platformer games to change the challenge type from
a dexterity based execution challenge to a cerebral plan- Juul, J. 2011. Half-Real: Video Games between Real Rules
ning challenge. SMES does this by recording the actions of and Fictional Worlds. MIT Press.
a player, and then presents it to them in a visual interface Klepek, P. 2019. The small but important
where they can make changes and replay the updated play- change ’celeste’ made to its celebrated assist mode.
traces. We also discussed how playtraces can be operated on https://www.vice.com/en us/article/43kadm/celeste-assist-
via AI systems with the goal of assisting and empowering mode-change-and-accessibility. Accessed: 2020-5-8.
players. We hope that with SMES we can contribute to the Matt Makes Games. 2018. Celeste. Windows PC version.
ongoing conversation of how we can use AI techniques to Neal, D. 2016. Designing AI for killer instinct. https://www.
assist our player. youtube.com/watch?v=9yydYjQ1GLg.
Nintendo. 2017. Super mario odyssey. Nintendo Switch.
References
Organization, W. H. 2020. Games industry unites to pro-
Aarseth, E. 2014. I fought the law: Transgressive play and mote world health organization messages against COVID-
the implied player. In Segal, N., and Koleva, D., eds., From 19; launch #playaparttogether campaign.
Literature to Cultural Literacy. London: Palgrave Macmil-
Pitaru, A. 2008. E is for everyone: The Case for inclu-
lan UK. 180–188.
sive game design. MacArthur Foundation Digital Media and
Adams, E. 2014. Fundamentals of Game Design. Pearson Learning Initiative.
Education.
Sethfors, H. 2018. I used a switch control for a day - 24 ac-
Aytemiz, B., and Smith, A. M. forthcoming. Proceedings cessibility. https://www.24a11y.com/2018/i-used-a-switch-
of the 15th International Conference on the Foundations of control-for-a-day/. Accessed: 2020-7-24.
Digital Games. Taylor, C. 2013. Able gamers includification.
Beeston, J.; Power, C.; Cairns, P.; and Barlet, M. 2018. Ac- Team Meat. 2010. Super meat boy. Windows PC version.
cessible player experiences (APX): The players. In Comput-
ers Helping People with Special Needs, 245–253. Springer Zhan, Z.; Aytemiz, B.; and Smith, A. M. 2018. Taking the
International Publishing. scenic route: Automatic exploration for videogames. arXiv
preprint arXiv:1812.03125.
Cairns, P.; Power, C.; Barlet, M.; and Haynes, G. 2019. Fu-
ture design of accessibility in games: A design vocabulary.
International Journal of Human.
Compton, K., and Mateas, M. 2015. Casual creators. In
ICCC, 228–235. axon.cs.byu.edu.
Du, Y.; Tiomkin, S.; Kiciman, E.; Polani, D.; Abbeel, P.; and
Dragan, A. 2020. AvE: Assistance via empowerment.
Ecoffet, A.; Huizinga, J.; Lehman, J.; Stanley, K. O.; and
Clune, J. 2019. Go-Explore: a new approach for Hard-
Exploration problems.
Englard, K. 2018. Microsoft has created the most accessi-
ble gaming controller ever made. https://www.vice.com/en
us/article/d3kvx7/microsoft-adaptive-controller. Accessed:
2020-7-24.
Gee, J. P., and Hayes, E. 2012. Nurturing affinity spaces and
game-based learning.
Grammenos, D.; Savidis, A.; Georgalis, Y.; and Stephanidis,
C. 2006. Access invaders: Developing a universally accessi-