The process of application modernization consumes the efort of software development teams charged with upgrading legacy applications to modern technology, architecture, and design. While some tools exist to aid in these eforts, the modernization of an application's user experience is an arduous and primarily manual undertaking. Through a process of user research and design exploration, we investigate how generative AI models might be used to assist software development teams in modernizing the user experience of legacy applications. Our goal is to identify opportunities for further research to aid teams involved in user experience modernization eforts.
1. Introduction We also provide detail on the many unmet needs we have cation modernization projects. For topical completeness learned about in Table 1. and consistency, we guided each informant through a
Next, we begin crafting a vision for how generative series of open-ended questions. We balanced these intenAI technologies can address some of the unmet needs in tional topics with ample opportunity for informants to Section 5.1, and map those onto specific kinds of genera- introduce their own concerns, and to educate us about tive models that could be trained to support these needs the complexities of their work. in Section 5.2. Each interview took place online, and lasted approxi
We begin to identify opportunities to develop new mately 60 minutes. Informants were invited to show as kinds of generative models, based on feasibility (e.g. the well as tell, by sharing their screen to illustrate the materiaccessing specific data) and potential for impact (e.g. als of their practices (e.g., screenshots from applications) meeting critical needs). Our philosophy is to decompose and the ways that they worked with those materials (e.g., the large, generally-specified tasks within UX modern- their own notes and proposals for further work).Each ization into a suite of purpose-built generative models interview was led by the same researcher, and at least that can be used by designers and software engineers to one other researcher attended as note-taker. In addition accomplish specific key tasks. to the manual notes, we recorded all interviews (with informants’ permission), and we used the e-meeting software to generate a speech-to-text transcript. We used the 2. Background transcript as a guide to what the informants had said, and we referred back to the recordings as necessary when UX modernization involves both the redesign of user fac- the transcripts may have contained speech-recognition ing elements of a legacy application and the underlying errors. architecture and technology that supports it. The goal is The majority of those we spoke to were members of to provide users with an improved experience in terms of a Data integration application, a Facility management eficiency, productivity, responsiveness, ease of use, and application, and Asset management application product enjoyment [15]. The task is complicated and constrained teams who are currently engaged in large modernization by the user habits and expectations that result from their projects. These informants gave us a high degree of acexperience with earlier versions of the application. cess to other project stakeholders and the content of their
Recent work has applied a variety of generative AI and modernization projects, which helped us gain a more conmachine learning techniques to a number of software crete understanding of the people, processes, tools, and development tasks, such as code completion [16, 17, 18, artifacts involved in modernization. By forming a deep 19], documentation and test generation [20, 21, 22, 23], understanding of their work practices, we were able to and computer language translation [24]. Allamanis et al. identify pain points that AI technologies could help alle[25] provide a comprehensive review of the use of AI viate, as well as new opportunities for which generative and machine learning within software engineering. In AI technologies could be a useful technological approach. the realm of application modernization, a variety of tools have been developed that help in the analysis and rearchitecture of legacy systems [26, 27, 3]. Tools to address 4. Results the modernization of the user experience, however, are notably lacking. 4.1. Identifying UX Modernization as a
Since application user interfaces typically have a vi- particular challenge sual component to them, recent work in image analysis, generation, and style transfer [28, 29, 30, 31, 32] establish One area in which all three of these product groups are a baseline of capabilities that may be adapted to the UX currently making massive, multi-year investments is in modernization domain. the modernization of the UX. In some of our first inter
Finally, since UX modernization activities will typi- views we learned that the UX transformation part of the cally involve much iteration, work on Human-AI Collab- modernization process was particularly dificult and time oration with generative models [33, 34, 35, 36] and the consuming, and it did not seem to provide many opportutypes of interaction made possible by the emergence of nities for salvaging or re-use of legacy UI designs or code. generative AI techniques for natural language, such as One informant who is an architect called their efort “a GPT-2 [37] and GPT-3 [38] may also prove to be relevant. complete re-write,” indicating that nearly all of the legacy code driving the UI had to be re-written to match a completely re-designed UX. A Program Director for the Data 3. Method Integration tool volunteered that modernizing the UI is “the biggest pain point” in building their next generation application because it requires a full UX design team and
design leads who are currently working on legacy appliUI re-architecture, which would take as much time as as needed, until the modernization process is complete. designing a whole new application from scratch. This Overall, this process is highly manual and incurs tremenobservation piqued our interest because it seemed that dous coordination costs across designers, architects, softa legacy product’s UX ought to provide a starting point ware engineers, and product managers. All tasks – large for creating a modern re-design, rather than designing or small – require human attention and manual work. and building a new UX from scratch. Very little legacy code or design artifacts are reused, if
In subsequent interviews, we confirmed that these they are even present. In some cases, design artifacts large-scale UX modernization projects require high ef- for the legacy application need to be created before artifort, large teams, and multiple years to complete. Some facts for the transformed application can be created, due of the major challenges are similar to those faced when to the need to catalog and preserve the functionality of modernizing back-end systems: a lack of access to people the legacy application. Then, once modernized design who created the original product3; a lack of product doc- specification mockups have been created 4 the process of umentation, and a lack of functional specifications (e.g. implementing them in code is also completely manual, validation logic of input fields). There are also additional even though certain aspects of the UX, such as layout challenges due to the visual and interactive nature of and UI component usage, have been precisely specified the front-end: a lack of malleable design assets makes it in the design files. Hence, the overall UX modernization dificult to conduct re-design work; a lack of user stories process for an enterprise application is time consuming: makes it dificult to understand core functionality; a lack each of the 3 UX modernization projects we studied has of flow diagrams makes it dificult to understand the in- a 1-3 year roadmap for completion. teraction design; a lack of a UI catalog makes it dificult In speaking with our informants about their as-is proto understand how UI components are used across the cess for conducing UX modernization, we identified a entirety of the UX; and the requirements to use modern, large number of opportunities where AI technologies (not web-based, front-end UI frameworks makes it dificult to just generative) could provide support. After compiling understand how to re-write legacy code (e.g. porting a this list, we filtered it through the lens of generative technative Windows application to a Web application). niques (as consistent with our broader research question of potential uses of generative AI) to focus on new areas 4.2. As-Is: Current Process for UX and directions for generative machine learning research.
At a high level, we find UX modernization to be inter
Modernization esting from a machine learning perspective because of The Data Integration, Facility Management, and Asset the need to simultaneously co-transform multiple media Management application modernization projects we stud- types to conform to a desired state: the legacy UX design ied are each briefly introduced in Figures 1, 2, and 3. (as represented by images or design files) and the legacy Although each project has its own unique goals and tech- UX implementation (as represented by code) need to be nical and design challenges, we found that the phases of transformed into modernized UX design and modernized work required across all of them shared some common UX implementation. traits. We show a high-level overview of this process in We enumerate the subset of unmet needs we learned Figure 4. about in our user research that might be supported by
The start of each modernization project is character- generative AI innovations in Table 1, organized by the ized by an initial period of discovery in which product phases of the modernization process. We identify how managers, designers and software architects learn about those unmet needs may be addressed by motivating new the functionality of the legacy product (Step 1). This generative AI model development. process can be dificult due to a lack of product knowledge on the team and a lack of comprehensive product 5. Discussion documentation. Much of this learning happens through trial-and-error usage of the product, as well as conducting Internet searches and watching online video tutorials 5.1. AI-Supported UX Modernization (e.g. YouTube video demonstrations of the product). After gaining insight into the characteristics and un
The next phase involves planning which existing func- met needs of the existing UX modernization process, we tionality will be moved into the next generation product sought to re-envision a new, AI-supported UX moderniza(Step 2). Next, small functional units are specified (Step tion process. Our aim is to apply generative technologies 3), designed (Step 4), and implemented (Step 5) in an iter- in an assistive or augmentative fashion, turning UX modative fashion with integrated functional and user testing ernization into a co-creative process in which human
dates back 20+ years.
such as Sketch [39] and Figma [40]. attention is focused on work requiring creativity and mockups or textual descriptions into implemented code judgement, and generative models are used to automate (e.g., [25, 46]), or vice versa (e.g., [47]). tedious and manual work (e.g., [41]). Settu and Raj describe the challenges in automating modernization, and 5.2. Generative Models to explore for UX pNriolspsoosneegxupildoerleidnerastfioornaplreascftoicramlcoadseersn[iz9a].tiIonntahnedirftahcetosirss, Modernization to consider when planning a modernization project [42]. In Table 1, we identified the subset of unmet needs that Examples of modernization and automation approaches we felt could be addressed by new generative models. We appear in [4, 43, 27]. expand on these ideas below by describing how we might
We also desire to increase the reuse of legacy assets. implement such models to support those unmet needs. Already, designers and architects need to capture and Our observation is that UX modernization involves a preserve details of the legacy applications in order to gen- large number of diverse “micro-tasks” that may or may erate functional specifications, but aspects of this work not be needed, depending on the availability of informaare highly automatable, such as crawling UI screens and tion or documentation of the legacy product5. Thus, we cataloging the use of UI elements, and separating busi- assert that a single UX modernization assistive tool will ness logic from presentation logic. An important aspect not be suficient for addressing the myriad of challenges of this work is to transform legacy assets into the kinds faced by practitioners. Rather, we take an approach moof malleable intermediate representations used by design- tivated by Unix philosophy [48]: build small tools that ers and architects in the redesign process. For example, do one task well, and enable users to chain these tools rather than having designers re-create representations of together to achieve greater behaviors. We demonstrate the legacy UI in a drawing tool such as Sketch to prepare how an AI-supported designer might work in this new it for transformation, generative models might transform fashion in Figure 5. screenshots or legacy UI code into a “first cut” representation a designer can then work with [44, 45]. Or, generative models might perform transformative operations between representations, such as converting design
cataloging the UI is not necessary.
style and layout conventions. It requires a lot of manual efort to transform legacy styles & layouts to product-specific modernized styles & layouts.
Provide tools that convert legacy UX representations into a modern design system (e.g. transform UX screenshots to look like Carbon). This functionality may be especially useful in cases in which a minimal amount of layout and functional changes are planned, or where a quick what-if view would help the planning process.
Provide tools that can be used to restyle UI elements and layouts to fit the style of a specified design system and usage pattern. Note that design systems are generic, but layout and usage conventions are specific to product standards, and these should be a part of the transformation target.
UI-Style-Transfer, UI-FrameworkTranslation Screen-to-Mockup UI-Style-Transfer Mockup-toImplementation UI-FrameworkTranslation UI-CodeDisentanglement
A Screen-to-Mockup model would use a screenshot of A UI-Style-Transfer model would be able to take a screena UI as input and generate an editable UX mock-up rep- shot of a visual UI mockup as input and transform the resentation as output (e.g. Sketch file). Where available, styling of UI components to match a particular design it would be helpful to include legacy source code to im- system. For example, renderings using a legacy design prove the quality of the transformation. The Screen2Vec system could be transformed to use a next generation research by Li et al. [32] as well as the wireframe gener- design system. Additionally, any detailed mockup could ation research by Gajjar et al. [49] suggests this goal is be rendered as low fidelity style wireframes to facilitate feasible. discussions of functionality when UI detail is not a useful focus. Existing style transfer mechanisms (e.g. StyleGAN [50]) may be able to accomplish this task through
UI code as input (e.g. Java Swing, JSP) and transform it to a new UI framework as output (e.g. React, SwiftUI). Existing state-of-the-art generative code models (e.g. [54, 55, 24]) demonstrate how to translate source code between languages, but provide very limited support for translation at the level of packages and libraries.
code as input (e.g. PHP) and transform it into two modules: code that drives display & presentation logic (e.g. the Controller in a Model-View-Controller architecture) and code that drives business logic (e.g. the Model in a Model-View-Controller architecture). ifne tuning. Layout optimization capabilities such as the three projects described in section 4.2. Within the those described by Rahman et al. [45] would be required scope of these projects, we identified UX modernization for style transforms that include layout practices. Gener- as a challenging and time consuming part of the overative GUI design creation capabilities are also explored all modernization process that requires a lot of manual in research by Zhao et al. [51]. steps by multiple modernization team members including project managers, UX designers, and software engi5.2.3. Mockup-to-Implementation neers as illustrated in Figure 4. Current approaches to modernization [3, 2, 3, 4, 8, 56, 57, 58] revealed that UX Given a UI design representation (e.g. Sketch file) as in- modernization is not as well addressed as other modernput, a Mockup-to-Implementation model would be able ization tasks such as transforming core code into microto generate UI code as output (e.g. React). Existing rule- services [43, 27, 3], and potentially holds promise as a based systems have shown this is desirable functional- good area to consider generative AI technology help. We ity [52]; we hypothesize that generative models may be identified a number of pain points within the UX modable to improve the transformation process by incorporat- ernization process observed that could potentially be ing transformed UI logic (e.g. form validation code) from addressed by new generative AI technology in the form legacy code. The pix2code research by Beltramelli [47] of a set of generative AI models (described in Table 1 and as well as the SynZ research by Sermuga Pandian et al. in Section 5.2). Finally, in Figure 5, we used a scenario [53] show that deep learning models can be leveraged to visualize what the future of UX modernization work for this purpose as well. might be like if we were able to create and deploy those models. 5.2.4. UI-Framework-Translation Our contributions are as follows:
models and applications depicted with prospective users. 6. Conclusion At the same time we will explore feasibility from a data collection and training perspective. We share these ideas We conducted user research that enabled us to under- with this workshop in the hope of inviting critique, disstand goals, challenges, and work practices of teams en- cussion, and generative AI experimentation in the area gaged in application modernization work exemplified by of assisting the UX modernization process. • Discovery of a previously undescribed pain-point and opportunity for application modernization, in the dificulty of UX modernization. This aspect of our work extends previous analyses of legacy application modernization [3, 2, 3, 4, 56, 57, 58], supplementing previous generative AI work in other aspects of software engineering [16, 19, 20, 21, 22, 23, 24]. • Documentation of three UX modernization usecases, which require significant amounts of time and human efort. • Proposals for new tooling that can apply contemporary AI approaches [17, 18, 19, 21, 22, 24] to
UX modernization. 3. Next, he opens Sketch (a vectorbased drawing tool that he uses to prototype UI designs). There he right clicks to access a Generative AI powered plugin utility that allows him to paste an editable version of the legacy page he just copied onto the Sketch canvas.
Several different conversion style options are available. He chooses to convert the screen “as is”.
4. This converts the image into editable text and UI graphics where text and images can be examined, copied, pasted, and subsequently edited. The conversion process enabled by this paste was facilitated by the multiple image and code inputs captured earlier in the deep copy.
*Screen-to-Mockup
Sketch
Board 1 Board 2
Board 2
Board 2 2. He will use a number of new generative AI modernization utilities as he works today. One of these is a browser plug-in that allows him to do a “deep capture” of a UX screen. He uses it to capture an image of the page as well as the HTML representation of the screen, hierarchical Domain Object Mode model (DOM), text content, and any associated frontend code that can be located.
Sketch Board 1 6. This allows him to generate a second screen mock-up next to the legacy mock-up with the same UX features mapped to their Carbon design system equivalents.
The conversion is not perfect but gives Joe a head-start on different changes he needs to make to comply with style patterns that his group is using as well as functional changes specified by the product manager. *UI-Style-Transfer 7. He replaces some radio button choices with check boxes, puts the slide-out panel in context of a new drag and drop workflow area of the tool that was not present in the legacy version, and he adds accept/cancel buttons that got lost in the transfer. After Joe has evaluated and refined the design with users and other stakeholders, he attaches the completed Sketch file to a GitHub work item already prepared by the product manager.
Deep Capture
Convert as is Convert as wireframe Convert in Carbon style
Sketch Board 1
Sketch Board 1
Export as React Export as BootStrap
Export as original framework 8. Eva, a software engineer on the development team gets the implementation task assignment.
She starts by opening Joes's Sketch file that Joe prepared using the Generative AI plugin to convert the new page mockup-in sketch to a coded version using the React framework of the next gen application. The export capability uses the new design as well as details such as field names and validation in the legacy framework to produce the resulting code file. *Mockup-to-Implementation *UI-Framework-Translation 1. Joe is a UX designer on a team that is modernizing a legacy web application. Today he is going to design and specify an updated Preferences page. He begins by opening the legacy preferences page and reading the functional user story as well as links to legacy documentation provided by the product manager.
Sketch Board 1 VS Code
Convert as is Convert as wireframe
Convert in Carbon style 5. In this case, Joe is just going to use the legacy screen for reference while he converts the functionality they plan to preserve into a new design. He right clicks to access a menu of style transfer capabilities offered by the plugin.
He chooses to convert the style to
Carbon, which is the design
system of the next gen version of
the application.
5. Eva opens the resulting code file
in her IDE where she reviews what
is there and tries displaying and
testing the page. It’s not perfect
but it’s a good head start. She has
to fix some problems and also
needs to chase down some legacy
business logic that was not picked
up in the conversion because it
resided in a back-end part of the
code. She finds that code using
another AI Utility that helps her to
find and extract it.
*UI-Code-Disentanglement
10. When her work is done the first
version of the modernized page is
ready for testing. The AI tools
used along the way accelerated
the time it took to get to this point
compared with the way the team
used to do all steps manually. Eva
estimates it might make the
process faster by 20% or even
more!
[26] T. Tritchew, A multi-step incremental approach to URL: https://doi.org/10.1145/302979.303030.
modernizati
IEEE, 2018, pp. 76–83. [41] D. Friedmana, D. Pollaka, Image co-creation by
non[29] K. Gregor, I. Danihelka, A. Graves, D. Rezende, programmers and generative adversarial networks
D. Wierstra, Draw: A recurrent neural network (2021). URL:
http://ceur-ws.org/Vol-2903/IUI21WSfor image generation, in: F. Bach, D. Blei (Eds.), Pro- HAIGEN-4.pdf .
ceedings of the 32nd International Conference on [42] S. Nilsson, Application modernization:
ApMachine Learning, volume 37 of Proceedings of Ma- proaches, problems and evaluation,
Maschine Learning Research, PMLR, Lille, France, 2015, ter’s thesis, Umeå University, 2015. URL:
pp. 1462–1471. URL: https://proceedings.mlr.press/ https://www.diva-portal.org/smash/get/diva2:
v37/gregor15.html. 875375/FULLTEXT01.pdf .
[30] L. A. Gatys, A. S. Ecker, M. Bethge, Image style [43] IBM, Ibm mono2micro, 2021. URL: https:
transfer using convolutional neural networks, in: //www.ibm.com/cloud/mono2micro, accessed:
Proceedings of the IEEE C
Vision and Pattern Recognition (CVPR), 2016. [44] M. Laine, Y. Zhang, S. Santala, J. P. Jokinen, [31] Y. Aslam, S. N, A review of deep learning ap- A. Oulasvirta, Responsive and personalized web proaches for image analysis, in: 2019 International layouts with integer programming, Proceedings of Conference on Smart Systems and Inventive Tech- the ACM on Human-Computer Interaction 5 (2021) nology (ICSSIT), 2019, pp. 709–714. doi:10.1109/ 1–23.
ICSSIT46314.2019.8987922. [45] S. Rahman, V. P. Sermuga Pandian, M. Jarke, Ruite:
[32] T. J.-J. Li, L. Popowski, T. Mitchell, B. A. Myers, Refining ui layout aesthetics using transformer
Screen2vec: Semantic embedding of gui screens encoder, in: 26th International Conference on
and gui components, in: Proceedings
Journal of Human–Computer Interaction 36 (2020) [47] T. Beltramelli, pix2code: Generating code from a 495–504. graphical user interface screenshot, in: Proceedings [35] J. M. Bradshaw, P. J. Feltovich, M. Johnson, Human– of the ACM SIGCHI Symposium on Engineering agent interaction, in: The handbook of human- Interactive Computing Systems, 2018, pp. 1–6. machine interaction, CRC Press, 2017, pp. 283–300. [48] Wikipedia contributors, Unix philoso[36] E. Horvitz, Principles of mixed-initiative user phy — Wikipedia, the free encyclopedia, interfaces, in: Proceedings of the SIGCHI https://en.wikipedia.org/w/index.php?title= Conference on Human Factors in Computing Unix_philosophy&oldid=1036396134, 2021. Systems, CHI ’99, Association for Computing [Online; accessed 6-August-2021].
Machinery, New York, NY, USA, 1999, p. 159–166.