Opportunities for Generative AI in UX Modernization Stephanie Houde1,5 , Steven I. Ross1,5 , Michael Muller1,5 , Mayank Agarwal1 , Fernando Martinez4 , John Richards2 , Kartik Talamadupula3 and Justin D. Weisz2 1 IBM Research AI, Cambridge, MA, USA 2 IBM Research AI, Yorktown Heights, NY, USA 3 IBM Research AI, Seattle, WA, USA 4 IBM Argentina, La Plata, Buenos Aires, AR 5 Authors contributed equally to the work Abstract The process of application modernization consumes the effort of software development teams charged with upgrading legacy applications to modern technology, architecture, and design. While some tools exist to aid in these efforts, the modernization of an application’s user experience is an arduous and primarily manual undertaking. Through a process of user research and design exploration, we investigate how generative AI models might be used to assist software development teams in modernizing the user experience of legacy applications. Our goal is to identify opportunities for further research to aid teams involved in user experience modernization efforts. Keywords application modernization, user experience, generative AI models, software development, design 1. Introduction effort involving multiple roles [10]. As part of a broader evaluation of potential uses of gen- Software development has been an ongoing human activ- erative AI, in this paper we report on user research and ity for over 70 years [1]. While new applications continue design inspiration that explores how generative AI mod- to be created, software development teams are now of- els might assist software engineers, product managers, ten faced with the task of modernizing existing or legacy and user experience designers engaged in modernizing applications [2, 3, 4]. These applications may continue to the UX of legacy products. In this kind of modernization have large and dedicated user bases, but the technology effort, the design and implementation of a user inter- that they were based upon and the languages and tools face must undergo a simultaneous transformation: UX that were used to construct them may now be obsolete, designs need to be transformed to fit a modern design and the user interfaces that they present may be old- system1 , and UX code implementations need to be trans- fashioned and out-dated [5]. Attempting to bring these formed to utilize a modern UI framework2 . Our aim is applications up to modern standards is further hampered to inspire development of new machine learning tech- by inadequate or missing documentation and the loss niques that enable software engineers and designers to of institutional memory that results from employee at- work across a multitude of UX representations, including trition and retirement [6, 7]. While all aspects of the screenshots, design mock-ups, and user interface imple- application modernization effort present challenges, the mentations. task of modernizing the user experience (UX) is particu- We have identified UX modernization as a unique op- larly daunting [7] – but also particularly necessary for portunity area through our deliberate effort to identify customer satisfaction [8, 9] – and requires a great deal of technological challenges through user research. We de- scribe the process we used to identify specific pain points Joint Proceedings of the ACM IUI Workshops 2022, March 2022, Helsinki, Finland and unmet needs in UX modernization in Section 4.1 and $ Stephanie.Houde@ibm.com (S. Houde); our overall observation is that it has a broad scope with steven_ross@us.ibm.com (S. I. Ross); michael_muller@us.ibm.com many different constituent tasks and sub-tasks. We de- (M. Muller); Mayank.Agarwal@ibm.com (M. Agarwal); scribe the as-is state of UX modernization in Section 4.2 martferc@ar.ibm.com (F. Martinez); ajtr@us.ibm.com (J. Richards); and show a high-level overview of the process in Figure 4. krtalamad@us.ibm.com (K. Talamadupula); jweisz@us.ibm.com (J. D. Weisz) 1 A design system is a set of visual and interactive layout, UI  0000-0002-0246-2183 (S. Houde); 0000-0002-2533-9946 component, color and other standards designed to create consistent (S. I. Ross); 0000-0001-7860-163X (M. Muller); 0000-0002-8442-2651 UX often across multiple displays such as web pages and mobile (M. Agarwal); 0000-0001-7172-4805 (F. Martinez); application screens. Examples of popular design systems include 0000-0001-8489-2170 (J. Richards); 0000-0002-4628-3785 Material [11] and Carbon [12] 2 (K. Talamadupula); 0000-0003-2228-2398 (J. D. Weisz) A UI framework is a library of code that facilitates front-end © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). web and mobile application development. Popular UI frameworks CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) include Bootstrap [13] and React [14] We also provide detail on the many unmet needs we have cation modernization projects. For topical completeness learned about in Table 1. and consistency, we guided each informant through a Next, we begin crafting a vision for how generative series of open-ended questions. We balanced these inten- AI technologies can address some of the unmet needs in tional topics with ample opportunity for informants to Section 5.1, and map those onto specific kinds of genera- introduce their own concerns, and to educate us about tive models that could be trained to support these needs the complexities of their work. in Section 5.2. Each interview took place online, and lasted approxi- We begin to identify opportunities to develop new mately 60 minutes. Informants were invited to show as kinds of generative models, based on feasibility (e.g. the well as tell, by sharing their screen to illustrate the materi- accessing specific data) and potential for impact (e.g. als of their practices (e.g., screenshots from applications) meeting critical needs). Our philosophy is to decompose and the ways that they worked with those materials (e.g., the large, generally-specified tasks within UX modern- their own notes and proposals for further work).Each ization into a suite of purpose-built generative models interview was led by the same researcher, and at least that can be used by designers and software engineers to one other researcher attended as note-taker. In addition accomplish specific key tasks. to the manual notes, we recorded all interviews (with informants’ permission), and we used the e-meeting soft- ware to generate a speech-to-text transcript. We used the 2. Background transcript as a guide to what the informants had said, and we referred back to the recordings as necessary when UX modernization involves both the redesign of user fac- the transcripts may have contained speech-recognition ing elements of a legacy application and the underlying errors. architecture and technology that supports it. The goal is The majority of those we spoke to were members of to provide users with an improved experience in terms of a Data integration application, a Facility management efficiency, productivity, responsiveness, ease of use, and application, and Asset management application product enjoyment [15]. The task is complicated and constrained teams who are currently engaged in large modernization by the user habits and expectations that result from their projects. These informants gave us a high degree of ac- experience with earlier versions of the application. cess to other project stakeholders and the content of their Recent work has applied a variety of generative AI and modernization projects, which helped us gain a more con- machine learning techniques to a number of software crete understanding of the people, processes, tools, and development tasks, such as code completion [16, 17, 18, artifacts involved in modernization. By forming a deep 19], documentation and test generation [20, 21, 22, 23], understanding of their work practices, we were able to and computer language translation [24]. Allamanis et al. identify pain points that AI technologies could help alle- [25] provide a comprehensive review of the use of AI viate, as well as new opportunities for which generative and machine learning within software engineering. In AI technologies could be a useful technological approach. the realm of application modernization, a variety of tools have been developed that help in the analysis and re- architecture of legacy systems [26, 27, 3]. Tools to address 4. Results the modernization of the user experience, however, are notably lacking. 4.1. Identifying UX Modernization as a Since application user interfaces typically have a vi- particular challenge sual component to them, recent work in image analysis, One area in which all three of these product groups are generation, and style transfer [28, 29, 30, 31, 32] establish a baseline of capabilities that may be adapted to the UXcurrently making massive, multi-year investments is in modernization domain. the modernization of the UX. In some of our first inter- Finally, since UX modernization activities will typi-views we learned that the UX transformation part of the cally involve much iteration, work on Human-AI Collab- modernization process was particularly difficult and time oration with generative models [33, 34, 35, 36] and the consuming, and it did not seem to provide many opportu- types of interaction made possible by the emergence of nities for salvaging or re-use of legacy UI designs or code. generative AI techniques for natural language, such as One informant who is an architect called their effort “a GPT-2 [37] and GPT-3 [38] may also prove to be relevant.complete re-write,” indicating that nearly all of the legacy code driving the UI had to be re-written to match a com- pletely re-designed UX. A Program Director for the Data 3. Method Integration tool volunteered that modernizing the UI is “the biggest pain point” in building their next generation We interviewed 12 product managers, architects, and application because it requires a full UX design team and design leads who are currently working on legacy appli- UI re-architecture, which would take as much time as as needed, until the modernization process is complete. designing a whole new application from scratch. This Overall, this process is highly manual and incurs tremen- observation piqued our interest because it seemed that dous coordination costs across designers, architects, soft- a legacy product’s UX ought to provide a starting point ware engineers, and product managers. All tasks – large for creating a modern re-design, rather than designing or small – require human attention and manual work. and building a new UX from scratch. Very little legacy code or design artifacts are reused, if In subsequent interviews, we confirmed that these they are even present. In some cases, design artifacts large-scale UX modernization projects require high ef- for the legacy application need to be created before arti- fort, large teams, and multiple years to complete. Some facts for the transformed application can be created, due of the major challenges are similar to those faced when to the need to catalog and preserve the functionality of modernizing back-end systems: a lack of access to people the legacy application. Then, once modernized design who created the original product3 ; a lack of product doc- specification mockups have been created4 the process of umentation, and a lack of functional specifications (e.g. implementing them in code is also completely manual, validation logic of input fields). There are also additional even though certain aspects of the UX, such as layout challenges due to the visual and interactive nature of and UI component usage, have been precisely specified the front-end: a lack of malleable design assets makes it in the design files. Hence, the overall UX modernization difficult to conduct re-design work; a lack of user stories process for an enterprise application is time consuming: makes it difficult to understand core functionality; a lack each of the 3 UX modernization projects we studied has of flow diagrams makes it difficult to understand the in- a 1-3 year roadmap for completion. teraction design; a lack of a UI catalog makes it difficult In speaking with our informants about their as-is pro- to understand how UI components are used across the cess for conducing UX modernization, we identified a entirety of the UX; and the requirements to use modern, large number of opportunities where AI technologies (not web-based, front-end UI frameworks makes it difficult to just generative) could provide support. After compiling understand how to re-write legacy code (e.g. porting a this list, we filtered it through the lens of generative tech- native Windows application to a Web application). niques (as consistent with our broader research question of potential uses of generative AI) to focus on new areas 4.2. As-Is: Current Process for UX and directions for generative machine learning research. At a high level, we find UX modernization to be inter- Modernization esting from a machine learning perspective because of The Data Integration, Facility Management, and Asset the need to simultaneously co-transform multiple media Management application modernization projects we stud- types to conform to a desired state: the legacy UX design ied are each briefly introduced in Figures 1, 2, and 3. (as represented by images or design files) and the legacy Although each project has its own unique goals and tech- UX implementation (as represented by code) need to be nical and design challenges, we found that the phases of transformed into modernized UX design and modernized work required across all of them shared some common UX implementation. traits. We show a high-level overview of this process in We enumerate the subset of unmet needs we learned Figure 4. about in our user research that might be supported by The start of each modernization project is character- generative AI innovations in Table 1, organized by the ized by an initial period of discovery in which product phases of the modernization process. We identify how managers, designers and software architects learn about those unmet needs may be addressed by motivating new the functionality of the legacy product (Step 1). This generative AI model development. process can be difficult due to a lack of product knowl- edge on the team and a lack of comprehensive product documentation. Much of this learning happens through 5. Discussion trial-and-error usage of the product, as well as conduct- ing Internet searches and watching online video tutorials 5.1. AI-Supported UX Modernization (e.g. YouTube video demonstrations of the product). After gaining insight into the characteristics and un- The next phase involves planning which existing func- met needs of the existing UX modernization process, we tionality will be moved into the next generation product sought to re-envision a new, AI-supported UX moderniza- (Step 2). Next, small functional units are specified (Step tion process. Our aim is to apply generative technologies 3), designed (Step 4), and implemented (Step 5) in an iter- in an assistive or augmentative fashion, turning UX mod- ative fashion with integrated functional and user testing ernization into a co-creative process in which human 3 4 For some of the products we examined, their original design Visual specification mockups are often created in applications dates back 20+ years. such as Sketch [39] and Figma [40]. Figure 1: Data integration application UX Modernization project. The legacy version of the data integration application we studied was originally written 25 years ago. It offers a direct manipulation solution for defining Extract/Transfer/Load (ETL) data processing jobs. A team is currently working to build a next generation version of the product while, in parallel, continuing to support the still widely used legacy product. UX modernization plans (now partially implemented) include changing from a desktop Windows application framework to the web-based React framework [14] with the Carbon design system [12]. The team also seeks to identify pre-built platform components that can replace existing functionality in the Windows application. This project is estimated to take 1-2 years. Figure 2: Facility management application UX modernization project. The legacy version of the facility management product we studied was originally written 20 years ago. A team is currently working to modernize its web UI while making as few functional changes as possible. The legacy web UI was implemented as Java Server Pages (JSP) with a legacy web design system.The team is working to re-implement it using the React framework and the Carbon design system. This work will be done incrementally on the live product, transforming the existing product while it is still in use. Incremental implementation has already begun and this project is estimated to be completed in 2-3 years. Figure 3: Asset management application UX modernization project. The legacy version of the enterprise facility management product we studied is used heavily as a mobile application for equipment maintenance in the field. The focus of their UX modernization efforts are on re-designing and re-implementing their legacy mobile applications as a new web applications using the mobile version of the Carbon design system. New features and improved navigation will also be added. The modernization process is being executed on one major feature at a time. In the UI re-implementation, the architecture of front-end code is being dramatically changed to adopt a new XML-based declarative system. executed in multiple waves 1 2 3 4 5 Legacy UX Modernization Functional Design Next Gen UX Discovery Planning Specification Specification Implementation Product architect works Product architect, Product architect does a UX Designers define the Software engineers to understand the legacy engineering architect, detailed review of the next gen UI given the implement the next gen application both inside design leads and multiple legacy system next in line user and functional user experience given the and out. stakeholders work to for modernization, requirements. The legacy specifications and define a general strategy documents existing UI may be used as a requirements. The legacy for how the old functionality, modifies to visual reference during code base is used as a functionality should be remove and add features this work as they create reference while mapped to the new UX, according to next gen next gen mockups that functionality is and in what order given plans, and develops a restyle but also replicate reimplemented in the the next gen user story to describe some of the existing next gen framework. requirements. these capabilities in work functionality and items for design and evaluate them with development. users. Figure 4: A high level view of the UX modernization process. All three projects first required discovering and inventorying the user-facing capabilities presented in the legacy product (Step 1). Next, strategic plans are made (Step 2) to choose the highest impact feature(s) to modernize first, choose the functional group to modernize first, or figure out how old functionality can be replaced by a new application suite context. Once the priorities have been established, functional specifications (Step 3), design specifications (Step 4), and UX implementations (Step 5) occur in an iterative fashion, akin to traditional product development. We learned the common steps of this process from our informants. It is also documented similarly and in more technical detail in Tritchew [26]. attention is focused on work requiring creativity and mockups or textual descriptions into implemented code judgement, and generative models are used to automate (e.g., [25, 46]), or vice versa (e.g., [47]). tedious and manual work (e.g., [41]). Settu and Raj de- scribe the challenges in automating modernization, and 5.2. Generative Models to explore for UX propose guidelines for practical cases [9]. In their thesis, Nilsson explored rationales for modernization and factors Modernization to consider when planning a modernization project [42]. In Table 1, we identified the subset of unmet needs that Examples of modernization and automation approaches we felt could be addressed by new generative models. We appear in [4, 43, 27]. expand on these ideas below by describing how we might We also desire to increase the reuse of legacy assets. implement such models to support those unmet needs. Already, designers and architects need to capture and Our observation is that UX modernization involves a preserve details of the legacy applications in order to gen- large number of diverse “micro-tasks” that may or may erate functional specifications, but aspects of this work not be needed, depending on the availability of informa- are highly automatable, such as crawling UI screens and tion or documentation of the legacy product5 . Thus, we cataloging the use of UI elements, and separating busi- assert that a single UX modernization assistive tool will ness logic from presentation logic. An important aspect not be sufficient for addressing the myriad of challenges of this work is to transform legacy assets into the kinds faced by practitioners. Rather, we take an approach mo- of malleable intermediate representations used by design- tivated by Unix philosophy [48]: build small tools that ers and architects in the redesign process. For example, do one task well, and enable users to chain these tools rather than having designers re-create representations of together to achieve greater behaviors. We demonstrate the legacy UI in a drawing tool such as Sketch to prepare how an AI-supported designer might work in this new it for transformation, generative models might transform fashion in Figure 5. screenshots or legacy UI code into a “first cut” repre- sentation a designer can then work with [44, 45]. Or, generative models might perform transformative opera- 5 tions between representations, such as converting design For example, if a UI catalog exists for the legacy product, then cataloging the UI is not necessary. Activity Unmet Need Solution Idea Model Idea Modernization It takes a long time and special design Provide tools that convert legacy UX UI-Style-Transfer, Planning and implementation skills to visualize representations into a modern design UI-Framework- the modernized UX for stakeholder system (e.g. transform UX screen- Translation evaluation. shots to look like Carbon). This func- tionality may be especially useful in cases in which a minimal amount of layout and functional changes are planned, or where a quick what-if view would help the planning pro- cess. Design All controls in the legacy application Provide tools that transform legacy Screen-to-Mockup Specification have to be redrawn in a design tool screens into editable representations to produce an editable version of the in UX (e.g. as Sketch or Figma design UX. files). Design Different products have different UI Provide tools that can be used to UI-Style-Transfer Specification style and layout conventions. It restyle UI elements and layouts to requires a lot of manual effort to fit the style of a specified design sys- transform legacy styles & layouts to tem and usage pattern. Note that de- product-specific modernized styles & sign systems are generic, but layout layouts. and usage conventions are specific to product standards, and these should be a part of the transformation tar- get. Next Gen UX Even though design specifications Provide tools that generate UI code Mockup-to- Implementation are pixel-perfect, software engineers that match the intended design to Implementation need to implement them from save time by offering a starting point scratch in code. for further editing. Next Gen UX It’s difficult to re-purpose or re-use Provide tools that can translate de- UI-Framework- Implementation UI code written in a legacy UI frame- sign systems from one framework to Translation work. another as a starting point for further editing. Next Gen UX It’s difficult to re-purpose or re-use Provide tools that can disentangle UI-Code- Implementation UI code written in a legacy UI frame- code that manages the UI from code Disentanglement work, when code for the UI is mixed that manages business logic. with code that implements business logic. Table 1 Unmet needs discovered through our interviews with UX modernization practitioners. Each unmet need is mapped to a potential solution, which may (or may not) involve an application of AI. For some unmet needs, there are clear mappings to potential generative models (shown in bold italic); these are extracted and discussed in further detail in section 5.2. 5.2.1. Screen-to-Mockup 5.2.2. UI-Style-Transfer A Screen-to-Mockup model would use a screenshot of A UI-Style-Transfer model would be able to take a screen- a UI as input and generate an editable UX mock-up rep- shot of a visual UI mockup as input and transform the resentation as output (e.g. Sketch file). Where available, styling of UI components to match a particular design it would be helpful to include legacy source code to im- system. For example, renderings using a legacy design prove the quality of the transformation. The Screen2Vec system could be transformed to use a next generation research by Li et al. [32] as well as the wireframe gener- design system. Additionally, any detailed mockup could ation research by Gajjar et al. [49] suggests this goal is be rendered as low fidelity style wireframes to facilitate feasible. discussions of functionality when UI detail is not a useful focus. Existing style transfer mechanisms (e.g. Style- GAN [50]) may be able to accomplish this task through fine tuning. Layout optimization capabilities such as the three projects described in section 4.2. Within the those described by Rahman et al. [45] would be required scope of these projects, we identified UX modernization for style transforms that include layout practices. Gener- as a challenging and time consuming part of the over- ative GUI design creation capabilities are also explored all modernization process that requires a lot of manual in research by Zhao et al. [51]. steps by multiple modernization team members includ- ing project managers, UX designers, and software engi- 5.2.3. Mockup-to-Implementation neers as illustrated in Figure 4. Current approaches to modernization [3, 2, 3, 4, 8, 56, 57, 58] revealed that UX Given a UI design representation (e.g. Sketch file) as in- modernization is not as well addressed as other modern- put, a Mockup-to-Implementation model would be able ization tasks such as transforming core code into micro- to generate UI code as output (e.g. React). Existing rule- services [43, 27, 3], and potentially holds promise as a based systems have shown this is desirable functional- good area to consider generative AI technology help. We ity [52]; we hypothesize that generative models may be identified a number of pain points within the UX mod- able to improve the transformation process by incorporat- ernization process observed that could potentially be ing transformed UI logic (e.g. form validation code) from addressed by new generative AI technology in the form legacy code. The pix2code research by Beltramelli [47] of a set of generative AI models (described in Table 1 and as well as the SynZ research by Sermuga Pandian et al. in Section 5.2). Finally, in Figure 5, we used a scenario [53] show that deep learning models can be leveraged to visualize what the future of UX modernization work for this purpose as well. might be like if we were able to create and deploy those models. 5.2.4. UI-Framework-Translation Our contributions are as follows: A UI-Framework-Translation model would take legacy • Discovery of a previously undescribed pain-point UI code as input (e.g. Java Swing, JSP) and transform and opportunity for application modernization, it to a new UI framework as output (e.g. React, Swif- in the difficulty of UX modernization. This aspect tUI). Existing state-of-the-art generative code models of our work extends previous analyses of legacy (e.g. [54, 55, 24]) demonstrate how to translate source application modernization [3, 2, 3, 4, 56, 57, 58], code between languages, but provide very limited sup- supplementing previous generative AI work in port for translation at the level of packages and libraries. other aspects of software engineering [16, 19, 20, 21, 22, 23, 24]. 5.2.5. UI-Code-Disentanglement • Documentation of three UX modernization use- A UI-Code-Disentanglement model would take legacy UI cases, which require significant amounts of time code as input (e.g. PHP) and transform it into two mod- and human effort. ules: code that drives display & presentation logic (e.g. • Proposals for new tooling that can apply contem- the Controller in a Model-View-Controller architecture) porary AI approaches [17, 18, 19, 21, 22, 24] to and code that drives business logic (e.g. the Model in a UX modernization. Model-View-Controller architecture). In future work we will evaluate the desirability of the models and applications depicted with prospective users. 6. Conclusion At the same time we will explore feasibility from a data collection and training perspective. We share these ideas We conducted user research that enabled us to under- with this workshop in the hope of inviting critique, dis- stand goals, challenges, and work practices of teams en- cussion, and generative AI experimentation in the area gaged in application modernization work exemplified by of assisting the UX modernization process. Sketch Sketch Board 1 Convert as is Deep Capture Convert as wireframe Convert in Carbon style 1. Joe is a UX designer on a team 2. He will use a number of new 3. Next, he opens Sketch (a vector- 4. This converts the image into that is modernizing a legacy web generative AI modernization based drawing tool that he uses to editable text and UI graphics application. Today he is going to utilities as he works today. One of prototype UI designs). There he where text and images can be design and specify an updated these is a browser plug-in that right clicks to access a Generative examined, copied, pasted, and Preferences page. He begins by allows him to do a “deep capture” AI powered plugin utility that subsequently edited. The opening the legacy preferences of a UX screen. He uses it to allows him to paste an editable conversion process enabled by page and reading the functional capture an image of the page as version of the legacy page he just this paste was facilitated by the user story as well as links to legacy well as the HTML representation of copied onto the Sketch canvas. multiple image and code inputs documentation provided by the the screen, hierarchical Domain Several different conversion style captured earlier in the deep copy. product manager. Object Mode model (DOM), text options are available. He chooses content, and any associated front- to convert the screen “as is”. *Screen-to-Mockup end code that can be located. Sketch Sketch Sketch Sketch Board 1 Board 1 Board 2 Board 1 Board 2 Board 1 Board 2 Convert as is Convert as wireframe Export as React Convert in Carbon style Export as BootStrap Export as original framework 5. In this case, Joe is just going to 6. This allows him to generate a 7. He replaces some radio button 8. Eva, a software engineer on the use the legacy screen for second screen mock-up next to choices with check boxes, puts the development team gets the reference while he converts the the legacy mock-up with the same slide-out panel in context of a new implementation task assignment. functionality they plan to preserve UX features mapped to their drag and drop workflow area of the She starts by opening Joes's into a new design. He right clicks Carbon design system equivalents. tool that was not present in the Sketch file that Joe prepared using to access a menu of style transfer The conversion is not perfect but legacy version, and he adds the Generative AI plugin to convert capabilities offered by the plugin. gives Joe a head-start on different accept/cancel buttons that got lost the new page mockup-in sketch to He chooses to convert the style to changes he needs to make to in the transfer. After Joe has a coded version using the React Carbon, which is the design comply with style patterns that his evaluated and refined the design framework of the next gen system of the next gen version of group is using as well as functional with users and other stakeholders, application. The export capability the application. changes specified by the product he attaches the completed Sketch uses the new design as well as manager. file to a GitHub work item already details such as field names and prepared by the product manager. validation in the legacy framework *UI-Style-Transfer to produce the resulting code file. *Mockup-to-Implementation *UI-Framework-Translation VS Code 5. Eva opens the resulting code file 10. When her work is done the first in her IDE where she reviews what version of the modernized page is is there and tries displaying and ready for testing. The AI tools testing the page. It’s not perfect used along the way accelerated but it’s a good head start. She has the time it took to get to this point to fix some problems and also compared with the way the team needs to chase down some legacy used to do all steps manually. Eva business logic that was not picked estimates it might make the up in the conversion because it process faster by 20% or even resided in a back-end part of the more! code. She finds that code using another AI Utility that helps her to find and extract it. *UI-Code-Disentanglement Figure 5: Future UX modernization "to-be" scenario. In this scenario, a UX designer’s work is augmented via a suite of generative models that make it easier to move between UI representations (e.g. Screen-to-Mockup and UI Style transfer in Steps 4 and 6) as well as hand-off work to software engineers (e.g. Mockup-to-Implementation, UI Framework Translation, and UI Code Disentanglement in steps 8 and 9). *Names of future generative models, shown in bold italic text, described in more detail in Section 5.2. References URL: https://github.com/jbranchaud/awesome- react-design-systems, accessed: 2021-12-20. [1] M. Yost, A brief history of software [15] J. Lasarte, Principles to guide your ux mod- development (2018). URL: https://https: ernization (2018). URL: https://medium.com/ //medium.com/@micahyost/a-brief-history- headspring-ux-team/principles-to-guide-your- of-software-development-f67a6e6ddae0. ux-modernization-d1d7ee56270d, [Online; [2] S. Jain, I. Chana, Modernization of legacy systems: accessed 16-Dec-2021]. A generalised roadmap, in: Proceedings of the [16] A. Hindle, E. T. Barr, Z. Su, M. Gabel, P. Devanbu, Sixth International Conference on Computer and On the naturalness of software, in: 2012 34th In- Communication Technology 2015, 2015, pp. 62–67. ternational Conference on Software Engineering [3] A. K. Kalia, J. Xiao, C. Lin, S. Sinha, J. Rofrano, (ICSE), IEEE, 2012, pp. 837–847. M. Vukovic, D. Banerjee, Mono2micro: an ai-based [17] V. Raychev, M. Vechev, A. Krause, Predicting pro- toolchain for evolving monolithic enterprise appli- gram properties from" big code", in: Proceedings cations to a microservice architecture, in: Proceed- of the 42nd Annual ACM SIGPLAN-SIGACT Sym- ings of the 28th ACM Joint Meeting on European posium on Principles of Programming Languages, Software Engineering Conference and Symposium 2015, pp. 111–124. on the Foundations of Software Engineering, 2020, [18] M. Bruch, M. Monperrus, M. Mezini, Learning from pp. 1606–1610. examples to improve code completion systems, in: [4] M. Mishra, S. Kunde, M. Nambiar, Cracking the Proceedings of the 7th joint meeting of the Euro- monolith: Challenges in data transitioning to cloud pean software engineering conference and the ACM native architectures, in: Proceedings of the 12th Eu- SIGSOFT symposium on the foundations of soft- ropean Conference on Software Architecture: Com- ware engineering, 2009, pp. 213–222. panion Proceedings, 2018, pp. 1–4. [19] A. Svyatkovskiy, S. K. Deng, S. Fu, N. Sundaresan, [5] M. Fauscette, R. Perry, Simplifying it to drive better Intellicode compose: Code generation using trans- business outcomes and improved roi: Introducing former, arXiv preprint arXiv:2005.08025 (2020). the it complexity index, International Data Corpo- [20] M. Tufano, D. Drain, A. Svyatkovskiy, S. K. Deng, ration (2014). N. Sundaresan, Unit test case generation with trans- [6] M. Kangasaho, et al., Legacy application modern- formers, arXiv preprint arXiv:2009.05617 (2020). ization with rest wrapping (2016). [21] X. Guo, Towards automated software testing [7] J. D. Weisz, M. Muller, S. Houde, J. Richards, S. I. with generative adversarial networks, in: 2021 Ross, F. Martinez, M. Agarwal, K. Talamadupula, 51st Annual IEEE/IFIP International Conference Perfection not required? human-ai partnerships in on Dependable Systems and Networks - Supple- code translation, in: 26th International Conference mental Volume (DSN-S), IEEE Computer Society, on Intelligent User Interfaces, 2021, pp. 402–412. Los Alamitos, CA, USA, 2021, pp. 21–22. URL: [8] J. Johnson, H. Mulder, Endless modernization https://doi.ieeecomputersociety.org/10.1109/ (2020). URL: https://www.researchgate.net/profile/ DSN-S52858.2021.00021. doi:10.1109/DSN- Hans-Mulder-2/publication/348849361_Endless_ S52858.2021.00021. Modernization_How_Infinite_Flow_Keeps_ [22] L. Moreno, J. Aponte, G. Sridhara, A. Marcus, L. Pol- Software_Fresh/links/60132878299bf1b33e30c29e/ lock, K. Vijay-Shanker, Automatic generation of Endless-Modernization-How-Infinite-Flow- natural language summaries for java classes, in: Keeps-Software-Fresh.pdf. 2013 21st International Conference on Program [9] R. Settu, P. Raj, Cloud application modernization Comprehension (ICPC), IEEE, 2013, pp. 23–32. and migration methodology, in: Cloud Computing, [23] A. Y. Wang, D. Wang, J. Drozdal, M. Muller, Springer, 2013, pp. 243–271. S. Park, J. D. Weisz, X. Liu, L. Wu, C. Dugan, [10] L. Tahlawi, Combining legacy modernization ap- Themisto: Towards automated documentation gen- proaches for OO and SOA, Ph.D. thesis, University eration in computational notebooks, arXiv preprint of New Brunswick., 2012. arXiv:2102.12592 (2021). [11] Google, Material design, n.d. URL: https: [24] B. Roziere, M.-A. Lachaux, L. Chanussot, G. Lam- //material.io/, n.d. ple, Unsupervised translation of programming lan- [12] IBM, Carbon design system, 2021. URL: https:// guages., in: NeurIPS, 2020. www.carbondesignsystem.com/, accessed: 2021- [25] M. Allamanis, E. T. Barr, P. Devanbu, C. Sutton, A 12-20. survey of machine learning for big code and natu- [13] O. source community, Get bootstrap, n.d. URL: ralness, ACM Computing Surveys (CSUR) 51 (2018) https://getbootstrap.com/, n.d. 1–37. [14] Anonymous, Awesome react design systems, 2021. [26] T. Tritchew, A multi-step incremental approach to URL: https://doi.org/10.1145/302979.303030. modernization (2020). URL: https://medium.com/ doi:10.1145/302979.303030. @tedt_39153/a-multi-step-incremental-approach- [37] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, to-modernization-e6264318b462. I. Sutskever, Language models are unsupervised [27] IBM, Ibm cloud transformation advisor, 2021. URL: multitask learners, OpenAI blog 1 (2019) 9. https://www.ibm.com/garage/method/practices/ [38] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Ka- learn/ibm-transformation-advisor, accessed: plan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sas- 2021-12-20. try, A. Askell, et al., Language models are few-shot [28] Z. Zhao, X. Ma, A compensation method of two- learners, arXiv preprint arXiv:2005.14165 (2020). stage image generation for human-ai collaborated [39] Anonymous, Sketch, 2021. URL: https: in-situ fashion design in augmented reality envi- //www.sketch.com/, accessed: 2021-12-20. ronment, in: 2018 IEEE International Conference [40] Anonymous, Figma, n.d. URL: https: on Artificial Intelligence and Virtual Reality (AIVR), //www.figma.com/, accessed: 2022-12-20. IEEE, 2018, pp. 76–83. [41] D. Friedmana, D. Pollaka, Image co-creation by non- [29] K. Gregor, I. Danihelka, A. Graves, D. Rezende, programmers and generative adversarial networks D. Wierstra, Draw: A recurrent neural network (2021). URL: http://ceur-ws.org/Vol-2903/IUI21WS- for image generation, in: F. Bach, D. Blei (Eds.), Pro- HAIGEN-4.pdf. ceedings of the 32nd International Conference on [42] S. Nilsson, Application modernization: Ap- Machine Learning, volume 37 of Proceedings of Ma- proaches, problems and evaluation, Mas- chine Learning Research, PMLR, Lille, France, 2015, ter’s thesis, Umeå University, 2015. URL: pp. 1462–1471. URL: https://proceedings.mlr.press/ https://www.diva-portal.org/smash/get/diva2: v37/gregor15.html. 875375/FULLTEXT01.pdf. [30] L. A. Gatys, A. S. Ecker, M. Bethge, Image style [43] IBM, Ibm mono2micro, 2021. URL: https: transfer using convolutional neural networks, in: //www.ibm.com/cloud/mono2micro, accessed: Proceedings of the IEEE Conference on Computer 2021-08-05. Vision and Pattern Recognition (CVPR), 2016. [44] M. Laine, Y. Zhang, S. Santala, J. P. Jokinen, [31] Y. Aslam, S. N, A review of deep learning ap- A. Oulasvirta, Responsive and personalized web proaches for image analysis, in: 2019 International layouts with integer programming, Proceedings of Conference on Smart Systems and Inventive Tech- the ACM on Human-Computer Interaction 5 (2021) nology (ICSSIT), 2019, pp. 709–714. doi:10.1109/ 1–23. ICSSIT46314.2019.8987922. [45] S. Rahman, V. P. Sermuga Pandian, M. Jarke, Ruite: [32] T. J.-J. Li, L. Popowski, T. Mitchell, B. A. Myers, Refining ui layout aesthetics using transformer Screen2vec: Semantic embedding of gui screens encoder, in: 26th International Conference on and gui components, in: Proceedings of the 2021 Intelligent User Interfaces - Companion, IUI ’21 CHI Conference on Human Factors in Computing Companion, Association for Computing Machin- Systems, 2021, pp. 1–15. ery, New York, NY, USA, 2021, p. 81–83. URL: https: [33] T. Weber, H. Hußmann, Z. Han, S. Matthes, Y. Liu, //doi.org/10.1145/3397482.3450716. doi:10.1145/ Draw with me: Human-in-the-loop for image 3397482.3450716. restoration, in: Proceedings of the 25th Interna- [46] D. K. Palani, Statistical Machine Translation of En- tional Conference on Intelligent User Interfaces, glish Text to API Code Usages: A comparison of 2020, pp. 243–253. Word Map, Contextual Graph Ordering, Phrase- [34] B. Shneiderman, Human-centered artificial intelli- based, and Neural Network Translations, Ph.D. the- gence: Reliable, safe & trustworthy, International sis, Concordia University, 2018. Journal of Human–Computer Interaction 36 (2020) [47] T. Beltramelli, pix2code: Generating code from a 495–504. graphical user interface screenshot, in: Proceedings [35] J. M. Bradshaw, P. J. Feltovich, M. Johnson, Human– of the ACM SIGCHI Symposium on Engineering agent interaction, in: The handbook of human- Interactive Computing Systems, 2018, pp. 1–6. machine interaction, CRC Press, 2017, pp. 283–300. [48] Wikipedia contributors, Unix philoso- [36] E. Horvitz, Principles of mixed-initiative user phy — Wikipedia, the free encyclopedia, interfaces, in: Proceedings of the SIGCHI https://en.wikipedia.org/w/index.php?title= Conference on Human Factors in Computing Unix_philosophy&oldid=1036396134, 2021. Systems, CHI ’99, Association for Computing [Online; accessed 6-August-2021]. Machinery, New York, NY, USA, 1999, p. 159–166. [49] N. Gajjar, V. P. Sermuga Pandian, S. Suleri, detectors, in: 26th International Conference on M. Jarke, Akin: Generating ui wireframes Intelligent User Interfaces - Companion, IUI ’21 from ui design patterns using deep learning, Companion, Association for Computing Machinery, in: 26th International Conference on Intelli- New York, NY, USA, 2021, p. 67–69. URL: https: gent User Interfaces - Companion, IUI ’21 Com- //doi.org/10.1145/3397482.3450725. doi:10.1145/ panion, Association for Computing Machinery, 3397482.3450725. New York, NY, USA, 2021, p. 40–42. URL: https: [54] Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, //doi.org/10.1145/3397482.3450727. doi:10.1145/ L. Shou, B. Qin, T. Liu, D. Jiang, et al., Codebert: 3397482.3450727. A pre-trained model for programming and natural [50] T. Karras, S. Laine, T. Aila, A style-based generator languages, arXiv preprint arXiv:2002.08155 (2020). architecture for generative adversarial networks, [55] D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, in: Proceedings of the IEEE/CVF Conference on L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu, et al., Computer Vision and Pattern Recognition, 2019, Graphcodebert: Pre-training code representations pp. 4401–4410. with data flow, arXiv preprint arXiv:2009.08366 [51] T. Zhao, C. Chen, Y. Liu, X. Zhu, Guigan: Learning (2020). to generate gui designs using generative adver- [56] H. Knoche, W. Hasselbring, Using microservices sarial networks, in: Proceedings of the 43rd for legacy software modernization, IEEE Software International Conference on Software Engineering, 35 (2018) 44–49. ICSE ’21, IEEE Press, 2021, p. 748–760. URL: [57] H. Knoche, W. Hasselbring, Experience with mi- https://doi.org/10.1109/ICSE43902.2021.00074. croservices for legacy software modernization, Soft- doi:10.1109/ICSE43902.2021.00074. ware Engineering and Software Management 2019 [52] Sketch2React, Build react & html prototypes in (2019). sketch, 2021. URL: http://sketch2react.io. [58] T. Arachchi, Process of Conversion Monolithic Ap- [53] V. P. Sermuga Pandian, S. Suleri, M. Jarke, Synz: plication to Microservices Based Architecture, Ph.D. Enhanced synthetic dataset for training ui element thesis, 2021.