Periodic Reporting for period 4 - COMPUTED (Computational User Interface Design)
Reporting period: 2019-10-01 to 2020-03-31
Despite extensive research on human-computer interaction (HCI), the problem of how to best design usable user interfaces (UIs) is unsolved. That is, no method exists that guarantees the optimal or even a provably good design. Instead, idea-generation in UI design is driven by heuristics and iterative improvements informed by empirical data collection. This heuristic–iterative approach is costly and at times ineffective, because UI design often involves very large design spaces with multiple objectives and constraints and complex user behavior.
COMPUTED aims to establish the foundations for solving UI design problems by combinatorial optimization methods that deploy mathematical models of user behavior as objective functions. Given objectives and constraints, a UI is automatically optimized. Previous work in UI optimization shows significant improvements to usability, but the scope has been restricted to virtual keyboards and widget layouts. COMPUTED researches methods that could vastly expand the scope and permit solutions to any well-defined UI design problem. First, objective functions are currently limited to models of sensorimotor performance. COMPUTED develops algorithmic support for acquiring more comprehensive models that cover the main human factors. Second, current work has formally defined only one UI optimization problem, the letter assignment problem. To combat a more relevant set of design problems with appropriate optimization methods, COMPUTED formally analyzes recurring design problems. Third, previous work has followed the “fire-and-forget” approach where the problem is completely predefined for an optimizer. COMPUTED develops a novel interactive UI optimization paradigm that promotes fast convergence to good results even in the face of uncertainties and incomplete preknowledge. The novel capabilities are demonstrated in four hard cases: 1) universal keyboard layout, 2) web applications, 3) hand gesture input, and 4) interactive dashboards in cars.
COMPUTED aims to establish the foundations for solving UI design problems by combinatorial optimization methods that deploy mathematical models of user behavior as objective functions. Given objectives and constraints, a UI is automatically optimized. Previous work in UI optimization shows significant improvements to usability, but the scope has been restricted to virtual keyboards and widget layouts. COMPUTED researches methods that could vastly expand the scope and permit solutions to any well-defined UI design problem. First, objective functions are currently limited to models of sensorimotor performance. COMPUTED develops algorithmic support for acquiring more comprehensive models that cover the main human factors. Second, current work has formally defined only one UI optimization problem, the letter assignment problem. To combat a more relevant set of design problems with appropriate optimization methods, COMPUTED formally analyzes recurring design problems. Third, previous work has followed the “fire-and-forget” approach where the problem is completely predefined for an optimizer. COMPUTED develops a novel interactive UI optimization paradigm that promotes fast convergence to good results even in the face of uncertainties and incomplete preknowledge. The novel capabilities are demonstrated in four hard cases: 1) universal keyboard layout, 2) web applications, 3) hand gesture input, and 4) interactive dashboards in cars.
"The project has develop new methodology to support model-based combinatorial optimization. This work consists of three four main activities:
1. Mathematical definition of design problems and their efficient solution by known means in combinatorial optimization. In 2015-2017, we have defined several common UI problems not defined before, in particular in menu optimization, functionality selection, keyboard design, biomechanical design, and functionality selection. Some problems are defined at the level of a decision problem, which then can be solved using black-box optimizers. Others are defined using integer programming and permit the computation of guarantees for solution quality. We have also defined and presented first real-time optimizers for web page sketches and started to work on deep learning based optimization of web page layouts.
2. Interactive optimization in design. In 2015-2017, we have defined new methods that base on real-time optimization, robust optimization, and machine learning. They allow supporting the designer while avoiding overloading and providing sufficient means for steering optimization. COMPUTED has developed the first zero-effort approach to interactive optimization. This means that the designer's goal is inferred automatically and used to generate a diverse set of ideas on on alternative designs. Presently, we are working on the problem of how to infer regions of a graphical UI from its image only. This will help generalizing the use of these approaches beyond sketching to, for example, web pages etc.
3. Modeling of interactive behavior: We have advanced modeling of interactive behavior, in particular in the area of computational rationality. In this approach, we defined bounds/capacity limitations of humans and estimate the best achievable performance with a candidate design using Q-learning or other reinforcement learning approaches. This has been thus far applied in menus, information search, and visual layouts. One fundamental problem the project has looked at is how to infer models of users from realistic data. To this end, we have worked on inverse modeling of computational rationality models. The idea is to use ABC (approximate bayesian computation) to infer model parameters from naturalistic user data, such as clickstreams. We showed that parameters of the human sensorimotor systems can be obtained like this in menu interaction. This will enable an optimizer to learn models of users without running experiments. These models can then be used to optimize or adapt the UI for that user group.
4. Demonstrators: Using results from #1-#3, we have shown new results in challenging UI design problems from keyboards to menus and over to functionality selection, gestural input, web layouts, and biomechanics.
"
1. Mathematical definition of design problems and their efficient solution by known means in combinatorial optimization. In 2015-2017, we have defined several common UI problems not defined before, in particular in menu optimization, functionality selection, keyboard design, biomechanical design, and functionality selection. Some problems are defined at the level of a decision problem, which then can be solved using black-box optimizers. Others are defined using integer programming and permit the computation of guarantees for solution quality. We have also defined and presented first real-time optimizers for web page sketches and started to work on deep learning based optimization of web page layouts.
2. Interactive optimization in design. In 2015-2017, we have defined new methods that base on real-time optimization, robust optimization, and machine learning. They allow supporting the designer while avoiding overloading and providing sufficient means for steering optimization. COMPUTED has developed the first zero-effort approach to interactive optimization. This means that the designer's goal is inferred automatically and used to generate a diverse set of ideas on on alternative designs. Presently, we are working on the problem of how to infer regions of a graphical UI from its image only. This will help generalizing the use of these approaches beyond sketching to, for example, web pages etc.
3. Modeling of interactive behavior: We have advanced modeling of interactive behavior, in particular in the area of computational rationality. In this approach, we defined bounds/capacity limitations of humans and estimate the best achievable performance with a candidate design using Q-learning or other reinforcement learning approaches. This has been thus far applied in menus, information search, and visual layouts. One fundamental problem the project has looked at is how to infer models of users from realistic data. To this end, we have worked on inverse modeling of computational rationality models. The idea is to use ABC (approximate bayesian computation) to infer model parameters from naturalistic user data, such as clickstreams. We showed that parameters of the human sensorimotor systems can be obtained like this in menu interaction. This will enable an optimizer to learn models of users without running experiments. These models can then be used to optimize or adapt the UI for that user group.
4. Demonstrators: Using results from #1-#3, we have shown new results in challenging UI design problems from keyboards to menus and over to functionality selection, gestural input, web layouts, and biomechanics.
"
Results from the first period of COMPUTED show that combinatorial UI optimization offers a strong complement to existing design methods. The results have shown formalizations of design problems that allow applying combinatorial optimization methods. This can significantly increase the cost-efficiency of UI design and improve the outputs. COMPUTED has shown that these methods support even novel, ill-defined design problems, where they support designers by allowing them to delegate combinatorially hard sub-problems to a computer. It supports decision-making and offers a objective criteria that can prevent costly and futile iteration. On the applied side, we have worked with AFNOR, a French standardization agency, to apply our methods to the redesign of the Azerty keyboard layout, and in particular to improve its performance and usability by remapping the special characters. This practical problem required advances in modelling shortcut behavior and ergonomics, and solving a large (quadratic assignment) problem. We have found further improvements to state-of-the-art QAP algorithms. We have also presented several breakthrough results in gestural interaction, including the first model-based approach to gesture optimization, as well as optimization approaches to real-time hand tracking. Another application worth mentioning is scatterplot optimization: we apply models of human perception to tune the design of scatterplot for data analysis tasks such as correlation estimation and cluster detection. It is the first time models of human perception are used for this purpose.