Marc Satkowski

We investigate hybrid user interfaces (HUIs), aiming to establish a cohesive understanding and to adopt consistent terminology for this nascent research area. HUIs combine heterogeneous devices in complementary roles, leveraging the distinct benefits of each. Our work focuses on cross-device interaction between 2D devices and mixed reality environments, which are particularly compelling, leveraging the familiarity of traditional 2D platforms while providing spatial awareness and immersion. Although prior work has prominently explored such HUIs in the context of mixed reality, we still lack a cohesive understanding of the unique design possibilities and challenges of such combinations, resulting in a fragmented research landscape. We conducted a systematic survey and present a taxonomy of HUIs that combine conventional display technology and mixed reality environments. Based on this, we discuss past and current challenges, the evolution of definitions, and prospective opportunities to tie together the past 30 years of research with our vision of future HUIs.

Survey Website

Labels, textual annotations attached to virtual or real-world objects, play a crucial role in Mixed Reality (MR) by providing guidance, instruction, and additional information. However, accurately placing and managing labels in MR environments is a challenging problem. Despite extensive research on label placement algorithms, little attention has been given to adapting MR labeling systems for real-world applications or developing accessible authoring tools for non-technical users. To address this gap, this work introduces considerations and concepts for context-aware labeling that go beyond static annotations, incorporating interactive and dynamic label behaviors that adjust to context and user interactions. Furthermore, an intuitive authoring tool was developed that enables users to confgure and deploy MR labeling experiences without specialized programming knowledge. This work lays a foundation for more accessible, adaptable, and interactive MR labeling systems by combining context-sensitive MR labeling considerations with practical content creation.

Poster | GitHub Repository

Augmented Reality (AR) promises to enhance daily office activities involving numerous textual documents, slides, and spreadsheets by expanding workspaces and enabling more direct interaction. However, there is a lack of systematic understanding of how knowledge workers can manage multiple documents and organize, explore, and compare them in AR environments. Therefore, we conducted a user-centered design study (N = 21) using predefined spatial document layouts in AR to elicit interaction techniques, resulting in 790 observation notes. Thematic analysis identified various interaction methods for aggregating, distributing, transforming, inspecting, and navigating document collections. Based on these findings, we propose a design space and distill design implications for AR document arrangement systems, such as enabling body-anchored storage, facilitating layout spreading and compressing, and designing interactions for layout transformation. To demonstrate their usage, we developed a rapid prototyping system and exemplify three envisioned scenarios. With this, we aim to inspire the design of future immersive offices.

Video Figure | Video Presentation

GitHub Repository

This position paper addresses the fallacies associated with the improper use of affordances in the opportunistic design of augmented reality (AR) applications. While opportunistic design leverages existing physical affordances for content placement and for creating tangible feedback in AR environments, their misuse can lead to confusion, errors, and poor user experiences. The paper emphasizes the importance of perceptible affordances and properly mapping virtual controls to appropriate physical features in AR applications by critically reflecting on four fallacies of facilitating affordances, namely, the subjectiveness of affordances, affordance imposition and reappropriation, properties and dynamicity of environments, and mimicking the real world. By highlighting these potential pitfalls and proposing a possible path forward, we aim to raise awareness and encourage more deliberate and thoughtful use of affordances in the design of AR applications.

In this position paper, we propose researching the combination of Augmented Reality (AR) and Artificial Intelligence (AI) to support conversations, inspired by the interfaces of dialogue systems commonly found in videogames. AR-capable devices are becoming more powerful and conventional in looks, as seen in head-mounted displays (HMDs) like the Snapchat Spectacles, the XREAL glasses, or the recently presented Meta Orion. This development reduces possible ergonomic, appearance, and runtime concerns, thus allowing a more straightforward integration and extended use of AR in our everyday lives, both in private and at work. At the same time, we can observe an immense surge in AI development (also at CHI). Recently notorious Large Language Models (LLMs) like OpenAI's o3-mini or DeepSeek-R1 soar over their precursors in their ability to sustain conversations, provide suggestions, and handle complex topics in (almost) real time. In combination with natural language recognition systems, which are nowadays a standard component of smartphones and similar devices (including modern AR-HMDs), it is easy to imagine a combined system that integrates into daily conversations and provides various types of assistance. Such a system would enable many opportunities for research in AR+AI, which, as stated by Hirzle et al., remains scarce. In the following, we describe how the design of a conversational AR+AI system can learn from videogame dialogue systems, and we propose use cases and research questions that can be investigated thanks to this AR+AI combination.

Haptic feedback devices bear great potential in multiuser Mixed Reality experiences, allowing for multisensory virtual experiences and facilitating interpersonal touch in distributed spaces. However, they also present significant ethical and safety concerns. In this paper, we describe three facets of a serious threat: the transmission of unwanted and harmful haptic stimulation, and propose strategies for developers and designers of Mixed Reality applications to reduce this risk.

In this paper, we present a research platform to support studying collaboration in hybrid and co-located scenarios. Mixed-presence collaboration includes various novel and exciting use cases, such as situated and immersive data analysis by multiple users. However, research in this emerging field is hindered by the technical complexity of the setups and often requires re-implementation of common features. We address this issue by contributing a toolkit and research platform for mixed-presence collaboration that serves as an extensible baseline implementation and enables fast prototyping for user studies in collaborative mixed reality. Furthermore, our platform provides adjustable parameters, such as types of avatars, audio source placement, or the amount of simulated network latency. This way, developers are supported in making design choices regarding typical, re-occurring technical challenges.

GitHub Repository

AR enables combining virtual data spaces with real-world environments through visual augmentations, transforming everyday environments into user interfaces of arbitrary type, size, and content. In the past, the development of AR was mainly technology-driven. This made head-mounted MR devices more common in research, industrial, or personal use cases. However, such devices are always human-centered, making it increasingly important to closely investigate and understand human factors within such applications and environments. The growing data availability, amount, and complexity amplified the need and wish to generate insights through such visualizations. Those, in turn, can utilize human visual perception and ARs natural interactions, the potential to display 3D data, or the stereoscopic display. In my thesis, I aim to deepen the understanding of how AR applications must be designed to optimally adhere to human factors and ergonomics, especially in the area of visual data analysis. To address this challenge, I ground my thesis on three research questions: (1) How can we design such applications in a human-centered way? (2) What influence does the real-world environment have within such applications? (3) How can AR applications be combined with existing systems and devices? To answer those research questions, I explore different human properties and real-world environments that can affect the same environment’s augmentations. For human factors, I investigate the competence in working with visualizations as visualization literacy, the visual perception of visualizations, and physical ergonomics like head movement. Regarding the environment, I examine two main factors: the visual background’s influence on reading and working with immersive visualizations and the possibility of using alternative placement areas in AR. Lastly, to explore future AR systems, I designed and implemented Hybrid User Interfaces and authoring tools for immersive environments. [...]

The recent growth of Artificial Intelligence (AI) based systems considerably widespread their use in many application areas and our daily lives. For instance, AI models are nowadays imbued into web search engines, self-autonomous vehicles, recommendation systems, games, and healthcare. Accordingly, the demand for eXplainable AI (XAI) has risen to help users cope with the growing complexity and opaqueness of the emerging generation of AI models. By allowing users to perceive and make sense of the behaviors and outcomes of such models, XAI enables users of diverse levels of expertise to trust, manage, design, inspect, and develop AI models. Since visualizations are the most common form of explanations, one of the most prominent ways to achieve XAI is by leveraging Visual Analytics (VA). This is also presented by hundreds of examples discussed in recent surveys. VA uses interactive data visualization techniques to support users in understanding, reasoning, and making decisions based on large and complex datasets. However, as the amount and complexity of data on VA tasks increase, a single device can be insufficient to cover the user needs. In response to this, hybrid user interfaces (HUIs) provide a meaningful combination of devices and modalities to increase the quality of interaction and the display possibilities. HUIs allow users to benefit from a multi-device ecology for a given task in a specific context. Furthermore, complementing traditional displays with mixed reality (MR) devices provides advantages such as stereoscopic rendering, unlimited display space, and natural interaction. Considering that XAI deals with large, complex, multi-faceted datasets and given evidence that it can be explored in MR, using hybrid or complementary interfaces for XAI is a natural combination to consider. [...]

Video Presentation

Hybrid User Interfaces (HUIs) describe a combination of several devices into one system. Such a combination promises to alleviate the disadvantages of one device by adding features of another. The term HUIs was first coined by Feiner and Shamash in 1991. Since then, many research projects have focused on exploring possible systems within this interface category, showing the importance and interest of this term. However, the broadness of the follow-up research projects can be seen as “wild growth”, which caused the term HUI to become unspecific and fragmented - as also the call for submission of this workshop states. Additionally, a set of related terms have been used since then, which partially overlap with HUI, like cross-device interfaces or augmented displays. In this position paper, we want to take a closer look at some examples of the current corpus of literature to explore the current HUI term. Then, we present some edge cases and (counter) examples to ground our claim of an “unspecific” definition and highlight where the current definition(s) do not hold or do not contribute to clarity. Lastly, we discuss a few possible refinements to the definition of HUIs, to help sharpen it.

Video Presentation

This paper presents Pearl, a mixed-reality approach for the analysis of human movement data in situ. As the physical environment shapes human motion and behavior, the analysis of such motion can benefit from the direct inclusion of the environment in the analytical process. We present methods for exploring movement data in relation to surrounding regions of interest, such as objects, furniture, and architectural elements. We introduce concepts for selecting and filtering data through direct interaction with the environment, and a suite of visualizations for revealing aggregated and emergent spatial and temporal relations. More sophisticated analysis is supported through complex queries comprising multiple regions of interest. To illustrate the potential of Pearl, we developed an Augmented Reality-based prototype and conducted expert review sessions and scenario walkthroughs in a simulated exhibition. Our contribution lays the foundation for leveraging the physical environment in the in-situ analysis of movement data.

30s Video Teaser | Video Figure | Video Presentation

Appendix | GitHub Repository

Mixed Reality (MR) popularizes numerous situated applications where virtual content is spatially integrated into our physical environment. However, we only know little about what properties of an environment influence the way how people place digital content and perceive the resulting layout. We thus conducted a preliminary study (N = 8) examining how physical surfaces affect organizing virtual content like documents or charts, focusing on user perception and experience. We found, among others, that the situated layout of virtual content in its environment can be characterized by the level of spatial as well as semantic coupling. Consequently, we propose a two-dimensional design space to establish the vocabularies and detail their parameters for content organization. With our work, we aim to facilitate communication between designers or researchers, inform general MR interface design, and provide a first step towards future MR workspaces empowered by blending digital content and its real-world context.

30s Video Teaser | Video Presentation

Poster

Manual point cloud registration is often a crucial step during the mapping of 3D point clouds and usually performed on a conventional desktop setup with mouse interaction. Since 3D point clouds are inherently spatial, these 2D applications suffer from impaired depth perception and inconvenient interaction. Nonetheless, there are few efforts to improve the usability of these applications. To address this, we propose an alternative setup, consisting of a stereoscopic display and an external hand tracker, allowing for enhanced depth perception and natural interaction without the need for body-worn devices or handheld controllers. We developed interaction techniques for point cloud alignment in 3D space, including visual feedback during alignment, and implemented a proof-of-concept prototype in the context of a surgical use case. We describe the use case, design and implementation of our concepts and outline future work. Herewith we provide a user-centered alternative to desktop applications for manual point cloud registration.

30s Video Teaser | Video Presentation

Poster | Supplemental Material

Augmented Reality (AR) interfaces support users by providing access to digital content within real-world environments. However, displaying content at the users’ eye level might result in the occlusion of the real world. Therefore, it requires finding AR content placement areas that free the users’ field of vision. In this work, we systematically investigate two content placement areas beyond the users’ eye level: the ceiling and floor. To understand how potential users perceive virtual content on the ceiling and floor and how the content should be placed on these areas, we conducted two user studies. While the first exploratory study showed the general usefulness of either area, the second quantitative study allowed us to define optimal placement parameters regarding visibility and comfort. With insights from our studies, we provide design recommendations for future AR applications that support 2D content presentation on the ceiling and the floor.

Video Figure | Video Presentation

Appendix | Supplemental Material

The ability to read, understand, and comprehend visual information representations is subsumed under the term visualization literacy (VL). One possibility to improve the use of information visualizations is to introduce adaptations. However, it is yet unclear whether people with different VL benefit from adaptations to the same degree. We conducted an online experiment (n = 42) to investigate whether the effect of an adaptation (here: De-Emphasis) of visualizations (bar charts, scatter plots) on performance (accuracy, time) and user experiences depends on users' VL level. Using linear mixed models for the analyses, we found a positive impact of the De-Emphasis adaptation across all conditions, as well as an interaction effect of adaptation and VL on the task completion time for bar charts. This work contributes to a better understanding of the intertwined relationship of VL and visual adaptations and motivates future research.

Appendix | Supplemental Material

To leverage the full potential of cyber-physical production systems (CPPS) in terms of flexibility and adaptability, the development of such systems must go beyond digitization and modularization. During the engineering and operation of CPPS, humans are essential enablers for the system’s changeability. In this paper, we propose a model of an iterative conducive design process that incorporates perspectives and competencies from several research disciplines such as process control, industrial engineering, computer science, and instructional and cognitive psychology. The goal of this approach is to enhance human-machine interaction and to realize efficient functioning of the system via a combination of the unique potentials provided by humans and the system. The proposed iterative approach is exemplified on a practical level in the engineering of a demonstration plant that tests safety systems for modular plants.

Besides the ability to utilize visualizations, the process of creating and authoring them is of equal importance. However, for visualization environments beyond the desktop, like multi-display or immersive analytics environments, this process is often decoupled from the place where the visualization is actually used. This separation makes it hard for authors, developers, or users of such systems to understand, what consequences different choices they made will have for the created visualizations. We present an extended visualization authoring model for Post-WIMP interfaces, which support designers by a more seamless approach of developing and utilizing visualizations. With it, our emphasis is on the iterative nature of creating and configuring visualizations, the existence of multiple views in the same system, and requirements for the data analysis process.

30s Video Teaser

Poster

Augmented Reality (AR) has been shown to enhance the data visualization and analysis process by supporting users in their immersive exploration of data in a real-world context. However, authoring such visualizations still heavily relies on traditional, stationary desktop setups, which inevitably separates users from the actual working space. To better support the authoring process in immersive environments, we propose the integration of spatially-aware mobile devices. Such devices also enable precise touch interaction for data configuration while lowering the entry barriers of novel immersive technologies. We therefore contribute an initial set of concepts within a scenario for authoring AR visualizations. We implemented an early prototype for configuring visualizations in-situ on the mobile device without programming and report our first impressions.

Video Figure

Poster

We present Marvis, a conceptual framework that combines mobile devices and head-mounted Augmented Reality (AR) for visual data analysis. We propose novel concepts and techniques addressing visualization-specific challenges. By showing additional 2D and 3D information around and above displays, we extend their limited screen space. AR views between displays as well as linking and brushing are also supported, making relationships between separated visualizations plausible. We introduce the design process and rationale for our techniques. To validate Marvis’ concepts and show their versatility and widespread applicability, we describe six implemented example use cases. Finally, we discuss insights from expert hands-on reviews. As a result, we contribute to a better understanding of how the combination of one or more mobile devices with AR can benefit visual data analysis. By exploring this new type of visualization environment, we hope to provide a foundation and inspiration for future mobile data visualizations.

30s Video Teaser | Video Figure | Video Presentation

Appendix

In this work we report on two comprehensive user studies investigating the perception of Augmented Reality (AR) visualizations influenced by real-world backgrounds. Since AR is an emerging technology, it is important to also consider productive use cases, which is why we chose an exemplary and challenging industry 4.0 environment. Our basic perceptual research focuses on both the visual complexity of backgrounds as well as the influence of a secondary task. In contrast to our expectation, data of our 34 study participants indicate that the background has far less influence on the perception of AR visualizations. Moreover, we observed a mismatch between measured and subjectively reported performance. We discuss the importance of the background and recommendations for visual real-world augmentations. Overall, our results suggest that AR can be used in many visually challenging environments without losing the ability to productively work with the visualizations shown.

30s Video Teaser | Video Figure | Video Presentation

Study Material | Study Data | Statistical Analysis

The research field of augmented reality (AR) is of increasing popularity, as seen, among others, in several recently published surveys. To produce further advancements in AR, it is not only necessary to create new systems or applications, but also to evaluate them. One important aspect in regards to the evaluation is the general understanding of how users experience a given AR application, which can also be seen by the increased number of papers focusing on this topic that were published in the last years. With the steadily growing understanding and development of AR in general, it is only a matter of time until AR devices make the leap into the consumer market where such an in-depth user understanding is even more essential. Thus, a better understanding of factors that could influence the design and results of user experience studies can help us to make them more robust and dependable in the future. In this position paper, we describe three challenges which researchers face while designing and conducting AR users studies. We encountered these challenges in our past and current research, including papers that focus on perceptual studies of visualizations, interaction studies, and studies exploring the use of AR applications and their design spaces.

Video Presentation

Large high-resolution displays (LHRD) can benefit information visualization due to their size and resolution. As users often move back and forth in front of the display, we need to investigate what tasks should be supported from varying distances. In this work, we combine touch interaction on a LHRD with casual, ‘eyes-free’ device interaction using smartphones from a distance. We describe early insights and discuss important questions based on experiences with our prototype implementation.