diff --git a/docs/_data/navigation.json b/docs/_data/navigation.json index 2eb6266e..0881e69e 100644 --- a/docs/_data/navigation.json +++ b/docs/_data/navigation.json @@ -457,6 +457,81 @@ } ] }, + { + "name": "Case Studies", + "path": "/knowledge/case-studies/", + "file": "overview.html", + "type": "page", + "content": + [ + { + "name": "DepthTouch (2012)", + "path": "/knowledge/case-studies/", + "file": "depthtouch.html", + "type": "page" + }, + { + "name": "FlexiWall (2014)", + "path": "/knowledge/case-studies/", + "file": "flexiwall.html", + "type": "page" + }, + { + "name": "DEEP (2015)", + "path": "/knowledge/case-studies/", + "file": "deep.html", + "type": "page" + }, + { + "name": "Interactive Animations (2016)", + "path": "/knowledge/case-studies/", + "file": "interactive-animations.html", + "type": "page" + }, + { + "name": "Zoomable Product Browser (2017)", + "path": "/knowledge/case-studies/", + "file": "zoomable-product-browser.html", + "type": "page" + }, + { + "name": "Glyphboard (2017)", + "path": "/knowledge/case-studies/", + "file": "glyphboard.html", + "type": "page" + }, + { + "name": "DeepZoom (2018)", + "path": "/knowledge/case-studies/", + "file": "deep-zoom.html", + "type": "page" + }, + { + "name": "DisPlay (2018)", + "path": "/knowledge/case-studies/", + "file": "display.html", + "type": "page" + }, + { + "name": "Construction Progress Visualization (2020)", + "path": "/knowledge/case-studies/", + "file": "bim.html", + "type": "page" + }, + { + "name": "Layers (2022)", + "path": "/knowledge/case-studies/", + "file": "layers.html", + "type": "page" + }, + { + "name": "Other Prototypes", + "path": "/knowledge/case-studies/", + "file": "other-prototypes.html", + "type": "page" + } + ] + }, { "name": "Models", "path": "/knowledge/scientific/", diff --git a/docs/assets/img/kb/case-studies/bim_title.jpg b/docs/assets/img/kb/case-studies/bim_title.jpg new file mode 100755 index 00000000..d6eb1ce6 Binary files /dev/null and b/docs/assets/img/kb/case-studies/bim_title.jpg differ diff --git a/docs/assets/img/kb/case-studies/others-escape-room.png b/docs/assets/img/kb/case-studies/others-escape-room.png new file mode 100755 index 00000000..bbb75412 Binary files /dev/null and b/docs/assets/img/kb/case-studies/others-escape-room.png differ diff --git a/docs/assets/img/kb/case-studies/others-flightsim.jpg b/docs/assets/img/kb/case-studies/others-flightsim.jpg new file mode 100755 index 00000000..b11919a1 Binary files /dev/null and b/docs/assets/img/kb/case-studies/others-flightsim.jpg differ diff --git a/docs/assets/img/kb/case-studies/others-mireflex.jpg b/docs/assets/img/kb/case-studies/others-mireflex.jpg new file mode 100755 index 00000000..b6b08ea1 Binary files /dev/null and b/docs/assets/img/kb/case-studies/others-mireflex.jpg differ diff --git a/docs/knowledge/case-studies/bim.md b/docs/knowledge/case-studies/bim.md index e69de29b..cc6797a3 100644 --- a/docs/knowledge/case-studies/bim.md +++ b/docs/knowledge/case-studies/bim.md @@ -0,0 +1,164 @@ +--- +title: "Case Study: Construction Progress Visualization" +--- + +# {{ page.title }} + + +## Table of Contents + +1. [Research Questions](#research-questions) +2. [Features and Limitations](#features-and-limitations) +3. [Scenario](#scenario) +4. [Findings and Discussion](#findings-and-discussion) + +![Case Study: Construction Progress Visualization]({{ site.baseurl }}/assets/img/kb/case-studies/bim_title.jpg){:.content__title-image} + + +| Profile field | Description | +| ------------- | ----------- | +| Title | Layered Maps | +| Year | 2020 | +| Focus | Concept, Implementation | +| Format | Elastic Display wall | +| Framework | dSense | +| Sensor | Microsoft Kinect 2 | +| Architecture | Client-server (.NET Core / WPF) | +| Scenario | Construction progress visualization | +| Features | - combining layered data with zoomable user interfaces
- mode switching through an external foot switch
- gesture development | +| Publication | *Müller, M., Lier, E., Groh, R. & Hannß, F.* (2020): **A Tangible Concept for Layered Map Visualizations: Supporting on-site Civil Engineering Construction Consultations using Elastic Displays**. In: C. Hansen, A. Nürnberger & B. Preim (Eds.): Mensch und Computer 2020 - Workshopband. Bonn, Germany: Gesellschaft für Informatik e. V. (GI). DOI: [10.18420/muc2020-ws121-368](https://doi.org/10.18420/muc2020-ws121-368). | + +The eighth case study explored **combining zoomable maps and layered information for construction planning and consultation**. The profile describes the historical dSense implementation, following the conventions in the [case study overview](overview.md#case-study-profiles). The source does not list an associated repository. + +## Research Questions + +Earlier studies suggested that layered content and zoomable user interfaces could complement one another. [DeepZoom](deep-zoom.md) had taken an initial step by adding information overlays to magnified images. Layered Maps investigated this combination in a practical civil-engineering scenario, using **zoom and layer navigation as separate interaction modes**. + +The study addressed four main questions: + +- How can users switch smoothly between magnifying a map and exploring its semantic layers? +- How can a selected view persist across mode changes, and what happens when its virtual state no longer corresponds to the physical deformation? +- Does the surface's return to rest still provide a meaningful natural reset when the application retains earlier interaction states? +- How can maps, schedules, and cost information support discussion of construction progress and the resolution of conflicts between dependent tasks? + +A retained maximum zoom level on an undeformed surface illustrated the tension between physical and virtual state. At the same time, retaining a view could let users step back from the wall and inspect it without continuing to press. + +The scenario was informed by a detailed requirements and data analysis. The analysis and initial concepts were discussed with a domain expert to align the design with civil-engineering problems. + +**[⬆ back to top](#table-of-contents)** + +## Features and Limitations + +### Extending the Existing Zoom Client + +The implementation followed DeepZoom's **WPF/.NET client-server approach**, using dSense and a Microsoft Kinect 2 to capture surface deformation. Only minor adaptations were made to the framework and server. Most changes were on the client, where the existing zoom implementation was extended to handle layered data. + +An external **foot switch** provided a dedicated mode-switching input. Alternatives used a specific surface gesture or the number of detected contacts. Gesture recognition took place **on the client**, so the case study did not establish a general gesture-recognition component in the framework. + +### Context-Dependent Depth Interaction + +Deformation depth controlled either the map's magnification or the selected data layer, depending on the active mode. Zooming and panning affected the full map; layer navigation used a local **magic lens**. The lens supplied feedback about the active mode and context, including the zoom level and/or current process layer. + +The two-hand lens used a fixed size. Early trials found that continuously changing its diameter while users pressed the surface interfered with layer selection and made the lens appear unstable. Alternative size controls remained proposals for further development. + +### Scope of the Application + +The study focused on construction execution, particularly progress monitoring, schedule updates, and cost control within a Building Information Modeling (BIM) context. It developed a map-based visualization and interaction concept rather than addressing the full project lifecycle. + +Its layer model related construction phases to their physical depth. This suited the chosen civil-engineering scenario, but constrained transfer to projects with simultaneous work at different heights. The source describes map interaction, gesture development, and planning visualizations; the reported comparative user test specifically concerns mode switching. + +**[⬆ back to top](#table-of-contents)** + +## Scenario + +### Supporting Construction Consultations + +The application was intended to help participants communicate project status, examine dependent processes, and negotiate responses to delays or conflicts. The **construction plan occupied the centre of the display**, with compact schedule and cost summaries along the top and left edges. Expanding these summaries provided access to detailed planning views. + +The visualization concept combined semantic zoom, interactive lenses, and an extended Gantt chart. The **PlanningLineGlyph** approach provided a basis for representing task hierarchies, dependencies, temporal tolerances, shifts, and delays. + +### Navigating Maps and Construction Layers + +Semantic zoom kept the initial map uncluttered and progressively revealed more detail as users pressed into the surface. Pressing near the map's edges panned the view. + +Semantic layers filtered the map by construction process. In the selected scenario, work in deeper ground layers generally preceded work above them; completed layers were sealed before subsequent work began. The ordering of underground services, road construction, and landscaping therefore allowed physical depth and process sequence to share a layer representation. + +Full-map navigation and local layer inspection were separated visually: the map could be magnified and moved, while the magic lens exposed a selected process layer within its surrounding context. Deformation depth selected the layer displayed inside the lens. + +Three ways of switching between these interactions were explored: + +| Mode-switching variant | Interaction | +| ---------------------- | ----------- | +| **Foot switch** | Use a separate physical control to switch between map and lens interaction. | +| **Number of contacts** | Navigate the map with one hand; adding a second hand opens the layer lens between the hands and redirects input to layer navigation. | +| **Together-and-Apart** | Bring the hands together and move them apart again within a short interval, using shallow deformation, to open or close the lens. | + +### Schedule and Cost Views + +The compact timeline showed deviations from the original schedule for individual tasks and the project phase as a whole. It also related current progress to the planned state and showed a forecast completion date. **Blue indicated time savings; red indicated delays.** Start and end markers restricted the period shown in the map. + +Pressing the timeline expanded it into a **hierarchical Gantt chart with three levels**, allowing processes to be expanded or collapsed. Dependencies were represented using the critical-path method. PlanningLineGlyph features communicated temporal tolerances and changes, including critical changes that caused dependency conflicts or delayed the overall project. + +Pressing a task provided semantic zoom through three information levels: + +| Detail level | Information shown | +| ------------ | ----------------- | +| Default | Task identifier and name | +| Intermediate | A short label, quantities such as material or working hours, or prices | +| Maximum | A detailed description | + +The cost concept followed a similar structure with a vertical layout and cost bars. Its overview compared planned and actual costs, task-level and overall savings or overruns, and the projected total. Map elements likewise used blue for savings or earlier completion and red for additional costs or delays. + +Cost ranges were omitted because, in the scenario, these figures were not normally communicated between clients and contractors. Interaction development concentrated on schedule adjustment; the cost overview supported assessment of alternative conflict resolutions. + +### Manipulating Tasks and Resetting the Application + +The **Shift gesture** adapted the two-contact translation concept from [Glyphboard](glyphboard.md). Contacts near a task bar were interpreted according to their positions relative to it: + +- **Move a task:** Press on either side of the bar. It moves towards the deeper contact, as though sliding into the depression. +- **Change its duration:** Place one hand on the bar to fix that side. Pressing with the other hand extends the bar towards that hand; pulling outward shortens it. + +This context-sensitive interpretation distinguished moving an entire interval from changing its length. A separate **Pull-and-Release gesture**, implemented by pulling the surface outward and releasing it shortly afterwards, reset the application. + +**[⬆ back to top](#table-of-contents)** + +## Findings and Discussion + +### Comparing Mode-Switching Techniques + +The source reports a user test documented in *Lier (2019, pp. 146 ff.)*. The foot switch showed **significant advantages over Together-and-Apart** in task completion, completion time, usability, and user experience. Differences between the foot switch and the contact-count-based two-hand lens were smaller and **not statistically significant in that test**. + +The excerpt provides no participant count, effect sizes, or detailed statistics. The result therefore supports a comparison of the tested techniques without establishing a general superiority of external controls or an evaluation of the entire planning workflow. + +### Persistent Views and Physical State + +A dedicated mode switch offered a further benefit: retaining the current view allowed users to step away from the wall and see the complete visualization. This addressed the difficulty of inspecting a large display while holding a deformation. + +Persistence also complicated the direct correspondence between surface shape and application state. Returning to a saved view could restore a zoom level that the current deformation did not express. The study raised this issue alongside the limits of the natural reset and introduced an explicit reset gesture, but did not establish a general solution for reconciling physical and retained virtual states. + +### Two-Hand Roles and Collaborative Use + +Using contact count to select a mode constrained collaboration. Detected pressure points could not be assigned explicitly to individual users, so simultaneous input from several people automatically activated the lens. This prevented other combinations of collaborative actions, such as zooming and marking. + +Both user tests and observations indicated that future concepts should consider **distinct roles for the two hands**. The number of detected contacts alone did not reliably express users' intended tasks, especially when several people shared the display. + +### Lens Size and Independent Spatial Navigation + +Users repeatedly requested adjustable lens size. Two alternatives were proposed to avoid the instability of continuous resizing: switching between predefined sizes only after a substantial change in hand spacing, or setting the diameter from the initial distance between the hands when entering lens mode. Neither was reported as the implemented solution in this study. + +The coupling of construction phase and physical depth also had a clear boundary. Multi-storey buildings can involve different trades working simultaneously on different floors. Supporting such projects would require **spatial navigation independent of process sequence**, extending the chosen layer concept. + +### Implications for Framework Development + +For the forthcoming [requirements analysis](../../software/repo/requirements.md), the study provides evidence for the following areas: + +- **Combining input sources:** Allow surface interaction to work alongside external controls such as a foot switch. +- **Application-specific gesture interpretation:** Expose multiple contact positions and deformation depths so clients can interpret gestures according to their current mode and target objects. +- **Explicit interaction state:** Support clear mode feedback, retained views, and deliberate reset behaviour when releasing the surface no longer restores the complete initial state. +- **Two-hand and collaborative interaction:** Account for contact roles and the ambiguity between one person's two hands and input from several people. +- **Stable parameter control:** Separate layer selection from lens-size adjustment when simultaneous mappings interfere with one another. +- **Independent data dimensions:** Avoid assuming that spatial depth and temporal process order always coincide when generalizing layered navigation. + +These implications connect the prototype and its observed limitations to later design decisions. They do not imply that framework-level gesture recognition, user identification, or independent spatial and temporal navigation were implemented in this case study. + +**[⬆ back to top](#table-of-contents)** diff --git a/docs/knowledge/case-studies/deep-zoom.md b/docs/knowledge/case-studies/deep-zoom.md index e69de29b..2c2e4966 100644 --- a/docs/knowledge/case-studies/deep-zoom.md +++ b/docs/knowledge/case-studies/deep-zoom.md @@ -0,0 +1,146 @@ +--- +title: "Case Study: DeepZoom - Gigapixel Visualizations" +--- + +# {{ page.title }} + + +## Table of Contents + +1. [Research Questions](#research-questions) +2. [Features and Limitations](#features-and-limitations) +3. [Scenario](#scenario) +4. [Findings and Discussion](#findings-and-discussion) + +| Profile field | Description | +| ------------- | ----------- | +| Title | DeepZoom | +| Year | 2018 | +| Focus | Integration, Framework | +| Format | Elastic Display wall and tabletop | +| Framework | dSense, ReFlex | +| Sensor | Intel RealSense | +| Architecture | Client-server (.NET/WPF), communicating via TCP | +| Scenario | Visualization and exploration of gigapixel imagery | +| Features | - magic lens for magnification
- additional information layers during zoom
- efficient display of high-resolution images | +| Publication | *Müller, M., Lier, E. & Gründer, T.* (2018): **Zoomable User Interfaces für Elastic Displays**. In: R. Dachselt & G. Weber (Eds.): Mensch und Computer 2018 - Workshopband. Bonn, Germany: Gesellschaft für Informatik e. V. (GI). DOI: [10.18420/muc2018-ws05-0502](https://doi.org/10.18420/muc2018-ws05-0502). | +| Repository | [reflex-deepzoom](https://github.com/visualengineers/reflex-deepzoom) | + +The sixth case study explored **zoomable image visualizations across multiple application contexts**. The profile covers the historical dSense and ReFlex iterations, following the conventions in the [case study overview](overview.md#case-study-profiles). The published repository may include later adaptations. Although gigapixel imagery motivated the study, the implementation's image-size limit kept the displayed images below one gigapixel. + +## Research Questions + +DeepZoom continued the investigation of zoomable user interfaces begun with [Glyphboard](glyphboard.md). Glyphboard concentrated on one information-visualization scenario and continuous movement through a data space. DeepZoom broadened the application contexts and explored other ways of relating an overview to a magnified detail view. + +The study addressed four main questions: + +- How can surface deformation control magnification in high-resolution images, using alternatives to continuous movement through a data space? +- How can local lenses and full-screen views support navigation and preserve spatial orientation? +- How can images from different sources be prepared and displayed efficiently, and which limits arise from their representation and storage? +- How can magnification reveal additional information or alternative image layers, combining geometric and semantic zoom? + +Interactive lenses offered a central alternative to immersive navigation. Like a magnifying glass on a map table, a lens could expose detail while preserving the surrounding overview. + +**[⬆ back to top](#table-of-contents)** + +## Features and Limitations + +### Separating Tracking and Visualization + +The implementation used **.NET and WPF** in a client-server architecture. Despite sharing the same technology platform, the visualization client remained strictly separate from the server responsible for hardware access and surface-interaction detection. The applications exchanged data through a **TCP interface**, and the original prototype captured surface deformation with an Intel RealSense sensor. + +This continued Glyphboard's separation of tracking from application logic, while applying it to a native desktop visualization client. + +### Mapping Deformation to Magnification + +The visualization combined two images: a preview at the application's native resolution, typically **1920 × 1080 pixels**, and a high-resolution detail image. The magnified view was centred on the detected finger position. Deformation depth determined how far the detail image was scaled down from its full resolution: maximum deformation exposed the full available detail, while shallower deformation reduced magnification. + +Depth therefore controlled the **magnification level directly**, whereas Glyphboard used depth to control zoom speed. DeepZoom implemented both full-screen magnification and a local lens with a configurable offset from the contact position. A mini-map provided overview context in the full-screen presentation. + +An optional third image supplied annotations. It matched the detail image's resolution and was composited through its alpha channel. In the final iteration, the overlay's overall opacity also depended on magnification, allowing annotations to appear gradually. + +### Image Size and Memory Limits + +The implementation stored pixel colour values in a byte array. The source reports a limit of **2,147,483,591 elements**, corresponding to approximately **30,893 × 17,377 pixels** for a 16:9 image with four RGBA channels. These dimensions remained below one gigapixel but provided approximately **16-fold magnification** relative to the preview, which was sufficient to demonstrate the interaction concepts. + +This was a limit of the implementation described in the study. Loading complete detail images also constrained possible extensions: in that implementation, a second independent lens would load another copy of the entire detail image into memory. Effective multi-touch use and collaborative viewing with multiple lenses were not addressed. + +**[⬆ back to top](#table-of-contents)** + +## Scenario + +### Lens Placement and Overview Context + +The study considered five presentation concepts, illustrated in Figure 5-39 of the source: + +| Presentation | Relationship between detail and context | +| ------------ | --------------------------------------- | +| **Lens at the contact position** | Keeps the magnified region close to its original location, but the hand obscures content and deformation introduces projection distortion. | +| **Fisheye view** | Could blend magnification into its surroundings according to the actual surface deformation; the benefit of this direct mapping remained an open question. | +| **Offset lens** | Reduces hand occlusion and deformation-related projection distortion, but makes it harder to relate enlarged elements to their surroundings. | +| **Dedicated detail region** | Would reserve a separate area for magnification, combining aspects of local lenses and full-screen zoom. | +| **Full-screen view with mini-map** | Uses the main display for detail while retaining a small overview for orientation. | + +The implemented modes were **a lens with configurable offset and full-screen zoom**. The fisheye and dedicated detail-region concepts were alternatives considered during design, rather than implemented results reported by the study. + +### Panoramas, Maps, and Event Plans + +Several kinds of content tested the same interaction approach: + +- **Astronomical imagery:** A high-resolution view of the Milky Way from ESA's Gaia project supported exploration of large-scale structure and local detail. The source credits ESA/Gaia/DPAC and identifies the image licence as CC BY-SA 3.0 IGO. +- **City panoramas:** A Dresden panorama assembled from photographs taken for the project provided a familiar setting for inspecting buildings and landmarks. +- **Maps and aerial imagery:** Pre-rendered Google Maps satellite images at different resolutions and historical aerial photographs of Dresden from 1953 connected a city-wide overview with individual streets and buildings. +- **Event plans:** Exhibition plans supplied as vector graphics were used for OUTPUT and Fit4Congress. The overview showed room and stand silhouettes; the detailed image added room names and event information. + +The event plans combined **geometric zoom**, enlarging the image, with **semantic zoom**, revealing additional content. They suggested an information-terminal scenario in which visitors could explore both a venue's layout and its programme. However, embedding all labels in the detail image made small text appear at low magnification, where it was difficult to read and could distract from the overview. + +### Comparing Different Image Layers + +Another variation returned to the Ghent Altarpiece restoration example explored in [FlexiWall](flexiwall.md#painting-and-image-analysis). Here, overview and detail represented different states of the same artwork: the base image showed the inner altarpiece before restoration, while the detail image showed it afterwards. + +The lens made colour changes and alterations to pictorial elements visible within the surrounding pre-restoration image. This demonstrated a connection between zoomable interfaces and layered content: magnification could reveal another state of an object as well as more pixels. + +### Gradually Revealing Annotations + +The final iteration addressed the readability problem by separating annotations from the underlying detail image. A semi-transparent information layer became progressively more opaque as magnification increased, keeping fine detail unobtrusive at low zoom levels. + +For the Dresden panorama, the overlay labelled significant buildings at their image positions, resembling an augmented-reality view. The same approach was tested with other panoramas and with traffic information overlaid on maps. It worked in both the full-screen and lens presentations. + +**[⬆ back to top](#table-of-contents)** + +## Findings and Discussion + +### Combining Zoom and Information Layers + +The source reports demonstration feedback and implementation experience, without a participant count or quantitative evaluation. Its findings therefore provide exploratory design insights. + +The restoration comparison and fading annotations demonstrated two ways to enrich zoomable images with additional information. Maps could support multiple semantic overlays, while restoration imagery suggested opportunities to explain or document changes to an artwork. Developing interaction concepts that combine zoom with navigation through several information layers remained a central research question. + +Lens placement also exposed a practical trade-off: keeping detail at its source position supported spatial context, while moving it away reduced hand occlusion and projection distortion. The appropriate presentation depended on the content and viewing task. + +### Navigation and Feed-Forward + +Feedback on map exploration identified **panning** as a natural extension. Glyphboard's two-contact panning gesture was a candidate, but efficient loading of additional image data would also need to be addressed. + +A **hover effect** inspired by Glyphboard was proposed to provide feed-forward before zooming. This raised a content-design question: which information should appear as an overlay during magnification, and which should be shown as a separate hover annotation? Neither panning nor hover was reported as an implemented DeepZoom capability. + +### Multi-Touch and Collaboration + +Collaborative use suggested a second zoom lens, but duplicating complete detail images made this technically demanding. Other proposed uses of a second hand included controlling lens size or overlay opacity, navigating between information layers, or revealing an overlay during full-screen zoom, potentially as a local or context-sensitive lens. + +These proposals expanded the possible interaction vocabulary without establishing an implemented multi-touch solution. They also showed that application memory management would need to evolve alongside interaction capabilities. + +### Implications for Framework Development + +For the forthcoming [requirements analysis](../../software/repo/requirements.md), the study provides evidence for the following areas: + +- **Independent tracking and application layers:** Maintain a clear communication boundary so that surface input can drive a separately implemented visualization client. +- **Continuous parameter mappings:** Make position and deformation depth available for direct control of magnification, alongside mappings to speed such as those used in Glyphboard. +- **Configurable presentation and feedback:** Allow applications to adjust lens offset and provide overview context while accounting for occlusion and projection distortion. +- **Layered content and annotation visibility:** Support application concepts that relate magnification to alternative imagery and gradual information disclosure. +- **Resource-aware interaction design:** Account for image-size limits, memory duplication, and data-loading costs when extending navigation or adding simultaneous lenses. +- **Extensible gesture interpretation:** Provide a basis for exploring panning and second-hand controls while keeping proposed gestures distinct from demonstrated functionality. + +These implications are derived from the prototype's capabilities and limitations. They connect the case study to later framework decisions without implying that image streaming, multiple lenses, or general navigation through information layers were implemented here. + +**[⬆ back to top](#table-of-contents)** diff --git a/docs/knowledge/case-studies/display.md b/docs/knowledge/case-studies/display.md index e69de29b..89b92fda 100644 --- a/docs/knowledge/case-studies/display.md +++ b/docs/knowledge/case-studies/display.md @@ -0,0 +1,148 @@ +--- +title: "Case Study: DisPlay - Game Concepts" +--- + +# {{ page.title }} + + +## Table of Contents + +1. [Research Questions](#research-questions) +2. [Features and Limitations](#features-and-limitations) +3. [Scenario](#scenario) +4. [Findings and Discussion](#findings-and-discussion) + +| Profile field | Description | +| ------------- | ----------- | +| Title | DisPlay | +| Year | 2018 | +| Focus | Implementation, Integration | +| Format | Elastic Display tabletop | +| Framework | dSense | +| Sensor | Microsoft Kinect 2 | +| Architecture | Client-server (.NET/WPF + Unreal Engine 4) | +| Scenario | Game concepts for physics-based interaction | +| Features | - plugin for Unreal Engine 4
- exhibition operation
- development tools | + +The seventh case study explored **surface deformation as a means of influencing virtual environments through physical forces**. The profile describes the historical dSense implementation, following the conventions in the [case study overview](overview.md#case-study-profiles). The source does not list an associated publication or repository. + +## Research Questions + +DisPlay returned to the playful exploration of physical effects introduced by [DepthTouch](depthtouch.md). Studies such as [DEEP](deep.md) and the [Zoomable Product Browser](zoomable-product-browser.md) had mapped simulated forces to abstract actions such as selection and filtering. DisPlay instead investigated how deformation could influence objects in a virtual environment, ideally making simple physical principles understandable through interaction. + +The study addressed four main questions: + +- How can gravity, friction, inertia, velocity, and acceleration support interaction in spatial environments? +- Does the simulation require an accurate vector field derived from the surface relief, or can detected deformation extrema provide a sufficient approximation? +- How can the framework integrate with a game engine for complex applications beyond web visualizations? +- Which development tools, setup procedures, and user guidance are needed for an application intended for public exhibitions? + +The tabletop format connected pressing into a horizontal surface with gravity-like attraction. The scenarios were designed to support collaboration around the table while remaining usable by one person. + +**[⬆ back to top](#table-of-contents)** + +## Features and Limitations + +### Integrating Unreal Engine 4 + +The implementation combined a **.NET/WPF server using dSense** with a client built in **Unreal Engine 4**. A plugin connected detected depth interactions to the engine's integrated physics simulation. Microsoft Kinect 2 captured surface deformation, and multi-touch processing was optimized for stability and performance: an immediate response was important for making the relationship between deformation and physical effects understandable. + +Communication used **WebSockets**, with an additional intermediary layer to support the **Socket.IO** interface. This extended the separation of tracking and application logic explored in [Glyphboard](glyphboard.md) and [DeepZoom](deep-zoom.md) to a game-engine client. + +### Approximating Deformation with Point Forces + +The source reports that the Unreal Engine integration did not offer a native way to feed dynamic vector fields into the physics simulation. The implementation therefore approximated deformation through **radial force actors**: point-force sources positioned at detected contacts, with their strength determined by deformation depth. + +This exposed multiple contacts to the simulation without reconstructing the full surface relief. It also imposed a conceptual boundary: the resulting force field did not necessarily match the shape or combined effects of the physical deformation. + +### Emulation, Debugging, and Configuration + +A simple **mouse-based emulator** generated depth-interaction events to make development more efficient. The Unreal Engine client included a debugging overlay showing connection status, received interactions, and their coordinates. A separate configuration overlay exposed game parameters, which were stored as a saved game and restored when the application started. + +These tools supported testing and diagnosis across the tracking and visualization boundary, alongside adjustment of application behaviour. + +### Automated Startup + +Exhibition operation motivated an **autostart function in the framework**. Starting the server loaded the selected sensor, resolution, filtering and processing settings, and connection configuration, then started the corresponding components. + +A separate utility with a simple configuration interface launched the programs in the correct order, including the communication intermediary. Framework startup and application settings therefore addressed different parts of the installation's setup. + +**[⬆ back to top](#table-of-contents)** + +## Scenario + +### Tabletop Interaction and Exhibition Flow + +Three scenarios were implemented as separate Unreal Engine levels. They switched after a configurable interval, but only when no interaction was detected. Active interaction postponed a transition by **30 seconds**, allowing visitors to continue without an immediate interruption. + +Idle overlays invited visitors to interact, and the space-game overlay explained its mechanics through animation. In the ball simulation, the simulation continued beneath a blur effect, with a countdown indicating the next level transition. + +Apart from short idle-screen texts, the scenarios had no fixed reading direction, allowing people to gather around the table. Although outward pulling was detected, all levels were designed to work through **inward pressing alone**. Collaboration was intended to add possibilities without being required. + +### Ball Simulation: Revisiting DepthTouch + +The first level adapted DepthTouch's circles into **three-dimensional balls**. Shading made their volume visible, while textures revealed rotation. The physics engine added size-dependent mass, inertia, friction, and more complex collisions. + +Moving to a three-dimensional simulation also allowed balls to move vertically and stack. Invisible boundaries formed a virtual glass box: a floor and side walls contained the balls, while a ceiling slightly above the diameter of the largest ball limited vertical movement. This reduced excessive stacking while retaining some of the additional behaviour introduced by the engine. + +The scenario thus explored both the visual appeal and interaction consequences of replacing a simplified simulation with richer physical behaviour. + +### Particle Simulation + +The second level used a particle effect as a proof of concept for interactive force-field visualization. Pressing created a force source at the hand position, and deformation depth controlled its strength. Particles responded by changing their trajectories, producing an interactive snowstorm-like effect. + +Magnetic fields and interactions between them motivated the concept, but the prototype **did not simulate magnetic field lines**. Producing those patterns would have required additional constraints on particle motion, which remained outside the implemented scenario. + +### Space Game: Steering Objects through Forces + +The third level developed a game around indirect control. An initial projectile concept would have mapped pressure to launch speed and contact position to firing direction. It was set aside because it favoured single-touch interaction and fitted the idea of simultaneous gravitational sources less well. + +The implemented game instead featured a spaceship moving along a predefined, irregular path while continuously losing energy. Power-ups entered from the screen edges at regular intervals with random directions and speeds. Players created force fields to steer these objects towards the ship and replenish its energy. The ship itself was unaffected by the fields, and all game elements occupied a single depth plane. + +| Interaction | Effect in the game | +| ----------- | ------------------ | +| Press inward | Attract power-ups towards the contact position. | +| Pull outward | Repel power-ups from the contact position. | +| Vary deformation depth | Adjust force strength to redirect, accelerate, or capture objects. | +| Combine several contacts | Influence trajectories through multiple force sources, including trapping objects between them. | + +Visual feedback identified contact positions, highlighted incoming power-ups and their directions, and represented declining ship energy through increasing smoke. Restoring full energy triggered a success message. Players influenced power-ups through forces rather than directly dragging them onto the ship. + +**[⬆ back to top](#table-of-contents)** + +## Findings and Discussion + +### Physical Realism and Interaction Efficiency + +The source reports demonstrations and discussions with users in several contexts, without a participant count or quantitative evaluation. Its findings provide exploratory observations about the interaction concepts and implementation. + +The balls appeared less agile than DepthTouch's circles because of friction and inertia, although their rendering and collisions were more convincing. Stacking offered an additional effect, but greater physical realism did not automatically improve interaction. + +For abstract tasks such as selection and filtering, the study judged simplified physical metaphors sufficiently understandable and more efficient. Detailed simulation was most appropriate when the physical behaviour itself mattered to the scenario. + +### Limits of the Point-Force Approximation + +Point forces were **not an adequate general replacement for a surface-derived vector field**. Their strength decreased with distance differently from the actual surface deformation. Multiple sources could also overlap and reinforce one another in ways that did not reflect the membrane's shape. Extended surface features, such as an inclined plane, could not be represented by this approach. + +These differences were particularly problematic for balls intended to behave as though they were rolling over the deformed surface. For particles and the space game, however, the same transformation offered useful additional interactions, such as trapping objects between several sources. The suitability of the approximation therefore depended on whether the application sought to reproduce surface behaviour or use deformation to control an independent simulation. + +### Learning Indirect Control + +In the space game, users initially tried to push objects directly onto the ship. Influencing trajectories indirectly required familiarization. The delayed effect of forces also made control difficult and could catapult power-ups out of the play area. + +A boundary preventing objects from escaping was proposed, but would weaken the consistency of the space setting. Further possibilities included obstacles around which power-ups would need to be guided and additional object types with different, potentially negative effects. These remained proposed extensions. + +### Implications for Framework Development + +For the forthcoming [requirements analysis](../../software/repo/requirements.md), the study provides evidence for the following areas: + +- **Integration across technologies:** Deliver interaction data to game-engine clients through a clear communication boundary, accounting for protocol adapters and dependent processes. +- **Stable, responsive multi-touch:** Preserve multiple contact positions and deformation depths so applications can map them to simultaneous forces with little delay. +- **Development and diagnosis:** Provide emulated interaction input and visibility into connection state, received events, and coordinate mappings. +- **Reproducible exhibition setup:** Restore sensor, processing, and connection settings automatically and coordinate startup across components. +- **Application configuration and guidance:** Persist game parameters, explain unfamiliar interactions, and make automatic transitions sensitive to ongoing input. +- **Explicit limits of input representations:** Distinguish contact-based force control from full-surface reconstruction when choosing how to connect deformation to a simulation. + +These implications connect implemented tools and observed limitations to later framework decisions. They do not imply that accurate surface-derived force fields, magnetic-field simulation, or the proposed game extensions were implemented in this case study. + +**[⬆ back to top](#table-of-contents)** diff --git a/docs/knowledge/case-studies/glyphboard.md b/docs/knowledge/case-studies/glyphboard.md index e69de29b..603f8bd2 100644 --- a/docs/knowledge/case-studies/glyphboard.md +++ b/docs/knowledge/case-studies/glyphboard.md @@ -0,0 +1,150 @@ +--- +title: "Case Study: Glyphboard" +--- + +# {{ page.title }} + + +## Table of Contents + +1. [Research Questions](#research-questions) +2. [Features and Limitations](#features-and-limitations) +3. [Scenario](#scenario) +4. [Findings and Discussion](#findings-and-discussion) + +| Profile field | Description | +| ------------- | ----------- | +| Title | Glyphboard | +| Year | 2017 | +| Focus | Integration, Framework | +| Format | Elastic Display wall and tabletop | +| Framework | dSense, ReFlex | +| Sensor | Intel RealSense | +| Architecture | Client-server (.NET/WPF + Angular) | +| Scenario | Interactive visualization of high-dimensional data | +| Features | - separation of the interaction layer
- control of a web application
- zoomable user interfaces: zoom and pan | +| Publication | *Kammer, D., Keck, M., Müller, M., Gründer, T. & Groh, R.* (2017): **Exploring Big Data Landscapes with Elastic Displays**. In: M. Burghardt, R. Wimmer, C. Wolff & C. Womser-Hacker (Eds.): Mensch und Computer 2017 - Workshopband. Spielend einfach interagieren. Bonn, Germany: Gesellschaft für Informatik e. V. (GI). DOI: [10.18420/muc2017-ws08-0342](https://doi.org/10.18420/muc2017-ws08-0342). | +| Repository | [reflex-glyphboard](https://github.com/visualengineers/reflex-glyphboard) | + +The fifth case study explored **zoomable user interfaces for information visualization**. The profile covers the historical dSense and ReFlex iterations, following the conventions in the [case study overview](overview.md#case-study-profiles). The published repository may include later adaptations. + + + +## Research Questions + +Glyphboard returned to the exploration of complex information spaces after earlier studies had investigated simpler content and navigation concepts. [FlexiWall](flexiwall.md) had identified zoomable user interfaces as a promising scenario, but its layer-based interaction model did not readily support continuous navigation. The single-touch implementation from [Interactive Animations](interactive-animations.md) provided the initial technical basis. + +The application context was **Glyphboard**, a web application developed at the Chair of Media Design at Technische Universität Dresden. It uses dimensionality reduction to visualize multidimensional data as a scatterplot and reveals individual attributes through more detailed glyphs as magnification increases. Although Glyphboard supports both magnifying lenses and full-screen enlargement, this case study concentrated on immersive exploration through continuous full-screen zoom. + +Three questions guided the work: + +- How can surface deformation control zooming in, zooming out, and resetting the view, while supporting meaningful levels of detail through semantic zoom? +- How can users switch between zoom and pan without a separate, explicit mode-switching action? +- How can an Elastic Display become an optional interaction layer for an existing web application, separating tracking from visualization and mapping continuous deformation to event-based application behaviour? + +The central idea was to use deformation to control continuous movement through a virtual data space. This complemented the local selection and inspection concepts explored in the [Zoomable Product Browser](zoomable-product-browser.md). + +**[⬆ back to top](#table-of-contents)** + +## Features and Limitations + +### Separating Tracking from the Web Application + +The main technical contribution was the complete separation of tracking and analysis from the application layer. Components originating in the FlexiWall framework were adapted to a **client-server architecture**, connecting a .NET/WPF tracking system to an Angular web application. + +This required an exchange format for surface pressure points, termed **interactions**. The first version contained a timestamp and a list of normalized spatial coordinates. Later iterations added touch identifiers, confidence values, and a classification distinguishing inward pushes from outward pulls. + +Filtering and smoothing were initially handled by the framework. Gesture recognition was implemented as a separate module in the client application, where detected interactions could be interpreted for the visualization's navigation controls. + +### Communication and Configuration + +Interaction data was transmitted through **WebSockets**. The earliest implementation only sent data from the server to the client. Later iterations added bidirectional exchange, particularly for configuration such as calibration data and interaction-area dimensions, as part of the framework's extension with a REST API. + +The architecture allowed Glyphboard's existing visualization and core behaviour to be reused. Adaptation was confined to an optional Elastic Display interaction layer in the client. The richer interaction format and configuration exchange were subsequent developments, rather than capabilities of the first implementation. + +### From Single-Touch to Multi-Touch + +The initial prototype used single-touch detection for its greater robustness and accuracy and lower latency. This was sufficient for continuous zoom, but restricted the available gestures when panning was introduced. An initial division into central zoom and peripheral pan regions exposed limitations in surface elasticity, sensor coverage, and directional control. + +Later iterations therefore used multi-touch detection to distinguish one-point zoom from two-point panning. This expanded the interaction vocabulary, while making reliable contact detection and low latency essential for avoiding unintended mode changes. + +**[⬆ back to top](#table-of-contents)** + +## Scenario + +### Data Overview and Semantic Zoom + +The starting view was a scatterplot of a preprocessed, clustered dataset. Spatial proximity indicated related items, while colour identified broader group membership. Earlier FlexiWall experiments had compared clustering results; Glyphboard instead supported further exploratory analysis within a complex dataset. + +Semantic zoom progressively revealed the dimensions of individual items at configurable magnification thresholds: + +| Detail level | Data representation | +| ------------ | ------------------- | +| **1: Overview** | Coloured points showed the distribution of items and their cluster membership. | +| **2: Attributes** | Flower glyphs or star plots exposed individual data dimensions. | +| **3: Detailed inspection** | The glyph representation gained data axes to support closer inspection. | + +### Continuous Zoom Control + +Pressing into the surface moved continuously into the point cloud. Deformation depth controlled **zoom speed**, with an exponential mapping allowing fine adjustments under shallow deformation and rapid enlargement under deeper deformation. Depth therefore controlled the rate of movement, rather than selecting a fixed magnification directly. + +Pulling the membrane outward zoomed out. Initially, pulling beyond a depth threshold also reset the zoom factor to its default value. A later gesture replaced this reset mechanism to reduce the need to grasp and pull the fabric. + +### Panning with One or Two Contacts + +In the first panning implementation, contact position selected the operation: pressing the centre zoomed, while pressing near an edge moved the view towards that edge. Panning was continuous, with speed controlled by deformation depth. Higher fabric tension at the edges left little depth available for speed adjustment, and the implementation restricted movement to one axis at a time. + +The multi-touch version switched to panning when **two pressure points** were detected. The shallower contact acted as an anchor. The vector from this anchor to the deeper contact determined direction, and the deeper contact's depth controlled speed. + +Moving a contact laterally adjusted direction; reversing the contacts' relative depths reversed movement. Adding or removing a second contact enabled transitions between pan and zoom without selecting a separate interface control. + +### Hover, Feedback, and Reset + +The refined interaction introduced a shallow-deformation range before zoom began. Within this range, the application displayed details for the nearest data point in the vicinity of the detected position, providing a **hover effect**. Further deformation crossed the zoom threshold. + +Visual feedback identified the current operation: a cursor and contextual information for hover, an anchor and directional arrows indicating movement and speed for panning, and a magnifying-glass icon for zoom. + +The later reset gesture used **three rapid presses**. When deformation exceeded a configurable amplitude three times within a configurable time interval, both zoom and pan returned to their initial values. This offered a way to return to the overview and explore another region without pulling the membrane. + +**[⬆ back to top](#table-of-contents)** + +## Findings and Discussion + +### Integration and Reuse + +The source reports implementation experience, demonstrations, and user observations, without a participant count or quantitative comparison. The findings therefore provide exploratory design insights. + +Separating tracking from the application demonstrated that an existing web visualization could be adapted through an optional interaction layer. Defining a shared interaction format was central to this reuse and established a concrete basis for communication between framework and application. + +### Navigation and Physical Constraints + +Edge-based panning was simple and robust to detect, but peripheral fabric tension, incomplete sensor coverage, and reduced tracking precision constrained its usefulness. Reserving edges for panning also reduced the area available for zoom, particularly with the limited image height of a 16:9 projection. Changing direction could require moving a hand to another side of the table. + +Using a vector between the edge contact and the display centre was proposed to remove the axis restriction, but would not resolve these ergonomic limitations. + +The two-handed gesture was also reported as robust and usable after brief familiarization. With practice, users could switch smoothly between zoom and pan, make small directional adjustments, and reverse movement. Direction changes between approximately 30 and 150 degrees nevertheless produced circular movement, making it preferable to restart the gesture. The distance between contacts was identified as a possible additional control parameter, rather than a demonstrated mapping. + +Unintended zoom occurred when one contact was lost or the relatively passive anchor hand fell below the hover threshold. These observations underline the importance of low latency and reliable interpretation of transitions between one and two contacts. + +### Hover and Gesture Ambiguity + +Hover received positive feedback. Selecting the nearest nearby data point reduced targeting errors. However, hover could also appear during an intended zoom gesture, especially as the hand was released. At the start of a press, latency introduced by confidence filtering and smoothing often caused the shallow hover range to be skipped. + +Restricting hover to the inward phase of a press was proposed as a remedy for unwanted activation on release. This was a conceptual improvement, not an implemented result reported by the study. + +The repeated-press reset was easy and ergonomic to perform, but recognition was not consistently successful. Individual differences in speed and the masking of presses by confidence filtering affected detection. Overlap with other gestures also produced distracting visual responses. More precise **feed-forward** during execution was proposed to guide timing and help distinguish reset from other interactions. + +### Implications for Framework Development + +For the forthcoming [requirements analysis](../../software/repo/requirements.md), the study provides evidence for the following areas: + +- **Independent interaction and visualization layers:** Allow existing applications to consume Elastic Display input through an optional adapter, while retaining their rendering and application logic. +- **Shared interaction data:** Communicate normalized positions and timing, with identifiers, confidence, and push/pull classification supporting more demanding client interactions. +- **Communication and configuration:** Support continuous interaction delivery and the exchange of calibration and interaction-area settings across component boundaries. +- **Configurable processing:** Balance noise suppression and stability against latency and the preservation of short gesture events. +- **Application-level gesture interpretation:** Support continuous parameter control, configurable thresholds and timing, and reliable transitions between contact counts and gesture phases. +- **Feedback and physical constraints:** Make interaction state visible and account for uneven deformability, peripheral sensing limitations, and the difficulty of pulling the surface. + +These implications connect the prototype's observed capabilities and limitations to framework design. They distinguish implemented iterations from proposed improvements and provide evidence for deriving requirements. + +**[⬆ back to top](#table-of-contents)** diff --git a/docs/knowledge/case-studies/layers.md b/docs/knowledge/case-studies/layers.md index e69de29b..4f128406 100644 --- a/docs/knowledge/case-studies/layers.md +++ b/docs/knowledge/case-studies/layers.md @@ -0,0 +1,176 @@ +--- +title: "Case Study: Layers - Layered Data" +--- + +# {{ page.title }} + + +## Table of Contents + +1. [Research Questions](#research-questions) +2. [Features and Limitations](#features-and-limitations) +3. [Scenario](#scenario) +4. [Findings and Discussion](#findings-and-discussion) + +| Profile field | Description | +| ------------- | ----------- | +| Title | Layers | +| Year | 2022 | +| Focus | Concept, Framework | +| Format | Elastic Display wall and tabletop | +| Framework | ReFlex | +| Sensor | Microsoft Azure Kinect | +| Architecture | Client-server (ASP.NET Core + Angular) | +| Scenario | Layered data: maps, MRI data, and paintings | +| Features | - multiple lenses and layer annotations
- layer blending modes
- streaming of sensor data | +| Repository | [reflex-layers](https://github.com/visualengineers/reflex-layers) | + +The ninth case study combined **layer blending, multi-touch detection, and interactive lenses** in a web-based successor to the [FlexiWall experimentation environment](flexiwall.md). The profile describes the ReFlex implementation reported in the source, following the conventions in the [case study overview](overview.md#case-study-profiles). The published repository may include later adaptations. The source does not list an associated publication. + +This account documents the research questions, interaction concepts, and findings. Dataset configuration, texture conversion, controls, and development instructions are maintained in the [Layers application documentation](../../software/apps/layers.md). + + + +## Research Questions + +Layers revisited FlexiWall's exploration of layered content using ReFlex and web technologies. Its contribution went beyond porting the earlier visualization: detecting individual pressure points made it possible to combine direct depth-image blending with lenses and contextual information about the selected layers. + +Two technical questions guided the work: + +- How can datasets with substantially more depth layers be loaded dynamically, overcoming the restrictive limits of individually supplied shader textures? +- How can the depth image be streamed to a separate visualization client with sufficiently low latency for continuous interaction? + +The study also aimed to retain FlexiWall's usefulness as an experimentation environment, particularly the simple addition of content and configuration at runtime. Conceptually, it investigated simultaneous lenses, information displayed at each lens, and visual guidance that helps users locate their current position within a data volume. + +**[⬆ back to top](#table-of-contents)** + +## Features and Limitations + +### Web Client and Rendering Approaches + +The implementation used ReFlex's **client-server architecture**, combining an ASP.NET Core server with an Angular application and Microsoft Azure Kinect sensing. The client used **three.js** for graphics and could optionally be packaged as an Electron desktop application. + +Two rendering approaches offered different capabilities: native web technologies and CSS for image-based lenses, and **WebGL pixel shaders** for per-pixel blending and texture-array lenses. These differences affected both the supported interaction modes and the available lens annotations. + +The application reused ReFlex's separate **emulator tool** instead of reimplementing FlexiWall's integrated emulator. Embedding that tool in a frame allowed it to be used as an overlay during development. + +### Layer Formats and Capacity + +The study supported two usable representations of layer content: + +| Representation | Purpose and limitations in the reported implementation | +| -------------- | ----------------------------------------------------- | +| **Texture2D** | Individual JPG or PNG images could be loaded without conversion and displayed through web elements or a shader. Pixel Blending supported a maximum of 15 separately supplied layer textures; this was a restriction of the shader path, not a general limit on image-based lens navigation. | +| **TextureArray** | Equal-sized layers were packaged as raw data or a KTX2 texture array. Arrays made many more layers accessible to the shader. Raw data support retained compatibility with existing datasets; a workflow using the KTX Software Tools prepared new arrays. | + +The source describes a 256-layer limit for its raw-data representation. KTX2 avoided that particular restriction, but usable array sizes still depended on the graphics implementation, hardware, and available resources. The goal of supporting an arbitrary number of layers should therefore be understood as removing the earlier small fixed limit, rather than providing unlimited capacity. + +**3D textures were investigated but not retained as a usable option**, because support in the three.js implementation used by the study was not reliable. Existing FlexiWall datasets stored as 3D textures consequently required conversion. Texture arrays and 3D textures are distinct representations, even when both hold volumetric content. + +### Streaming and Server Control + +Retrieving depth images through individual **REST requests** offered a simple interface, but its latency was too high for the study's real-time visualization. The implementation therefore provided **native WebSocket and SignalR interfaces** for continuous transmission. + +Clients could obtain different representations according to their needs: an unprocessed depth image encoded in grayscale, a point cloud in spatial coordinates, or a calibrated point cloud mapped to screen coordinates. Pixel Blending used a grayscale depth texture derived from the selected input, while lens interaction used detected pressure points. + +The REST API was also extended to exchange configuration data, start and stop the sensor, and let clients explicitly enable or disable streams. Layers thus exercised both continuous data delivery and application-driven control of the tracking server. + +**[⬆ back to top](#table-of-contents)** + +## Scenario + +### Three Presentation Concepts + +Layers provided three ways to explore the same content. Magic Lens had separate single-touch and multi-touch variants: + +| Mode | Mapping from deformation to visualization | +| ---- | ---------------------------------------- | +| **Pixel Blending** | Each display pixel showed content selected or blended according to the corresponding value in the depth texture, reproducing FlexiWall's direct relationship between surface shape and data layers. Optional smoothing softened transitions. | +| **Magic Lens** | A local lens revealed the layer associated with a detected contact's depth. Single-touch used the contact furthest from the surface's resting plane. Multi-touch displayed a lens for each contact, subject to a configurable maximum per dataset. | +| **Layer Navigation** | The maximum deformation selected one layer for the entire display, carrying forward the frame-navigation concept from [Interactive Animations](interactive-animations.md). | + +In Pixel Blending, the grayscale values corresponding to the resting plane and maximum deformation were configurable. This allowed the dataset to be mapped, for example, entirely to the depth range reached by pressing inward. + +The modes were not available in every combination. **Texture2D supported all four selectable variants, whereas TextureArray supported Pixel Blending, a single Magic Lens, and Layer Navigation. Multiple texture-array lenses were not implemented in the reported study.** + +### Lens Appearance and Orientation + +Lens size and horizontal and vertical offset were configurable for both data representations, extending ideas from [DeepZoom](deep-zoom.md). Other options depended on how the lens was rendered: + +| Lens option | Texture2D | TextureArray | +| ----------- | --------- | ------------ | +| Size and offset | Supported | Supported | +| Grayscale blending mask | Not implemented | Supported | +| Lens widget with layer information | Supported | Not implemented | +| Layer widget at the display edge | Supported | Supported | + +For texture arrays, a **grayscale mask** interpolated between the layer at the current pressure point and the resting view. Linear and exponential gradients produced different transitions, including a fisheye-like appearance. This combined local lens interaction with layer blending; it did not reproduce the measured surface relief inside the lens. + +For image-based lenses, the implemented metadata consisted of a **configurable layer title**. Figure 5-52 in the source also illustrates a fan-shaped arrangement of layers at the lens edge. Richer annotations, relationships between layers, and highlighting of related elements were proposed extensions. + +A separate **layer widget** represented the available depth range as a vertical axis, marking the layers and the depths of detected contacts. It was particularly useful for lenses and full-screen navigation, where it showed the current position in the layer stack. During Pixel Blending, contact markers indicated how far users had deformed the surface into the data volume, but did not describe the complete distribution of visible layers. + +### Datasets and Application Contexts + +The experimentation environment adapted earlier FlexiWall content and explored larger layer stacks: + +| Content | Exploration supported by the study | +| ------- | --------------------------------- | +| **Thematic maps** | Simultaneous lenses exposed different map layers while preserving the surrounding map as context. Layer titles helped identify the content of each lens. | +| **Paintings** | Alternative image layers could be inspected locally, with lens widgets providing context about the selected layer. | +| **Scientific animation** | The Cahn-Hilliard visualization from Interactive Animations was ported to the new environment, linking deformation to progression through an image sequence. | +| **MRI volume** | The dataset identified in the source as *Bruce Goochs Brain* comprised **156 layers at 256 × 256 pixels** each. | +| **Anatomical sections** | A head subset from the National Library of Medicine's *Visible Human Project* comprised **217 layers at 4096 × 2700 pixels** each. The exploration concept drew on the Visible Human Explorer. | +| **Historical maps** | A sequence of **47 layers** represented Europe between **1500 and 2008**, making time accessible through depth navigation. | + +These examples illustrate semantic, spatial, and temporal interpretations of a layer stack. The medical datasets demonstrated exploration of volumetric content; the source does not report clinical evaluation. + +### Configuration for Reusable Demonstrations + +JSON configuration described datasets, their formats, and their resources. Optional **dataset-specific presets** selected the interaction mode and associated presentation settings automatically when content was loaded. Separate configuration supplied keyboard shortcuts, lens-mask textures, and fallback settings for datasets without their own presets. + +Global options also covered depth-range mapping, calibration, debugging views, and an idle mode with an optional logo. Together with runtime controls, these settings made it possible to adapt the same application to different content and demonstration contexts. The [application reference](../../software/apps/layers.md#app-configuration) documents the configuration details. + +**[⬆ back to top](#table-of-contents)** + +## Findings and Discussion + +### Combining Earlier Results + +Layers brought together several strands of the preceding studies: + +- **[FlexiWall](flexiwall.md):** Layer interaction and a configurable experimentation environment. +- **[Interactive Animations](interactive-animations.md):** Longer image sequences and preview animation. +- **[Glyphboard](glyphboard.md):** Web integration and lens-interaction concepts. +- **[DeepZoom](deep-zoom.md):** Configurable lens size and offset. +- **[Construction Progress Visualization](bim.md):** Layer annotations and contextual information. + +The source reports implementation experience and conceptual findings, without participant counts, quantitative usability results, or measured streaming benchmarks. Its conclusions therefore document capabilities and design insights rather than a controlled comparison of interaction techniques. + +### Layer Context and Multiple Lenses + +The main conceptual advance was connecting **detected pressure points with information beyond the layer image itself**. Layer titles and a visible position in the stack made the selected content explicit, while simultaneous lenses allowed different layers to be inspected together. + +Each lens could be controlled through its contact's lateral position and deformation depth. The source identifies this simultaneous depth control as a distinctive opportunity of Elastic Displays relative to conventional multi-touch interaction. Comparing map layers demonstrated the concept; further combinations of lenses and richer metadata remained opportunities for future work. + +### Rendering Trade-offs + +The two rendering paths exposed a practical trade-off. Image-based web elements supported multiple lenses and interface widgets, while texture arrays enabled larger shader-accessible layer stacks and blending masks. Their capabilities were complementary, so choosing a data format also meant choosing an interaction feature set. + +A **histogram of visible layers** was proposed as a richer overview for Pixel Blending. It was not implemented: the existing shader computed output independently per pixel on the GPU and did not provide a global summary. The source anticipated that moving those calculations to the CPU would compromise real-time performance. This was a limitation of the chosen implementation, not evidence that such summaries are generally impossible. + +### Implications for Framework Development + +Layers provided an application-level validation of the ReFlex architecture by combining depth textures, pressure points, calibration, and server settings in one client, with configuration also communicated back to the server. For the forthcoming [requirements analysis](../../software/repo/requirements.md), it provides evidence for the following areas: + +- **Multiple input representations:** Expose both surface-wide depth data and discrete contacts so applications can combine blending with lenses and annotations. +- **Low-latency streaming:** Support continuous delivery independently of request-based configuration and control. +- **Calibration and coordinate mapping:** Make raw, processed, and screen-mapped data available according to the visualization's needs. +- **Bidirectional server integration:** Allow clients to retrieve and change configuration, control sensor operation, and select active streams. +- **Reusable configuration:** Separate content and dataset-specific interaction presets from application code, enabling rapid changes between scenarios. +- **Multiple contacts and contextual feedback:** Provide the position and depth information needed for independently controlled lenses and orientation within a layer stack. +- **Reusable development tools:** Support application testing through the shared emulator instead of duplicating its functionality in each client. + +These implications connect the implemented capabilities and their limits to framework design. Texture preparation, GPU rendering, and the choice of lens widgets remained application responsibilities. + +**[⬆ back to top](#table-of-contents)** diff --git a/docs/knowledge/case-studies/other-prototypes.md b/docs/knowledge/case-studies/other-prototypes.md index e69de29b..a6d7bf7d 100644 --- a/docs/knowledge/case-studies/other-prototypes.md +++ b/docs/knowledge/case-studies/other-prototypes.md @@ -0,0 +1,127 @@ +--- +title: "Case Studies: Other Prototypes" +--- + +# {{ page.title }} + + +## Table of Contents + +1. [EscapeRoom](#escaperoom) +2. [FlightSim](#flightsim) +3. [MIReFlex](#mireflex) + +These smaller prototypes explored specific technological and conceptual approaches, addressing open questions identified during model development. They also served to validate the ReFlex framework in practical use and were demonstrated at selected events to gather user feedback. + +![EscapeRoom: Game concepts with vibrotactile feedback]({{ site.baseurl }}/assets/img/kb/case-studies/others-escape-room.png){:.content__title-image} + +## EscapeRoom + +| Profile field | Description | +| ------------- | ----------- | +| Title | EscapeRoom | +| Year | 2022 | +| Focus | Concept, Integration | +| Format | Elastic Display tabletop | +| Framework | ReFlex | +| Sensor | Microsoft Azure Kinect | +| Architecture | Client-server (ASP.NET Core + Angular + TactJam) | +| Scenario | Spatial interaction in a game scenario | +| Features | - spatial interaction metaphors
- vibrotactile feedback | +| Publication | *Müller, M. & Kammer, D.* (2022): **Augmenting Elastic Displays with Active Vibrotactile Feedback**. In: C. Marky, U. Grünefeld & T. Kosch (Eds.): Mensch und Computer 2022 - Workshopband. Bonn, Germany: Gesellschaft für Informatik e. V. (GI). DOI: [10.18420/muc2022-mci-ws04-359](https://doi.org/10.18420/muc2022-mci-ws04-359). | + +### Research Questions + +EscapeRoom explored **spatial interaction metaphors** in a web-based game. It combined lateral movement, inward pressing, and outward pulling with active vibrotactile cues to support gesture execution and communicate interaction outcomes. + +### Features and Limitations + +The implementation combined an ASP.NET Core server with an Angular client. An interface between **ReFlex and TactJam** controlled vibration motors mounted along the edges of the fabric. + +Vibrations served as **feed-forward during gesture execution**, as feedback when a gesture was completed, and as cues suggesting physical resistance. The integration therefore extended the display's interaction with actively generated tactile signals. + +### Scenario + +Players manipulated objects through several deformation-based gestures: + +| Interaction | Effect in the game | +| ----------- | ------------------ | +| Move the pressure point laterally | Pull an object open through a spatial swipe gesture. | +| Repeatedly press inward and pull outward ("pumping") | Dig objects free. | +| Pull outward | Pick up objects. | +| Press lightly | Touch objects. | +| Press more deeply | Continuously manipulate objects to move or break them open. | + +Vibrotactile cues accompanied these gestures, connecting their execution and completion with the game's spatial interaction metaphors. + +**[⬆ back to top](#table-of-contents)** + +![MIReFlex: Spatial Audio Placement in Multisensory Interaction Room]({{ site.baseurl }}/assets/img/kb/case-studies/others-flightsim.jpg){:.content__title-image} + +## FlightSim + +| Profile field | Description | +| ------------- | ----------- | +| Title | FlightSim | +| Year | 2022 | +| Focus | Concept, Integration | +| Format | Elastic Display wall | +| Framework | ReFlex | +| Sensor | Microsoft Azure Kinect | +| Architecture | Client-server (ASP.NET Core + three.js) | +| Scenario | Immersive flight control | +| Features | - gestures for steering and controlling the speed of flying objects
- partitioning of the interaction space | +| Publication | *Müller, M., Kammer, D., Grimm, L., Fabian, K. & Simon, D.* (2022): **ReFlex Framework: Rapid Prototyping for Elastic Displays**. In: P. Bottoni & E. Panizzi (Eds.): AVI 2022: Proceedings of the 2022 International Conference on Advanced Visual Interfaces. New York, NY, USA: ACM. DOI: [10.1145/3531073.3534482](https://doi.org/10.1145/3531073.3534482). | + +### Research Questions + +FlightSim explored remote control of flying or hovering objects through an Elastic Display. Its interaction concept investigated how **different deformation-depth ranges** could distinguish looking around from controlling movement. + +### Features and Limitations + +The prototype used ReFlex with an ASP.NET Core server and a three.js client. A procedurally generated landscape provided an immersive view from the flying object's perspective, with flight along the virtual space's depth axis. + +The control concept assumed that the object **did not need to move backwards**. It partitioned the interaction space into two depth ranges, assigning different controls to shallow and deeper pressing. + +### Scenario + +| Interaction | Effect in the simulation | +| ----------- | ------------------------ | +| Press lightly | Change the viewing direction without changing the direction of movement. | +| Press more deeply | Control speed and flight direction. | + +Steering followed a **joystick-like mapping**: the direction vector was the difference between the centre of the display and the pressure-point position, considering only displacement within the display plane. + +**[⬆ back to top](#table-of-contents)** + +![MIReFlex: Spatial Audio Placement in Multisensory Interaction Room]({{ site.baseurl }}/assets/img/kb/case-studies/others-mireflex.jpg){:.content__title-image} + +## MIReFlex + +| Profile field | Description | +| ------------- | ----------- | +| Title | MIReFlex | +| Year | 2023 | +| Focus | Concept, Integration | +| Format | Elastic Display tabletop | +| Framework | ReFlex | +| Sensor | Microsoft Azure Kinect | +| Architecture | Client-server (ASP.NET Core + Unreal Engine 5) | +| Scenario | Control of spatial audio sources | +| Features | - integration into a multisensory interaction room
- linking different user perspectives | + +### Research Questions + +MIReFlex explored using an Elastic Display to position spatial audio sources within a **multisensory interaction room** (*Mehrsinnlicher Interaktionsraum*, [MIR](https://www.htw-dresden.de/hochschule/fakultaeten/info-math/labore-und-lehrplattformen/mir)). It connected a tabletop view of the virtual environment with an immersive surrounding projection. + +### Features and Limitations + +The installation combined an Elastic Display tabletop with a **270° surrounding projection**. The implementation used a distributed multiplayer application in Unreal Engine 5, with different roles assigning the views and rendering paths for the surrounding environment and the tabletop. Content was synchronized between the two systems. + +The Elastic Display was integrated through the [Unreal Engine plugin]({{ site.baseurl }}/software/templates/ue.md) developed as part of ReFlex. + +### Scenario + +The tabletop displayed a stylized overhead view of the virtual environment. Users positioned audio objects on this three-dimensional map by pressing into the surface; **deformation depth controlled their height in the virtual scene**. The tabletop thus provided a spatial control view linked to the room's immersive presentation. + +**[⬆ back to top](#table-of-contents)** diff --git a/docs/software/apps/layers.md b/docs/software/apps/layers.md index a39b1724..28748d47 100644 --- a/docs/software/apps/layers.md +++ b/docs/software/apps/layers.md @@ -1,5 +1,7 @@ # reflex-layers +This page documents application setup, content preparation, and controls. For the research questions, interaction concepts, and findings of the historical implementation, see [Case Study: Layers - Layered Data](../../knowledge/case-studies/layers.md). + ## Table-of-contents @@ -18,7 +20,7 @@ ## User Interface -The application features basically three visualization / interaction modes (see also [Interaction Modes](#interaction-modes)). +The application provides three presentation concepts. Magic Lens has separate single-touch and multi-touch variants; availability depends on the texture format. See [Interaction Modes](#interaction-modes) for behaviour and configuration options. ### Magic Lens (single touch and multi-touch) @@ -54,14 +56,15 @@ Pixel Blending Options include developer / debugging options, whether to interpo 1. `Texture2D` * multiple images in common file formats (preferably `png` and `jpg`) - * constraint: maximum 15 images can be used + * Pixel Blending supports a maximum of 15 separately supplied layer textures in the case-study implementation; this limit does not apply generally to image-based lens navigation 2. `TextureArray`: - * Number of images of equal size provided ina single file - * either in RAW format (raw bit data) or in `Khronos KTX` texture format, could contain 256 or more images + * multiple images of equal size provided in a single file + * either in RAW format (raw byte data) or in `Khronos KTX2` texture format + * the case study describes a 256-layer limit for its raw-data representation; KTX2 avoids that restriction, but usable array sizes still depend on the graphics implementation and hardware 3. `Texture3D`: * number of images of equal size, encoded as volumetric data structure * either in `DDS` format (raw bit data) or in `Khronos KTX` texture format - * __REMARK:__ currently not working / not implemented as `three.js` _can_ read and load DDS textures, but binding to a `Texture3d` is not working (it seems that three.js focuses on Texture Arrays) + * __REMARK:__ not a usable option in the implementation described by the case study: 3D-texture integration with three.js was unreliable, so existing FlexiWall 3D textures required conversion. This is distinct from the supported `TextureArray` formats. ### Common Properties for TextureResource @@ -112,9 +115,9 @@ There is an optional parameter `config` that can be set for each dataset. It spe "showLenseUI": true, // show position in the current layer on the side "showLayerUI": false, - // selected mask for Magic Lens (only 3D Textures) + // selected mask for Magic Lens (TextureArray only) "defaultLensMaskIdx": 1, - // selected border color for Magic Lens (only 3D Textures) - RGBA (Hex) + // selected border color for Magic Lens (TextureArray only) - hex color "lensBorderColor": "#ffff00", // scaling factor of lens "lensSize": 1.0, @@ -305,6 +308,15 @@ Example Config: ## Interaction Modes +The case-study implementation supports the following combinations. These describe the reported prototype; later repository versions may extend them. + +| Mode | Texture2D | TextureArray | +| ---- | --------- | ------------ | +| Pixel Blending | Supported, with the layer limit described under [Texture Formats](#texture-formats) | Supported | +| Magic Lens (single-touch) | Supported | Supported | +| Magic Lens (multi-touch) | Supported | Not implemented | +| Layer Navigation | Supported | Supported | + ### Pixel Blending * Blends the image based on the color value from the depth sensor @@ -320,21 +332,21 @@ Example Config: ### Magic Lens (Single or Multi-Touch) * Displays a lens at the finger position containing the content based on the deformation at the fingertip -* In Single-touch mode, the first touch is used to determine the lens position, in multi-touch mode, a lens for every finger is displayed +* In single-touch mode, the contact furthest from the resting plane determines the lens position and layer. In multi-touch mode, each contact controls a lens, up to the dataset's configured `maxNumLenses` (Texture2D only in the case-study implementation). * Options: * __Lens Size__: Scaling factor for the lens (_Default:_ __1.0__) * __Lens Offset X__: move the lens from the fingertips in horizontal direction (_Default:_ __0.0__) * __Lens Offset Y__: move the lens from the fingertips in vertical direction (_Default:_ __0.0__) - * __Show Lens UI__: show current layer at the border of the lens (_Default:_ __true__) + * __Show Lens UI__: show current layer at the border of the lens (Texture2D only; _Default:_ __true__) * __Show Layer UI__: show layers and current position of the lenses at the side of the screen (_Default:_ __true__) ### Layer Navigation -* go through layers by deforming the screen, complete layer is displayed based on the deepest finger position +* Navigate through layers by deforming the screen. The contact furthest from the resting plane selects the layer displayed across the entire screen. * Options: * __Show Layer UI__: show layers and current position at the side of the screen (_Default:_ __true__) -When using KTX-Arrays, __Lens Modes__ offer an additional option to specify the lens mask. his can be customized to achieve effects such as a fisheye effect. Also, the border color of the lens can be specified. +For texture-array lenses, a grayscale mask can interpolate between the selected layer and the resting view, producing effects such as a fisheye-like transition. The lens border color is also configurable. Lens size, offset, and the layer widget at the screen edge are available for both texture formats; widgets directly on the lens are limited to Texture2D in the case-study implementation. ## Project Setup and Development