ShadAR

Real-time Prompting On-demand Visual Augmentation in Augmented Reality

Yanni Mei Samuel Hartenstein Jonas Wombacher Florian Mueller Jan Gugenheimer

TU Darmstadt

UIST '26 · Detroit, MI, USA

A user wearing a Meta Quest 3 asks to turn a book into dark mode; the system segments the book, generates a shader and shows the book in inverted colors. Further examples: red-green color blindness, pencil sketch style, an enlarged stop sign, and a line measuring the distance between two hands.
ShadAR turns a spoken request such as “Turn the book into dark mode” into a pixel shader that an object detector aims at the right region of the passthrough view. Further examples: (B) simulating red-green color blindness, (C) a pencil-sketch room, (D) a larger stop sign, (E) a line showing the distance between two hands.

Abstract

Augmented Reality can reshape our visual perception (e.g., simulating color blindness) yet such effects are typically pre-programmed by experts, making visual augmentation static and inflexible. We introduce ShadAR, an AR system that allows users to describe expected visual augmentations (e.g., "Blur all people") for changing their vision in real time. ShadAR can graphically manipulate AR passthrough real-time via shader injection. When receiving a user’s prompt, ShadAR uses LLMs (GPT5.2) to interpret the user’s intent, analyzes the user’s view with object detection (YOLOv26), generates a pixel shader, and applies it to the targeted region on AR passthrough. To understand usage pattern, we invited 21 HCI researchers to use ShadAR, identifying four application domains (design, health, productivity, and social interaction) and two prompt-framing patterns: procedural prompts specifying what to modify and how (“Highlight the book.”), and end-goal prompts conveying desired mental/behavioral states (“Make me laugh.”). Such end-goal prompts suggest a future in which users specify outcomes (e.g., “Make me not miss my flight”) while the system determines how to realize them, though often with inherently indeterminate results. We discuss design opportunities for ShadAR and ethical concerns.

Video

How it works

The headset records the spoken prompt and sends it, transcribed, to an LLM that writes an HLSL pixel shader. A separate Unity instance on a PC compiles the shader into an asset bundle, which the headset loads at runtime. In parallel, every passthrough frame is streamed to an object detector on the PC; its boxes and segmentation masks come back as GPU buffers, so the shader can restrict its effect to exactly the objects the prompt is about.

Pipeline diagram: voice command and one frame go through speech transcription, the shader generator and the shader compiler; each passthrough frame goes through object detection and a detection-result translator; both feed the reprogrammable AR view on the Quest 3.
Shader generation (top) and real-world detection (bottom) run independently and meet in the reprogrammable AR view on the Quest 3.

What people asked for

In a workshop, 21 HCI researchers used ShadAR in four domains: as a design tool, for health interventions, for personal productivity and for social interaction.

Ten workshop examples with their prompts, among them mirroring the world upside down, recoloring boxes, summer foliage, making cake less attractive than broccoli, focusing on a laptop, finding a phone, blurring a person, a heart bubble around two people, and pixelating a screen.
Prompts from the workshop, grouped by domain. Faces in (H) and (I) were anonymized by the authors.

Besides procedural prompts that say what to change, participants wrote end-goal prompts that only describe the desired state, leaving the effect itself to the system.

Four generated effects: neon halos with fluid distortion, a kaleidoscopic double vision, googly eyes on a television, and a jittering, misaligned view.
End-goal prompts: two different results for “Make me feel as if I am on an LSD trip” (A, B), “Make me laugh” (C) and “Surprise me” (D).

Code

The Unity app for Meta Quest 3, the detection server and setup instructions are on GitHub. The repository has been developed further since the paper (stereo rendering, open-vocabulary detection); its README lists the differences and how to approximate the setup from the paper.

Citation

@inproceedings{mei2026shadar,
  author    = {Mei, Yanni and Hartenstein, Samuel and Wombacher, Jonas and Mueller, Florian and Gugenheimer, Jan},
  title     = {ShadAR: Real-time Prompting On-demand Visual Augmentation in Augmented Reality},
  booktitle = {The 39th Annual ACM Symposium on User Interface Software and Technology (UIST '26)},
  year      = {2026},
  month     = nov,
  location  = {Detroit, MI, USA},
  publisher = {ACM},
  address   = {New York, NY, USA},
  numpages  = {14},
  doi       = {10.1145/3830398.3830534}
}