Agentic AI for XR — an open-source foundation for multimodal, real-time conversational AI in the NVIDIA CloudXR ecosystem.
XR AI connects web, Android, iOS/visionOS, and native clients to a shared media hub, GPU-accelerated AI services, tool-using agents, and optional CloudXR remote rendering. Agents can see and hear what a participant experiences, call native tools, and return audio or data to the same participant.
This project is a public beta. APIs and behavior may change as it evolves.
Refer to the versioned documentation for setup, requirements, credentials, networking, architecture, and troubleshooting. The landing page opens the newest complete documentation set; a release takes precedence once it contains the current entry points. Start with:
- Run the browser demo with a coding agent (
run-demo) - Set up for development with a coding agent
- Build your own application
- Manual quickstart
- System requirements
- Architecture
The site also publishes the current main branch and release-tagged versions.
For multilingual OCR with local Hugging Face inference or an external NIM endpoint, refer to
OCR serving.
Build an application in the XR AI source tree so its unpublished SDK
dependencies resolve from their repository paths. Start from
agent-samples/simple-vlm-example/, copy it to apps/<your-app>/, keep only
the SDK layers and services the application needs, and follow the
build-your-application guide
through scaffolding, startup, and hardware-free verification.
| Sample | Purpose |
|---|---|
| Model servers beside each agent sample | Convenience launchers for each sample’s required local models; refer to the workflow |
| NIM endpoints | Configure and validate independently deployed models |
simple-vlm-example |
Voice and text questions about the current camera frame |
conversation-example |
Routed conversation, current-view questions, one-shot visual alerts, and note-taking |
procedure-capture-guide |
Capture narrated demonstrations and follow approved guides with step verification |
lab-instrument-monitoring |
Marker-associated visual monitoring with a foreground voice agent |
tea-making-sample |
Guided workflow with visual evidence and background observations |
xr-render-demo |
Voice-driven CloudXR scene manipulation |
Each sample README gives the shortest runnable command sequence. The linked documentation contains architecture, behavior, configuration, output contracts, evaluation, and adaptation guidance.
| Directory | Contents |
|---|---|
client-samples/ |
Platform clients and shared StreamKit implementations |
agent-sdk/ |
Hub IPC, model clients, runtime, tools, voice, and web events |
agent-samples/ |
Runnable agent stacks |
apps/ |
Application workspaces outside repository sample tooling |
services/ |
Hub, model servers, and typed capability services |
utils/ |
Launcher, logging, VAD, vLLM, and voice-gate utilities |
tests/ |
Cross-package and integration tests |
docs/source/ |
Canonical user and contributor documentation |
skills/ |
Coding-agent skills for setup, demos, application development, and reviews |
For repository constraints and dependency boundaries, refer to
AGENTS.md and DEPENDENCIES.md.
Refer to CONTRIBUTING.md for the contribution process and
tests/README.md for the shortest test commands. Refer to
SECURITY.md to report security issues.
XR AI is licensed under Apache-2.0.
Third-party license and attribution texts are preserved under
third_party_licenses/. These files retain their
upstream terms; refer to License texts
for details.
Refer to THIRD_PARTY_NOTICES.md for each component's
applicable license and third_party_licenses/
for the preserved texts.
