Human-Centered Physical AI & Orchestration

How should people and autonomous agents interact with intelligent systems embedded in the physical world? A user should be able to ask a high-level question or specify a goal without knowing which sensor, model, or processing pipeline is required. A robot should similarly be able to translate a mission into concrete perception and decision tasks.

We develop reasoning and orchestration mechanisms that bridge this gap. The system interprets intent, determines what evidence is needed, invokes the appropriate sensing and inference pipelines, and combines the resulting observations into a grounded answer or action. A single query may require RF localization, visual scene understanding, radar-based motion analysis, or several of these together. The orchestration layer must therefore decide what to sense, where to sense, and how to reconcile evidence across modalities.

People may communicate through language, speech, or gesture, while autonomous agents operate through goals, plans, and actions. In both cases, the interaction remains grounded in measurable evidence from the environment rather than relying only on abstract model knowledge.

This leads to the next question: what sensing capabilities and spatial representations are required to provide such evidence?

Multimodal Sensing & RF-Aware Digital Twins

No single sensor provides a complete description of the physical world. Cameras offer rich semantic information but are limited by occlusion, poor lighting, and field of view. GPS is unavailable indoors and unreliable in many operational settings. RF, radar, thermal, acoustic, and inertial sensors reveal complementary properties, but each produces only a partial and often indirect view of the environment.

We combine these modalities to recover spatial, physical, and semantic context. Our systems localize people and assets, map GPS-denied spaces, image occluded regions, monitor activity, and estimate how objects and materials affect wireless propagation. We use these observations to construct RF-aware digital twins that represent both the physical geometry of an environment and its radio characteristics.

These twins provide persistent context for the orchestration layer. They allow a system to reason about where objects and people are, and about how sensing quality, connectivity, and localization accuracy change as the environment evolves. New measurements update these representations as layouts, mobility, and network conditions change.

Representative systems: UbiqMap, SkyScale, Argos, and EcoVis.

This raises a practical challenge: how can sensing and inference operate continuously on resource-constrained devices?

Edge Intelligence & Application-Specific Acceleration

Multimodal sensing can generate large and continuous data streams, but many applications operate on embedded platforms with limited power, memory, bandwidth, and computation. Transmitting every measurement to a distant cloud is often too slow, expensive, or unreliable, and can expose raw sensor data unnecessarily.

We therefore co-design sensing, inference, and hardware so that useful evidence is produced close to where data is captured. Our work includes task-aware measurement selection, model and data compression, TinyML, distributed edge processing, FPGA acceleration, and domain-specific architectures for RF, radar, vision, and multimodal workloads.

The objective is not merely to reduce model size. We redesign the path from acquisition to inference so that the system performs only the computation required for the current task. This allows the orchestration layer to invoke sensing pipelines dynamically while maintaining predictable latency and bounded energy and communication costs.

Representative systems: WISDOM and AutoCompress.