Rhythmic Pixel Regions rearchitects the visual pipeline around what each part of a scene actually needs, turning a uniform camera stream into an adaptive sensing system that preserves high-fidelity detail where it matters while reducing the cost of moving pixels everywhere else.
Introduction: Rethinking Uniform Visual Sensing
Low resolution frames can reduce pixel memory throughput, but also reduce the opportunities for high-precision visual sensing. Our intuition is that not all parts of a scene need to be captured at a uniform resolution. Selectively and opportunistically reducing resolution for different regions of image frames can yield high-precision visual computing at energy-efficient memory data rates.
System Architecture: Adaptive Pixel-Region Pipeline
We develop a visual sensing pipeline architecture that flexibly allows application developers to dynamically adapt the spatial resolution and update rate of different rhythmic pixel regions in the scene. The system ingests pixel streams from commercial image sensors with their standard raster-scan pixel read-out patterns, but only encodes relevant pixels before storing them in memory.
We also present streaming hardware to decode the stored rhythmic pixel region stream into traditional frame-based representations to feed standard computer vision algorithms.
Results: High-Fidelity Vision at Lower Data Rates
We integrate our encoding and decoding hardware modules into existing video pipelines around Xilinx's ZCU102. On top of this, we develop runtime support allowing developers to flexibly specify the region labels. With rhythmic pixel regions, the system can achieve 50% of 4K throughput while maintaining around the same accuracy as 4K.
