I Built a Real-Time Perception Stack on CARLA. Here Is What Actually Happened.

Short summary

A developer shares lessons from three weeks integrating YOLOv11, ByteTrack, StrongSORT, CARLA, and ROS 2 into a real-time perception stack. Key issues include timestamp mismatches causing tracker instability, blocking inference callbacks dropping frames, and the gap between model benchmark speed and robotics middleware requirements. Solutions include proper ROS 2 clock stamping and decoupling ingestion from inference via bounded queues and dedicated threads.

• •Missing ROS 2 clock timestamps on image headers caused erratic tracker behavior — always stamp headers
• •Blocking inference in subscription callbacks drops frames; decouple with a bounded queue and dedicated inference thread
• •Model benchmark speed differs from robotics middleware speed — threading and timing matter as much as raw inference latency

Generated with AI, which can make mistakes.