Learning multi-granular worker intentions from incomplete visual observations for worker-robot collaboration in construction

Recognizing a worker’s intentions is an important prerequisite to enable smooth human-robot collaboration in construction. However, the highly dynamic construction workplace and long-horizon construction tasks prevent robots from obtaining long-term observations, which is detrimental to information accumulation and intention disambiguation. We present (1) a data and knowledge fusion strategy that combines visual contextual information and task knowledge to reduce the information loss caused by incomplete observations, and (2) a multi-granularity worker intent recognition model to explore the optimal granularity by comparing the intent modeling capabilities of different granularities. Results show that the proposed method can recognize multi-granular worker intentions with macro-F1 scores higher than 0.85, and that the intermediate activity is the best-suited granularity as it strikes a good balance between intention recognition accuracy and intention modeling capability. For more details, please read our paper.