BAO Jinsong, LI Jianjun, YUAN Yi, Lü Chaofan, WANG Sen. Augmented Reality Assembly Method Assisted by Large Language Models[J]. Aeronautical Manufacturing Technology, 2024, 67(16): 107-116.
BAO Jinsong, LI Jianjun, YUAN Yi, Lü Chaofan, WANG Sen. Augmented Reality Assembly Method Assisted by Large Language Models[J]. Aeronautical Manufacturing Technology, 2024, 67(16): 107-116. DOI: 10.16080/j.issn1671-833x.2024.16.107.
Augmented Reality Assembly Method Assisted by Large Language Models
Augmented reality-based assembly guidance systems superimpose digital information onto physical scenes to effectively guide complex assembly tasks. However
the gap between humans and the physical world in the assembly environment is huge
and the information to be fused into the physical world needs to be prepared in advance and triggered manually during the assembly process. The study of real-time and ubiquitous prompts is a hot research topic for complex assembly in augmented reality environment. In this paper
we propose an augmented reality assembly method based on the assistance of large language models (LLMs)
the core of which is to use LLMs as another brain in the assembly process
providing ubiquitous assembly guidance and prompts support for technique information. Firstly
a system of LLMs-assisted augmented reality assembly method is established
and the elements and interrelationships of the system are analysed. Secondly
a matching process information model is constructed for LLMs environment. Then
the assisted guidance decision-making method and process based on LLMs are given. Finally
combined with a cable assembly expertise
a professional question answering system is developed to realize the smart assisted guidance of LLMs. Results show that the assembly pass rate increased by 15% and the effectiveness of the method is verified through several cases.