Beyond Code Assistance with GPT-4: Leveraging GitHub Copilot and ChatGPT for Peer Review in VSE Engineering

Abstract

Most companies are Very Small Entities (VSEs), meaning they have fewer than 25 employees. Primarily domain specialists, these companies lack in-house expertise in important areas such as security and reliability engineering, process improvement, Quality Management (QM) and Systems Engineering (SE). VSEs struggle to adhere to Stan- dard Operating procedures (SOP), and research has shown that con- tractual obligations to follow industry standards and best practices have little effect on actual engineering. This paper describes a case study that explored the potential of Large Language Models (LLMs) to support engineering best practices at a VSE by taking on the role of an expert peer in areas where the company had a skills gap. Aiwell, a Norwegian producer of building automation equipment, used ChatGPT, GitHub Copilot and GPT-4 to assess the quality of their system and stakeholder requirements. A GPT-4 foundation model with no additional training was given links to reference materials on requirements engineering pro- duced by The International Council on Systems Engineering (INCOSE) and allowed to participate in discussions on the same digital collabora- tion platform as the human engineers. The study found that AI-assisted requirement reviews immediately and positively impacted the entire en- gineering process, supporting the feasibility of integrating advanced AI technologies in VSEs, even with limited training and resources. Partici- pants highlighted the complementary nature of human intelligence and AI, where LLMs augmented human judgment through dialogue, leading to enriched engineering practices. Ethical and data privacy considera- tions also emerged as central themes, emphasising the need for proactive measures.