Multimodal Large Language Model–Powered Learning Companions for Interactive and Process-Based Education

This project proposes a pedagogically aligned Multimodal Large Language Model (MLLM) system featuring lifelike, voice-enabled avatars to enhance interactive and process-based learning. The system addresses critical educational challenges—such as limited authentic practice and insufficient personalized feedback—by offering virtual patient interactions for medical consultations, AI-guided topic review construction, and an AI reviewer for tailored feedback. A key innovation involves fine-tuning MLLMs on expert Socratic dialogues to guide students without directly providing answers, thereby fostering deeper learning and maintaining academic integrity. The system will be piloted across three testbed courses: COMP 5423 (Deep Learning for Medical Image Analysis), COMP 4421 (Image Processing), and LIFS 1980 (Guided Study on Biomedical and Health Sciences – Virtual Patient Clinical Communication Module). While initially deployed in computer science and life sciences, the project is designed for flexible expansion across all HKUST disciplines, with a strategic focus on supporting the university’s forthcoming medical school.