Multimodal AI for Architecture Education
Research Assistant, Team Lead, Generative AI Engineer & Researcher
This project plans to train a general multi-modal model for architecture, localize it for HKU, and build AI tools for formative assessment and gamified learning.LLM，AWS，React
The project will first curate existing architectural cases from the web and convert them into multi-turn Q&A tasks for AI training, with the goal of developing a multimodal expert model in architecture. The model will be trained across multiple dimensions—image understanding, integrated reasoning tied to design intent, and expert-level analysis, critique, and reflection on case studies—to cultivate a refined architectural “taste.”Next, we will draw on tens of thousands of student projects collected by the architecture school over the past two decades, select representative samples, and localize the expert model to critique student work. This will help students understand the current level of their projects and identify concrete avenues for improvement.We also plan to introduce gamified design elements (such as points and dashboards) to quantify each student’s positioning and progress along different personalized paths in architecture, thereby encouraging consistent learning. By incorporating RAG (Retrieval-Augmented Generation), the system will surface targeted reference cases to support students’ day-to-day study.Finally, leveraging the expert model’s interpretive capabilities, we aim to evolve from a basic architectural Q&A chatbot into a context-aware teaching assistant that can understand multimodal student work and learning states, and provide guidance accordingly.As the project lead and primary developer, I will curate the architectural training corpus, train and localize the expert model, and co-develop a scalable, maintainable product in collaboration with a senior engineer at IBM. I will also partner with researchers from PolyU and HKU to conduct user studies and experiments. We plan to publish two to three papers spanning education, HCI, and AI.
Curios
Research Assistant, Full-stack developer
HKU’s metaverse concept-related immersive learning platform.React, Firebase
Curios offers 5 selectable classroom scenes where students can customize their appearance and control a 3D avatar to greet others in a virtual environment. They can also make calls and send private or group messages.Students collaborate on group assignment whiteboards, editing content and adding 3D decorations. Professors, visitors, and students can like and comment on assignments.The project includes flexible controls for smooth facilitation—for example, quick spatial teleportation and convenient instructor tools to modify group setups: swapping group assignment panels, adding members, and adjusting management permissions.A rich set of hooks captures in‑scene interaction data (such as editing durations and dwell time). This data is stored in a cloud database and powers features like in‑platform leaderboards, as well as research use.To ensure a seamless user experience, the codebase is substantial: at least 100 scripts with 500+ lines each, spanning frontend and backend deployment, user authentication and authorization, researcher-facing control pages, end‑to‑end system logic from R3F (React Three Fiber) scene rendering to UI interactions, and cloud functions that interface with the database and use caching to accelerate loading.I was the sole maintainer and developer on the project. This was my first transition from Unity to web development, and I resolved 150+ issues across the stack. Additional contributions include upgrading rendering across all five scenes to a more efficient system, optimizing the existing UI, and implementing group broadcasting and communication features. In a previously sparsely documented codebase, I added hundreds of comments, compiled an index of code entry points for major features, and wrote a concise handover guide to help the next developer continue the work.
Digital Reality Painter
Nvidia Joint Lab STEM Intern, Technical Lead and concept designer
VR painting tool integrating AIGC methods for 3D, 2D, and projection-based art creation.Unity (Oculus), ComfyUI
Hong Kong International Licensing Show, 2025
PolyVerse - Culture and Art Tech Exhibition 2025
Paper under submission (IEEE VR 2026)
The project integrates AI across three layers of VR painting: 3D spatial strokes generate 3D models, 2D surface strokes generate 2D decals, and lighting strokes drive an AI-powered projector that casts dynamic lighting. All generative features are driven by ComfyUI workflows.The system includes a suite of editing tools, such as grouping selected objects for unified scaling and translation, duplication and deletion, and a magnet-based deformation tool (similar to a “liquify” effect in digital painting).The project supports both hand-tracking gestures and controllers.Development took just one week. A senior from PolyU validated the core logic for 3D strokes and magnet deformation, while I completed the remaining development, including system refinement and all AI workflow implementation.
Beauty of Canton Porcelain
Nvidia Joint Lab STEM Intern, Generative AI Engineer & Unity Developer
The project utilizes Lora to learn intangible cultural heritage patterns and integrates AI into collaborative art creation.Unity, ComfyUI, GPT-API
Journal of Human Computer Interaction (second author)
Featured on television channels such as Channels News Asia
PolyVerse - Culture and Art Tech Exhibition 2025
The project began with a senior from PolyU gathering image samples of Guangcai porcelain and validating the basic interaction interface.Based on these samples, I re-cropped and optimized the dataset and trained a LoRA model suitable for creative production.I then refactored the interaction system from the ground up to improve performance and deliver smoother mouse-and-keyboard control, and I developed GPT–ComfyUI workflow integrations.I also added editing features for inserting decals with scaling and rotation, as well as an AI-powered visual analysis function for porcelain.
GeneyMAP
Part-time Research Assistant, Sole developer
Tool for designing GenAI-empowered user journey mapping.Unity, GPT-API
CHI 2025 Late-Breaking Work (co-first author)
As one of the earliest projects at the PolyU Design School to integrate the GPT-API, I first explored the capability boundaries of GPT-4 for creating User Journey Maps. I proposed a table-based editing workflow and an AI submission/retrieval mechanism, and validated a system where stable JSON structures enable both granular and global table adjustments via AI.Over the next two iterations, working from a colleague’s Figma designs at PolyU, I implemented complex, precise UI interactions in Unity (though a web stack would have been more suitable). Features included AI-suggested or custom table dimensions, collapsible/expandable sections, adding and removing rows and columns with reflow, rapid AI corrections for individual cells, and AI-generated reference imagery for inspiration.
GPT-World
Master’s thesis project, Supervisor: Prof. Henry Duh
An AI-driven DnD game featuring text-based role-playing, with JSON-based object generation, multi-agent AI integration, quantified game mechanics, and automated data retrieval via LLM.Unity (DIY Framework, FEEL), GPT-API
This is the project I’m most proud of and passionate about. Players start by describing the world they want to explore and defining their own character. The system then automatically generates class templates needed for the game (for example, Place and Character). Each class comes with well-defined attributes; a Character might have dimensions like Magic, Strength, Wisdom, Values, and Secrets. The system also creates Events and instantiates entities under each class, such as concrete locations and NPCs.Compared to AI Dungeon (aidungeon.com), the gameplay is more structured and engaging. The system proactively introduces challenges, and player choices have quantifiable, systemic effects on entities. For instance, successfully cultivating land increases a location’s Prosperity, rather than just advancing a narrative beat. It’s more than a storytelling simulator—it’s a dynamic, stateful game world.The original goal was to build a fully AI-generated and AI-driven homage to Taiko Risshiden V. Over five iterations, I developed a stable JSON-based content pipeline. The system evolved from directly generating entities to first producing flexible class templates, enabling a modular, extensible architecture. A multi-agent setup now handles different subsystems, progressively increasing flexibility. As the project evolved from GPT-3.5 to GPT-4o, the game became notably more adaptable and fun.
The rhythm of life
Master’s Assignment
An art installation aims to express vibrant vitality through the rhythm of sound and visuals.Visitors can use various methods to rotate or shake the water bottles in their hands, and our device will provide auditory and visual feedback based on their interactions.Ableton & Max, Microcontroller
Cɑs1n0
Bachelor’s thesis project
This project focuses on the four elements of "Interaction," "Metaverse," "Fun," and "AI" and is dedicated to helping players improve their adaptability and observation skills in virtual environments while learning the basics of classical cryptography.Unity (DIY Framework, Inworld AI)
GridSystem
Tech Intern - Unity, Supervisor: Zhang Liming (Tech Director)
This Demo project is my work as an intern in Unity, aiming to verify the feasibility and effect of grid system construction.
By rewriting the plug-in -- Grid Placement System, more complex construction functions are realized, such as withdrawing instructions, archiving, area operations, etc.Unity (Rewrite Plugin)
Mono
Leader & Developer & Artist
In this AR game, white squares in bright light are separated from opposite black shadows. Players need to build blocks in a way that matches the given puzzle projection. Players can switch between black and white environments to observe the existence of blocks or shadows. The game runs through a fairy tale full of philosophical sayings.Unity (Vuforia, DOTween)
Best AR Game, Global Game Jam 2022, Hong Kong
Li QingZhao
Leader & Developer
A VR visual novel game that tells the life of Li Qingzhao, a famous ancient female lyricist. The game includes small games such as answering questions and escaping.Unity (DOTween, Fungus)