CodeMap

Abstract

Understanding an unfamiliar codebase is an essential task for developers in various scenarios, such as during the onboarding process. Especially when the codebase is large and time is limited, achieving a decent level of comprehension remains challenging for both experienced and novice developers, even with the assistance of large language models (LLMs). Existing studies have shown that LLMs often fail to support users in understanding code structures or to provide user-centered, adaptive, and dynamic assistance in real-world settings.

To address this, we propose learning from the perspective of a unique role, code auditors, whose work often requires them to quickly familiarize themselves with new code projects on a weekly or even daily basis. We recruited and interviewed 8 code auditing practitioners to understand how they master codebase understanding. We identified four design opportunities for an LLM-based codebase understanding system: supporting cognitive alignment through automated codebase information extraction, decomposition, and representation, as well as reducing manual effort and conversational distraction through interaction design.

To validate these four design opportunities, we designed a system prototype, CodeMap, that provides dynamic information extraction and representation aligned with the human cognitive flow and enables interactive switching among hierarchical codebase visualizations. We conducted a user study with nine experienced developers and six novice developers. Our results demonstrate that CodeMap significantly improved users' perceived intuitiveness, ease of use, and usefulness in supporting code comprehension, while reducing their reliance on reading and interpreting LLM responses by 79% and increasing map usage time by 90% compared to the static visualization analysis tool.

Research Questions

Research Pipeline

System Architecture

CodeMap consists of three core components working together: a Prompt Generator that assembles task-specific prompts using templates, a Retrieval Augmented Generation (RAG) System with a vector store and LLM that produces multi-level responses (global, local, and detail levels), and a Map Interaction Client featuring a graph viewer, textual explanation viewer, and chatbot for interactive exploration.

🏗️ Architecture diagram placeholder

System architecture of CodeMap: Prompt Generator assembles requests, RAG System retrieves and generates multi-level responses, and the Map Interaction Client provides graph visualization, textual explanations, and chatbot interaction.

• Prompt Generator: Combines prompt templates with prompt assembler to create structured requests from user queries and codebase context.
• RAG System: Uses a vector store for retrieval and LLM for generation, producing responses at global, local, and detail levels.
• Graph Viewer: Renders received dot language as interactive graphs, allowing node selection and exploration.
• Textual Explanation Viewer: Displays structured natural language explanations alongside the graph.
• Chatbot: Collects user queries and selected node context for follow-up interaction.

Prompt Design

CodeMap uses carefully designed prompt templates that decompose complex code understanding tasks into step-by-step instructions. Each prompt includes a task description, content and format requirements, and few-shot examples with generic placeholders that work across languages and projects. The system outputs both dot-format graphs for visualization and JSON-format structured textual explanations.

📝 Prompt template diagram placeholder

Prompt design template: Input (user request + retrieved codebase content) → Prompt Template (task description, content requirements, few-shot examples) → Output (dot format graph + JSON structured textual explanation).

User Study

We conducted comprehensive user studies to validate CodeMap. In the summative evaluation, 9 experienced developers and 6 novice developers completed three code understanding tasks using different tool configurations: CodeMap + VSCode, ChatGPT + VSCode, and Understand + VSCode. Each participant spent approximately 1.5 to 2 hours, with each task taking around 25 minutes followed by an interview and questionnaire.

👥 User study procedure placeholder

Summative user study procedure: Participants complete three tasks (~25 min each) with different tools, followed by an interview and questionnaire (~10 min). Total commitment: 1.5–2 hours per participant.