Research Paper

Understanding Codebase
like a Professional!

Human-AI Collaboration for Code Comprehension
Jie Gao1, Yue Xue2, Xiaofei Xie3, Junming Cao4, SoeMin Thant5, Erika Lee6, Bowen Xu7
1Johns Hopkins University·2Metatrust Labs·3Singapore Management University·4Fudan University
5Singapore-MIT Alliance for Research and Technology·6UC San Diego·7NC State

Abstract

Understanding an unfamiliar codebase is an essential task for developers in various scenarios, such as during the onboarding process. Especially when the codebase is large and time is limited, achieving a decent level of comprehension remains challenging for both experienced and novice developers, even with the assistance of large language models (LLMs). Existing studies have shown that LLMs often fail to support users in understanding code structures or to provide user-centered, adaptive, and dynamic assistance in real-world settings.

To address this, we propose learning from the perspective of a unique role, code auditors, whose work often requires them to quickly familiarize themselves with new code projects on a weekly or even daily basis. We recruited and interviewed 8 code auditing practitioners to understand how they master codebase understanding. We identified four design opportunities for an LLM-based codebase understanding system: supporting cognitive alignment through automated codebase information extraction, decomposition, and representation, as well as reducing manual effort and conversational distraction through interaction design.

To validate these four design opportunities, we designed a system prototype, CodeMap, that provides dynamic information extraction and representation aligned with the human cognitive flow and enables interactive switching among hierarchical codebase visualizations. We conducted a user study with nine experienced developers and six novice developers. Our results demonstrate that CodeMap significantly improved users' perceived intuitiveness, ease of use, and usefulness in supporting code comprehension, while reducing their reliance on reading and interpreting LLM responses by 79% and increasing map usage time by 90% compared to the static visualization analysis tool.

Research Questions

RQ1. How do auditors cognitively navigate and understand an unfamiliar codebase? What information are they seeking during the process?

RQ2. What types of assistance do they seek from ChatGPT during the process? What challenges do they encounter? What suggestions do they have?

RQ3. How does our expert-inspired system affect experienced and novice developers' perceived usefulness, intuitiveness, ease of use, and others?

RQ4. How are experienced and novice developers' system usage behaviors affected, and why?

Research Pipeline

Stage 1

Formative Interview

Semi-structured interviews with 8 code auditors to understand cognitive navigation strategies and ChatGPT usage patterns during code understanding.

Stage 2

Design Opportunities

Identified key cognitive support needs: information extraction, decomposition, representation, and decreased manual effort through interaction.

Stage 3

Prototype Design

Built CodeMap with dynamic information extraction, visualization via interactive dot graphs, and natural language interaction powered by RAG + LLM.

Stage 4

Prototype Validation

Summative user studies with 9 experienced and 6 novice developers comparing CodeMap + VSCode, ChatGPT + VSCode, and Understand + VSCode.

Stage 1

Formative Interview

Key findings from semi-structured interviews with 8 code auditing practitioners, organized by understanding level.

Interview Transcripts & Qualitative Analysis

Participant Demographics

ID Occupation Onboarding Freq. Audited Projects LLM Usage Coding Exp. Auditing Exp.
P1Smart Contract AuditorDaily~Weekly200-250Daily7 yrs3 yrs
P2Smart Contract AuditorWeekly100-150Daily2 yrs2 yrs
P3EngineerWeekly100-150Weekly3 yrs1.5 yrs
P4Smart Contract AuditorWeekly100-150Daily2.5 yrs2.5 yrs
P5Smart Contract AuditorWeekly50-100Daily2 yrs2 yrs
P6CS Master StudentMonthly0-50Daily3 yrs1 yr
P7Smart Contract AuditorWeekly~Monthly0-50Daily4 yrs1 yr
P8Smart Contract AuditorMonthly0-50Monthly3 yrs0.5 yr

Key Findings

Understanding Level ChatGPT Assistance, Challenges & Suggestions Example Quotes
Global
Project Overview
Gather project overview from README files and documentation, including key file names and project background.
Assist in high-level explanation of the codebase
Assist in evaluating codebase complexity
Lack of automated guidance for codebase understanding (requires manual prompting)
LLM tools should provide pre-defined domain-specific prompts
"After obtaining the project, I will read its written content, including white paper and README. I only need to know what each specific contract is for." P1
Global
Business Logic Flow
Understand business logic and function-call flow: main components, relationships, key files, imports, inheritance, and data flows.
Assist in understanding which components use a given variable
Lack of automated guidance (requires manual prompting)
Lack of automated information decomposition (requires manual decomposition)
Lack of filtering for irrelevant code, leading to redundant analysis
LLM tools should provide pre-defined domain prompts
LLM tools should provide codebase structure visualization
"You have a general basic [structure] framework... Some companies will draw a diagram, such as architecture diagram." P6
"Especially when I want to understand the relationships between multiple files, it probably won't be able to help me." P7
Local
Local Level
Locate components, files, variables, and specific contents of interest: important variables, dependency order, key functions and call relationships.
Assist in extracting key information from individual files
Lack of support for cross-file variable relationships
Lack of automated information decomposition (requires manual decomposition)
LLM tools should help users identify which parts to focus on
"There might be an issue [at a certain point], and I'll make a note... Then, after finishing looking at this [overall structure], I will come back and focus on this marked part." P8
Detail
Detailed Level
Seek clarification of specific code snippets, verifying whether the code aligns with current understanding.
Assist in explaining code snippets
Assist in confirming users' understanding
Lack of continuous understanding flow -- clarification questions often interrupt the ongoing reasoning process
"After giving ChatGPT a code snippet, [I may have] many questions. After you ask question a, when you then ask question b, there is a process interruption." P5
Stage 2

Design Opportunities

Four design opportunities identified from the formative interview for an LLM-powered codebase understanding system.

DO 1

Information Extraction Aligned with Understanding Flow

Support a structured three-level chain of comprehension across global, local, and detailed levels, guiding users through sequential stages rather than displaying all codebase information at once.

DO 2

Information Decomposition to Align with Mental Models

Automate information decomposition to reduce manual effort. LLM systems should break codebases into functional components automatically, helping users grasp different parts without expertise-driven manual work.

DO 3

Information Representation Across Abstraction Layers

Provide structural representations across multiple abstraction layers, visually connecting relationships across different levels to enable easier and more efficient comprehension.

DO 4

Analytical Structure to Reduce Manual Effort

Provide predefined information structures or commonly requested details that are automatically displayed, allowing users to access insights without manually formulating queries.

Stage 3

Prototype Design

CodeMap provides an interactive interface for exploring codebase structure through visualization and natural language.

View on GitHub

System Demo

System Architecture

CodeMap consists of three core components working together: a Prompt Generator that assembles task-specific prompts using templates, a Retrieval Augmented Generation (RAG) System with a vector store and LLM that produces multi-level responses (global, local, and detail levels), and a Map Interaction Client featuring a graph viewer, textual explanation viewer, and chatbot for interactive exploration.

Prompt Design

P1R1-P2R2 Business Component (Global Map)
Analyze this project based on the vector store and provide the following information:
1. A JSON format output containing:
  a. Project overview, including a brief summary, the main entry point of the project, and a simple description of how to run the project.
  b. Key modules or components of the project and their basic functions.
  c. A step-by-step guide for understanding the project's architecture and functionality.
2. A DOT language graph representing the project structure and relationships.
3. One module can contain multiple components

For the JSON output:
- Each description should be concise and easy to understand.
- Avoid technical jargon and complex explanations.
- Focus on providing practical, actionable information for a Python beginner.

For the DOT graph:
- Use the dot language to create the graph.
- Annotate the purpose of each edge.
- Use "components: (component description)" to show the main component names.
- Output important functions, variables, and file paths.
- Show the business relationship and business flow between components in detail.

Ensure that:
1. There is no duplication in the output content, and each JSON field contains unique information.
2. The output paths related to files follow the information in the task1_filepath.md.
3. The output includes all the files in the vector database, with no exceptions.
4. The Modules scope must include all code in the file, you must not omit any Modules.
5. If the projectArchitectureGuide in the JSON output include a filename, the filename must output link with the module name.
6. Do not leave any ambiguous description, and all steps in the projectArchitectureGuide must have a clear logical connection.
P3R3 Business Local Graph
Generate a local map under the {selectedNode} business component:
1. Using DOT language
2. Annotate the purpose of each edge
3. Output important functions, variables, and file path

[Includes DOT structure with classes, functions, variables subgraphs and relationship connections]
P4R4 Business Flow Analysis
Analyze the highlighted business flow for {selectedNode} and provide the following information in JSON format:
{
  "Component Flow Overview": "A brief overview of the business flow.",
  "Role of the Highlighted Component and its flow in the Overall Framework": "Explanation of how this business flow fits into the larger system.",
  "Roles of Each Sub Component Under This Component": {
    "Component1": "Role and purpose of Component1",
    "Component2": "Role and purpose of Component2"
  }
}
Provide the output strictly in this JSON format without any additional text or explanations.
P5R5 Business Component Relation Analysis
Provide the following information in JSON format:
{
  "Component Name": "Name of the Selected Component",
  "Neighbourhood Components in the Project level": [
    {
      "name": "Component Name",
      "role": "Description of the neighbourhood component role in the project level",
      "interactions with selected component": "How the selected component interacts with the neighbourhood component"
    }
  ]
}
Provide the output strictly in this JSON format without any additional text or explanations.
P6R6-P7R7 Function Call (Global Map)
Analyze the project, focus on all above files, and return the following information:
1. JSON format analysis:
{
  "Overview": "A brief overview of the entire project structure",
  "Modules": [
    {
      "Module": "Module Name: Brief description of the module's purpose",
      "classes": [
        {"Class": "ClassName1: Concise description of the class, including its purpose and relationships with other classes"}
      ],
      "files": [
        {"File": "filename1: Detailed description of the file's contents and purpose"}
      ]
    }
  ],
  "Relationships": [
    "0. This whole project starts from ClassX, and then goes into ClassA.",
    "1. ClassA uses ClassB by calling functions X and Y, in order to share data between ClassA and ClassB (e.g., varM, varN).",
    "2. After ClassA uses ClassB, the results of ClassB will be passed into ClassC, which then..."
  ]
}

2. Inheritance graph using DOT language with modules, class descriptions, key functions, key variables, file paths, and detailed relationship annotations.

Ensure that:
1. The information in the dot graph aligns with the JSON.
2. Class names are explicitly shown as "ClassName.Class".
3. Descriptions are easily understandable by beginners.
4. Output must cover all code and files provided.
5. Relationships provide a numbered list (starting from 0) describing the project flow.
P9R9 Function Call Local Graph
In this inheritance graph, provide me with the function call flow of the [selected node] as dot file content, keeping the main functions and their call relationships in the graph, while presenting secondary functions or complex call relationships in the form of comments.

{selectedNode}

Output format: using DOT language
- Example call flow
- Explanation of each function in each node

[Includes few-shot example with main classes, methods, relationships, subgraphs for method flows, and notes]
P8R8 Function Call Local Description
Explain the details for the selected node {selectedNode}. Provide the output in the following JSON format:
{
  "highlightedInheritance": [
    {
      "name": "Class Overview",
      "value": "Brief overview of the Selected Class and its flow"
    },
    {
      "name": "Role of the Class in the Overall CodeBase",
      "value": "Description of how this class fits in the overall codebase"
    },
    {
      "name": "Roles of the key functions under this Class",
      "value": {
        "function1": "Role and purpose of function1",
        "function2": "Role and purpose of function2"
      }
    }
  ]
}
Ensure the output is strictly in this JSON format without any additional text or explanations.
P10R10 Function Call Local Explain
Explain the relevant inheritance flow for the selected node {selectedNode}. Provide the output in the following JSON format:
{
  "relevantInheritance": [
    {
      "name": "Class Name",
      "key": "inheritance_flow_overview",
      "value": "Name of the Selected Class"
    },
    {
      "name": "Neighbourhood Classes in the Project level",
      "key": "function_roles",
      "value": {
        "name": "Class Name",
        "role": "Description of the neighbourhood Class role in the project level",
        "interactions with selected Class": "How the selected Class interacts with the neighbourhood Class"
      }
    }
  ]
}
Ensure the output is strictly in this JSON format without any additional text or explanations.
Iteration Iteration Prompt
[output from last round]
[file list]
Based on this file list, continue to supplement and improve the dot graph according to the same requirements, including all [N] files in the dot graph.
Stage 4

Prototype Validation

We conducted summative user studies with 9 experienced developers and 6 novice developers. Each participant completed three code understanding tasks using different tool configurations: CodeMap + VSCode, ChatGPT + VSCode, and Understand + VSCode. Compared to the static visualization tool (Understand + VSCode), CodeMap reduced participants' reliance on reading and interpreting LLM responses by 79% and increased map usage time by 90%.

What Participants Said

"It allows me to intuitively understand the entire project by dividing into modules and shows what each module does."
-- Expert P7
"This [project architecture understanding guide] is so amazing because when you handle a project for the first time and have no idea what it is about, this is a very good guide."
-- Novice P5
"ChatGPT just analyzes for me whatever I tell it to analyze. So I need to know what I want to analyze."
-- Expert P3, on ChatGPT limitations
"If you want to learn something new and difficult, it is easier to learn through a diagrammatic approach, like a UML class diagram, which ChatGPT fails to give."
-- Novice P6
"If you just read the code by yourself, you might miss some details like some key functions... this global understanding can provide me with a blueprint-like view."
-- Novice P1
"In CodeMap, a set of standard questions is already present, such as 'What is the business problem about?', 'What are the relationships between components?'"
-- Novice P6, on predefined prompts

BibTeX

@misc{gao2026understandingcodebaselikeprofessional, title={Understanding Codebase like a Professional! Human-AI Collaboration for Code Comprehension}, author={Jie Gao and Yue Xue and Xiaofei Xie and SoeMin Thant and Erika Lee and Bowen Xu}, year={2026}, eprint={2504.04553}, archivePrefix={arXiv}, primaryClass={cs.HC}, url={https://arxiv.org/abs/2504.04553}, }