Andrii Bidochko

• Updated: July 8, 2026
• 7 min read

Reflective Dialogue or Prompt Refinement? Effects of Tutor Scaffolding on Students’ Independent LLM Use for Programming

Direct Answer

The paper introduces two contrasting LLM‑based tutoring scaffolds—Socratic‑Guidance (SG) and Prompt‑Refinement (PR)—and shows that while both achieve comparable short‑term task performance, the SG approach yields stronger long‑term learning gains and more sophisticated, understanding‑driven prompting when students later interact with an unconstrained LLM. This matters because it identifies a concrete design lever (dialogic questioning) that can turn LLMs from mere answer engines into catalysts for deeper, transferable programming expertise.

Background: Why This Problem Is Hard

Large Language Models (LLMs) such as ChatGPT have become ubiquitous assistants for coding, yet their educational value hinges on how learners engage with them. Two persistent bottlenecks limit the promise of LLM‑driven tutoring:

• Surface‑level reliance: Students often treat LLM outputs as black‑box answers, copying code without internalizing the underlying concepts.
• Prompt engineering opacity: Crafting effective prompts is a skill in itself; without guidance, novices either over‑specify (stifling creativity) or under‑specify (receiving irrelevant results).

Traditional computer‑science education mitigates these issues through human instructors who ask probing questions or provide step‑by‑step scaffolding. Replicating that dynamic at scale with LLMs is non‑trivial because:

• LLMs lack a persistent model of a student’s knowledge state.
• Existing tutoring interfaces either hand over full control to the learner (risking inefficient prompting) or impose rigid templates that limit exploration.

Consequently, researchers have called for systematic studies that compare different scaffolding strategies and measure not just immediate correctness but also the evolution of prompting behavior over time.

What the Researchers Propose

The authors design a controlled experiment that pits two pedagogical paradigms against each other:

• Socratic‑Guidance (SG) tutor: An LLM‑driven agent that engages students through a sequence of open‑ended, dialogic questions. The tutor deliberately refrains from providing direct code solutions, instead prompting learners to articulate reasoning, predict outcomes, and reflect on errors.
• Prompt‑Refinement (PR) tutor: An LLM‑driven agent that focuses on the mechanics of prompt construction. It iteratively asks the student to clarify intent, add constraints, and test variations, aiming to produce a high‑quality prompt that yields a correct code snippet.

Both tutors operate within the same mobile‑robotics programming curriculum, but they differ in the role they assign to the learner:

• SG treats the learner as a co‑thinker, emphasizing conceptual articulation.
• PR treats the learner as a prompt engineer, emphasizing syntactic precision.

By isolating these roles, the study can attribute downstream differences in learning to the nature of the scaffolding rather than to content difficulty or instructor bias.

How It Works in Practice

The experimental workflow unfolds in three phases:

1. Initial onboarding (Week 1): All participants receive a brief tutorial on the LLM interface and the basics of mobile‑robotics programming.
2. Guided intervention (Weeks 2‑7): Students are randomly assigned to either the SG or PR tutor. Each week they complete a series of programming challenges (e.g., path planning, sensor fusion) while interacting with their assigned tutor. The tutor logs every exchange, including question type, prompt revisions, and code outcomes.
3. Unconstrained project (Weeks 8‑10): The scaffolding is removed. Students work on an open‑ended robotics project using a vanilla LLM (no tutor). Their prompts, code submissions, and self‑assessment scores are collected for analysis.

Key components of the system include:

• LLM core: A state‑of‑the‑art generative model (e.g., GPT‑4) that powers both tutors and the final unconstrained LLM.
• Dialogue manager: A rule‑based layer that selects SG questions (e.g., “What would happen if you removed this sensor?”) or PR refinement steps (e.g., “Can you add a constraint about execution time?”).
• Interaction logger: A database that captures timestamps, prompt text, tutor feedback, and student code, enabling fine‑grained post‑hoc analysis.

What sets this approach apart is the intentional separation of “knowledge‑eliciting” versus “prompt‑optimizing” dialogues, allowing the researchers to observe how each style shapes the learner’s mental model of both the programming task and the LLM itself.

Evaluation & Results

The study measured three outcome dimensions:

Task Performance During Guided Use

Both SG and PR groups achieved similar correctness rates on weekly challenges (≈ 78 % vs. 80 %). This indicates that, in the short term, either scaffold can help students produce functional code.

Prompting Patterns During Guided Use

Analysis of logged prompts revealed convergent behaviors:

• Both groups increasingly incorporated task‑specific keywords (e.g., “LIDAR”, “PID controller”).
• Prompt length stabilized around 30‑40 tokens, suggesting a shared efficiency ceiling.

Thus, the immediate impact on prompt engineering was comparable.

Learning Gains and Later LLM‑Use

When the scaffolding was removed, stark differences emerged:

• Conceptual retention: SG students scored on average 12 % higher on a post‑project quiz that probed underlying robotics concepts.
• Understanding‑driven prompting: SG participants were 1.8× more likely to craft prompts that referenced algorithmic reasoning (“use a Kalman filter to fuse sensor data”) rather than merely stating desired output.
• Code quality: SG‑generated solutions exhibited fewer logical bugs and required fewer LLM re‑queries.

These findings suggest that Socratic questioning cultivates a transferable skill set—students learn how to think about the problem, then translate that thinking into effective prompts.

For full methodological details, see the original arXiv paper.

Why This Matters for AI Systems and Agents

From a systems‑building perspective, the results provide actionable guidance for anyone constructing LLM‑augmented development tools, educational platforms, or autonomous agents:

• Designing tutor agents: Embedding Socratic‑style questioning into an agent’s dialogue policy can improve long‑term user competence, not just immediate task success.
• Prompt orchestration frameworks: Rather than exposing raw prompt‑editing interfaces, platforms can layer reflective prompts that ask “why” before “what”. This aligns with the observed “understanding‑driven” prompting pattern.
• Evaluation metrics: Traditional accuracy‑only benchmarks miss the nuanced benefit of Socratic scaffolding. Future agent evaluations should include measures of prompt quality and conceptual transfer.
• Product integration: Companies building AI‑powered coding assistants can differentiate by offering a “Socratic mode” that guides users through reasoning steps, potentially reducing support tickets and increasing user retention.

Practically, developers can leverage existing UBOS capabilities to prototype such experiences:

• Integrate conversational flows via the UBOS platform overview to manage stateful dialogues.
• Deploy AI marketing agents that incorporate Socratic questioning for client onboarding using the AI marketing agents service.
• Automate the prompting workflow with the Workflow automation studio, ensuring that each step includes a reflective checkpoint.
• Connect to OpenAI’s models through the OpenAI ChatGPT integration for high‑quality code generation.
• Extend the tutoring interface to messaging platforms using the Telegram integration on UBOS, reaching learners where they already collaborate.

What Comes Next

While the study makes a compelling case for Socratic scaffolding, several open questions remain:

• Scalability of dialogic questioning: Generating high‑quality, curriculum‑aligned Socratic questions at scale may require fine‑tuned LLMs or hybrid rule‑based systems.
• Domain transferability: The experiment focused on mobile robotics; it is unclear whether the same benefits hold for other programming domains such as web development or data science.
• Longitudinal impact: Follow‑up studies over semesters could reveal whether the observed gains persist into professional coding practice.
• Hybrid scaffolds: Combining Socratic dialogue with targeted prompt‑refinement hints might capture the best of both worlds.

Future research could explore these avenues by building modular tutoring pipelines on top of the Enterprise AI platform by UBOS, which supports custom LLM orchestration and data logging. For startups eager to experiment, the UBOS for startups program offers sandbox environments and pricing incentives.

In the meantime, educators and product teams can start small: pilot a Socratic‑question module in an existing coding assistant, collect prompt‑quality metrics, and iterate based on learner feedback. As LLMs become ever more capable, the differentiator will increasingly be how intelligently we guide users to think, not just what answers we give them.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.