This pilot study explores the comparative effectiveness of ChatGPT versus Traditional Online Search (TOS) methods in supporting novice second-year undergraduates with no prior R or RStudio experience in completing a data visualization assignment. Participants (n=20) were randomly assigned to two groups (n=10 each) and given approximately 1 hour and 50 minutes across two class periods to load a built-in dataset, explore its structure, and create visualizations using either ChatGPT or TOS. Assignments were graded on code accuracy, use of RStudio features, visualization quality, documentation, and presentation. Post-assignment surveys assessed perceived ease of use, clarity of guidance, engagement, and understanding of R programming and visualization techniques. Preliminary results indicate the ChatGPT group significantly outperformed the TOS group in assignment scores and survey ratings (e.g., higher means in ease of use and understanding), aligning with expected effects of large language models in reducing cognitive load through personalized guidance. Qualitative feedback underscored ChatGPT's interactivity versus TOS's information overload. Distinct from broader meta-analyses, this pilot provides initial classroom-specific insights into AI advantages for short-term programming learning. Implications include hybrid AI-TOS frameworks to address overreliance, but these frameworks have limitations, including a small sample size and no long-term assessment. Future studies should be scaled with controls and pretests.

Authors:
Amit Lal, Unvieristy of the Pacific, United States

About the Presenter(s)
Dr. Amit Lal is a Professor of Information Systems teaching computer science and business analytics at Modesto Junior College and the University of the Pacific. He studies how AI supports learning and is developing frameworks for responsible AI use.

Connect on Linkedin
https://www.linkedin.com/in/proflal

See this presentation on the full schedule – On Demand Schedule

Bookmark