Lectures

Week 1

  • Course Introduction. We will discuss the topics covered in the course, the evaluation methods and marking scheme.
  • How Often Are LLMs Used In Software Engineering? This week we will explore the breadth of the applications of Large Language Models (LLMs) in Software Engineering research. To explore the breadth we will survey the most recent instances of Software Engineering research conferences including: ICSE, ASE, FSE, ICST, ISSTA. We will divide into groups of 2 and each group will review the papers published in a particular conference. Each paper that is an LLM-based technique will be recorded and we will then discuss the results.

Week 2

  • An Overview of Large Language Models (LLMs). We’ll review the state-of-the-art in LLMs. Consider how LLMs are trained and pre-trained. We’ll also learn about different kinds of LLMs including encoder-only, encoder-decoder and decoder-only models.
    • How Large Language Models Work [YouTube]
  • Group Paper Discussion. We will discuss the following paper in class:

Week 3

WEEK 4

  • Student Presentations on LLMs in Software Testing
    • Zhe Liu, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Xing Che, Dandan Wang, and Qing Wang. 2024. Make LLM a Testing Expert: Bringing Human-like Interaction to Mobile GUI Testing via Functionality-aware Decisions. In Proc. of the IEEE/ACM 46th International Conference on Software Engineering (ICSE ’24). pp. 1–13. [presented by: Mosarrat Rumman]
    • S. Rahman, A. Baz, S. Misailovic and A. Shi. 2024. Quantizing Large-Language Models for Predicting Flaky Tests. In Proc. of the IEEE Conference on Software Testing, Verification and Validation (ICST 2024). pp. 93-104. [presented by Abdul Wahab]
    • Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. 2024. ChatUniTest: A Framework for LLM-Based Test Generation. In Companion Proc. of the 32nd ACM International Conference on the Foundations of Software Engineering (FSE 2024). pp. 572–576. [presented by Bridget Green]
    • R. Santos, I. Santos, C. Magalhaes and R. de Souza Santos. 2024. Are We Testing or Being Tested? Exploring the Practical Applications of Large Language Models in Software Testing. In Proc. of the IEEE Conference on Software Testing, Verification and Validation (ICST 2024). pp. 353-360. [presented by: Sara Saljoughi]
    • Nadia Alshahwan, Jubin Chheda, Anastasia Finogenova, Beliz Gokkaya, Mark Harman, Inna Harper, Alexandru Marginean, Shubho Sengupta, and Eddy Wang. 2024. Automated Unit Test Improvement using Large Language Models at Meta. In Companion Proc. of the 32nd ACM International Conference on the Foundations of Software Engineering (FSE 2024). pp. 185–196. [presented by Ankita Mukherjee]
    • R. Feldt, S. Kang, J. Yoon and S. Yoo. 2023. Towards Autonomous Testing Agents via Conversational Large Language Models. In Proc. of the 38th IEEE/ACM International Conference on Automated Software Engineering (ASE 2023). pp. 1688-1693. [presented by Dale McInnis]
    • Simone Mezzaro, Alessio Gambi, and Gordon Fraser. 2024. An Empirical Study on How Large Language Models Impact Software Testing Learning. In Proc. of the 28th International Conference on Evaluation and Assessment in Software Engineering (EASE ’24). pp. 555–564. [presented by Stacey Koornneef]
    • Zhe Liu, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Zhilin Tian, Yuekai Huang, Jun Hu, and Qing Wang. 2024. Testing the Limits: Unusual Text Inputs Generation for Mobile App Crash Detection with Large Language Model. In Proc. of the IEEE/ACM 46th International Conference on Software Engineering (ICSE ’24). pp. 1–12. [presented by Emon Roy]
    • Chunqiu Steven Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang. 2024. Fuzz4All: Universal Fuzzing with Large Language Models. In Proc. of the IEEE/ACM 46th International Conference on Software Engineering (ICSE ’24). pp.1–13. [presented by William Conley]
    • Arghavan Moradi Dakhel, Amin Nikanjam, Vahid Majdinasab, Foutse Khomh, and Michel C. Desmarais. 2024. Effective test generation using pre-trained Large Language Models and mutation testing. Inf. Softw. Technol. 171, C (Jul 2024). [presented by Anushka Zaman in a future lecture]
    • Shengcheng Yu, Chunrong Fang, Yuchen Ling, Chentian Wu, Zhenyu Chen. 2023. LLM for Test Script Generation and Migration: Challenges, Capabilities, and Opportunities. In Proc. of the 23rd IEEE International Conference on Software Quality, Reliability, and Security (QRS 2023). pp. 206-217. [presented by Parsa Memarzadehsaghezi in a future lecture]

Week 5

  • Comparing Few-Shot Learning and Fine Tuning for the Detection and Classification of Flaky Tests
    Guest Speaker: Riddhi More, MSc Student, Ontario Tech University

Week 6

  • …

Week 7

  • Project Proposal Presentations. Each project group will present their proposed course project.

Week 8

  • …

Week 9

  • …

Week 10

  • …

Week 11

  • …
  • Updates on Course Projects. Student groups will provide informal updates on their course projects. A discussion of the final presentation and final deliverables will take place.
    • Video: [Google Drive]