Empirical Methods Should Guide the Development of New Software Engineering Tools

The following quote is one of my favorite quotes regarding the right way to conduct Software Engineering (SE) research. It summarizes the importance of utilizing empirical methods to inspire and guide the development of new SE tools and techniques:

“In all fields of SE, empirical methods should enable the development of scientific knowledge about how useful different SE technologies are, for different kinds of actors, performing different kinds of activities, on different kinds of systems. Such scientific knowledge should guide the development of new SE technology and be a major input to important SE decisions in industry and services.”

– Dag I. K. Sjoberg, Tore Dyba, Magne Jorgensen. The future of empirical methods in software engineering research. In Proc. of ICSE 2007, Future of Soft. Eng. (FOSE ’07), pages 358-378, 2007.

This quote is five years old but is still relevant today. One only needs to look at the proceedings of any recently held SE conference (e.g., ICSE 2012) to see the importance that is placed on empirical methods. In research there is no substitute for sound empirical results and one should always avoid shortcuts that downplay the importance of empirical research methods.

Predicting Mutation Scores

Last week my MSc student, Kevin Jalbert, presented his early thesis results at the Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (RAISE 2012). The workshop took place in Zurich Switzerland and was colocated with ICSE 2012. The title of the presentation (and the paper that appears in the proceedings) was “Predicting Mutation Score Using Source Code and Test Suite Metrics.” The paper was awarded the Best Paper Award at the workshop.

Mutation testing can be used to evaluate the effectiveness of test suites and can also be used as an oracle during the creation or improvement of test suites. Mutation testing works by creating many versions of a program each with a single syntactic fault. These program versions are created using mutation operators which are based on an existing fault taxonomy (i.e., a set of known fault types that we are trying to find during testing). One mutation operator, Relational Operator Replacement (ROR), could create a new mutant version of the program in which one of the instance of a relational operator (e.g., <) is replaced with a different operator. For example, line 3 of the following Java source  code:Continue reading “Predicting Mutation Scores”

Good Resources for Learning to Program with Concurrency

With the increase in multicore processors there has been an increase in demand for concurrent programming and an increase in books and resources that focus on programming with concurrency. Below I will outline some of the books on my own bookshelf that have been useful.
Continue reading “Good Resources for Learning to Program with Concurrency”

The Importance of Incremental Research

Last year Bertrand Meyer authored a post titled Long Live Incremental Research! at BLOG@CACM. Rather then writing my own post on the topic, I instead want to encourage people to read Meyer’s post as I think he does a great job of summarizing the importance of approaching research incrementally and not aiming for the next great breakthrough. A few quotes from his post:

“First, 99.97% of all research (precise statistic derived from my own ground-breaking research, funding for its continuation would be welcome) is incremental. Second, when a “breakthrough” does happen — the remaining 0.03%  — it was often not planned as a breakthrough.”

Continue reading “The Importance of Incremental Research”

Static Analysis Tools for Concurrency

Top 4 Reasons to Study Computer Science at UOIT

[Note: This is an old post and while many of the reasons remain true some of the data is old. A new post from 2017 is not available! Top Reasons to Study Computer Science at UOIT]

This Saturday, March. 3, 2012 from 10:00am-2:00pm UOIT will be hosting its annual Winter Open House for prospective students interested in studying at UOIT. This year myself and Dr. Christopher Collins will be coordinating the Computer Science lab tour. As the undergraduate program director for Computer Science, I will be providing a 30 minute overview of the Computer Science programs (comprehensive program, digital media specialization, digital forensics specialization) while Chris will be in the undergraduate lab where a number of our undergraduate students. The students will be available to give demos, answer questions and showcase Computer Science at UOIT.

I have participated in UOIT recruitment events for the past 5 years and I have answered a lot of questions from potential students and their parents. One of the most common questions that I get asked every year is:

Why should I choose to study Computer Science at UOIT?

Continue reading “Top 4 Reasons to Study Computer Science at UOIT”

Software Testing Tools for Concurrency

As the use of concurrency in software increases there are more software testing tools being developed in both industry an academia. Below is a fairly comprehensive list of tools (mainly for Java):

  • ConTest – IBM concurrent testing tool that inserts random delays into Java bytecode
  • ConAn – a thread testing tool for Java
  • CHESS – a Microsoft tool for concurrent testing in Visual Studio
  • CalFuzzer – an active testing framework
  • ConMAn – mutation testing tool for concurrent programs written in Java
  • ConcurrencyCloner – clone detection tool that detects user specified patterns (intended for project-specific bug detection). Can be combined with ConTest tool.
  • Coverity Dynamic Analysis – commercial tool that detects bugs in multithreaded Java programs [http://scan.coverity.com/]
  • Chord – static and dynamic analysis tool for Java (listed below as well).
  • Pluralism – Modular object protocol checking for Java
  • Flashlight Dynamic Analysis – a commercial tool from SureLogic that is currently available in early release.
  • MPI testing tools: TotalView, Intel Message Checking, MPI Check, Marmot
  • JFuzz – testing tool built on Java PathFinder
  • MutMut – concurrency mutation testing tool built on Java PathFinder
  • AtomRacer – detection of data races and atomicity violations
  • Eraser – detection of data races by tracking sets of locks that are acquired/held during execution.

Welcome!

This blog is a place where I will…

  • discuss ongoing research projects in the Software Quality Research Group Lab
  • discuss interesting state-of-the-art work  being done in the areas of software testing, software quality assurance, concurrent software systems, distributed software systems.
  • comment on the current state of Software Engineering — both research and practice.
  • mention anything else I find interesting or funny.

If you have any comments or thoughts on my posts I’d be interested in hearing what you have to say.