Sara’s testing thoughts
On “marking your own homework”
In software development there is a notion that a person making a change shouldn’t be the same as the person testing a change and yet time and resource constraints inevitably lead to this being exactly the case. Being forced into a position where you have to sacrifice your ideals is not pleasant and can make you feel out of control. But what if, in this case, it is the ideal of “don’t mark your own homework” itself which is wrong?
We use the phrase “don’t mark your own homework” when talking about testing software that we’ve written ourselves but we’re not at school, there is no teacher to tell us whether or not we got the right answer and we’re not completing an assignment alone on a topic we only just started learning about. We are experienced professionals. So what is the purpose of this analogy?
It stems from the idea that a person that made a change can’t be trusted to verify that the change was right. That they are prejudiced into thinking one-dimensionally about the change and assuming that it is correct which leads them to writing and running tests which verify what they already believe to be true.
Traditional schools of thought say that a person testing an application shouldn’t know anything about how the application is built or the precise change that was made because it might prejudice their approach, force them into unwise assumptions and hence lead them into missing bugs. This is known as Black Box testing and is commonly seen when software engineers produce a set of changes then hand over testing to a dedicated test team to “mark their homework” and act as a bug net. This can work well if you batch the changes, have a test team with a lot of time and experience in using the program (or a dedicated group of specialists testing for security vulnerabilities or accessibility issues) and also don’t mind getting back information on defects potentially months after they were developed.
It doesn’t work well however in a fast paced, continuous delivery scenario when you are pressed for resource because time and resource are dedicated evenly to both areas that have no change and areas affected by the change. Critical testing can be missed and any defects that are found are found too late to be quickly and efficiently resolved. To prevent this, the change-maker may guide the tester as to which area they expect to be affected but then bias has been introduced and the tests are no longer black box. The tester may try to understand the change in order to better focus their tests but they can’t possibly understand all the nuances of the system and the change if they weren’t there when it was made! If defects are found, they’re still found after the change-maker has moved onto other work.
In my experience, I’ve found that when a person makes a change to a piece of software, they are in an optimal position to consider all the potential impacts of that change and the best places to measure and assure that change. By considering how to prove a change whilst also implementing that change, a person is forced to think about boundary conditions and edge cases which in turn leads to potential defects being caught before they are even written. It’s a different way of working and does take some time and experience to get right but ultimately making a change and testing that change are activities which are best done in tandem.
However this doesn’t mean that one person should be making, testing and signing off on a change all alone. You may encounter problems in things such as miscommunication of the requirements, inexperience in the area of the software you are changing, not finding all the edge cases or even just having an off-day. This can be mitigated through practices such as upfront refinement of a piece of work with unambiguous acceptance criteria and by pairing upon making and assuring a change so that there’s always another point of view.
The moral of the story? The software you develop isn’t “homework” and you don’t need anyone else to “mark it”. The people implementing a change to software are the ones that are in the best position to say it is “the right answer”. Work with your team to consider testing at every stage of your development process, you will end up with software that is a pleasure to work with.
