07 Jun, 2006
White/Black box testing and shades of gray
A great debate arose at work about what White and Black box testing, what they were for etc.
For starters here is my simple definition of testing in general
- Testing: Checking that the solution meets the requirements of system users.
From this we then split testing into two groups
- Blackbox: Testing that is not aware of how it works. Just that it works. (e.g. user testing)
- Whitebox: Testing that uses knowledge of how it works to help testing (e.g. looking in the database to see if the data really saved).
This sounds quite a simple distinction.The debate at work was about whether unit testing can be (or should be) black box. I would argue it can and at least a good chunk of it should.
Unit testing is about testing the public interface of an object. You are testing to see if the object meets its contract (i.e. its requirements). When you write your unit tests (obviously first!) you should ideally check it meets the contract. I would argue that this sort of test is black box testing - as the user (the calling class) is checking without any necessary knowledge of the class, it meets the requirements.
We do tend to enter shades of grey when it comes to really testing a system as running a black box test can often make testing hard. The main problems tend to be
- Getting suitable mock objects (or real objects) to test the interface
- The database (what state is it in).
- The API of the object may allow us to do everything it is meant to do. However it may not allow us to test is has done them. e.g. a void doIt() function.
- Speed (normally related to the previous two items).
So we then muddle our clean black box test to fix these issues. Normally this involves a combination of
- Calling/adding protected methods
- Stubbing database implementations
This is where we tend to get a grey test. The tests value however can only be preseved if you keep in mind your focus is to test it meets the contract.
Finally there are valid situations where we do want to white box unit test - however I would argue these are a minority. An example would be a complex logic to generate a random string suitable for use as a username or password. This should not be exposed via a public interface (getUserName() getPassword() would be appropiate), but to prove it is working you would need to test the implementation.
In conclusion I think you need to be clear about what you test is testing and you tests should be categorised into
- Black Box (with shades of grey): testing the contract of the interface
- White Box: testing the implementation.
Alternative Title: why Mike and Gaz are wrong