End Of The Green Bar Paradise? The Overestimation Of Unit Testing In Distributing Computing 📎
Unit Testing is great, integration / functional testing even better. Unit Tests are aimed to verify an isolated chunk of functionality, integration tests same thing in more realistic environment. They are coarser in general. However all of these techniques test the application in sequential way. A test is executed after a test. This is really odd in distributed environment. However this approach gets more and more popular - mostly because of the "metrics driven development". Its getting even worse - what I observe is the misuse of mocking to build an own perfect world and just abstract the invonvenient "truth" (legacy servers, deployments etc).
The problem with the sequential testing of a distributed system is mostly the misinterpretation of the results. The "Green Bar" (in Eclipse), or "Green Text" (in Netbeans :-)), is just the indicator, that all you sequential tests are passed. This is not enough for the most applications in general (except single-threaded fat clients). "Green Bar, 90% test coverage" projects can really fail. Some samples:
- In JMS messages are only ordered, in sequential case. Under load, they are mostly unordered
- Optimistic Lock Exceptions mostly happen in concurrent environment (only then, different threads (users) will access the same entity)
- Deadlock will never occur during unit testing
- Inconsistencies (like direct access to the EntityManager from a webapp) are only visible under load
- In Unit Testing there is no difference between the transactions, and "auto commit" mode.
- Memory Leaks - only visible during long running tests
- Resource allocation (data source, jms factories, JCA connectors, pooled components) works mostly during unit testing