6.2 Is the archive up & running and functional?
An archive is a service for the end-users. It has to be available 24/7. It requires complex cooperations between different layers, standalone services. All these layers have to work and communicate smoothly. Otherwise, the user won't get the answer to her question.
For example, if the archive application is up & running, but the storage device which persists the data got dismounted from the network (NAS), then the archive is not functional.
First of all, setting up monitoring of all services is essential. It's a standard task for system administration, and it should be quick to get it done.
To make sure all services cooperate nicely, it's recommended to setup functional testing also. If the archive's user interface is web-based, then any web application test framework can be used.
We can also use an interface (for ex. REST) of the archive system directly bypassing the GUI. It has to be able to execute a search and get back the result.
This way we implement regression tests. Our test cases are search definitions, and the test results are search results. Both search definitions and results can be serialized as XML or JSON and persisted, perhaps in a version control repository. We can build a simple framework which executes all search definitions one by one and compares the result with the expected one.
Of course, this is a data subset test. It does not test everything, only specific cases. Still, it's powerful test, and we can consciously pick some corner cases.
Usually, we try to find the golden mean, not to define too few nor too many uses cases, but since the data is immutable in the archive, having many test cases do not indicate high maintenance work.
Beyond getting the correct result, the test can also measure the performance of the system.
When the automated tests should be executed?
- During development - The developer must test the searches. For finding test cases, he has to dig the data which takes time. Instead of testing it manually, when likely the test cases will be shortly forgotten, it's the best to add the test cases to the regression test system in the first place. Then the test cases will be saved, and the test can be repeated at any time without human interaction.
- System upgrades - In case the archive system (or any other underlying services) will be added, then the automated test will come handy. It can spare a lot of working hours and lead to more steady operation.
- Scheduled - It can be run as a scheduled task on the server as part of the monitoring. If it's a heavy load on the machine, then make sure to run it nightly or outside of business hours.