Add ability for ROSE projects to specify their own tests
Now that QEMU-based Tests are a thing we'll want to allow projects to provide their own use-case centered test cases.
A fixed set of test-classes will make it easier to define certain contexts, dependencies, etc. for the tests. The following test classes come to mind:
- So first there are "outside" tests like ssh. But there could be more for e.g. HTTP, firewall, attack scenarios or other protocols
- Then there are startup and system tests checking systemd, bootlog, dmesg (more a real hardware test thing), (container) services
- For those there might also be performance and profiling tests (probably also more a real hardware thing, but with offset one can roughly estimate)
- Then there are tests on migrations and updates. For those we'll probably have to setup a certain filesystem state by manipulating the image before boot (to simulate certain common configuration changes etc.)
- Last we also want to test certain update / upgrade / rebase situations with rpm-ostree
A fixed set of tests provided by default in ROSE should be extendible. For this to work the iteration of tests has to become dynamic as well as the test scripts. It should be possible to mount custom script files / a custom script directory into the ROSE container.
The provided framework will at first just make sure that:
- Custom tests can be provided
- Tests are executed in order
- Failing tests don't interfere with other tests
- Test classes exist with their own set of dependencies, environment, etc.
- A summary is composed with the test-results
Maybe re-inventing the wheel is not what we want or even have to do. There might be generic test-frameworks we can use.
Todos
-
Research available, generic test frameworks