CI Agent/Tooling: How to enable unit testing for different Machine Learning frameworks

[h2floh]

Current Situation:
A Azure DevOps pipelines will leverage a custom docker image as the CI "runtime". It is possible to add Python related conda and pip dependencies (Machine Learning frameworks) via this file it will be used in the Docker build for that "container build-agent" and has it's own CI/CD pipeline.
Currently Tensorflow is installed there to enable unit tests for the training scripts.

Problem Statement:
How to enable unit testing for other ML framework examples (in this repo) as well while keeping the footprint of the build agent(s) images small.

Technical Background/Restrictions:
The unit tests are currently executed in the code-quality-pipeline at repo root without any naming restrictions. This means that currently all unit tests will be executed regardless of the (sub)project/example being modified. This means also that all frameworks needed within the unit test have to be in place.

Solution options:

1. Splitting the code-quality pipeline into several pipelines for each ML framework and only execute the relevant test files. This would introduce the need for a test file naming convention. The used CI agent images will be built separately each with another conda dependency file.
2. Splitting the code-quality pipeline into several pipelines for each ML framework and only execute the relevant test files. This would introduce the need for a test file naming convention. In contrast to Option 1 a slim build-agent image is used and the conda dependencies are installed when the pipeline is run (high likely will increase runtime of the pipeline)
3. Install several conda environments into the CI build agent image and activate the right one before unit tests are started. This will also require the code-quality pipeline to be split. This approach is in contrast to keeping a low container image footprint but it simplifies the CI/CD approach for the build-agent(s).
4. many more please add your ideas!

Pragmatic approach:

• Let's be aware of this topic, but I don't want to overengineer now. As long as it will be possible to add other ML frameworks to the current CI dependency file without conflicts I would leave as is. The code-quality pipeline will then be able to run all ML scripts related unit tests in one pass. If we slide into dependency conflicts we need to split anyway with one of the options mentioned above