[Feature Request]: Support unit testing with multiple configurations (VSC-1627)

[jdbaptista]

Is your feature request related to a problem? Please describe.

I have found it necessary to use multiple configurations with the "ESP-IDF: Select Project Configuration" command in order to develop features for two versions of hardware alongside each other. Unfortunately, using these configurations means that the command "ESP-IDF: Unit Test: Build and Flash Unit Test App for Testing" fails because it can't find the configuration files, which are present in the main folder and not the generated unity-app folder. I can place the configuration files in the unity-app folder and successfully build the testing app, however when I go to run a test I get errors (shown in additional context section).

Edit: The unity-app folder is also missing necessary partition files by default, which can be added manually.

Describe the solution you'd like

I would like the command "ESP-IDF: Unit Test: Build and Flash Unit Test App for Testing" to place a copy of the currently selected sdkconfig.defaults file in the unity-app folder, which I am currently doing manually, and for the issue that occurs when running tests after building the app to be fixed.

Describe alternatives you've considered

I don't believe there are alternatives to the feature I am requesting, as it is more of a request for support between two pre-existing features.

Additional context

Edit: My workaround is to simply make a new project that uses unity and, in the top-level CMakeLists.txt, sets "set(TEST_COMPONENTS ...)". This will do everything except for integrate testing with the VSCode testing tab. Good news is that its actually a lot faster to run tests directly than through the tab, bad news is that there are no pretty colors.

Error when running tests built as explained above:

=========================================== test session starts ===========================================
platform win32 -- Python 3.11.2, pytest-8.3.3, pluggy-1.5.0
rootdir: C:\Users\bapti\Documents\traffic-pcb-sb\software\esp-firmware\unity-app      
plugins: embedded-1.11.8, ignore-test-results-0.2.2, rerunfailures-14.0, timeout-2.3.1
collected 1 item

test_unit_case.py E

================================================= ERRORS ================================================== 
_______________ ERROR at setup of test_unit_test[signedDistanceFromDiagLine_atypical_angle] _______________ 

args = ()
kwargs = {'_fixture_classes_and_options': ClassCliOptions(classes={'app': <class 'pytest_embedded_idf.app.IdfApp'>, 'serial': <...app.IdfApp object at 0x00000135F3126190>, 'msg_queue': <pytest_embedded.log.MessageQueue object at 0x00000135F3109E50>}
_close_or_terminate = <function multi_dut_generator_fixture.<locals>.wrapper.<locals>._close_or_terminate at 0x00000135F311AA20>
res = None

    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        def _close_or_terminate(obj):
            if obj is None:
                del obj
                return

            try:
                if isinstance(obj, (subprocess.Popen, multiprocessing.process.BaseProcess)):
                    obj.terminate()
                    obj.kill()
                elif isinstance(obj, io.IOBase):
                    try:
                        obj.close()
                    except Exception as e:
                        logging.debug('file %s closed failed with error: %s', obj, str(e))
                else:
                    try:
                        obj.close()
                    except AttributeError:
                        try:
                            obj.terminate()
                        except AttributeError:
                            pass
                    except Exception as e:
                        logging.debug('Not properly caught object %s: %s', obj, str(e))
            except Exception as e:
                logging.debug('%s: %s', obj, str(e))
                return  # swallow up all error
            finally:
                referrers = gc.get_referrers(obj)
                for _referrer in referrers:
                    if isinstance(_referrer, list):
                        for _i, val in enumerate(_referrer):
                            if val is obj:
                                _referrer[_i] = None
                    elif isinstance(_referrer, dict):
                        for key, value in _referrer.items():
                            if value is obj:
                                _referrer[key] = None
                del obj

        if _COUNT == 1:
            res = None
            try:
>               res = func(*args, **kwargs)

C:\Users\bapti\.espressif\python_env\idf5.3_py3.11_env\Lib\site-packages\pytest_embedded\plugin.py:490:     
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
C:\Users\bapti\.espressif\python_env\idf5.3_py3.11_env\Lib\site-packages\pytest_embedded\plugin.py:1111: in 
serial
    return serial_gn(**locals())
C:\Users\bapti\.espressif\python_env\idf5.3_py3.11_env\Lib\site-packages\pytest_embedded\dut_factory.py:431: in serial_gn
    return cls(**_drop_none_kwargs(kwargs))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pytest_embedded_idf.serial.IdfSerial object at 0x00000135F0C048D0>
app = <pytest_embedded_idf.app.IdfApp object at 0x00000135F3126190>, target = None
confirm_target_elf_sha256 = False, erase_nvs = False
kwargs = {'baud': 115200, 'esp_flash_force': False, 'esptool_baud': 921600, 'meta': Meta(logdir='C:\\Users\\bapti\\AppData\\Loc...t[signedDistanceFromDiagLine_atypical_angle]', port_target_cache={}, port_app_cache={}, logfile_extension='.log'), ...}

    def __init__(
        self,
        app: IdfApp,
        target: Optional[str] = None,
        confirm_target_elf_sha256: bool = False,
        erase_nvs: bool = False,
        **kwargs,
    ) -> None:
        self.app = app
        self.confirm_target_elf_sha256 = confirm_target_elf_sha256
        self.erase_nvs = erase_nvs

        if not hasattr(self.app, 'target'):
>           raise ValueError(f"Idf app not parsable. Please check if it's valid: {self.app.binary_path}")   
E           ValueError: Idf app not parsable. Please check if it's valid: None

C:\Users\bapti\.espressif\python_env\idf5.3_py3.11_env\Lib\site-packages\pytest_embedded_idf\serial.py:37: ValueError
------------------------------------------- Captured log setup -------------------------------------------- 
WARNING  root:app.py:108 config\sdkconfig.json doesn't exist. Skipping...
-- generated xml file: C:\Users\bapti\Documents\traffic-pcb-sb\software\esp-firmware\unity-app\test.xml --- 
============================================== Failed Cases =============================================== 
================ you can use --ignore-result-files or --ignore-result-cases to ignore them ================ 
test_unit_case.py::test_unit_test[signedDistanceFromDiagLine_atypical_angle]
========================================= short test summary info ========================================= 
ERROR test_unit_case.py::test_unit_test[signedDistanceFromDiagLine_atypical_angle] - ValueError: Idf app not parsable. Please check if it's valid: None
============================================ 1 error in 0.91s =============================================

[brianignacio5]

Hi @jdbaptista, so the only missing part is to copy sdkconfig defaults to unity-app project ?

I would like the command "ESP-IDF: Unit Test: Build and Flash Unit Test App for Testing" to place a copy of the currently selected sdkconfig.defaults file in the unity-app folder

If that is all, I can quickly create a fix for this, otherwise I need more information.

[jdbaptista]

Hello, thank you for looking into this. I have created an example project that is a simplified version of my current multiple configuration setup to illustrate exactly what I mean above. In the README.md I have added some instructions to replicate the issues that are seen when attempting to build the unity app.

I don't believe that the only missing part is to copy sdkconfig.defaults into the unity-app project, as explained in the repo. I'm not sure what the pytest errors are stating are missing either, unfortunately.

One related issue that became clear is that, when using multiple configurations, the unity app does not build for the configuration target, rather for an esp32. I explain this in more detail in the README.md of the example project below.

Repo: https://github.com/jdbaptista/Issue1478_Example.

Edit: Please note that, while not included in the example project, custom partition files and associated folders (ex. partitions/partitionsV1.csv and partitions/partitionsV2.csv) are also missing from the initial unity build.