fix: match python version in agent test workflow to version on agent #258
Conversation
The Python version currently installed on (some of) the agents is 3.8.10. The agents test workflow installs 3.8 -> 3.8.18, which will lead to version-sensitive code passing when it shouldn't.
The provisioning and firmware update phases will be triggered when `provision_data` or `firmware_update` data are not empty, which is the case when attachments are provided for these phases. In order to minimise changes to the agent code and keep its logic exactly the same as before (for now), attachment data is *stripped* from the job data dict as soon as the attachments are unpacked. This eliminates any interference that the presence of attachments may have to the execution of each phase.
There was a problem hiding this comment.
patching things up like this is certainly not my favorite thing to do, but I'm ok with it as a temporary measure if we don't have a better approach. I was really hoping we could use the get_members workaround that I suggested before, but you're right that there's a lot going on in the existing data_filter and get_filtered_attrs that we would miss if we just did that.
There was a problem hiding this comment.
I didn't check whether agent/testflinger_agent/tarfile_patch.py is a proper subset of Python 3.8's tarfile.py, but trust that somebody else will verify that :)
Added one more comment about simplifying the nested try..except..else blocks, but other than that, I think this is fine. It might also make sense to add some comment along the lines of "if you read this comment and the minimum supported Python version is >= X.Y.Z, the tarfile_patch.py file can be removed and the import changed back to tarfile".
Description and resolved issues
Merging #250 and deploying to staging uncovered certain issues.
filterargument oftarfile.extractallwhich is not supported in the Python version installed on the agent, i.e.3.8.10.3.8), resulting in the latest patch version being installed, i.e.3.8.18). Thefilterargument was added in version3.8.17, so it is supported in the environment used for testing in the workflow but not in the actual agent.provision_dataandfirmware_update_datafields. The code assumes that any data under these fields suggests that the corresponding stages should be triggered.Changes
3.8.12) is specified in the agent testing workflow. It does not match the Python version installed on the agent because that version is not available on our self-hosted runners, but it is sufficient for testing because the filtering feature was introduced in3.8.17.filterfunctionality provides an important safety feature. Instead of re-implementing what would necessarily be a subset of that functionality, I decided to use a minimal subset of the implementation provided in themainbranch of the Python standard library. A check in the imports section decides whether or not to use thetarfilethat comes with the current Python installation (if it supportsfilter) or the alternative one otherwise. The major advantages to this solution is that (a) the actual unpacking code remains unchanged and uses thefilterfunctionality and (b) this is not a version-sensitive solution, i.e. whenever it can be guaranteed that the agents will run a Python version supportingfilter, the unpacking code will remain the same and only the conditional import will be removed.Provision DataandFirmware update datasections of the Job page on Testflinger and (b) it is yet unclear how that attachment data might be used by a device connector later down the line.