[META] Testing scenarios for Elastic Agent E2E testing framework

[jlind23]

This issue will act as a placeholder which contains a list of all the E2E testing scenarios that should be implemented as part of the Elastic Agent E2E testing framework.

Tasks

Preview Give feedback

1. Generalize TestUpgradeFleetManagedElasticAgent for various upgrade scenarios #2641
   Team:Elastic-Agent Testing +2
   
2. Create tests to ensure that Elastic Agent's upgrade are working as expected #2176
   Team:Elastic-Agent +1
   
3. [E2E][Linux][Windows][Mac] Upgrade managed Elastic Agent 8.X and 7.X #2429
   Team:Elastic-Agent +1
   
4. [E2E][Linux ][Windows][Mac] Upgrade standalone Elastic Agent 8.X and 7.X #2430
   Team:Elastic-Agent +1
   
5. [E2E][Linux][Windows][Mac] Install Elastic Agent 8.X with tags --tag=production,OS #2431
   Team:Elastic-Agent +1
6. [E2E][Linux] Install Elastic Agent 8.X with the linux integration #2432
   Team:Elastic-Agent +1
7. https://github.com/elastic/ingest-dev/issues/1652
8. https://github.com/elastic/ingest-dev/issues/1455
9. Add a new E2E test to confirm that Elastic Agent component logs can be shipped to Fleet #2402
   Team:Elastic-Agent +1
   
10. Revive the Fleet and Upgrade E2E tests #1174
    8.6-candidate Team:Elastic-Agent-Control-Plane ci:end-to-end estimation:Week +4
11. Add automated test coverage for service based components #1357
    8.6-candidate Team:Elastic-Agent-Control-Plane V2-Architecture +3
12. Add automated test coverage for 8.4 to 8.5+ upgrades on MacOS #1399
    8.6-candidate Team:Elastic-Agent-Control-Plane ci:end-to-end +3
13. Include an E2E test that emulates an air gapped installation without external connectivity.
14. [E2E][Cloud] Spin up a cloud stack on cloud testing region and make sure an Hosted Fleet server is available fleet-server#2484
    Team:Fleet +1
    
15. Create an integration test for the fleet proxy set by --proxy-url and in the policy #2482
    Team:Elastic-Agent +1
    
16. Integration test for the passphrase support - SSL private key #2501
    Team:Fleet +1
    
17. Add an integration test for the upgrade rollback proces #2705
    Team:Elastic-Agent +1
18. Create an E2E test that installs Elastic Defend and ensures the again remains healthy #2948
    Team:Elastic-Agent +1
    

• Add an end to end test to ensure the agent can process actions, including signed actions. The simplest example is the settings action to change the log level. For the signed actions (as of March 7, 2023), the only actions that can be signed by kibana are the endpoint actions. The agent validates the signature for all INPUT_ACTION types of actions though, so potentially one could sign osquery actions outside of the kibana if they want.
  Also there are two things that are getting signed: 1) protection configuration of the policy 2) input actions. For comprehensive testing of all the error cases the tests would need to explore the cases where the signed policy configuration or the signed actions are being tempered with, such as: invalid signature validation key, corrupted signed data, policy id mismatch for the policy signed data, action id and the agent id mismatch for the action signed data. Basically tests different types of tampering use cases.

[jlind23]

@pierrehilbert can you make sure this list is complete and that all test cases are planned for one of the upcoming sprint?

[cmacknz]

Added #2705

[cmacknz]

Added #2948

[ycombinator]

Closing in favor of individual issues for pending tests:

Add an end to end test to ensure the agent can process actions, including signed actions. The simplest example is the settings action to change the log level. For the signed actions (as of March 7, 2023), the only actions that can be signed by kibana are the endpoint actions. The agent validates the signature for all INPUT_ACTION types of actions though, so potentially one could sign osquery actions outside of the kibana if they want.
Also there are two things that are getting signed: 1) protection configuration of the policy 2) input actions. For comprehensive testing of all the error cases the tests would need to explore the cases where the signed policy configuration or the signed actions are being tempered with, such as: invalid signature validation key, corrupted signed data, policy id mismatch for the policy signed data, action id and the agent id mismatch for the action signed data. Basically tests different types of tampering use cases.

#5216

Create tests to ensure that Elastic Agent's upgrade are working as expected

#5217
#5219
#5218