chore: Add e2e tests for playwright templates and pip based templates #1136
+137
−37
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
t-tooling
|
Example test run: https://github.com/apify/crawlee-python/actions/runs/14246372360 |
There was a problem hiding this comment.
Pull Request Overview
This PR adds end-to-end tests for both Playwright and pip-based templates while addressing an issue with dependency installations and aligning the maximum allowed crawl requests to documentation examples.
- Added new e2e test parametrizations and markers for various template types and package managers.
- Refactored the function patching the crawlee version to support multiple package managers and updated related tests accordingly.
- Adjusted Dockerfile template and crawler configuration files to use a lower max_requests_per_crawl value.
Reviewed Changes
Copilot reviewed 7 out of 10 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/e2e/project_template/utils.py | Updated function to conditionally patch projects based on package manager and modified Dockerfile/requirements handling. |
| tests/e2e/project_template/test_static_crawlers_templates.py | Updated test parametrizations and function calls to reflect the new patching function and limits. |
| tests/e2e/conftest.py | Added pytest markers configuration for new test parameters. |
| src/crawlee/project_template/templates/main_playwright.py | Lowered max_requests_per_crawl to align with documentation and test expectations. |
| src/crawlee/project_template/templates/main_parsel.py | Lowered max_requests_per_crawl to align with documentation and test expectations. |
| src/crawlee/project_template/templates/main_beautifulsoup.py | Lowered max_requests_per_crawl to align with documentation and test expectations. |
| .github/workflows/templates_e2e_tests.yaml | Updated the workflow matrix and commands to incorporate the new test parameters and optimize runner utilization. |
Files not reviewed (3)
- Makefile: Language not supported
- src/crawlee/project_template/{{cookiecutter.project_name}}/Dockerfile: Language not supported
- src/crawlee/project_template/{{cookiecutter.project_name}}/requirements.txt: Language not supported
Comments suppressed due to low confidence (1)
tests/e2e/project_template/utils.py:25
- Using [0] on the result of re.findall may raise an IndexError if no match is found. Consider using a safer retrieval method such as using next(iter(...), '') or verifying that the list is non-empty.
crawlee_extras = re.findall(r'crawlee(\[.*\])', requirements)[0] or ''
|
More parallelized run with properly ordered tests to lower overall tests time: https://github.com/apify/crawlee-python/actions/runs/14260627276 6.5 minutes |
vdusek
reviewed
Apr 4, 2025
Comment on lines
-5
to
-8
| secrets: | ||
| APIFY_TEST_USER_API_TOKEN: | ||
| description: API token of the Python testing user on Apify | ||
| required: true |
There was a problem hiding this comment.
Why is this secret no longer required? The template tests are not running on Apify?
| @@ -1,5 +1,5 @@ | |||
| # % if cookiecutter.crawler_type == 'playwright-camoufox' | |||
| camoufox | |||
| camoufox[geoip]>=0.4.5 | |||
There was a problem hiding this comment.
Rather major version upper bound?
Comment on lines
+3
to
+4
|
|
||
|
|
There was a problem hiding this comment.
3 empty lines? :)
Suggested change
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
max_requests_per_crawlto 10 in all templates to be aligned with doc examples and to make the tests faster.Issues