PoC: new Github CI test labels and markers #771
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dorotat-nv
requested review from
jstjohn,
pstjohn,
trvachov,
malcolmgreaves,
jwilber,
sichu2023,
skothenhill-nv,
jomitchellnv and
cspades
as code owners
dorotat-nv
changed the title
Signed-off-by: dorotat <dorotat@nvidia.com>
Signed-off-by: dorotat <dorotat@nvidia.com>
Signed-off-by: Peter St. John <pstjohn@nvidia.com> Signed-off-by: dorotat <dorotat@nvidia.com>
- Update evo2 readme to add Quickstart section (previously unfinished), linking to notebooks. - Move `assets/` out of `examples/` to `bionemo-evo2` root. - Update finetuning notebook image link to use new `assets/` dir. --------- Signed-off-by: Jared Wilber <jwilber@nvidia.com> Co-authored-by: Peter St. John <pstjohn@nvidia.com> Signed-off-by: dorotat <dorotat@nvidia.com>
### Description This PR: 1. Aligns location of tensorboard logs for * ESM2 from `result_dir/experiment_name/lightining_logs/dev` to `result_dir/experiment_name/dev/` and unit test it in `sub-packages/bionemo-esm2/tests/bionemo/esm2/scripts/test_train_esm2.py::test_main_runs` * Evo2 from `experiment_name/dummy/lightining_logs/` to `result_dir/experiment_name/dev/` and unit test it in `sub-packages/bionemo-evo2/tests/bionemo/evo2/run/test_train.py::test_train_evo2_runs` 3. Adds tflops callback to ESM2 training and unit tests it in `sub-packages/bionemo-esm2/tests/bionemo/esm2/scripts/test_train_esm2.py::test_main_runs` 4. Adds resume training unit tests for * Evo2 in `sub-packages/bionemo-evo2/tests/bionemo/evo2/run/test_train.py::test_train_evo2_stop_at_max_steps_and_continue` (the tests fails on L40 in the internal CI, skipped test and added issue: #769 ) * ESM2 in `sub-packages/bionemo-esm2/tests/bionemo/esm2/scripts/test_train_esm2.py::test_main_stop_at_num_steps_and_continue` (the trainings does not resume, added xfail with issue #757) Fixed   ### Type of changes <!-- Mark the relevant option with an [x] --> - [ ] Bug fix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [x] Refactor - [ ] Documentation update - [ ] Other (please describe): ### CI Pipeline Configuration Configure CI behavior by applying the relevant labels: - [SKIP_CI](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#skip_ci) - Skip all continuous integration tests - [INCLUDE_NOTEBOOKS_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_notebooks_tests) - Execute notebook validation tests in pytest - [INCLUDE_SLOW_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_slow_tests) - Execute tests labelled as slow in pytest for extensive testing > [!NOTE] > By default, the notebooks validation tests are skipped unless explicitly enabled. ### Usage <!--- How does a user interact with the changed code --> ```python python sub-packages/bionemo-evo2/src/bionemo/evo2/run/train.py --mock-data --model-size 1b_nv --num-layers 4 --hybrid-override-pattern SDH* --no-activation-checkpointing --add-bias-output --warmup-steps 1 --seq-length 128 --hidden-dropout 0.1 --attention-dropout 0.1 --create-tensorboard-logger --wandb-project fix-tensorboard-logs --val-check-interval 50 --max-steps 100 python sub-packages/bionemo-esm2/src/bionemo/esm2/scripts/train_esm2.py --train-cluster-path=/data/train_clusters.parquet --train-database-path=/data/train.db --valid-cluster-path=/data/valid_clusters.parquet --valid-database-path=/data/validation.db --micro-batch-size=16 --num-nodes=1 --num-gpus=1 --val-check-interval=50 --limit-val-batches=1 --min-seq-length=1024 --max-seq-length=1024 --num-layers=33 --hidden-size=1280 --num-attention-heads=20 --ffn-hidden-size=5120 --create-tensorboard-logger --wandb-project=fix-tensorboard-logs --val-check-interval 50 --create-tflops-callback --num-steps=100 --resume-if-exists ``` ### Pre-submit Checklist <!--- Ensure all items are completed before submitting --> - [x] I have tested these changes locally - [x] I have updated the documentation accordingly - [x] I have added/updated tests as needed - [x] All existing tests pass successfully Signed-off-by: dorotat <dorotat@nvidia.com>
Signed-off-by: dorotat <dorotat@nvidia.com>
Signed-off-by: dorotat <dorotat@nvidia.com>
Signed-off-by: dorotat <dorotat@nvidia.com>
Signed-off-by: dorotat <dorotat@nvidia.com>
dorotat-nv
force-pushed
the
dorotat/add-ciflow-scopes
branch
from
f3515f6 to
58376dd
Compare
|
/ok to test |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Introducing new PR labels and pytest markers for more efficient compartmentalization of tests in BioNeMo FW
Type of changes
CI Pipeline Configuration
Configure CI behavior by applying the relevant labels:
Note
By default, the notebooks validation tests are skipped unless explicitly enabled.
Authorizing CI Runs
We use copy-pr-bot to manage authorization of CI
runs on NVIDIA's compute resources.
automatically be copied to a pull-request/ prefixed branch in the source repository (e.g. pull-request/123)
/ok to testcomment on the pull request to trigger CI. This will need to be done for each new commit.Usage
Pre-submit Checklist