Skip to content

Commit 7ad4d47

Browse files
ankatiyarankatiyar
authored
ci: port lint, unit test, and e2e tests to Actions (kedro-org#155)
* Add unit test + lint test on GA * trigger GA - will revert Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Fix lint Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add end to end tests * Add cache key Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add cache action Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Rename workflow files Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Lint + add comment + default bash Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add windows test Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update workflow name + revert changes to READMEs Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add kedro-telemetry/RELEASE.md to trufflehog ignore Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add pytables to test_requirements remove from workflow Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Revert "Add pytables to test_requirements remove from workflow" This reverts commit 8203daa. * Separate pip freeze step Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
1 parent efe59db commit 7ad4d47

6 files changed

+200
-0
lines changed

‎.github/workflows/check-plugin.yml

+134
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
name: Running tests and linter
2+
3+
on:
4+
workflow_call:
5+
inputs:
6+
plugin:
7+
type: string
8+
9+
jobs:
10+
unit-tests:
11+
defaults:
12+
run:
13+
shell: bash
14+
strategy:
15+
matrix:
16+
os: [ ubuntu-latest, windows-latest ]
17+
python-version: [ "3.7", "3.8", "3.9", "3.10" ]
18+
runs-on: ${{ matrix.os }}
19+
steps:
20+
- name: Checkout code
21+
uses: actions/checkout@v3
22+
- name: Set up Python ${{matrix.python-version}}
23+
uses: actions/setup-python@v3
24+
with:
25+
python-version: ${{matrix.python-version}}
26+
- name: Cache python packages for Linux
27+
if: matrix.os == 'ubuntu-latest'
28+
uses: actions/cache@v3
29+
with:
30+
path: ~/.cache/pip
31+
key: ${{inputs.plugin}}-${{matrix.os}}-python-${{matrix.python-version}}
32+
restore-keys: ${{inputs.plugin}}
33+
- name: Cache python packages for Windows
34+
if: matrix.os == 'windows-latest'
35+
uses: actions/cache@v3
36+
with:
37+
path: ~\AppData\Local\pip\Cache
38+
key: ${{inputs.plugin}}-${{matrix.os}}-python-${{matrix.python-version}}
39+
restore-keys: ${{inputs.plugin}}
40+
- name: Install Kedro
41+
run: pip install git+https://github.com/kedro-org/kedro@main
42+
- name: Install dependencies
43+
run: |
44+
cd ${{ inputs.plugin }}
45+
pip install -r test_requirements.txt
46+
- name: Install pytables (only for kedro-datasets on windows)
47+
if: matrix.os == 'windows-latest' && inputs.plugin == 'kedro-datasets'
48+
run: pip install tables
49+
- name: pip freeze
50+
run: pip freeze
51+
- name: Run unit tests for Linux / all plugins
52+
if: matrix.os != 'windows-latest'
53+
run: make plugin=${{ inputs.plugin }} test
54+
- name: Run unit tests for Windows / kedro-airflow, kedro-docker, kedro-telemetry
55+
if: matrix.os == 'windows-latest' && inputs.plugin != 'kedro-datasets'
56+
run: |
57+
cd ${{ inputs.plugin }}
58+
pytest tests
59+
- name: Run unit tests for Windows / kedro-datasets / no spark sequential
60+
if: matrix.os == 'windows-latest' && inputs.plugin == 'kedro-datasets' && matrix.python-version == '3.10'
61+
run: |
62+
make test-no-spark-sequential
63+
- name: Run unit tests for Windows / kedro-datasets / no spark parallel
64+
if: matrix.os == 'windows-latest' && inputs.plugin == 'kedro-datasets' && matrix.python-version != '3.10'
65+
run: |
66+
make test-no-spark
67+
68+
lint:
69+
defaults:
70+
run:
71+
shell: bash
72+
runs-on: ubuntu-latest
73+
steps:
74+
- name: Checkout code
75+
uses: actions/checkout@v3
76+
- name: Set up Python 3.8
77+
uses: actions/setup-python@v3
78+
with:
79+
python-version: 3.8
80+
- name: Cache python packages
81+
uses: actions/cache@v3
82+
with:
83+
path: ~/.cache/pip
84+
key: ${{inputs.plugin}}-${{matrix.os}}-python-${{matrix.python-version}}
85+
restore-keys: ${{inputs.plugin}}
86+
- name: Install dependencies
87+
run: |
88+
cd ${{ inputs.plugin }}
89+
pip install git+https://github.com/kedro-org/kedro@main
90+
pip install -r test_requirements.txt
91+
pip freeze
92+
- name: Install pre-commit hooks
93+
run: |
94+
cd ${{ inputs.plugin }}
95+
pre-commit install --install-hooks
96+
pre-commit install --hook-type pre-push
97+
- name: Run linter
98+
run: make plugin=${{ inputs.plugin }} lint
99+
100+
e2e-tests:
101+
if: inputs.plugin != 'kedro-datasets'
102+
defaults:
103+
run:
104+
shell: bash
105+
strategy:
106+
matrix:
107+
os: [ ubuntu-latest ]
108+
python-version: [ "3.7", "3.8", "3.9", "3.10" ]
109+
runs-on: ${{ matrix.os }}
110+
steps:
111+
- name: Checkout code
112+
uses: actions/checkout@v3
113+
- name: Set up Python ${{matrix.python-version}}
114+
uses: actions/setup-python@v3
115+
with:
116+
python-version: ${{matrix.python-version}}
117+
- name: Cache python packages
118+
uses: actions/cache@v3
119+
with:
120+
path: ~/.cache/pip
121+
key: ${{inputs.plugin}}-${{matrix.os}}-python-${{matrix.python-version}}
122+
restore-keys: ${{inputs.plugin}}
123+
- name: Install dependencies
124+
run: |
125+
cd ${{ inputs.plugin }}
126+
pip install git+https://github.com/kedro-org/kedro@main
127+
pip install -r test_requirements.txt
128+
- name: pip freeze
129+
run: pip freeze
130+
- name: Run end to end tests
131+
# Custom shell to run kedro-docker e2e-tests because -it flag for `docker run`
132+
# isn't supported on Github Actions. See https://github.com/actions/runner/issues/241
133+
shell: 'script -q -e -c "bash {0}"'
134+
run: make plugin=${{ inputs.plugin }} e2e-tests

‎.github/workflows/kedro-airflow.yml

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
name: Run checks on kedro-airflow
2+
3+
on:
4+
push:
5+
paths:
6+
- "kedro-airflow/**"
7+
pull_request:
8+
paths:
9+
- "kedro-airflow/**"
10+
types: [ synchronize ]
11+
12+
jobs:
13+
airflow-test:
14+
uses: ./.github/workflows/check-plugin.yml
15+
with:
16+
plugin: kedro-airflow

‎.github/workflows/kedro-datasets.yml

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
name: Run checks on kedro-datasets
2+
3+
on:
4+
push:
5+
paths:
6+
- "kedro-datasets/**"
7+
pull_request:
8+
paths:
9+
- "kedro-datasets/**"
10+
types: [ synchronize ]
11+
12+
jobs:
13+
datasets-test:
14+
uses: ./.github/workflows/check-plugin.yml
15+
with:
16+
plugin: kedro-datasets

‎.github/workflows/kedro-docker.yml

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
name: Run checks on kedro-docker
2+
3+
on:
4+
push:
5+
paths:
6+
- "kedro-docker/**"
7+
pull_request:
8+
paths:
9+
- "kedro-docker/**"
10+
types: [ synchronize ]
11+
12+
jobs:
13+
docker-test:
14+
uses: ./.github/workflows/check-plugin.yml
15+
with:
16+
plugin: kedro-docker

‎.github/workflows/kedro-telemetry.yml

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
name: Run checks on kedro-telemetry
2+
3+
on:
4+
push:
5+
paths:
6+
- "kedro-telemetry/**"
7+
pull_request:
8+
paths:
9+
- "kedro-telemetry/**"
10+
types: [ synchronize ]
11+
12+
jobs:
13+
telemetry-test:
14+
uses: ./.github/workflows/check-plugin.yml
15+
with:
16+
plugin: kedro-telemetry

‎trufflehog-ignore.txt

+2
Original file line numberDiff line numberDiff line change
@@ -1 +1,3 @@
11
kedro-telemetry/README.md
2+
kedro-telemetry/RELEASE.md
3+
kedro-datasets/tests/tensorflow/test_tensorflow_model_dataset.py

0 commit comments

Comments
 (0)