Merge pull request #112 from Plant-Food-Research-Open/dev

Candidate for 0.5.0

Author: GallVp

.editorconfig

@@ -8,7 +8,7 @@ trim_trailing_whitespace = true
 indent_size = 4
 indent_style = space
 
-[*.{md,yml,yaml,html,css,scss,js}]
+[*.{md,yml,yaml,html,css,scss,js,cff}]
 indent_size = 2
 
 # These files are edited and tested upstream in nf-core/modules
@@ -58,3 +58,7 @@ indent_size = unset
 # ignore LICENSE
 [LICENSE]
 indent_size = unset
+
+# ignore bin/genepal_report
+[bin/genepal_report*]
+indent_size = unset

.github/CONTRIBUTING.md

@@ -16,7 +16,7 @@ If you'd like to write some code for plant-food-research-open/genepal, the stand
 1. Check that there isn't already an issue about your idea in the [plant-food-research-open/genepal issues](https://github.com/plant-food-research-open/genepal/issues) to avoid duplicating work. If there isn't one already, please create one so that others know you're working on this
 2. [Fork](https://help.github.com/en/github/getting-started-with-github/fork-a-repo) the [plant-food-research-open/genepal repository](https://github.com/plant-food-research-open/genepal) to your GitHub account
 3. Make the necessary changes / additions within your forked repository following [Pipeline conventions](#pipeline-contribution-conventions)
-4. Use `nf-core schema build` and add any new parameters to the pipeline JSON schema (requires [nf-core tools](https://github.com/nf-core/tools) >= 1.10).
+4. Use `nf-core pipelines schema build` and add any new parameters to the pipeline JSON schema (requires [nf-core tools](https://github.com/nf-core/tools) >= 1.10).
 5. Submit a Pull Request against the `dev` branch and wait for the code to be reviewed and merged
 
 If you're not used to this workflow with git, you can start with some [docs from GitHub](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests) or even their [excellent `git` resources](https://try.github.io/).
@@ -37,7 +37,7 @@ There are typically two types of tests that run:
 ### Lint tests
 
 `nf-core` has a [set of guidelines](https://nf-co.re/developers/guidelines) which all pipelines must adhere to.
-To enforce these and ensure that all pipelines stay in sync, we have developed a helper tool which runs checks on the pipeline code. This is in the [nf-core/tools repository](https://github.com/nf-core/tools) and once installed can be run locally with the `nf-core lint <pipeline-directory>` command.
+To enforce these and ensure that all pipelines stay in sync, we have developed a helper tool which runs checks on the pipeline code. This is in the [nf-core/tools repository](https://github.com/nf-core/tools) and once installed can be run locally with the `nf-core pipelines lint <pipeline-directory>` command.
 
 If any failures or warnings are encountered, please follow the listed URL for more documentation.
 
@@ -68,7 +68,7 @@ If you wish to contribute a new step, please use the following coding standards:
 2. Write the process block (see below).
 3. Define the output channel if needed (see below).
 4. Add any new parameters to `nextflow.config` with a default (see below).
-5. Add any new parameters to `nextflow_schema.json` with help text (via the `nf-core schema build` tool).
+5. Add any new parameters to `nextflow_schema.json` with help text (via the `nf-core pipelines schema build` tool).
 6. Add sanity checks and validation for all relevant parameters.
 7. Perform local tests to validate that the new code works as expected.
 8. If applicable, add a new test command in `.github/workflow/ci.yml`.
@@ -79,11 +79,11 @@ If you wish to contribute a new step, please use the following coding standards:
 
 Parameters should be initialised / defined with default values in `nextflow.config` under the `params` scope.
 
-Once there, use `nf-core schema build` to add to `nextflow_schema.json`.
+Once there, use `nf-core pipelines schema build` to add to `nextflow_schema.json`.
 
 ### Default processes resource requirements
 
-Sensible defaults for process resource requirements (CPUs / memory / time) for a process should be defined in `conf/base.config`. These should generally be specified generic with `withLabel:` selectors so they can be shared across multiple processes/steps of the pipeline. A nf-core standard set of labels that should be followed where possible can be seen in the [nf-core pipeline template](https://github.com/nf-core/tools/blob/master/nf_core/pipeline-template/conf/base.config), which has the default process as a single core-process, and then different levels of multi-core configurations for increasingly large memory requirements defined with standardised labels.
+Sensible defaults for process resource requirements (CPUs / memory / time) for a process should be defined in `conf/base.config`. These should generally be specified generic with `withLabel:` selectors so they can be shared across multiple processes/steps of the pipeline. A nf-core standard set of labels that should be followed where possible can be seen in the [nf-core pipeline template](https://github.com/nf-core/tools/blob/main/nf_core/pipeline-template/conf/base.config), which has the default process as a single core-process, and then different levels of multi-core configurations for increasingly large memory requirements defined with standardised labels.
 
 The process resources can be passed on to the tool dynamically within the process with the `${task.cpus}` and `${task.memory}` variables in the `script:` block.
 
@@ -96,7 +96,7 @@ Please use the following naming schemes, to make it easy to understand what is g
 
 ### Nextflow version bumping
 
-If you are using a new feature from core Nextflow, you may bump the minimum required version of nextflow in the pipeline with: `nf-core bump-version --nextflow . [min-nf-version]`
+If you are using a new feature from core Nextflow, you may bump the minimum required version of nextflow in the pipeline with: `nf-core pipelines bump-version --nextflow . [min-nf-version]`
 
 ### Images and figures
 

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -9,46 +9,34 @@ body:
       description: A clear and concise description of what the bug is.
     validations:
       required: true
+
   - type: textarea
     id: command_used
     attributes:
       label: Command used and terminal output
-      description: Steps to reproduce the behaviour. Please paste the command you used
-        to launch the pipeline and the output from your terminal.
+      description: Steps to reproduce the behaviour. Please paste the command you used to launch the pipeline and the output from your terminal.
       render: console
-      placeholder: "$ nextflow run ...
-
+      placeholder: |
+        $ nextflow run ...
 
         Some output where something broke
 
-        "
   - type: textarea
     id: files
     attributes:
       label: Relevant files
-      description: "Please drag and drop the relevant files here. Create a `.zip` archive
-        if the extension is not allowed.
-
-        Your verbose log file `.nextflow.log` is often useful _(this is a hidden file
-        in the directory where you launched the pipeline)_ as well as custom Nextflow
-        configuration files.
+      description: |
+        Please drag and drop the relevant files here. Create a `.zip` archive if the extension is not allowed.
+        Your verbose log file `.nextflow.log` is often useful _(this is a hidden file in the directory where you launched the pipeline)_ as well as custom Nextflow configuration files.
 
-        "
   - type: textarea
     id: system
     attributes:
       label: System information
-      description: "* Nextflow version _(eg. 23.04.0)_
-
+      description: |
+        * Nextflow version _(eg. 23.04.0)_
         * Hardware _(eg. HPC, Desktop, Cloud)_
-
         * Executor _(eg. slurm, local, awsbatch)_
-
-        * Container engine: _(e.g. Docker, Singularity, Conda, Podman, Shifter, Charliecloud,
-        or Apptainer)_
-
+        * Container engine: _(e.g. Docker, Singularity, Conda, Podman, Shifter, Charliecloud, or Apptainer)_
         * OS _(eg. CentOS Linux, macOS, Linux Mint)_
-
         * Version of plant-food-research-open/genepal _(eg. 1.1, 1.5, 1.8.2)_
-
-        "

.github/PULL_REQUEST_TEMPLATE.md

@@ -16,7 +16,7 @@ Learn more about contributing: [CONTRIBUTING.md](https://github.com/plant-food-r
 - [ ] This comment contains a description of changes (with reason).
 - [ ] If you've fixed a bug or added code that should be tested, add tests!
 - [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/plant-food-research-open/genepal/tree/master/.github/CONTRIBUTING.md)
-- [ ] Make sure your code lints (`nf-core lint`).
+- [ ] Make sure your code lints (`nf-core pipelines lint`).
 - [ ] Ensure the test suite passes (`nextflow run . -profile test,docker --outdir <OUTDIR>`).
 - [ ] Check for unexpected warnings in debug mode (`nextflow run . -profile debug,test,docker --outdir <OUTDIR>`).
 - [ ] Usage Documentation in `docs/usage.md` is updated.

.github/include.yaml

@@ -0,0 +1,10 @@
+".":
+  - ./.github/workflows/**
+  - ./nf-test.config
+tests:
+  - ./assets/*
+  - ./bin/*
+  - ./conf/*
+  - ./main.nf
+  - ./nextflow_schema.json
+  - ./nextflow.config

.github/version_checks.sh

@@ -1,8 +1,24 @@
 #!/usr/bin/env bash
 
-config_version=$(sed -n "s/.*version.*= '\(.*\)'.*/\1/p" nextflow.config)
+set -euo pipefail
 
-# Check CHANGELOG version
+config_version=$(sed -n "/^\s*version\s*=\s*'/s/version//p" nextflow.config | tr -d "=[:space:]'")
+cff_version=$(sed -n '/^version: /s/version: //p' CITATION.cff | tr -d '[:space:]')
+
+if [[ $config_version != $cff_version ]]; then
+    echo 'config_version != cff_version'
+    exit 1
+fi
 
-head -n 10 CHANGELOG.md | grep "## $config_version - " >/dev/null \
+# Check CHANGELOG version
+head -10 CHANGELOG.md | grep "## v$config_version - " >/dev/null \
     || (echo 'Failed to match CHANGELOG version'; exit 1)
+
+# Check .nf-core.yml version
+tail -5 .nf-core.yml | grep "version: $config_version" >/dev/null \
+    || (echo 'Failed to match .nf-core.yml version'; exit 1)
+
+# Check multiqc_config.yml version
+config_version_plain=$(echo $config_version | sed 's/.*\dev$/dev/; t; s/\+.*//')
+head -n 5 assets/multiqc_config.yml | grep "https://github.com/plant-food-research-open/genepal/blob/$config_version_plain/docs/usage.md" >/dev/null \
+    || (echo 'Failed to match multiqc_config.yml version'; exit 1)

.github/workflows/ci.yml

@@ -1,58 +1,100 @@
 name: nf-core CI
 # This workflow runs the pipeline with the minimal test dataset to check that it completes without any syntax errors
 on:
-  push:
-    branches:
-      - dev
   pull_request:
-  release:
-    types: [published]
 
 env:
   NXF_ANSI_LOG: false
+  NFT_WORKDIR: "nf-test-work"
+  NFT_DIFF: "pdiff"
+  NFT_DIFF_ARGS: "--line-numbers --expand-tabs=2"
 
 concurrency:
   group: "${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}"
   cancel-in-progress: true
-
 jobs:
+  nf-test-changes:
+    name: Check for changes
+    runs-on: ubuntu-latest
+    outputs:
+      nf_test_files: ${{ steps.list.outputs.components }}
+    steps:
+      - uses: actions/checkout@v4.2.1
+        with:
+          fetch-depth: 0
+
+      - name: List nf-test files
+        id: list
+        uses: adamrtalbot/detect-nf-test-changes@v0.0.4
+        with:
+          head: ${{ github.sha }}
+          base: origin/${{ github.base_ref }}
+          include: .github/include.yaml
+
+      - name: print list of nf-test files
+        run: |
+          echo ${{ steps.list.outputs.components }}
+
   test:
-    name: Run pipeline with test data
-    if: "${{ github.event_name != 'push' || (github.event_name == 'push' && github.repository == 'Plant-Food-Research-Open/genepal') }}"
+    name: ${{ matrix.nf_test_files }} ${{ matrix.profile }} NF-${{ matrix.NXF_VER }}
+    needs: [nf-test-changes]
+    if: needs.nf-test-changes.outputs.nf_test_files != '[]'
     runs-on: ubuntu-latest
     strategy:
+      fail-fast: false
       matrix:
         NXF_VER:
-          - "23.04.0"
-        TEST_PARAMS:
-          - minimal
-          - short
-          - nonmasked
-          - stub
-        OPTION_STUB:
-          - "-stub"
+          - "24.04.2"
+
+        nf_test_files: ["${{ fromJson(needs.nf-test-changes.outputs.nf_test_files) }}"]
+        profile:
+          - "docker"
 
     steps:
       - name: Check out pipeline code
-        uses: actions/checkout@0ad4b8fadaa221de15dcec353f45205ec38ea70b # v4
+        uses: actions/checkout@v4.2.1
 
       - name: Install Nextflow
         uses: nf-core/setup-nextflow@v2
         with:
           version: "${{ matrix.NXF_VER }}"
 
+      - uses: actions/setup-python@v5.2.0
+        with:
+          python-version: "3.11"
+          architecture: "x64"
+
+      - name: Install pdiff to see diff between nf-test snapshots
+        run: |
+          python -m pip install --upgrade pip
+          pip install pdiff
+
+      - uses: nf-core/setup-nf-test@v1.1.2
+        with:
+          version: 0.9.0
+
       - name: Disk space cleanup
-        uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be # v1.3.1
+        if: matrix.nf_test_files == 'tests/stub/main.nf.test'
+        uses: jlumbroso/free-disk-space@v1.3.1
+
+      - name: Run nf-test
+        run: |
+          nf-test test --verbose ${{ matrix.nf_test_files }} --profile "+${{ matrix.profile }}"
+
+  confirm-pass:
+    runs-on: ubuntu-latest
+    needs: [test]
+    if: always()
+    steps:
+      - name: All tests ok
+        if: ${{ !contains(needs.*.result, 'failure') }}
+        run: exit 0
+      - name: One or more tests failed
+        if: ${{ contains(needs.*.result, 'failure') }}
+        run: exit 1
 
-      - name: Run pipeline with test data
+      - name: debug-print
+        if: always()
         run: |
-          nextflow run \
-            ${GITHUB_WORKSPACE} \
-            -profile docker \
-            -params-file \
-            ./tests/${{ matrix.TEST_PARAMS }}/params.json \
-            ${{ matrix.OPTION_STUB }} \
-            --outdir ./results \
-            --max_cpus 2 \
-            --max_memory '6.GB' \
-            --max_time '2.h'
+          echo "toJSON(needs) = ${{ toJSON(needs) }}"
+          echo "toJSON(needs.*.result) = ${{ toJSON(needs.*.result) }}"