From 81bb17fbdbbf2cd0a2a17441f1774cfe46934789 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Levai=20Mackenzie=20=C3=81gb=C3=A0r=C3=A0?= <97461848+Mackenzie-OO7@users.noreply.github.com> Date: Wed, 19 Oct 2022 00:59:06 +0100 Subject: [PATCH 1/7] added Optional Inputs Parameters --- src/topics/inputs.md | 117 ++++++++++++++++++++++++++++++++++--------- 1 file changed, 94 insertions(+), 23 deletions(-) diff --git a/src/topics/inputs.md b/src/topics/inputs.md index 6733f47f..37dc4a90 100644 --- a/src/topics/inputs.md +++ b/src/topics/inputs.md @@ -11,7 +11,7 @@ and *null*; complex types are *array* and *record*; in addition there are special types *File*, *Directory* and *Any*. The following example demonstrates some input parameters with different -types and appearing on the command line in different ways. +types and how they can appear on the command line. First, create a file called `inp.cwl`, containing the following: @@ -31,7 +31,7 @@ Create a file called `inp-job.yml`: ````{note} You can use `cwltool` to create a template input object. That saves you from having -to type all the input parameters in a input object file: +to type all the input parameters in an input object file: ```{runcmd} cwltool --make-template inp.cwl :working-directory: src/_includes/cwl/inputs @@ -50,7 +50,7 @@ Next, create a whale.txt using [touch] by typing `touch whale.txt` on the comman $ touch whale.txt ``` -Now invoke `cwltool` with the tool description and the input object on the command line, +Now, invoke `cwltool` with the tool description and the input object on the command line, using the command `cwltool inp.cwl inp-job.yml`. The following boxed text describes these two commands and the expected output from the command line: @@ -67,9 +67,9 @@ the tools aren't accidentally accessing files that were not explicitly specified ``` -The field `inputBinding` is optional and indicates whether and how the -input parameter should appear on the tool's command line. If -`inputBinding` is missing, the parameter does not appear on the command +The field `inputBinding` is optional and indicates if and how the +input parameter should appear on the tool's command line. If +`inputBinding` is missing, the parameter does not appear on the command line. Let's look at each example in detail. ```cwl @@ -80,9 +80,9 @@ example_flag: prefix: -f ``` -Boolean types are treated as a flag. If the input parameter +Boolean types are treated as a flag. If the input parameter "example_flag" is "true", then `prefix` will be added to the -command line. If false, no flag is added. +command line. If false, no flag is added. ```cwl example_string: @@ -92,9 +92,9 @@ example_string: prefix: --example-string ``` -String types appear on the command line as literal values. The `prefix` +String types appear on the command line as literal values. The `prefix` is optional, if provided, it appears as a separate argument on the -command line before the parameter . In the example above, this is +command line before the parameter. In the example above, this is rendered as `--example-string hello`. ```cwl @@ -107,9 +107,9 @@ example_int: ``` Integer (and floating point) types appear on the command line with -decimal text representation. When the option `separate` is false (the +decimal text representation. When the option `separate` is false (the default value is true), the prefix and value are combined into a single -argument. In the example above, this is rendered as `-i42`. +argument. In the example above, this is rendered as `-i42`. ```cwl @@ -121,17 +121,17 @@ example_file: position: 4 ``` -File types appear on the command line as the path to the file. When the +File types appear on the command line as the path to the file. When the parameter type ends with a question mark `?` it indicates that the -parameter is optional. In the example above, this is rendered as -`--file=/tmp/random/path/whale.txt`. However, if the "example_file" +parameter is optional. In the example above, this is rendered as +`--file=/tmp/random/path/whale.txt`. However, if the "example_file" parameter were not provided in the input, nothing would appear on the command line. -Input files are read-only. If you wish to update an input file, you must +Input files are read-only. If you wish to update an input file, you must [first copy it to the output directory](staging-input-files.md). -The value of `position` is used to determine where parameter should +The value of `position` is used to determine where the parameter should appear on the command line. Positions are relative to one another, not absolute. As a result, positions do not have to be sequential, three parameters with positions 1, 3, 5 will result in the same command @@ -143,13 +143,85 @@ The `baseCommand` field will always appear in the final command line before the [touch]: http://www.linfo.org/touch.html +### Optional Input Parameters + +Optional `inputs` include `label` and `secondaryFiles`. +`label` is a short, human-readable description for the parameter. + +```cwl +inputs: + example_file: + type: File + label: Use case for label parameter + inputBinding: + position: 1 +``` + +`secondaryFiles` is an optional input parameter that provides a pattern of specifying files +or directories that must be included alongside the primary file. +The following example demonstrates the `secondaryFiles` input parameter' + +```cwl +inputs: + example_file: + type: file + inputBinding: + position: 1 + prefix: --file= + seperate: false + "secondaryFiles": [ + { + "class": "File", + "location": "example.txt", + "basename": "example.txt", + "nameroot": "example", + "nameext": ".txt", + } + ] +``` +If the value is an expression, the value of self in the expression must be the primary +input or output File object to which this binding applies. +The `basename`, `nameroot` and `nameext` fields must be present in self. + +Also, a file object listed in secondaryFiles may contain nested secondaryFiles as shown below: + +```cwl +inputs: + example_file: + type: file + inputBinding: + position: 1 + prefix: --file= + seperate: false + "secondaryFiles": [ + { + "class": "File", + "location": "example.fasta", + "basename": "example.fasta", + "secondaryFiles": [ + { + "class": "File", + "location": "example.fasta.fai", + "basename": "example.fasta.fai", + "nameroot": "example.fasta", + "nameext": ".fai", + } + ], + } + ] +``` + +Note that Secondary files are only valid when `type` has a value `File`, or is an array of `items: File`. +All listed secondary files must be present in the same directory as the primary file, +since an implementation may fail workflow execution if a listed secondary file is not present. + ## Array Inputs It is easy to add arrays of input parameters represented to the command line. There are two ways to specify an array parameter. First is to provide `type` field with `type: array` and `items` defining the valid data types that may appear in the array. Alternatively, brackets `[]` may be added after -the type name to indicate that input parameter is array of that type. +the `type: name` to indicate that the input parameter is an array of that type. ```{literalinclude} /_includes/cwl/inputs/array-inputs.cwl :language: cwl @@ -163,7 +235,7 @@ the type name to indicate that input parameter is array of that type. :name: array-inputs-job.yml ``` -Now invoke `cwltool` providing the tool description and the input object +Now invoke `cwltool` by providing the tool description and the input object on the command line: ```{runcmd} cwltool array-inputs.cwl array-inputs-job.yml @@ -191,7 +263,7 @@ You can specify arrays of arrays, arrays of records, and other complex types. ## Inclusive and Exclusive Inputs Sometimes an underlying tool has several arguments that must be provided -together (they are dependent) or several arguments that cannot be provided +together, (they are dependent) or several arguments that cannot be provided together (they are exclusive). You can use records and type unions to group parameters together to describe these two conditions. @@ -270,7 +342,7 @@ that accepts `null` (i.e. no value provided), or any value from an enum. Note how the JavaScript expression uses the value of the exclusive input parameter without taking into consideration a `null` value. If you provide a valid value, -such as “fasta” (one of the values of the enum), your command should execute +such as “fasta” (one of the values of the enum), your command should be executed successfully: ```{runcmd} cwltool exclusive-parameter-expressions.cwl --file_format fasta @@ -286,7 +358,7 @@ output field (a `string`), resulting in failure when running your workflow. :emphasize-lines: 5-10 ``` -To correct it, you must remember to use an or operator in your JavaScript expression +To correct it, you must remember to use an **or** operator in your JavaScript expression when using exclusive parameters, or any parameter that allows `null`. For example, the expression could be changed to `$(inputs.file_format || 'auto')`, to have a default value if none was provided in the command line or job input file. @@ -298,4 +370,3 @@ a default value if none was provided in the command line or job input file. % - Several ways of defining inputs/arguments to tools and workflows - https://github.com/common-workflow-language/user_guide/issues/33 % - Using an input output in another input - https://github.com/common-workflow-language/user_guide/issues/90 % - How to use linkMerge - https://github.com/common-workflow-language/user_guide/issues/117 (or maybe move to Advanced?) -% - Secondary files - https://github.com/common-workflow-language/common-workflow-language/issues/270 From 0496edced581f389609bcb48225e761b5157039f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Levai=20Mackenzie=20=C3=81gb=C3=A0r=C3=A0?= <97461848+Mackenzie-OO7@users.noreply.github.com> Date: Wed, 19 Oct 2022 08:03:36 +0100 Subject: [PATCH 2/7] Update inputs.md --- src/topics/inputs.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/src/topics/inputs.md b/src/topics/inputs.md index 37dc4a90..3c6fc27b 100644 --- a/src/topics/inputs.md +++ b/src/topics/inputs.md @@ -69,7 +69,7 @@ specified The field `inputBinding` is optional and indicates if and how the input parameter should appear on the tool's command line. If -`inputBinding` is missing, the parameter does not appear on the command +`inputBinding` is missing, the parameter does not appear on the command line. Let's look at each example in detail. ```cwl @@ -145,7 +145,7 @@ The `baseCommand` field will always appear in the final command line before the ### Optional Input Parameters -Optional `inputs` include `label` and `secondaryFiles`. +Optional input parameters must include `label` and `secondaryFiles`. `label` is a short, human-readable description for the parameter. ```cwl @@ -159,17 +159,17 @@ inputs: `secondaryFiles` is an optional input parameter that provides a pattern of specifying files or directories that must be included alongside the primary file. -The following example demonstrates the `secondaryFiles` input parameter' +The following example demonstrates the `secondaryFiles` input parameter. ```cwl inputs: example_file: - type: file + type: File inputBinding: position: 1 prefix: --file= seperate: false - "secondaryFiles": [ + secondaryFiles: [ { "class": "File", "location": "example.txt", @@ -179,21 +179,21 @@ inputs: } ] ``` -If the value is an expression, the value of self in the expression must be the primary +If the value is an expression, the value of `self` in the expression is the primary input or output File object to which this binding applies. -The `basename`, `nameroot` and `nameext` fields must be present in self. +The `basename`, `nameroot` and `nameext` fields are present in `self`. -Also, a file object listed in secondaryFiles may contain nested secondaryFiles as shown below: +Also, a file object listed in `secondaryFiles` may contain nested `secondaryFiles` as shown below: -```cwl +```{code-block} cwl inputs: example_file: - type: file + type: File inputBinding: position: 1 prefix: --file= seperate: false - "secondaryFiles": [ + secondaryFiles: [ { "class": "File", "location": "example.fasta", @@ -211,7 +211,7 @@ inputs: ] ``` -Note that Secondary files are only valid when `type` has a value `File`, or is an array of `items: File`. +Note that secondary files are only valid when `type` has a value `File`, or is an array of `items: File`. All listed secondary files must be present in the same directory as the primary file, since an implementation may fail workflow execution if a listed secondary file is not present. From 142c3c212aee71419978a9ba37112215c07a4db5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Levai=20Mackenzie=20=C3=81gb=C3=A0r=C3=A0?= <97461848+Mackenzie-OO7@users.noreply.github.com> Date: Thu, 20 Oct 2022 22:39:03 +0100 Subject: [PATCH 3/7] updated secondaryFiles parameter --- src/topics/inputs.md | 35 +++-------------------------------- 1 file changed, 3 insertions(+), 32 deletions(-) diff --git a/src/topics/inputs.md b/src/topics/inputs.md index 3c6fc27b..86b6bf15 100644 --- a/src/topics/inputs.md +++ b/src/topics/inputs.md @@ -165,23 +165,8 @@ The following example demonstrates the `secondaryFiles` input parameter. inputs: example_file: type: File - inputBinding: - position: 1 - prefix: --file= - seperate: false - secondaryFiles: [ - { - "class": "File", - "location": "example.txt", - "basename": "example.txt", - "nameroot": "example", - "nameext": ".txt", - } - ] + secondaryFiles: [example_file.txt] ``` -If the value is an expression, the value of `self` in the expression is the primary -input or output File object to which this binding applies. -The `basename`, `nameroot` and `nameext` fields are present in `self`. Also, a file object listed in `secondaryFiles` may contain nested `secondaryFiles` as shown below: @@ -189,24 +174,10 @@ Also, a file object listed in `secondaryFiles` may contain nested `secondaryFile inputs: example_file: type: File - inputBinding: - position: 1 - prefix: --file= - seperate: false secondaryFiles: [ { - "class": "File", - "location": "example.fasta", - "basename": "example.fasta", - "secondaryFiles": [ - { - "class": "File", - "location": "example.fasta.fai", - "basename": "example.fasta.fai", - "nameroot": "example.fasta", - "nameext": ".fai", - } - ], + example_file.txt + secondaryFiles: [example_file_2.txt] } ] ``` From 961cc8d45f10f050b380218ae3e0c84973a5ad70 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Levai=20Mackenzie=20=C3=81gb=C3=A0r=C3=A0?= <97461848+Mackenzie-OO7@users.noreply.github.com> Date: Fri, 28 Oct 2022 22:18:29 +0100 Subject: [PATCH 4/7] Update inputs.md --- src/topics/inputs.md | 137 +++++++++++++++++++++++++++++++++++++++---- 1 file changed, 124 insertions(+), 13 deletions(-) diff --git a/src/topics/inputs.md b/src/topics/inputs.md index 86b6bf15..731d6ad6 100644 --- a/src/topics/inputs.md +++ b/src/topics/inputs.md @@ -143,9 +143,9 @@ The `baseCommand` field will always appear in the final command line before the [touch]: http://www.linfo.org/touch.html -### Optional Input Parameters +### Optional fields on the Input Parameter -Optional input parameters must include `label` and `secondaryFiles`. +Optional fields on the input parameters include `label`, `doc` and `secondaryFiles`. `label` is a short, human-readable description for the parameter. ```cwl @@ -157,31 +157,142 @@ inputs: position: 1 ``` -`secondaryFiles` is an optional input parameter that provides a pattern of specifying files -or directories that must be included alongside the primary file. +`doc` is a documentation string for the object, or an array of strings which should be concatenated. + +```cwl +inputs: + example_file: + type: File + doc: This section demonstrates Optional fields on the Input Parameter + inputBinding: + position: 1 + ``` + +`secondaryFiles` is an optional field on the input parameter that provides a pattern of specifying files +or directories that must be included alongside the primary file. +Take for example, an HTML document containing an image; +the HTML document is the primary file, while the image is a secondary file. +The `src` attribute is used to reference the image path from your local directory, +which then stages the img file within the HTML document. + +Secondary files may be required or optional. When not explicitly specified, +secondary files specified for inputs are required while secondary files specified for outputs are optional. +To use `secondaryFiles`, make your primary file +(the script you will actually run) a default input and list the other dependencies as `secondaryFiles`. +An implementation must include matching Files and Directories in the `secondaryFiles` property of the primary file. The following example demonstrates the `secondaryFiles` input parameter. ```cwl inputs: example_file: type: File - secondaryFiles: [example_file.txt] + secondaryFiles: [.idx] ``` -Also, a file object listed in `secondaryFiles` may contain nested `secondaryFiles` as shown below: +The fields within secondary files are `pattern` and `required`. +Secondary files are specified using the following micro-DSL for secondary files: +- If the value is a string, it is transformed to an object with two fields `pattern` and `required`. +- By default, the value of `required` is null (this indicates default behavior, which may be based on the context). +- If the value ends with a question mark "?", the question mark is stripped off and the value of the field `required` is set to False. +- The remaining value is assigned to the field `pattern`. + +`pattern` in this context refers how we can reference secondary files, +with respect to the value, ie if it is an expression, or a string. +If the value is an expression, the value of self in the expression must be +the primary input or output File object to which this binding applies. +The basename, nameroot and nameext fields must be present in self. +For CommandLineTool outputs, the path field must also be present. +The expression must return a filename string relative to the path to the primary File, +a File or Directory object with either path or location and basename fields set, +or an array consisting of strings or File or Directory objects. +It is legal to reference an unchanged File or Directory object taken from input as a `secondaryFile`. +The expression may return "null" in which case there is no `secondaryFile` from that expression. + +To work on non-filename-preserving storage systems, +portable tool descriptions should avoid constructing new values from location, +but should construct relative references using basename or nameroot instead. + +If a value in `secondaryFiles` is a string that is not an expression, +it specifies that the following pattern should be applied to the path of the primary file +to yield a filename relative to the primary File: + +- If string begins with one or more caret "^" characters, for each caret, remove the last file extension from the path +(the last period . and all following characters). ```{code-block} cwl inputs: example_file: type: File - secondaryFiles: [ - { - example_file.txt - secondaryFiles: [example_file_2.txt] - } - ] + secondaryFiles: [^.idx] ``` +this will give "example_file.idx". + +- If string ends with "?" character, remove the last "?" and mark the resulting secondary file as optional. +- If there are no file extensions, the path is unchanged. +Append the remainder of the string to the end of the file path. + +For example, given the following schema: + +```{code-block} cwl +{ + "$graph": [ + { + "name": "SecondaryFilesDSLExample", + "type": "record", + "documentRoot": true, + "fields": [{ + "name": "secondaryFiles", + "type": "string", + "jsonldPredicate": { + _type: "@vocab", + "secondaryFilesDSL": true + } + }] + }] +} +Process the following example: + +[{ + "secondaryFiles": ".bai" +}, { + "secondaryFiles": ".bai?" +}, { + "secondaryFiles": { + "pattern": ".bai?" +}, { + "secondaryFiles": { + "pattern": ".bai?", + "required": true +}] +This becomes: + +[ + { + "secondaryFiles": { + "pattern": ".bai", + "required": null + }, + { + "secondaryFiles": { + "pattern": ".bai", + "required": false + }, + { + "secondaryFiles": { + "pattern": ".bai?" + }, + { + "secondaryFiles": { + "pattern": ".bai?", + "required": true + }, +] +``` + +An implementation must not fail workflow execution if `required` is set to false and the expected secondary file does not exist. +Default value for required field is true for secondary files on input, and false for secondary files on output. + Note that secondary files are only valid when `type` has a value `File`, or is an array of `items: File`. All listed secondary files must be present in the same directory as the primary file, since an implementation may fail workflow execution if a listed secondary file is not present. @@ -192,7 +303,7 @@ It is easy to add arrays of input parameters represented to the command line. There are two ways to specify an array parameter. First is to provide `type` field with `type: array` and `items` defining the valid data types that may appear in the array. Alternatively, brackets `[]` may be added after -the `type: name` to indicate that the input parameter is an array of that type. +the type name to indicate that the input parameter is an array of that type. ```{literalinclude} /_includes/cwl/inputs/array-inputs.cwl :language: cwl From d03dfaf43c6167954aed87cc17d06fc9b13da72e Mon Sep 17 00:00:00 2001 From: Mackenzie-OO7 Date: Fri, 13 Jan 2023 01:04:42 +0100 Subject: [PATCH 5/7] update documentation of secondaryFiles --- src/topics/inputs.md | 113 ++++++++++++++----------------------------- 1 file changed, 35 insertions(+), 78 deletions(-) diff --git a/src/topics/inputs.md b/src/topics/inputs.md index 731d6ad6..d652d02e 100644 --- a/src/topics/inputs.md +++ b/src/topics/inputs.md @@ -143,9 +143,8 @@ The `baseCommand` field will always appear in the final command line before the [touch]: http://www.linfo.org/touch.html -### Optional fields on the Input Parameter +### Documenting input parameters using `label` or `doc` -Optional fields on the input parameters include `label`, `doc` and `secondaryFiles`. `label` is a short, human-readable description for the parameter. ```cwl @@ -168,25 +167,38 @@ inputs: position: 1 ``` -`secondaryFiles` is an optional field on the input parameter that provides a pattern of specifying files -or directories that must be included alongside the primary file. -Take for example, an HTML document containing an image; -the HTML document is the primary file, while the image is a secondary file. -The `src` attribute is used to reference the image path from your local directory, -which then stages the img file within the HTML document. - -Secondary files may be required or optional. When not explicitly specified, -secondary files specified for inputs are required while secondary files specified for outputs are optional. -To use `secondaryFiles`, make your primary file -(the script you will actually run) a default input and list the other dependencies as `secondaryFiles`. + ````{note} + `label` and `doc` fields can also be used to document output parameters. + ```` + +### Attaching Secondary files to the inputs file + +Secondary files are input files that should be staged in the same directory as the input files. +The `secondaryFiles` field can define files or directories that can be automatically uploaded with the primary input file. +For example, if the primary input is an HTML file (.html), an image file (.png or .jpg) could be included as a secondary file. + +The use of secondary files are optional for inputs. However, if secondary files are defined for an input file, +they should exist. An implementation may fail workflow execution if an expected secondary file does not exist. + +Secondary files can also be used for output files. +When defining primary and secondary files in your CWL workflow, +use the input parameter to specify your primary files, and use the `secondaryFiles` property to define your secondary files. An implementation must include matching Files and Directories in the `secondaryFiles` property of the primary file. The following example demonstrates the `secondaryFiles` input parameter. ```cwl -inputs: - example_file: - type: File - secondaryFiles: [.idx] +{ + basename: example_file.bam + class: File + location: ../documents/example_file.bam + secondaryFiles: [ + { + basename: example_file.bai, + class: File, + location: ../documents/example_file.bam.bai + } + ] +} ``` The fields within secondary files are `pattern` and `required`. @@ -232,70 +244,15 @@ this will give "example_file.idx". - If there are no file extensions, the path is unchanged. Append the remainder of the string to the end of the file path. -For example, given the following schema: - -```{code-block} cwl -{ - "$graph": [ - { - "name": "SecondaryFilesDSLExample", - "type": "record", - "documentRoot": true, - "fields": [{ - "name": "secondaryFiles", - "type": "string", - "jsonldPredicate": { - _type: "@vocab", - "secondaryFilesDSL": true - } - }] - }] -} -Process the following example: - -[{ - "secondaryFiles": ".bai" -}, { - "secondaryFiles": ".bai?" -}, { - "secondaryFiles": { - "pattern": ".bai?" -}, { - "secondaryFiles": { - "pattern": ".bai?", - "required": true -}] -This becomes: - -[ - { - "secondaryFiles": { - "pattern": ".bai", - "required": null - }, - { - "secondaryFiles": { - "pattern": ".bai", - "required": false - }, - { - "secondaryFiles": { - "pattern": ".bai?" - }, - { - "secondaryFiles": { - "pattern": ".bai?", - "required": true - }, -] -``` - An implementation must not fail workflow execution if `required` is set to false and the expected secondary file does not exist. -Default value for required field is true for secondary files on input, and false for secondary files on output. +Default value for `required` field is true for secondary files on input, and false for secondary files on output. -Note that secondary files are only valid when `type` has a value `File`, or is an array of `items: File`. +````{note} +Secondary files are only valid when `type` has a value `File`, or is an array of `items: File`. All listed secondary files must be present in the same directory as the primary file, -since an implementation may fail workflow execution if a listed secondary file is not present. +since an implementation may fail workflow execution if a listed secondary file is not present. +It is an error for file names to be duplicated in `secondaryFiles`. +```` ## Array Inputs From f422a50809de61a040e023bff82c786505e4fe32 Mon Sep 17 00:00:00 2001 From: Mackenzie-OO7 Date: Thu, 26 Jan 2023 16:28:21 +0100 Subject: [PATCH 6/7] update secondaryFiles documentation --- src/topics/inputs.md | 69 ++++++++++++-------------------------------- 1 file changed, 18 insertions(+), 51 deletions(-) diff --git a/src/topics/inputs.md b/src/topics/inputs.md index d652d02e..a8356e5f 100644 --- a/src/topics/inputs.md +++ b/src/topics/inputs.md @@ -174,84 +174,51 @@ inputs: ### Attaching Secondary files to the inputs file Secondary files are input files that should be staged in the same directory as the input files. -The `secondaryFiles` field can define files or directories that can be automatically uploaded with the primary input file. +The `secondaryFiles` parameter defines files or directories that can be automatically uploaded with the primary input file. For example, if the primary input is an HTML file (.html), an image file (.png or .jpg) could be included as a secondary file. The use of secondary files are optional for inputs. However, if secondary files are defined for an input file, they should exist. An implementation may fail workflow execution if an expected secondary file does not exist. -Secondary files can also be used for output files. When defining primary and secondary files in your CWL workflow, -use the input parameter to specify your primary files, and use the `secondaryFiles` property to define your secondary files. -An implementation must include matching Files and Directories in the `secondaryFiles` property of the primary file. +use the input parameter to specify your primary files, and use the `secondaryFiles` parameter to define your secondary files. +An implementation must include matching files and directories in the `secondaryFiles` parameter of the primary file. The following example demonstrates the `secondaryFiles` input parameter. ```cwl -{ - basename: example_file.bam - class: File - location: ../documents/example_file.bam - secondaryFiles: [ - { - basename: example_file.bai, - class: File, - location: ../documents/example_file.bam.bai - } - ] -} +inputs: + example_file: + type: File + inputBinding: + prefix: -e + secondaryFiles: + - .idx ``` -The fields within secondary files are `pattern` and `required`. -Secondary files are specified using the following micro-DSL for secondary files: -- If the value is a string, it is transformed to an object with two fields `pattern` and `required`. -- By default, the value of `required` is null (this indicates default behavior, which may be based on the context). -- If the value ends with a question mark "?", the question mark is stripped off and the value of the field `required` is set to False. -- The remaining value is assigned to the field `pattern`. - -`pattern` in this context refers how we can reference secondary files, -with respect to the value, ie if it is an expression, or a string. -If the value is an expression, the value of self in the expression must be -the primary input or output File object to which this binding applies. -The basename, nameroot and nameext fields must be present in self. -For CommandLineTool outputs, the path field must also be present. -The expression must return a filename string relative to the path to the primary File, -a File or Directory object with either path or location and basename fields set, -or an array consisting of strings or File or Directory objects. -It is legal to reference an unchanged File or Directory object taken from input as a `secondaryFile`. -The expression may return "null" in which case there is no `secondaryFile` from that expression. - -To work on non-filename-preserving storage systems, -portable tool descriptions should avoid constructing new values from location, -but should construct relative references using basename or nameroot instead. - If a value in `secondaryFiles` is a string that is not an expression, it specifies that the following pattern should be applied to the path of the primary file to yield a filename relative to the primary File: -- If string begins with one or more caret "^" characters, for each caret, remove the last file extension from the path -(the last period . and all following characters). +- If the string begins with one or more caret "^" characters, for each caret, remove the last file extension from the path(the last period `.` and all following characters), and append the remainder of the string to the end of the file path. -```{code-block} cwl +```cwl inputs: example_file: type: File - secondaryFiles: [^.idx] + inputBinding: + prefix: -e + secondaryFiles: + - ^.idx ``` -this will give "example_file.idx". - -- If string ends with "?" character, remove the last "?" and mark the resulting secondary file as optional. -- If there are no file extensions, the path is unchanged. -Append the remainder of the string to the end of the file path. - -An implementation must not fail workflow execution if `required` is set to false and the expected secondary file does not exist. -Default value for `required` field is true for secondary files on input, and false for secondary files on output. +assuming the primary file above has a .txt extension (example_file.txt), the secondary file will be "example_file.idx". ````{note} Secondary files are only valid when `type` has a value `File`, or is an array of `items: File`. All listed secondary files must be present in the same directory as the primary file, since an implementation may fail workflow execution if a listed secondary file is not present. It is an error for file names to be duplicated in `secondaryFiles`. +Secondary files can also be used for output files. ```` ## Array Inputs From 6b7e8b7c6dc89f77b2fd98f692be44429725ae3c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Levai=20Mackenzie=20=C3=81gb=C3=A0r=C3=A0?= <97461848+Mackenzie-OO7@users.noreply.github.com> Date: Mon, 6 Feb 2023 12:34:22 +0100 Subject: [PATCH 7/7] update secondaryFiles documentation --- src/topics/inputs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/topics/inputs.md b/src/topics/inputs.md index a8356e5f..9d3c24dd 100644 --- a/src/topics/inputs.md +++ b/src/topics/inputs.md @@ -211,7 +211,7 @@ inputs: - ^.idx ``` -assuming the primary file above has a .txt extension (example_file.txt), the secondary file will be "example_file.idx". +assuming the primary file above has a .txt extension (`example_file.txt`), the secondary file will be `example_file.idx`. ````{note} Secondary files are only valid when `type` has a value `File`, or is an array of `items: File`.