Skip to content
This repository has been archived by the owner on Aug 20, 2024. It is now read-only.

Does not validate Azure or Google blob storage #171

Merged

Conversation

adamrtalbot
Copy link
Collaborator

Azure and Google blob storage failed to validate if it was a directory or path. This PR skips validating directories so it does raise an inappropriate error.

Azure and Google blob storage failed to validate if it was a directory or path. This PR skips validating directories so it does raise an inappropriate error.
Copy link
Collaborator

@nvnieuwk nvnieuwk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@nvnieuwk
Copy link
Collaborator

nvnieuwk commented Jul 5, 2024

(waiting for the tests, then I'll merge and release)

@adamrtalbot
Copy link
Collaborator Author

Before we merge - is there a way of checking my regex? I couldn't see a good test to add it to.

@nvnieuwk
Copy link
Collaborator

nvnieuwk commented Jul 5, 2024

Before we merge - is there a way of checking my regex? I couldn't see a good test to add it to.

You could add a dummy az:// and gs:// path as an input to a test where existence is checked. It doesn't matter if it exists, because it will be skipped

@adamrtalbot
Copy link
Collaborator Author

Before we merge - is there a way of checking my regex? I couldn't see a good test to add it to.

You could add a dummy az:// and gs:// path as an input to a test where existence is checked. It doesn't matter if it exists, because it will be skipped

yeah but, how do I test it? I've never used a custom plugin before 😅

@nvnieuwk
Copy link
Collaborator

nvnieuwk commented Jul 5, 2024

Ah hahahah you can copy paste a test in this file. Feel free to ask me if something is not clear in it :)

@adamrtalbot
Copy link
Collaborator Author

OK first attempt here: f70cecc

@nvnieuwk
Copy link
Collaborator

nvnieuwk commented Jul 5, 2024

The schema you used doesn't actually check for file existence :p can you use or create another one?

@adamrtalbot
Copy link
Collaborator Author

Corrected? 38b48b7

@ewels
Copy link
Member

ewels commented Jul 5, 2024

Should we also do this whilst we're here? nextflow-io#37

@nvnieuwk
Copy link
Collaborator

nvnieuwk commented Jul 5, 2024

There's also no exists in that schema: https://github.com/nextflow-io/nf-validation/blob/master/plugins/nf-validation/src/testResources/nextflow_schema_with_samplesheet.json

Also better to not use a schema that has the schema keyword in it, because it will try and read the file.

It does look like no exists tests were in nf-validation. Can you write a small new schema yourself? It needs to contain these keywords at least:

"type": "string",
"format": "file-path",
"exists": true

@nvnieuwk
Copy link
Collaborator

nvnieuwk commented Jul 5, 2024

Should we also do this whilst we're here? nextflow-io#37

Good idea!

@nvnieuwk
Copy link
Collaborator

nvnieuwk commented Jul 5, 2024

@adamrtalbot can you add this line to this file?

implementation 'org.json:json:20240303'

@adamrtalbot
Copy link
Collaborator Author

implementation 'org.json:json:20240303'

get yer own PR: #172

@adamrtalbot
Copy link
Collaborator Author

@nvnieuwk looks good now! can you give it a final check for me?

CHANGELOG.md Outdated Show resolved Hide resolved
Copy link
Collaborator

@nvnieuwk nvnieuwk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome looks good!

@nvnieuwk
Copy link
Collaborator

nvnieuwk commented Jul 5, 2024

Can I merge this?

@adamrtalbot
Copy link
Collaborator Author

Can I merge this?

I think so! I haven't been able to test for real yet 😬

@nvnieuwk
Copy link
Collaborator

nvnieuwk commented Jul 5, 2024

Eh it should be fine :)

@ewels
Copy link
Member

ewels commented Jul 5, 2024

Eh it should be fine :)

Given the history of this plugin, please test it 🙈 😆

@nvnieuwk
Copy link
Collaborator

nvnieuwk commented Jul 9, 2024

@adamrtalbot can you run a test? I don't have access to azure or google blob :)

@adamrtalbot
Copy link
Collaborator Author

adamrtalbot commented Jul 9, 2024

@adamrtalbot can you run a test? I don't have access to azure or google blob :)

I'll give it a try but I've never done plugin development so might take me some time. Of course you don't actually need a file, you just need a parameter of type file = az:// or gs://.

@adamrtalbot
Copy link
Collaborator Author

adamrtalbot commented Jul 9, 2024

@nvnieuwk are there any existing template repos or anything I can grab and modify instead of starting from scratch?

Dur, in ./examples: https://github.com/nextflow-io/nf-validation/tree/0edaedfd172d8d26f539b0f33a4443d3c0ac789c/examples/fromSamplesheetBasic

@adamrtalbot
Copy link
Collaborator Author

> ./gradlew check

FAILURE: Build failed with an exception.

* What went wrong:
Could not determine the dependencies of task ':plugins:nf-validation:test'.
> Could not resolve all task dependencies for configuration ':plugins:nf-validation:testRuntimeClasspath'.
   > Could not resolve org.codehaus.groovy:groovy:3.0.14.
     Required by:
         project :plugins:nf-validation
         project :plugins:nf-validation > org.codehaus.groovy:groovy-nio:3.0.14
      > Module 'org.codehaus.groovy:groovy' has been rejected:
           Cannot select module with conflict on capability 'org.codehaus.groovy:groovy:3.0.14' also provided by [org.apache.groovy:groovy:4.0.22(groovyRuntimeElements)]
   > Could not resolve org.codehaus.groovy:groovy-nio:3.0.14.
     Required by:
         project :plugins:nf-validation
      > Module 'org.codehaus.groovy:groovy-nio' has been rejected:
           Cannot select module with conflict on capability 'org.codehaus.groovy:groovy-nio:3.0.14' also provided by [org.apache.groovy:groovy-nio:4.0.22(groovyRuntimeElements)]
   > Could not resolve io.nextflow:nextflow:22.10.0.
     Required by:
         project :plugins:nf-validation
      > No matching variant of project :nextflow:nextflow was found. The consumer was configured to find a runtime of a library compatible with Java 17, packaged as a jar, preferably optimized for standard JVMs, and its dependencies declared externally but:
          - Variant 'apiElements' capability io.nextflow:nextflow:24.06.0-edge declares a library compatible with Java 11, packaged as a jar, and its dependencies declared externally:
              - Incompatible because this component declares an API of a component and the consumer needed a runtime of a component
              - Other compatible attribute:
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
          - Variant 'coverageDataElementsForTest' capability io.nextflow:nextflow:24.06.0-edge:
              - Incompatible because this component declares a component of category 'verification' and the consumer needed a library
              - Other compatible attributes:
                  - Doesn't say anything about how its dependencies are found (required its dependencies declared externally)
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
                  - Doesn't say anything about its target Java version (required compatibility with Java 17)
                  - Doesn't say anything about its elements (required them packaged as a jar)
                  - Doesn't say anything about its usage (required a runtime)
          - Variant 'mainSourceElements' capability io.nextflow:nextflow:24.06.0-edge declares a component, and its dependencies declared externally:
              - Incompatible because this component declares a component of category 'verification' and the consumer needed a library
              - Other compatible attributes:
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
                  - Doesn't say anything about its target Java version (required compatibility with Java 17)
                  - Doesn't say anything about its elements (required them packaged as a jar)
                  - Doesn't say anything about its usage (required a runtime)
          - Variant 'runtimeElements' capability io.nextflow:nextflow:24.06.0-edge declares a runtime of a library compatible with Java 11, packaged as a jar, and its dependencies declared externally:
              - Other compatible attribute:
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
          - Variant 'testFixturesApiElements' capability io.nextflow:nextflow-test-fixtures:24.06.0-edge declares a library, packaged as a jar, and its dependencies declared externally:
              - Incompatible because this component declares an API of a component compatible with Java 21 and the consumer needed a runtime of a component compatible with Java 17
              - Other compatible attribute:
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
          - Variant 'testFixturesRuntimeElements' capability io.nextflow:nextflow-test-fixtures:24.06.0-edge declares a runtime of a library, packaged as a jar, and its dependencies declared externally:
              - Incompatible because this component declares a component compatible with Java 21 and the consumer needed a component compatible with Java 17
              - Other compatible attribute:
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
          - Variant 'testResultsElementsForTest' capability io.nextflow:nextflow:24.06.0-edge:
              - Incompatible because this component declares a component of category 'verification' and the consumer needed a library
              - Other compatible attributes:
                  - Doesn't say anything about how its dependencies are found (required its dependencies declared externally)
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
                  - Doesn't say anything about its target Java version (required compatibility with Java 17)
                  - Doesn't say anything about its elements (required them packaged as a jar)
                  - Doesn't say anything about its usage (required a runtime)
   > Could not resolve io.nextflow:nf-commons:22.10.0.
     Required by:
         project :plugins:nf-validation
      > No matching variant of project :nextflow:nf-commons was found. The consumer was configured to find a runtime of a library compatible with Java 17, packaged as a jar, preferably optimized for standard JVMs, and its dependencies declared externally but:
          - Variant 'apiElements' capability io.nextflow:nf-commons:24.06.0-edge declares a library compatible with Java 11, packaged as a jar, and its dependencies declared externally:
              - Incompatible because this component declares an API of a component and the consumer needed a runtime of a component
              - Other compatible attribute:
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
          - Variant 'coverageDataElementsForTest' capability io.nextflow:nf-commons:24.06.0-edge:
              - Incompatible because this component declares a component of category 'verification' and the consumer needed a library
              - Other compatible attributes:
                  - Doesn't say anything about how its dependencies are found (required its dependencies declared externally)
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
                  - Doesn't say anything about its target Java version (required compatibility with Java 17)
                  - Doesn't say anything about its elements (required them packaged as a jar)
                  - Doesn't say anything about its usage (required a runtime)
          - Variant 'mainSourceElements' capability io.nextflow:nf-commons:24.06.0-edge declares a component, and its dependencies declared externally:
              - Incompatible because this component declares a component of category 'verification' and the consumer needed a library
              - Other compatible attributes:
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
                  - Doesn't say anything about its target Java version (required compatibility with Java 17)
                  - Doesn't say anything about its elements (required them packaged as a jar)
                  - Doesn't say anything about its usage (required a runtime)
          - Variant 'runtimeElements' capability io.nextflow:nf-commons:24.06.0-edge declares a runtime of a library compatible with Java 11, packaged as a jar, and its dependencies declared externally:
              - Other compatible attribute:
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
          - Variant 'testFixturesApiElements' capability io.nextflow:nf-commons-test-fixtures:24.06.0-edge declares a library, packaged as a jar, and its dependencies declared externally:
              - Incompatible because this component declares an API of a component compatible with Java 21 and the consumer needed a runtime of a component compatible with Java 17
              - Other compatible attribute:
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
          - Variant 'testFixturesRuntimeElements' capability io.nextflow:nf-commons-test-fixtures:24.06.0-edge declares a runtime of a library, packaged as a jar, and its dependencies declared externally:
              - Incompatible because this component declares a component compatible with Java 21 and the consumer needed a component compatible with Java 17
              - Other compatible attribute:
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
          - Variant 'testResultsElementsForTest' capability io.nextflow:nf-commons:24.06.0-edge:
              - Incompatible because this component declares a component of category 'verification' and the consumer needed a library
              - Other compatible attributes:
                  - Doesn't say anything about how its dependencies are found (required its dependencies declared externally)
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
                  - Doesn't say anything about its target Java version (required compatibility with Java 17)
                  - Doesn't say anything about its elements (required them packaged as a jar)
                  - Doesn't say anything about its usage (required a runtime)
   > Could not resolve org.apache.groovy:groovy:4.0.22.
     Required by:
         project :plugins:nf-validation > project :nextflow:nextflow
         project :plugins:nf-validation > project :nextflow:nextflow > project :nextflow:nf-commons
         project :plugins:nf-validation > project :nextflow:nextflow > project :nextflow:nf-httpfs
         project :plugins:nf-validation > project :nextflow:nextflow > org.apache.groovy:groovy-nio:4.0.22
         project :plugins:nf-validation > project :nextflow:nextflow > org.apache.groovy:groovy-xml:4.0.22
         project :plugins:nf-validation > project :nextflow:nextflow > org.apache.groovy:groovy-json:4.0.22
         project :plugins:nf-validation > project :nextflow:nextflow > org.apache.groovy:groovy-templates:4.0.22
         project :plugins:nf-validation > project :nextflow:nextflow > org.apache.groovy:groovy-yaml:4.0.22
         project :plugins:nf-validation > project :nextflow:nextflow > org.apache.groovy:groovy-xml:4.0.22 > org.apache.groovy:groovy-bom:4.0.22
      > Module 'org.apache.groovy:groovy' has been rejected:
           Cannot select module with conflict on capability 'org.codehaus.groovy:groovy:4.0.22' also provided by [org.codehaus.groovy:groovy:3.0.14(runtime)]
   > Could not resolve org.apache.groovy:groovy-nio:4.0.22.
     Required by:
         project :plugins:nf-validation > project :nextflow:nextflow
         project :plugins:nf-validation > project :nextflow:nextflow > project :nextflow:nf-commons
         project :plugins:nf-validation > project :nextflow:nextflow > project :nextflow:nf-httpfs
         project :plugins:nf-validation > project :nextflow:nextflow > org.apache.groovy:groovy-xml:4.0.22 > org.apache.groovy:groovy-bom:4.0.22
      > Module 'org.apache.groovy:groovy-nio' has been rejected:
           Cannot select module with conflict on capability 'org.codehaus.groovy:groovy-nio:4.0.22' also provided by [org.codehaus.groovy:groovy-nio:3.0.14(runtime)]

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 2s
18 actionable tasks: 9 executed, 9 up-to-date

@adamrtalbot
Copy link
Collaborator Author

> ./gradlew compileGroovy

> Task :nextflow:nextflow:compileGroovy
Note: /Users/adam.talbot/nextflow/modules/nextflow/src/main/groovy/nextflow/sort/BigSort.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :plugins:nf-validation:compileGroovy FAILED
startup failed:
/Users/adam.talbot/nf-validation/plugins/nf-validation/src/main/nextflow/validation/SamplesheetConverter.groovy: 157: [Static type checking] - Cannot call <K,V> org.codehaus.groovy.runtime.DefaultGroovyMethods#subMap(java.util.Map<K, V>, java.util.Collection<K>) with arguments [java.util.Map<java.lang.String, java.lang.String>, java.util.List<java.lang.Object>]
 @ line 157, column 59.
   String,String> newMap = (Map) row.subMap
                                 ^

/Users/adam.talbot/nf-validation/plugins/nf-validation/src/main/nextflow/validation/SchemaValidator.groovy: 867: [Static type checking] - Cannot call <K,V> org.codehaus.groovy.runtime.DefaultGroovyMethods#leftShift(java.util.Map<K, V>, java.util.Map<K, V>) with arguments [java.util.LinkedHashMap<java.lang.String, java.util.LinkedHashMap<java.lang.Object, java.lang.Object>>, java.util.LinkedHashMap<java.lang.Object, java.lang.Object>]
 @ line 867, column 16.
           return [ 'Core Nextflow options' : workflow_summary ] << params_summary as LinkedHashMap
                  ^

2 errors


FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':plugins:nf-validation:compileGroovy'.
> Compilation failed; see the compiler error output for details.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16s
28 actionable tasks: 10 executed, 18 up-to-date

@nvnieuwk
Copy link
Collaborator

nvnieuwk commented Jul 9, 2024

What is your java version?

@adamrtalbot
Copy link
Collaborator Author

What is your java version?

java 17.0.10-tem
groovy 4.0.22

Both installed with sdkman

@nvnieuwk
Copy link
Collaborator

nvnieuwk commented Jul 9, 2024

Can you try again with an older version (the plugin has been developed in java 11). Not sure that this will fix it but I've no idea at the moment why it would be failing

@adamrtalbot
Copy link
Collaborator Author

No dice! This is on a Mac using Sonoma 14.5 btw.

> sdk use java 11.0.21-tem

Using java version 11.0.21-tem in this shell.
> ./gradlew compileGroovy
Starting a Gradle Daemon, 3 incompatible Daemons could not be reused, use --status for details

> Task :nextflow:nextflow:compileGroovy
Note: /Users/adam.talbot/nextflow/modules/nextflow/src/main/groovy/nextflow/sort/BigSort.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :plugins:nf-validation:compileGroovy FAILED
startup failed:
/Users/adam.talbot/nf-validation/plugins/nf-validation/src/main/nextflow/validation/SamplesheetConverter.groovy: 157: [Static type checking] - Cannot call <K,V> org.codehaus.groovy.runtime.DefaultGroovyMethods#subMap(java.util.Map<K, V>, java.util.Collection<K>) with arguments [java.util.Map<java.lang.String, java.lang.String>, java.util.List<java.lang.Object>]
 @ line 157, column 40.
                       def newMap = (Map) row.subMap((List) [key] + (List) unique)
                                          ^

/Users/adam.talbot/nf-validation/plugins/nf-validation/src/main/nextflow/validation/SchemaValidator.groovy: 867: [Static type checking] - Cannot call <K,V> org.codehaus.groovy.runtime.DefaultGroovyMethods#leftShift(java.util.Map<K, V>, java.util.Map<K, V>) with arguments [java.util.LinkedHashMap<java.lang.String, java.util.LinkedHashMap<java.lang.Object, java.lang.Object>>, java.util.LinkedHashMap<java.lang.Object, java.lang.Object>]
 @ line 867, column 16.
           return [ 'Core Nextflow options' : workflow_summary ] << params_summary as LinkedHashMap
                  ^

2 errors


FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':plugins:nf-validation:compileGroovy'.
> Compilation failed; see the compiler error output for details.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 25s
28 actionable tasks: 24 executed, 4 up-to-date

@nvnieuwk
Copy link
Collaborator

nvnieuwk commented Jul 9, 2024

Yeah I've no idea if it's because of some Mac issue or something else :/. It's weird because the tests are running

Copy link
Collaborator

@nvnieuwk nvnieuwk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some logging adjustments

adamrtalbot and others added 3 commits August 1, 2024 12:56
…ators/DirectoryPathValidator.groovy

Co-authored-by: Nicolas Vannieuwkerke <101190534+nvnieuwk@users.noreply.github.com>
Co-authored-by: Nicolas Vannieuwkerke <101190534+nvnieuwk@users.noreply.github.com>
…ators/FilePathPatternValidator.groovy

Co-authored-by: Nicolas Vannieuwkerke <101190534+nvnieuwk@users.noreply.github.com>
@adamrtalbot
Copy link
Collaborator Author

Demonstrated to work:

main.nf:

include { validateParameters } from 'plugin/nf-validation'

workflow {
    validateParameters()
    println("Hi ${params.input}")
}

nextflow_schema.json:

{
  "$schema": "http://json-schema.org/draft-07/schema",
  "$id": "https://raw.githubusercontent.com/nf-core/testpipeline/master/nextflow_schema.json",
  "title": "nf-core/testpipeline pipeline parameters",
  "description": "this is a test",
  "type": "object",
  "definitions": {
    "file_patterns": {
      "title": "Input/output options",
      "type": "object",
      "fa_icon": "fas fa-terminal",
      "properties": {
        "input": {
          "type": "string",
          "format": "file-path",
          "exists": true,
          "default": "az://igenomes/"
        }
      }
    }
  },
  "allOf": [
    {
      "$ref": "#/definitions/file_patterns"
    }
  ]
}

nextflow.config:

params.input = 'az://igenomes/'

plugins {
    id 'nf-validation@1.1.4' // Or 1.1.3 to demonstrate the fix!
}

With nf-validation@1.1.3:

> nextflow run .
Nextflow 24.04.4 is available - Please consider updating your version to it
N E X T F L O W  ~  version 24.04.2
Launching `./main.nf` [shrivelled_morse] DSL2 - revision: a2030e4699
ERROR ~ ERROR: Validation of pipeline parameters failed!

 -- Check '.nextflow.log' file for details
The following invalid input values have been detected:

* --input: 'az://igenomes/' is not a file, but a directory (az://igenomes/)

 -- Check script 'main.nf' at line: 5 or see '.nextflow.log' file for more details

With nf-validation@1.1.4:

> nextflow run .
Nextflow 24.04.4 is available - Please consider updating your version to it
N E X T F L O W  ~  version 24.04.2
Launching `./main.nf` [admiring_hoover] DSL2 - revision: a2030e4699
Hi az://igenomes/

@adamrtalbot adamrtalbot merged commit 750a56d into nextflow-io:master Aug 7, 2024
3 checks passed
@adamrtalbot adamrtalbot deleted the do_not_validate_az_and_gcp_paths branch August 7, 2024 15:23
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants