Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix publish error when remote path contains double slash #5112

Closed
wants to merge 1 commit into from

Conversation

bentsherman
Copy link
Member

GCP doesn't accept paths that contain a double slash. Apparently this has been the case for a while, but we didn't realize it because publish errors were being ignored by default. Until recently when we made publish errors fatal.

The root cause appears to be that FileHelper.toCanonicalPath() does not normalize the path when it is remote. You can verify using this test case:

// local
outdir = 'results/'
filename = "${outdir}/bco.json"

println "local complete: ${(filename as Path).complete()}"
println "local canonical: ${nextflow.file.FileHelper.toCanonicalPath(filename)}"

// remote
outdir = 'gs://bucket/results/'
filename = "${outdir}/bco.json"

println "remote complete:  ${(filename as Path).complete().toUriString()}"
println "remote canonical: ${nextflow.file.FileHelper.toCanonicalPath(filename).toUriString()}"

Output:

local complete: /home/bent/projects/sketches/results/bco.json
local canonical: /home/bent/projects/sketches/results/bco.json
remote complete:  gs://bucket/results/bco.json
remote canonical: gs://bucket/results//bco.json

This will fix the specific issue with PublishDir, but long-term we should probably review the various ways in which strings are converted to paths across the codebase (which there are several) and try to use a uniform approach. The approach I used in this PR seems to be the most common one and the most robust.

See also: nextflow-io/nf-prov#34

Copy link

netlify bot commented Jul 3, 2024

Deploy Preview for nextflow-docs-staging canceled.

Name Link
🔨 Latest commit 34cc69d
🔍 Latest deploy log https://app.netlify.com/sites/nextflow-docs-staging/deploys/668583b5cfedac0008f3bd42

@@ -144,7 +144,7 @@ class PublishDir {
final resolved = value instanceof Closure ? value.call() : value
if( resolved instanceof String || resolved instanceof GString )
nullPathWarn = checkNull(resolved.toString())
this.path = FileHelper.toCanonicalPath(resolved)
this.path = (resolved as Path).complete()
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may be better to fix this at level of toCanonicalPath method

@pditommaso
Copy link
Member

Closing in favour of #5114

@pditommaso pditommaso closed this Jul 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants