Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test-infra: improve compiletest and run-make-support symlink handling #134659

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

jieyouxu
Copy link
Member

@jieyouxu jieyouxu commented Dec 22, 2024

I was trying to implement #134656 to port tests/run-make/incr-add-rust-src-component.rs, but found some blockers related to symlink handling, so in this PR I tried to resolve them by improving symlink handling in compiletest and run-make-support (particularly for native windows msvc environment).

Key changes:

  • I needed to copy symlinks (duplicate a symlink pointing to the same file), so I pulled out the copy symlink logic and re-exposed it as run_make_support::rfs::copy_symlink. This helper correctly accounts for the Windows symlink-to-file vs symlink-to-dir distinction (hereafter "Windows symlinks") when copying symlinks.
  • recursive_remove:
    • I needed a way to remove symlinks themselves (no symlink traversal). std::fs::remove_dir_all handles them, but only if the root path is a directory. So I wrapped std::fs::remove_dir_all to also handle when the root path is a non-directory entity (e.g. file or symlink). Again, this properly accounts for Windows symlinks.
    • I wanted to use this for both compiletest and run-make-support, so I put the implementation and accompanying tests in build_helper.
    • In this sense, it's a reland of compiletest: use std::fs::remove_dir_all now that it is available #129302 with proper test coverage.
    • It's a thin wrapper around std::fs::remove_dir_all (remove_dir_all correctly handles read-only entries on Windows). The helper has additional permission-setting logic for when the root path is a non-dir entry on Windows to handle read-only non-dir entry.

Fixes #126334.

@rustbot

This comment was marked as off-topic.

@rustbot rustbot added A-compiletest Area: The compiletest test runner A-run-make Area: port run-make Makefiles to rmake.rs A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Dec 22, 2024
@jieyouxu jieyouxu removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Dec 22, 2024
@rustbot

This comment was marked as off-topic.

@jieyouxu jieyouxu added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Dec 22, 2024
@jieyouxu jieyouxu removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Dec 22, 2024
@rust-log-analyzer

This comment has been minimized.

@ChrisDenton
Copy link
Member

Can I ask why you need to FORCE PERMISSIONS? remove_dir_all should remove readonly file on Windows 10 and 11.

@jieyouxu
Copy link
Member Author

jieyouxu commented Dec 22, 2024

Can I ask why you need to FORCE PERMISSIONS? remove_dir_all should remove readonly file on Windows 10 and 11.

... good question. For some reason I completely forgot about this. Let me revert it real quick.

EDIT: ah, but I will need to force perms when the root path is a non-dir entry. Does std::fs::remove_file handle read-only files? Based on locally testing, not remove_file (but remove_dir_all will).

@ChrisDenton
Copy link
Member

It should do. I'd be minded to try something like:

#[cfg(windows)]
use std::os::windows::fs::FileTypeExt;

pub fn recursive_remove<P: AsRef<Path>>(path: P) -> io::Result<()> {
    let path = path.as_ref();
    let metadata = fs::symlink_metadata(path)?;
    #[cfg(windows)]
    let is_dir = |meta: &fs::Metadata| meta.is_dir() || meta.file_type().is_symlink_dir();
    #[cfg(not(windows))]
    let is_dir = fs::Metadata::is_dir;

    if is_dir(&metadata) { fs::remove_dir_all(path) } else { fs::remove_file(path) }
}

@jieyouxu jieyouxu force-pushed the recursive-remove branch 2 times, most recently from 920e230 to bba8509 Compare December 22, 2024 17:30
@jieyouxu
Copy link
Member Author

It should do. I'd be minded to try something like:

#[cfg(windows)]
use std::os::windows::fs::FileTypeExt;

pub fn recursive_remove<P: AsRef<Path>>(path: P) -> io::Result<()> {
    let path = path.as_ref();
    let metadata = fs::symlink_metadata(path)?;
    #[cfg(windows)]
    let is_dir = |meta: &fs::Metadata| meta.is_dir() || meta.file_type().is_symlink_dir();
    #[cfg(not(windows))]
    let is_dir = fs::Metadata::is_dir;

    if is_dir(&metadata) { fs::remove_dir_all(path) } else { fs::remove_file(path) }
}

Thanks, that indeed is what I needed to do, just needed to additionally handle the special case of root path being a non-dir read-only entry.

@rust-log-analyzer

This comment has been minimized.

@jieyouxu
Copy link
Member Author

Chris said I can
r? @ChrisDenton

@jieyouxu jieyouxu marked this pull request as ready for review December 22, 2024 17:45
bors added a commit to rust-lang-ci/rust that referenced this pull request Dec 22, 2024
…omponent, r=<try>

Migrate `incr-add-rust-src-component` to rmake

**BLOCKED: this PR depends on rust-lang#134659**
**TODO: actually write some useful PR description**

r? `@ghost`

try-job: i686-msvc
try-job: x86_64-mingw-1
try-job: x86_64-msvc
try-job: aarch64-apple
@ChrisDenton
Copy link
Member

std should probably make remove_file just work on readonly files. But that's a job for another time.

@jieyouxu
Copy link
Member Author

@bors rollup=iffy (who knows what this will fail on)

@ChrisDenton
Copy link
Member

Ok, this version looks good to me. And I do like having a range of tests.

@bors r+

@bors
Copy link
Contributor

bors commented Dec 22, 2024

📌 Commit 5bb55fe has been approved by ChrisDenton

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Dec 22, 2024
@rust-log-analyzer

This comment has been minimized.

@ChrisDenton
Copy link
Member

@bors r-

I was so focused on Windows I missed the Linux, ha

@bors bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Dec 22, 2024
@jieyouxu jieyouxu force-pushed the recursive-remove branch 2 times, most recently from fbc60e6 to a001eff Compare December 22, 2024 18:41
@jieyouxu
Copy link
Member Author

jieyouxu commented Dec 22, 2024

I just added a #![allow(unused_variables)] to copy_symlink_raw, let me know if you prefer let _ = ty instead.

bors added a commit to rust-lang-ci/rust that referenced this pull request Dec 22, 2024
…omponent, r=<try>

Migrate `incr-add-rust-src-component` to rmake

**BLOCKED: this PR depends on rust-lang#134659**
**TODO: actually write some useful PR description**

r? `@ghost`

try-job: i686-msvc
try-job: x86_64-mingw-1
try-job: x86_64-msvc
try-job: aarch64-apple
@ChrisDenton
Copy link
Member

Hm, yeah, I'd prefer let _ in the long run even if it's a bit more verbose. Because when things change it's more obvious what was intended to be ignored and what wasn't.

jieyouxu and others added 4 commits December 23, 2024 03:25
`recursive_remove` is intended to be a wrapper around
`std::fs::remove_dir_all`, but which also allows the removal target to
be a non-directory entry, i.e. a file or a symlink. It also tries to
remove read-only attributes from filesystem entities on Windows for
non-dir entries, as `std::fs::remove_dir_all` handles the dir (and its
children) read-only cases.

Co-authored-by: Chris Denton <[email protected]>
`aggressive_rm_rf` does not correctly handle distinction between
symlink-to-file vs symlink-to-dir on Windows.
This facade is like other `run_make_support::fs` APIs that
panic-on-failure but includes the path that the operation was called on
in the panic message.
@jieyouxu
Copy link
Member Author

Changed to use let _ = ty; instead of allow(unused_variables).

@jieyouxu
Copy link
Member Author

@rustbot ready

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Dec 22, 2024
@ChrisDenton
Copy link
Member

@bors r+

@bors
Copy link
Contributor

bors commented Dec 22, 2024

📌 Commit 4d3bf1f has been approved by ChrisDenton

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Dec 22, 2024
tgross35 added a commit to tgross35/rust that referenced this pull request Dec 23, 2024
…Denton

test-infra: improve compiletest and run-make-support symlink handling

I was trying to implement rust-lang#134656 to port `tests/run-make/incr-add-rust-src-component.rs`, but found some blockers related to symlink handling, so in this PR I tried to resolve them by improving symlink handling in compiletest and run-make-support (particularly for native windows msvc environment).

Key changes:

- I needed to copy symlinks (duplicate a symlink pointing to the same file), so I pulled out the copy symlink logic and re-exposed it as `run_make_support::rfs::copy_symlink`. This helper correctly accounts for the Windows symlink-to-file vs symlink-to-dir distinction (hereafter "Windows symlinks") when copying symlinks.
- `recursive_remove`:
    - I needed a way to remove symlinks themselves (no symlink traversal). `std::fs::remove_dir_all` handles them, but only if the root path is a directory. So I wrapped `std::fs::remove_dir_all` to also handle when the root path is a non-directory entity (e.g. file or symlink). Again, this properly accounts for Windows symlinks.
    - I wanted to use this for both compiletest and run-make-support, so I put the implementation and accompanying tests in `build_helper`.
    - In this sense, it's a reland of rust-lang#129302 with proper test coverage.
    - It's a thin wrapper around `std::fs::remove_dir_all` (`remove_dir_all` correctly handles read-only entries on Windows). The helper has additional permission-setting logic for when the root path is a non-dir entry on Windows to handle read-only non-dir entry.

Fixes rust-lang#126334.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-compiletest Area: The compiletest test runner A-run-make Area: port run-make Makefiles to rmake.rs A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

compiletest: aggressive_rm_rf might not be aggressive enough
6 participants