Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Swift registry support #1016

Open
watt opened this issue Apr 15, 2024 · 11 comments · May be fixed by #1318
Open

Swift registry support #1016

watt opened this issue Apr 15, 2024 · 11 comments · May be fixed by #1318
Labels
enhancement New feature or request

Comments

@watt
Copy link
Contributor

watt commented Apr 15, 2024

It looks like there's currently no support for package dependencies resolved from a package registry. That is, packages defined with the id syntax:

dependencies: [
    .package(id: "mona.LinkedList", .upToNextMajor(from: "1.0.0")),
],

If Package.swift contains a dependency like this, the swift_update_pkgs task fails with this error:

gazelle: Identity could not be determined.

From a cursory look at the source it seems like registry dependencies aren't part of the data model. I'm not sure what additional changes would be necessary on top of that.

I'm guessing it might be a pain to build this out if one doesn't have access to a registry already, since there are no public Swift registries I am aware of. I'm happy to help test things or provide samples from a project resolved against a registry if it's useful.

@luispadron luispadron added the enhancement New feature or request label Apr 15, 2024
@luispadron
Copy link
Collaborator

Is this something you'd want to implement and PR @watt? We can answer questions in here if needed.

@watt
Copy link
Contributor Author

watt commented Apr 17, 2024

Is this something you'd want to implement and PR @watt? We can answer questions in here if needed.

I'm willing to try, though I will definitely need some guidance.

Aside from updating the data models that map to JSON files, any initial thoughts on what logic might need to be touched?

@luispadron
Copy link
Collaborator

I'm not super familiar with this part of the code so maybe @cgrindel could be more helpful.

I'd try updating this area first to support registry ID: https://github.com/cgrindel/rules_swift_package_manager/blob/main/gazelle/internal/swiftpkg/dependency.go and seeing how far we get. We likely just need to map into the package name which should be possible.

@cgrindel
Copy link
Owner

@watt Thanks for filing this issue. What is added to the Package.resolved after swift package update? Is it just a URL?

As a little background, I am currently working #924 which will radically change how external dependencies are downloaded. In essence, it relies on the information stored in the Package.resolved to download the external Swift packages. If the Package.resolved contains a URL for the resolved version, then supporting this should be fairly straightforward. 🤞

@watt
Copy link
Contributor Author

watt commented Apr 18, 2024

What is added to the Package.resolved after swift package update? Is it just a URL?

No, unfortunately, it's frustratingly light on data.

For this Package.swift:

// swift-tools-version: 5.7

import PackageDescription

let package = Package(
    name: "my-project",
    dependencies: [
        .package(id: "apple.swift-collections", from: "1.1.0"),
    ]
)

I get this Package.resolved:

{
  "pins" : [
    {
      "identity" : "apple.swift-collections",
      "kind" : "registry",
      "location" : "",
      "state" : {
        "version" : "1.1.0"
      }
    }
  ],
  "version" : 2
}

To reconstruct the remote URL we could read the registries.json config and figure out which registry this comes from, but this would require recreating a fair amount of behavior of SwiftPM.

Alternatively, the dependency shows up in the build directory as well, and it might be easier to just use that.

Screenshot 2024-04-18 at 11 39 47 AM

@cgrindel
Copy link
Owner

Unfortunately, we can't use the files downloaded in the build directory. The repository rule needs to be download the package from the information that is checked into client's repository. I think that we will need to read the registries.json, per your suggestion.

At a minimum, I presume that the registries.json contains a URL. Does it embed any authentication information, as well?

@watt
Copy link
Contributor Author

watt commented Apr 25, 2024

I drafted up an implementation in #1043. It'll surely need some work, but in the included example case, this works for me.

This draft hardcodes a registry URL rather than reading registries.json. I think it won't be too hard to read, but I wasn't sure if it would be more idiomatic to read the registry config early, during the swift_update_pkgs task, and generate build rules that look have a pre-resolved URL like this:

swift_registry_package(
    name = "swiftpkg_apple.swift_collections",
    dependencies_index = "@//:swift_packages_index.json",
    url = "https://artifactory.global.square/artifactory/api/swift/swift-test/apple/swift-collections/1.1.0.zip",
)

Or to read the registry config later, during build, and generate build rules that look like this:

swift_registry_package(
    name = "swiftpkg_apple.swift_collections",
    dependencies_index = "@//:swift_packages_index.json",
    id = "apple.swift-collections",
    version = "1.1.0",
)

The latter mirrors Package.swift more closely but the work involved will effectively be the same either way, I think? This draft generates rules like the former.

@watt
Copy link
Contributor Author

watt commented Apr 25, 2024

At a minimum, I presume that the registries.json contains a URL.

Yes. The format is described here. I think picking a registry is a matter of merging the user config with project config, and then choosing either a scoped registry or the default.

Does it embed any authentication information, as well?

Nope. Mine is stored in ~/.netrc.

@cgrindel
Copy link
Owner

I wasn't sure if it would be more idiomatic to read the registry config early

As part of #924, we are moving away from generating Bazel stuff in swift_update_pkgs. We will want to read it later.

I like your second option. However, I think that it will be something closer to this:

swift_registry_package(
    name = "swiftpkg_apple_swift_collections",
    registry_json = "@//:registries.json",
    id = "apple.swift-collections",
    version = "1.1.0",
)

The repository rule can read the registries file look up the URL and perform the download. WDT?

@watt
Copy link
Contributor Author

watt commented Apr 27, 2024

As part of #924, we are moving away from generating Bazel stuff in swift_update_pkgs. We will want to read it later.

Ah, OK. Does that mean the swift_update_pkgs task goes away entirely, and swift rules are generated dynamically on demand (if that's even possible in bazel)? Not that it affects this, just curious.

I like your second option. However, I think that it will be something closer to this:

swift_registry_package(
    name = "swiftpkg_apple_swift_collections",
    registry_json = "@//:registries.json",
    id = "apple.swift-collections",
    version = "1.1.0",
)

The repository rule can read the registries file look up the URL and perform the download. WDT?

Is @//:registries.json meant to be configurable, like you could reference a different JSON file? If we did that, I think it might differ from a bit from SwiftPM? SwiftPM normally looks in two places (and merges them):

  • user config in ~/.swiftpm/configuration/registries.json
  • project config in .swiftpm/configuration/registries.json

swift package has a --config-path option that seems like it might override the user path but I'm not sure you can alter the project config path.

@watt
Copy link
Contributor Author

watt commented May 1, 2024

I pushed an update that reads the registry config from starlark. For now, it uses a hardcoded path of @//:.swiftpm/configuration/registries.json rather than making the config location configurable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
3 participants