feat: add support for Swift package registries #1318

luispadron · 2024-10-30T17:02:03Z

This adds initial support for consuming packages using a Swift Package Registry.

The majority of the changes are in the new registry_swift_package rule which creates a repository to build the targets from a package downloaded via a registry.

Notably, because the identity of the package is used to derive the repository name, packages used via a registry have the following label: @swiftpkg_[org].[name], so for example @swiftpkg_swift_collections becomes @swiftpkg_apple.swift_collections if using the swift-collections package from the apple organization (as registries support multiple orgs).

The registry_swift_package rule is mostly just an updated version of #1043 (thanks @watt)

Open questions

Should the label replace . with _ in identity? E.g. @swiftpkg_apple.swift_collections vs. @swiftpkg_apple_swift_collections: NO

Follow-up

rule_swift_package_manager already has a bootstrapping issue, if there is no Package.resolved defined already then the Bazel repositories fail to generate. The addition of registry support adds another bootstrapping issue with the registries.json file (users will need to run swift package-registry commands to create it currently). We should address both of these by making bootstrapping a new project easier.

Caveats

There is no public Swift Package Registry (that I know of), to support testing this in CI I created a fake registry over at: https://github.com/luispadron/fake-swift-package-registry which is hosted at: https://fake-spm-registry.deno.dev
- Example: https://fake-spm-registry.deno.dev/apple/swift-collections
- I can maintain the fake, read-only registry until we have an alternative or it somehow starts costing me too much money 😅

Example

The following shows an example of a swift_binary target depending on an SPM dependency downloaded from a registry defined in registries.json.

`Package.swift`

let package = Package(
    name: "swift_package_registry_example",
    dependencies: [
        .package(id: "apple.swift-collections", exact: "1.1.3"),
    ]
)

`Package.resolved`

{
  "pins" : [
    {
      "identity" : "apple.swift-collections",
      "kind" : "registry",
      "location" : "",
      "state" : {
        "version" : "1.1.3"
      }
    }
  ],
  "version" : 2
}

`registries.json`

{
  "authentication" : {

  },
  "registries" : {
    "[default]" : {
      "supportsAvailability" : false,
      "url" : "https://fake-spm-registry.deno.dev/"
    }
  },
  "version" : 1
}

`MODULE.bazel`

swift_deps = use_extension(
    "@rules_swift_package_manager//:extensions.bzl",
    "swift_deps",
)
swift_deps.from_package(
    registries = "//:registries.json",
    resolved = "//:Package.resolved",
    swift = "//:Package.swift",
)
swift_deps.configure_swift_package(
    replace_scm_with_registry = True,
)
use_repo(
    swift_deps,
    "swift_package",
    "swiftpkg_apple.swift_collections",
)

`BUILD.bazel`

swift_binary(
    name = "swift_package_registry_example",
    srcs = ["main.swift"],
    module_name = "swift_package_registry_example",
    visibility = ["//visibility:public"],
    deps = ["@swiftpkg_apple.swift_collections//:Collections"],
)

swiftpkg/internal/bazel_repo_names.bzl

examples/swift_package_registry_example/registries.json

luispadron · 2024-11-12T16:46:16Z

Im testing in a real registry implementation now and realizing some issues, going to make some updates but the majority of the design is ready for review

brentleyjones · 2024-11-12T16:58:22Z

We should address both of these by making bootstrapping a new project easier.

💯 (as a follow-up).

brentleyjones · 2024-11-12T17:00:58Z

I can maintain the fake, read-only registry until we have an alternative or it somehow starts costing me too much money

Can we spin up a local webserver as part of CI instead?

cgrindel · 2024-11-12T17:24:06Z

Should the label replace . with _ in identity? E.g. @swiftpkg_apple.swift_collections vs. @swiftpkg_apple_swift_collections

I like the underscores version. However, it is not a strong preference.

cgrindel · 2024-11-12T17:27:15Z

I can maintain the fake, read-only registry until we have an alternative or it somehow starts costing me too much money
Can we spin up a local webserver as part of CI instead?

+1 to @brentleyjones suggestion. Again, I am not too familiar with Swift registry hosting. If it is static, can we use GitHub to host it? I presume the functionality just needs a URL.

examples/interesting_deps/MODULE.bazel

swiftpkg/internal/swift_package_tool_attrs.bzl

luispadron · 2024-11-12T18:21:43Z

I can maintain the fake, read-only registry until we have an alternative or it somehow starts costing me too much money

Can we spin up a local webserver as part of CI instead?

I tried this originally but local host doesn't support https and so the bazel download stuff fails because of insecure connection.

luispadron · 2024-11-12T18:22:10Z

I can maintain the fake, read-only registry until we have an alternative or it somehow starts costing me too much money

Can we spin up a local webserver as part of CI instead?

+1 to @brentleyjones suggestion. Again, I am not too familiar with Swift registry hosting. If it is static, can we use GitHub to host it? I presume the functionality just needs a URL.

It is not static, it needs to be a web server which follows the registry spec.

luispadron · 2024-11-12T18:56:42Z

Im testing in a real registry implementation now and realizing some issues, going to make some updates but the majority of the design is ready for review

Done! I fixed my fake registry implementation, the .zip needed to contain a single directory named scope.name where i had name-version

cgrindel · 2024-11-12T19:13:08Z

@luispadron When you are ready to merge, if you don't want to babysit the merge, you can specify @mergifyio queue in a comment. This will add it to the merge queue that we are using.

luispadron · 2024-11-12T19:54:57Z

Should the label replace . with _ in identity? E.g. @swiftpkg_apple.swift_collections vs. @swiftpkg_apple_swift_collections

I think the answer is here is: no? Thoughts? Doing the replacement would require manipulating the identity in a fair amount of the rules and im not sure it gives us much (plus the label is still different than non registry so we don't gain anything from doing the replacement)

watt

🎉

swiftpkg/internal/registry_swift_package.bzl

swiftpkg/internal/swift_package_tool_attrs.bzl

swiftpkg/internal/swift_package_tool_runner_template.sh

examples/swift_package_registry_example/Package.swift

brentleyjones · 2024-11-13T19:54:31Z

I think the answer is here is: no?

Yeah, let's not do the replacement.

luispadron · 2024-12-09T16:05:01Z

@cgrindel @watt @brentleyjones The PR has been updated, I believe I have addressed all comments here if folks want to give it another review. The biggest change was proper support for replace_scm_with_registry.

I also added a doc on using Swift package registries over here

brentleyjones · 2024-12-11T15:25:33Z

Firebase failure is unrelated to this change (cc: @cgrindel).

cgrindel · 2024-12-11T15:28:53Z

Firebase failure is unrelated to this change (cc: @cgrindel).

@brentleyjones That is a weird error. Is this an issue in rules_xcodeproj?

brentleyjones · 2024-12-11T15:31:34Z

Yes. Seems that rules_xcodeproj doesn't support Bazel 8 yet, in particular it looks like the change from ~ to + in Bzlmod labels.

cgrindel

Overall, LGTM. Most of my comments are related to formatting (e.g., line length).

cgrindel · 2024-12-11T15:32:06Z

swiftpkg/bzlmod/swift_deps.bzl

@@ -196,10 +206,23 @@ def _declare_pkg_from_dependency(dep, config_pkg):
            dependencies_index = None,
        )

+    elif dep.registry:
+        resolved = from_package.resolved if from_package else None
+        replace_scm_with_registry = config_swift_package.replace_scm_with_registry if config_swift_package else False


Can we format for line length?

cgrindel · 2024-12-11T15:32:56Z

swiftpkg/bzlmod/swift_deps.bzl

-            config_swift_package = config_swift_package
+        for config_swift_package_tag in mod.tags.configure_swift_package:
+            if config_swift_package:
+                fail("Expected only one `configure_swift_package` tag, but found multiple.")


Suggested change

fail("Expected only one `configure_swift_package` tag, but found multiple.")

fail("""\

Expected only one `configure_swift_package` tag, but found multiple.\

""")

cgrindel · 2024-12-11T15:35:18Z

swiftpkg/internal/pkginfos.bzl

@@ -499,7 +519,7 @@ def _new_dependency_identity_to_name_map(dump_deps):
    result = {}
    for dep in dump_deps:
        identity_provider_list = (
-            dep.get("sourceControl") or dep.get("fileSystem")
+            dep.get("sourceControl") or dep.get("fileSystem") or dep.get("registry")


Line length.

cgrindel · 2024-12-11T15:35:29Z

swiftpkg/internal/pkginfos.bzl

@@ -703,7 +723,7 @@ def _new_platform(name, version):

 # MARK: - External Dependency

-def _new_dependency(identity, name, source_control = None, file_system = None):
+def _new_dependency(identity, name, source_control = None, file_system = None, registry = None):


Line length.

cgrindel · 2024-12-11T15:36:46Z

swiftpkg/internal/registry_swift_package.bzl

+    # NOTE: because the Swift Package Registry spec does not define the allowable prefixes within the archive,
+    # we cannot rely on Bazel's `stripPrefix` feature to strip the parent directory reliably.
+    # Instead, we'll download and extract to a temporary location and then move the contents to the output.


Line length

cgrindel · 2024-12-11T15:36:57Z

swiftpkg/internal/registry_swift_package.bzl

+    # Instead, we'll download and extract to a temporary location and then move the contents to the output.
+    download_output = "{output}/download".format(output = output)
+
+    # NOTE: this requires Bazel 7.1 or later to use `headers` which must be set to reach this endpoint.


Please check the line length for this entire file.

cgrindel · 2024-12-11T15:39:19Z

swiftpkg/internal/swift_package_tool.bzl

    template_dict.add("%(enable_build_manifest_caching)s", "true" if ctx.attr.manifest_caching else "false")
    template_dict.add("%(enable_dependency_cache)s", "true" if ctx.attr.dependency_caching else "false")
    template_dict.add("%(manifest_cache)s", ctx.attr.manifest_cache)
+    template_dict.add("%(registries_json)s", registries.short_path if registries else "")
+    template_dict.add("%(replace_scm_with_registry)s", "true" if ctx.attr.replace_scm_with_registry else "false")


Line length.

cgrindel · 2024-12-11T15:39:58Z

swiftpkg/internal/swift_package_tool_attrs.bzl

+
+_swift_package_tool_config_attrs = {
+    "build_path": attr.string(
+        doc = "The relative path within the runfiles tree for the Swift Package Manager build directory.",


Please check the line length for this file.

cgrindel · 2024-12-11T15:49:10Z

Yes. Seems that rules_xcodeproj doesn't support Bazel 8 yet, in particular it looks like the change from ~ to + in Bzlmod labels.

Something is wrong. The .bazelversion for the repo is set to 7.4.1. 🤔 This seems to suggest that rules_bazel_integration_test is not using the correct version information.

watt

I attempted to test this locally with a Package.swift that has a dependency graph requiring the SCM replacement, and it didn't seem to work for me unfortunately. Haven't dug into exactly where it failed yet.

watt · 2024-12-12T01:10:08Z

swiftpkg/internal/registry_swift_package.bzl

+
+    return source_archive.get("checksum")
+
+def _get_resolved_pin_for_url(*, registry_url, repository_ctx, resolved_pkg_map, url):


Could there be a layer of caching in here somewhere, to avoid hitting the registry multiple times for the same URL?

watt · 2024-12-12T01:13:13Z

docs/swift_package_registry.md

Did you want to mention the Bazel 7 requirement in this file?

luispadron force-pushed the luis/add-swift-package-registry-support branch 2 times, most recently from a5adfda to 6739de1 Compare October 30, 2024 17:08

brentleyjones reviewed Oct 30, 2024

View reviewed changes

swiftpkg/internal/bazel_repo_names.bzl Outdated Show resolved Hide resolved

luispadron force-pushed the luis/add-swift-package-registry-support branch 4 times, most recently from 452f05d to ad96158 Compare November 5, 2024 07:15

luispadron force-pushed the luis/add-swift-package-registry-support branch 2 times, most recently from 671c25d to f921aba Compare November 12, 2024 02:41

luispadron marked this pull request as ready for review November 12, 2024 02:49

luispadron requested review from brentleyjones, cgrindel and jpsim November 12, 2024 02:51

luispadron force-pushed the luis/add-swift-package-registry-support branch 3 times, most recently from a541b18 to 9838fcd Compare November 12, 2024 04:50

luispadron commented Nov 12, 2024

View reviewed changes

examples/swift_package_registry_example/registries.json Outdated Show resolved Hide resolved

luispadron force-pushed the luis/add-swift-package-registry-support branch 4 times, most recently from 3d7452d to 01abd62 Compare November 12, 2024 16:45

cgrindel reviewed Nov 12, 2024

View reviewed changes

examples/interesting_deps/MODULE.bazel Outdated Show resolved Hide resolved

swiftpkg/internal/swift_package_tool_attrs.bzl Show resolved Hide resolved

luispadron force-pushed the luis/add-swift-package-registry-support branch from 01abd62 to 2a46f9e Compare November 12, 2024 18:17

cgrindel approved these changes Nov 12, 2024

View reviewed changes

watt approved these changes Nov 13, 2024

View reviewed changes

luispadron force-pushed the luis/add-swift-package-registry-support branch 3 times, most recently from 24c49a6 to 7ef0ec4 Compare November 24, 2024 23:35

luispadron force-pushed the luis/add-swift-package-registry-support branch 4 times, most recently from 0aebb42 to 00897e2 Compare December 9, 2024 16:03

feat: add support for Swift package registries

bc773b3

luispadron force-pushed the luis/add-swift-package-registry-support branch from 00897e2 to bc773b3 Compare December 11, 2024 05:55

cgrindel reviewed Dec 11, 2024

View reviewed changes

watt reviewed Dec 12, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add support for Swift package registries #1318

feat: add support for Swift package registries #1318

luispadron commented Oct 30, 2024 •

edited

Loading

luispadron commented Nov 12, 2024 •

edited

Loading

brentleyjones commented Nov 12, 2024

brentleyjones commented Nov 12, 2024

cgrindel commented Nov 12, 2024

cgrindel commented Nov 12, 2024

luispadron commented Nov 12, 2024

luispadron commented Nov 12, 2024

luispadron commented Nov 12, 2024

cgrindel commented Nov 12, 2024

luispadron commented Nov 12, 2024

watt left a comment

brentleyjones commented Nov 13, 2024

luispadron commented Dec 9, 2024

brentleyjones commented Dec 11, 2024

cgrindel commented Dec 11, 2024

brentleyjones commented Dec 11, 2024

cgrindel left a comment

cgrindel Dec 11, 2024

cgrindel Dec 11, 2024

cgrindel Dec 11, 2024

cgrindel Dec 11, 2024

cgrindel Dec 11, 2024

cgrindel Dec 11, 2024

cgrindel Dec 11, 2024

cgrindel Dec 11, 2024

cgrindel commented Dec 11, 2024

watt left a comment

watt Dec 12, 2024

watt Dec 12, 2024

-                fail("Expected only one `configure_swift_package` tag, but found multiple.")
+                fail("""\
+Expected only one `configure_swift_package` tag, but found multiple.\
+""")


		return source_archive.get("checksum")

		def _get_resolved_pin_for_url(*, registry_url, repository_ctx, resolved_pkg_map, url):

feat: add support for Swift package registries #1318

Are you sure you want to change the base?

feat: add support for Swift package registries #1318

Conversation

luispadron commented Oct 30, 2024 • edited Loading

Open questions

Follow-up

Caveats

Example

Package.swift

Package.resolved

registries.json

MODULE.bazel

BUILD.bazel

luispadron commented Nov 12, 2024 • edited Loading

brentleyjones commented Nov 12, 2024

brentleyjones commented Nov 12, 2024

cgrindel commented Nov 12, 2024

cgrindel commented Nov 12, 2024

luispadron commented Nov 12, 2024

luispadron commented Nov 12, 2024

luispadron commented Nov 12, 2024

cgrindel commented Nov 12, 2024

luispadron commented Nov 12, 2024

watt left a comment

Choose a reason for hiding this comment

brentleyjones commented Nov 13, 2024

luispadron commented Dec 9, 2024

brentleyjones commented Dec 11, 2024

cgrindel commented Dec 11, 2024

brentleyjones commented Dec 11, 2024

cgrindel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cgrindel commented Dec 11, 2024

watt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luispadron commented Oct 30, 2024 •

edited

Loading

`Package.swift`

`Package.resolved`

`registries.json`

`MODULE.bazel`

`BUILD.bazel`

luispadron commented Nov 12, 2024 •

edited

Loading