Design scheme for fetching yarn dependencies from arbitrary registries

[dboreham]

Currently yarn stores the host name used to fetch dependencies in yarn.lock. Since yarn.lock is in the git repo, this effectively prevents decentralized builds (see: yarnpkg/yarn#2566 and yarnpkg/rfcs#64). We need to devise some method for overriding the host name in yarn.lock so our users can build packages using their npm registry of choice.

[dboreham]

Trying this: https://github.com/kiegroup/lock-treatment-tool

[dboreham]

Some experimentation:

Redhat's lock-treatment-tool doesn't do what we need because its behavior can't be made scope-specific. Perhaps we can upgrade it to support scoping.

Running yarn remove @cerc-io/package then yarn add @cerc-io/package@version doesn't work because it updates the versions of transitive dependencies of "package".

[dboreham]

Trying this: https://stackoverflow.com/questions/49501749/is-it-possible-to-ignore-the-dependency-hash-validation-of-just-one-module-or-r

That didn't work either.

[dboreham]

It appears that we need to edit yarn.lock to replace whatever repository URL and checksum is there with the correct info for our local registry. Luckily yarn info can show us that info:

  dist: {
    integrity: 'sha512-wYwbQGqQKvZ6Lk5Ft6F7TshgFwKF5d8wyCwOf1Sg0YDtcctPEbaEhSCJrhUXcm6SkroLdiZx6xAtkpBcP5c1bw==',
    shasum: 'd5b7b7e96a4416117731dafc325c2b5c61fb5dc2',
    tarball: 'http://localhost:3000/api/packages/cerc-io/npm/%40cerc-io%2Flaconic-sdk/-/0.1.4/laconic-sdk-0.1.4.tgz'
  }

which we can use to make an edit like this:

diff --git a/yarn.lock b/yarn.lock
index 5066ad9..3b41ba5 100644
--- a/yarn.lock
+++ b/yarn.lock
@@ -4,8 +4,8 @@

 "@cerc-io/[email protected]":
   version "0.1.4"
-  resolved "https://git.vdb.to/api/packages/cerc-io/npm/%40cerc-io%2Flaconic-sdk/-/0.1.4/laconic-sdk-0.1.4.tgz#06159cb3c9c48325b53eba9bd642cf63420351d9"
-  integrity sha512-IsXUnz5S14zF+VPWydKy52PkCdyFqRJiMs9FCv8YEJVtUasXkU9xJLBmGaz7Nuo2MyNyKy2NHM7LoB1A75ZHAQ==
+  resolved "http://localhost:3000/api/packages/cerc-io/npm/%40cerc-io%2Flaconic-sdk/-/0.1.4/laconic-sdk-0.1.4.tgz#d5b7b7e96a4416117731dafc325c2b5c61fb5dc2"
+  integrity sha512-wYwbQGqQKvZ6Lk5Ft6F7TshgFwKF5d8wyCwOf1Sg0YDtcctPEbaEhSCJrhUXcm6SkroLdiZx6xAtkpBcP5c1bw==
   dependencies:
     "@cosmjs/amino" "^0.28.1"

Some testing suggests this works. Next step is to create some tooling to automate the process.

[dboreham]

Success. This works:

yarn_info_output=$(yarn info --json @cerc-io/[email protected] 2>/dev/null)
package_tarball=$(echo $yarn_info_output | jq -r .data.dist.tarball)
package_integrity=$(echo $yarn_info_output | jq -r .data.dist.integrity)
package_shasum=$(echo $yarn_info_output | jq -r .data.dist.shasum)
package_resolved=${package_tarball}#${package_shasum}
escaped_package_resolved=$(printf '%s\n' "$package_resolved" | sed -e 's/[\/&]/\\&/g')
if [ -n "$CERC_SCRIPT_VERBOSE" ]; then
    echo "Tarball: ${package_tarball}"
    echo "Integrity: ${package_integrity}"
    echo "Shasum: ${package_shasum}"
    echo "Resolved: ${package_resolved}"
fi
sed -i -e '/^\"@cerc-io\/laconic-sdk.*\":$/ , /^\".*$/ s/^\([[:space:]]\{1,\}resolved \).*$/\1'\"${escaped_package_resolved}\"'/' yarn.lock
sed -i -e '/^\"@cerc-io\/laconic-sdk.*\":$/ , /^\".*$/ s/^\([[:space:]]\{1,\}integrity \).*$/\1'${package_integrity}'/' yarn.lock

diff --git a/yarn.lock b/yarn.lock
index 5066ad9..3b41ba5 100644
--- a/yarn.lock
+++ b/yarn.lock
@@ -4,8 +4,8 @@

 "@cerc-io/[email protected]":
   version "0.1.4"
-  resolved "https://git.vdb.to/api/packages/cerc-io/npm/%40cerc-io%2Flaconic-sdk/-/0.1.4/laconic-sdk-0.1.4.tgz#06159cb3c9c48325b53eba9bd642cf63420351d9"
-  integrity sha512-IsXUnz5S14zF+VPWydKy52PkCdyFqRJiMs9FCv8YEJVtUasXkU9xJLBmGaz7Nuo2MyNyKy2NHM7LoB1A75ZHAQ==
+  resolved "http://localhost:3000/api/packages/cerc-io/npm/%40cerc-io%2Flaconic-sdk/-/0.1.4/laconic-sdk-0.1.4.tgz#d5b7b7e96a4416117731dafc325c2b5c61fb5dc2"
+  integrity sha512-wYwbQGqQKvZ6Lk5Ft6F7TshgFwKF5d8wyCwOf1Sg0YDtcctPEbaEhSCJrhUXcm6SkroLdiZx6xAtkpBcP5c1bw==
   dependencies:
     "@cosmjs/amino" "^0.28.1"
     "@cosmjs/crypto" "^0.28.1"