Migrate to Brewsci

[sjackman]

Let's migrate this tap to Brewsci!

It will use OpenBLAS on Linux.
It will support only OpenBLAS on macOS until Accelerate is fixed. (says @dpo) Note that Homebrew/core uses Accelerate, and we may have to address compatibility issues.
No formula should use the same name as a formula in Homebrew/core, which would create havoc.

Will it build bottles for both Linux and macOS? I can help you get set up with bottling on CircleCI.
Would you consider renaming the tap to brewsci/math, brewsci/num, brewsci/numerics, or similar name?

We can create a new, empty repo in Brewsci, and migrate each formula by opening a PR, ensuring CI passes, and building bottles for each formula as each PR is merged. Alternatively, we can move the repo using GitHub's Transfer ownership feature. I prefer the first. It involves some work, but ensures CI passes, and builds bottles. Are you up for that?

[sjackman]

Ping @schoeps

[dpo]

@sjackman Thanks for opening this issue. Things are a bit busy this week but I'll get to it. My understanding is that Accelerate will need a couple more releases to be fixed, so the current investment in OpenBLAS is not wasted.

Naming formulae is an issue. I don't know how to work around it. Currently, I do have some name clashes with core but I don't see it as a problem since a user wouldn't normally install core/ceres-solver (say) and dpo/openblas/ceres-solver at the same time, especially after the Accelerate vs. OpenBLAS issue is resolved. When all formulae depend on Accelerate, the only difference with core formulae will reside in the build options.

[sjackman]

My concern is that it's not possible to install both core/ceres-solver and dpo/openblas/ceres-solver and a package that depends on core/ceres-solver, like rawtoaces, and particularly its bottle may not work with dpo/openblas/ceres-solver. I'd suggest adding a suffix to dpo/openblas/ceres-solver such as ceres-solver-openblas or ceres-solver-brewsci. ceres-solver-openblas could either be keg-only, or it could conflict with ceres-solver.

[sjackman]

@dpo wrote…

I'm all for it but I'm a bit short of time again for the next two weeks or so.

No rush. It can wait a few weeks.

I'm also undecided regarding the names of formulae to avoid conflicts with core. Isn't brew tap-pin a way around the name clash problem?

Not really. It doesn't solve that problem that two formulae with the same name can't be installed at the same time. If a formula named ceres-solver were installed that wasn't Homebrew/core/ceres-solver, it may be not be possible to install Homebrew/core/rawtoaces, which depends on Homebrew/core/ceres-solver.

[dpo]

What do people think about FlexiBLAS (https://gitlab.mpi-magdeburg.mpg.de/software/flexiblas-release)? If formulae were built against it, users would have the option of switching the BLAS library at run time and we wouldn't have to worry about propagating the choice of BLAS library to all dependencies of a formula.

[sjackman]

Sounds neat. For Linuxbrew though I'd be more comfortable picking and using a single BLAS implementation that'd be good enough for 90% of users and avoid the additional complexity of FlexiBLAS.

[dpo]

The Flexiblas formula itself could install OpenBLAS (say) and use that as default BLAS. The advantage is that if someone has access to MKL and wants to try it out, there's no need to reinstall any package.

[sjackman]

It'd be your tap, so ultimately your call. I'd err on the side of simplicity.

[schoeps]

@dpo I get older and milder... I could survive with the naming convention "xxx-openblas.rb" if this unlocks a bigger user base via brewsci.

[dpo]

If FlexiBLAS pans out, it would resolve the issue with dependencies not being linked against the same BLAS and would allow users to use the MKL without any change to the brew formulae.

OR: we simply link against Accelerate, since it's the default on macOS and forget OpenBLAS. Few formulae need to wait for the forthcoming updates to Accelerate (notably julia).

It doesn't resolve the naming issue. Either way, xxx-openblas.rb is no longer the appropriate name.

[sjackman]

The formula name could be foo-brewsci, when foo is also found in Homebrew/core.

[dpo]

Alternatively, could a formula be keg only when it's also found in core? That would allow us to use the same name.

[sjackman]

Yes, it could be keg only, but that I don't believe that will enable you to use the same name. If a formula brewsci/openblas/foo were installed, it's not then possible to install homebrew/core/foo, which would break any formulae that depend on homebrew/core/foo.

[dpo]

Ok. That's a bit frustrating... Thanks.

[schoeps]

So we need to do both: use a different name and make it keg only. As far as I understand, our current solution may break user installations (e.g. we shadow suitesparse). This is bad. I think we should rename stuff and then I would not mind moving to brewsci. The question is whether we (openblas) support Linux at all since @sjackman already takes care in his repo....

Core should not have included the science formula :(

[dpo]

The science formulae in this repo provide more options, so I guess we should support linux. But you're right that there's definitely some redundancy.

[dpo]

@sjackman I'll (finally) have a little time in the coming week to migrate these formulae to Brewsci. Let's rename them brewsci-xxx.rb and name the tap brewsci/numerics as you suggested. We'll use OpenBLAS by default on macOS until they fix Accelerate. Then we'll use Accelerate.

How should we proceed?

[sjackman]

How about brewsci/num? I had originally called it brewsci/bioinformatics and found that I got really tired of typing bioinformatics. Or brewsci/numeric. The s in numerics seems unnecessary.

I can create the GitHub and Bintray repo and set up CI. Then you can submit a straight-forward formula to test that CI and bottling is working. Then you can start submitting and merging PRs.

Note that CircleCI has to pass to be able to merge a PR with bottles. It's not currently possible to ignore/override audit failures. The bottles are not uploaded when CircleCI fails.

Are you planning to support macOS and Linux or only macOS?

[dpo]

Sure, brewsci/num works. I'm planning to support both macOS and Linux. Do we agree that the formulae should be keg only? If a formula is both in core and here, they will install libraries, headers, etc., with the same names.

[sjackman]

Yes, if a formula is also in core, I agree that it should be keg only.

[sjackman]

I've set up the skeleton of the tap.
https://github.com/brewsci/homebrew-num
https://bintray.com/linuxbrew/bottles-num
https://circleci.com/gh/brewsci/workflows/homebrew-num

There's almost certainly more work to do before it's 💯 ready to go, but you can get started and open a pull request!

[sjackman]

Put formulae in the Formula subdirectory.

[go-ski]

Let me know if there was further FlexiBLAS development. I see FlexiBLAS now standard in Fedora and expanding on clusters for data science toolchains. Having this on macOS would simplify user transition from laptop to cluster in addition to having interchangeable BLAS libs and dynamic core allocation control in R, numpy, and Julia. I don't know enough to help here but do see this as a unifying future development.

[dpo]

Julia now uses https://github.com/JuliaLinearAlgebra/libblastrampoline. I haven't researched FlexiBLAS any further.