Add filter_map, dedup and some variants to the stdlib

[yannham]

Closes #1958. Adds filter_map, dedup, as well as two variants of dedup that avoid the quadratic behavior of the purely equality-based vanilla version, sort_dedup and hash_dedup.

[github-actions]

Bencher Report

Branch	stdlib/dedup-and-filter-map
Testbed	ubuntu-latest

Click to view all benchmark results

Benchmark	Latency	nanoseconds (ns)
fibonacci 10	📈 view plot 🚷 view threshold	492,600.00
foldl arrays 50	📈 view plot 🚷 view threshold	1,823,700.00
foldl arrays 500	📈 view plot 🚷 view threshold	7,162,400.00
foldr strings 50	📈 view plot 🚷 view threshold	7,330,100.00
foldr strings 500	📈 view plot 🚷 view threshold	64,605,000.00
generate normal 250	📈 view plot 🚷 view threshold	49,633,000.00
generate normal 50	📈 view plot 🚷 view threshold	2,367,600.00
generate normal unchecked 1000	📈 view plot 🚷 view threshold	3,643,400.00
generate normal unchecked 200	📈 view plot 🚷 view threshold	799,120.00
pidigits 100	📈 view plot 🚷 view threshold	3,234,800.00
pipe normal 20	📈 view plot 🚷 view threshold	1,523,500.00
pipe normal 200	📈 view plot 🚷 view threshold	10,979,000.00
product 30	📈 view plot 🚷 view threshold	860,070.00
scalar 10	📈 view plot 🚷 view threshold	1,550,800.00
sum 30	📈 view plot 🚷 view threshold	859,300.00

🐰 View full continuous benchmarking report in Bencher

[yannham]

It seems we now blow the stack on Windows (1MB), and it's can't be really tweaked from tests - I tried to wrap all main tests as a scoped thread with larger stack but it didn't work so I suspect this is the separate LSP command. This is annoying and also probably means the standalone LSP binary is also overflowing the stack on Windows.

[jneem]

Sorry, I just realized this has been hanging for weeks.

I think filter_map and dedup are totally fine. sort_dedup is fine too, but I proposed an alternative just for fun.

I'm not totally sold on hash_dedup, though...

[jneem]

I find the mention of probability misleading, especially when talking about a "hash" function: in a hash table, having a low collision probability only matters for performance, but here you absolutely need two distinct elements to have different "hash"es or else you'll get the wrong answer.

I wonder if it's worth baking in a notion of hashes and sets directly into nickel? It seems like almost as "core" a notion as equality, and we've had a feature request for sets for a while now...

[yannham]

You're right, it's a logic error to not separate unequal keys. In some sense the hash function is the notion of equality at hand. Maybe I shouldn't call this a hash function at all?

[yannham]

Though I don't have a better name in mind that would be discoverable as a function name (mathematically what we want is an injection T -> String), but dedup_inject doesn't sound so good.

For the motivation, this function is taken from a real use-case: this is how we check for duplicates uid in our build machine profiles. Also, it's the only way to deduplicate an array efficiently (in this context meaning O(nlog(n)) or less) without re-ordering it

[jneem]

Just brainstorming here, but what if we had dedup_sorted : forall a. -> Array a -> Array a instead (which would do what the inner go function does, basically)? Then you'd use it like xs |> sort cmp |> dedup_sorted, but it feels a bit more composable than sort_dedup because if you know that your array is already sorted then you don't need to sort it again.

[yannham]

Sure, we can factor out dedup_sorted. I would still keep sort_dedup as a handy alias, though.

[yannham]

After discussing with @jneem through other channels, we agreed that hash_dedup has a strange interface and will be split off this pull request. It should make a come back but without having to provide an explicit hash function; either using a built-in that we'll add to Nickel, or as a middle ground waiting for that, use std.serialize automatically.