Code

Support new OS models: Zephyr and Yi#392

Code

Draft

rmitsch wants to merge 58 commits intodevelopexplosion/spacy-llm:developfrom feat/new-os-modelsexplosion/spacy-llm:feat/new-os-modelsCopy head branch name to clipboard

+392-7

Commits

Commits on Oct 17, 2023

• Add context length info. Refactor BuiltinTask and models to facilitate this.

  rmitsch

  committed
  3aba660

• Merge branch 'develop' into feat/inf-doc-len

  Show description for 5699773
  rmitsch

  committed
  5699773

• Add token count estimator plumbing.

  rmitsch

  committed
  4213372

• Add plumbing for mapper and reducer.

  rmitsch

  committed
  f440ca4

Commits on Oct 18, 2023

• Add ShardMapper prototype.

  rmitsch

  committed
  e47f762

Commits on Oct 19, 2023

• Integrating mapping into prompt generation workflow.

  rmitsch

  committed
  89a5510

Commits on Oct 20, 2023

• Update response parsing and component to support sharding (WIP).

  rmitsch

  committed
  086dec9

Commits on Oct 27, 2023

• Fix shard & prompt flow.

  rmitsch

  committed
  23718fc

• Fix shard & prompt flow.

  rmitsch

  committed
  7ce670d

• Remove todo comments.

  rmitsch

  committed
  0d75ea8

• Fix Anthropic, Cohere, NoOp model tests.

  rmitsch

  committed
  9da7098

Commits on Oct 30, 2023

• Merge branch 'develop' into feat/inf-doc-len

  rmitsch

  authored
  0cb9afd

Commits on Oct 31, 2023

• Fix test_llm_pipe().

  rmitsch

  committed
  f368412

Commits on Nov 3, 2023

• Fix type checking test.

  rmitsch

  committed
  b1f111d

• Fix span parsing tests.

  rmitsch

  committed
  44a2787

• Fix internal tests.

  rmitsch

  committed
  6d8cdc7

• Fix _CountTask.

  rmitsch

  committed
  e712f41

• Fix sentiment and summarization tasks and tests.

  rmitsch

  committed
  985fd68

• Fix Azure connection URL. Fix Model test pings.

  rmitsch

  committed
  98842a2

• Fix Lemma parsing.

  rmitsch

  committed
  b54a3d9

• Start work on doc-to-shard property copying.

  rmitsch

  committed
  9bf365d

Commits on Nov 6, 2023

• Fix REL doc preprocessing.

  rmitsch

  committed
  dddfaab

• Remove comment on doc attribute handling during sharding, as this is done by spaCy's slicing directly.

  rmitsch

  committed
  3af21b5

Commits on Nov 8, 2023

• Add reducer implementations.

  rmitsch

  committed
  fee9ca7

Commits on Nov 14, 2023

• Implement outstanding task reducers.

  rmitsch

  committed
  e508499

• Resolve merge conflicts.

  rmitsch

  committed
  3218541

Commits on Nov 20, 2023

• Add shardable/non-shardable LLM task typing distinction. Add support for handling both types of tasks. Update tests.

  rmitsch

  committed
  c104387

Commits on Nov 21, 2023

• Merge branch 'develop' into feat/inf-doc-len

  Show description for 2c6d899
  rmitsch

  committed
  2c6d899

Commits on Nov 23, 2023

• Fix EL task.

  rmitsch

  committed
  2502c4d

• Fix EL tokenization and highlighting partially.

  rmitsch

  committed
  03055c5

Commits on Nov 24, 2023

• Fix tokenization and whitespaces for EL task.

  rmitsch

  committed
  4e4a2cd

• Fix merge conflicts.

  rmitsch

  committed
  865acec

• Add new registry handlers (with context length and arbitrary model names) for all REST models.

  rmitsch

  committed
  694d5da

• Add sharding test with simple count task.

  rmitsch

  committed
  5295400

• Fix sharding algorithm.

  rmitsch

  committed
  70e3643

Commits on Nov 27, 2023

• Add test with simple count task.

  rmitsch

  committed
  4321483

• Add context length as init arg in HF models.

  rmitsch

  committed
  ef6e738

Commits on Nov 28, 2023

• Fix tests. Don't stringify IO lists if sharded.

  rmitsch

  committed
  e3ff37d

Commits on Nov 29, 2023

• Fix tests.

  rmitsch

  committed
  056730a

• Add NER sharding test.

  rmitsch

  committed
  196c235

• Add REL and sentiment sharding tests.

  rmitsch

  committed
  1f51a4a

• Add summary sharding tests.

  rmitsch

  committed
  e18b302

• Add EL sharding task. Fix bug in shard mapper.

  rmitsch

  committed
  7c092ca

• Fix REL error with RELExample parsing.

  rmitsch

  committed
  358ba72

• Use regex for punctuation in REL conversion.

  rmitsch

  committed
  0c96fb6

Commits on Dec 1, 2023

• Maintain custom doc attributes, incl. test.

  rmitsch

  committed
  dc926bd

• Filter merge warnings in textcat reduction.

  rmitsch

  committed
  5585174

• Add Zephyr and Yi classes.

  rmitsch

  committed
  6d3a4c8

• Fix Yi model.

  rmitsch

  committed
  57acfe4

• Fix Yi model.

  rmitsch

  committed
  2f1a905

Commits on Dec 4, 2023

• Fix Yi and Zephyr processing.

  rmitsch

  committed
  9821063

• Remove deprecated comment.

  rmitsch

  committed
  98e3e6c

Commits on Dec 11, 2023

• Fix merge conflicts.

  rmitsch

  committed
  513c2fb

• Merge branch 'develop' into feat/new-os-models

  rmitsch

  committed
  482af35

• Change model used for Yi tests.

  rmitsch

  committed
  3747a2f

Commits on Dec 12, 2023

• Incorporate feedback.

  rmitsch

  committed
  b2dff8f

Commits on Dec 13, 2023

• Skip Yi test failing in CI, but suceeding locally.

  rmitsch

  committed
  dfe89ee

• Extend readme with links for Zephyr and Yi.

  rmitsch

  committed
  69c3c76