Code

Dev/manage meta#518

Code

Merged

BeachWang merged 180 commits intomainmodelscope/data-juicer:mainfrom dev/manage_metamodelscope/data-juicer:dev/manage_metaCopy head branch name to clipboardJan 3, 2025

+1,230-627

Commits

Commits on Oct 24, 2024

• add api call

  drcege

  committed
  63d430a

• add call_api ops

  drcege

  committed
  6720da4

Commits on Oct 29, 2024

• clean

  drcege

  committed
  8daa6e1

• minor update

  drcege

  committed
  ef11951

• more tests

  drcege

  committed
  5597d5c

• update tests

  drcege

  committed
  4b6e769

• Merge branch 'main' into dev/api_model

  drcege

  committed
  835be22

• update prompts

  drcege

  committed
  325a753

Commits on Oct 30, 2024

• fix unittest

  drcege

  committed
  4f04bdd

• update tests

  drcege

  committed
  0adbdcd

Commits on Nov 1, 2024

• add docs

  drcege

  committed
  0aa4069

• minor fix

  drcege

  committed
  f007532

Commits on Nov 5, 2024

• Merge branch 'main' into dev/api_model

  drcege

  committed
  9aa7390

• add API processor

  drcege

  committed
  ee4f461

• Merge branch 'main' into dev/api_model

  drcege

  committed
  9bbfe47

• refine API processor

  drcege

  committed
  b00b182

• refine

  drcege

  committed
  b718de7

Commits on Nov 6, 2024

• chunk and extract events

  BeachWang

  committed
  6d1d433

• fix bugs

  drcege

  committed
  4d1670f

• fix tests

  drcege

  committed
  9e11aa3

Commits on Nov 7, 2024

• extract attribute

  BeachWang

  committed
  cc40fc0

• Merge branch 'dev/api_model' of github.com:alibaba/data-juicer into dev/llm_info_extract

  BeachWang

  committed
  4c262ad

• refine tests

  drcege

  committed
  347bc0f

Commits on Nov 8, 2024

• extract nickname

  BeachWang

  committed
  c9d5051

• Merge branch 'dev/api_model' of github.com:alibaba/data-juicer into dev/llm_info_extract

  BeachWang

  committed
  8a128ca

• nickname test done

  BeachWang

  committed
  9262777

• merge main

  BeachWang

  committed
  58fc020

Commits on Nov 11, 2024

• lightRAG to OP

  BeachWang

  committed
  c7dc28e

• merge main

  BeachWang

  committed
  238869e

• doc done

  BeachWang

  committed
  0e51a43

• remove extra test

  BeachWang

  committed
  6d9d8a5

• relavant -> relevant

  BeachWang

  committed
  a637a64

• fix minor error

  BeachWang

  committed
  56e7988

Commits on Nov 12, 2024

• group by op done

  BeachWang

  committed
  03880b7

• ValueError -> Exception

  BeachWang

  committed
  23174fd

• merge main

  BeachWang

  committed
  e82cc06

• fix config_all error

  BeachWang

  committed
  20a8dee

Commits on Nov 13, 2024

• fix prepare_api_model

  BeachWang

  committed
  38a9511

• fix rank sample None

  BeachWang

  committed
  35f0eb3

• constant fix key

  BeachWang

  committed
  155d3dd

Commits on Nov 14, 2024

• aggregator op

  BeachWang

  committed
  f862897

• merge llm_info_extract

  BeachWang

  committed
  2d4da5e

Commits on Nov 20, 2024

• init python_lambda_mapper

  drcege

  committed
  7e66057

• set default arg

  drcege

  committed
  a61859b

Commits on Nov 21, 2024

• fix init

  drcege

  committed
  8031a31

• add python_file_mapper

  drcege

  committed
  67711f9

Commits on Nov 22, 2024

• support text & most relavant entities

  BeachWang

  committed
  cdeb692

Commits on Nov 25, 2024

• coverage ignore_errors

  drcege

  committed
  125a8f3

• index sample

  BeachWang

  committed
  0c68089

• role_playing_system_prompt_yaml

  BeachWang

  committed
  651789d

Commits on Nov 26, 2024

• merge python_file_mapper

  BeachWang

  committed
  c5d7b9e

• Merge branch 'main' of github.com:alibaba/data-juicer into dev/group_by_and_sum

  BeachWang

  committed
  cf6a53a

Commits on Nov 27, 2024

• system_prompt begin

  BeachWang

  committed
  222790e

• support batched

  drcege

  committed
  75f2911

• remove unforkable

  BeachWang

  committed
  11fa852

• support batched & add docs

  drcege

  committed
  4af2bfb

Commits on Nov 28, 2024

• Merge branch 'main' into op/python_lambda

  drcege

  committed
  8867580

• add docs

  drcege

  committed
  553d5ad

• fix docs

  drcege

  committed
  470ca19

• update docs

  drcege

  committed
  399a238

• Merge branch 'main' into op/python_file

  drcege

  committed
  706365f

• pre-commit done

  BeachWang

  committed
  115ab9a

Commits on Dec 2, 2024

• fix batch bug

  BeachWang

  committed
  ecb8635

• fix batch bug

  BeachWang

  committed
  03e3469

Commits on Dec 3, 2024

• merge fix_batch_bug

  BeachWang

  committed
  1788fa6

• Merge branch 'main' of github.com:alibaba/data-juicer into debug/fix_batch_bug

  BeachWang

  committed
  735ff4d

• fix filter batch

  BeachWang

  committed
  00ff624

• fix filter batch

  BeachWang

  committed
  8601519

• system prompt recipe done

  BeachWang

  committed
  eeefcab

• Merge branch 'main' of github.com:alibaba/data-juicer into dev/group_by_and_sum

  BeachWang

  committed
  6eaa50c

Commits on Dec 5, 2024

• not rank for filter

  BeachWang

  committed
  1575717

• limit pyav version

  BeachWang

  committed
  2c5c4a1

• Merge branch 'debug/fix_batch_bug' of github.com:alibaba/data-juicer into dev/group_by_and_sum

  BeachWang

  committed
  5c96dd5

• add test for op

  BeachWang

  committed
  49be467

• tmp

  BeachWang

  committed
  9ab02fe

• tmp

  BeachWang

  committed
  ba086de

• doc done

  BeachWang

  committed
  f712131

• Merge branch 'op/python_lambda' of github.com:alibaba/data-juicer into op/python_lambda

  BeachWang

  committed
  12b7616

• merge python_lambda

  BeachWang

  committed
  e57b64a

• merge python_lambda

  BeachWang

  committed
  5f463cd

Commits on Dec 6, 2024

• skip api test

  BeachWang

  committed
  a786070

• merge main

  BeachWang

  committed
  73f4e77

• merge main

  BeachWang

  committed
  4b6f0b9

• add env dependency

  BeachWang

  committed
  788a212

Commits on Dec 10, 2024

• install by recipe

  BeachWang

  committed
  10242c4

Commits on Dec 12, 2024

• dialog sent intensity

  BeachWang

  committed
  6a43eec

• add query

  BeachWang

  committed
  621a693

• change to dj_install

  BeachWang

  committed
  b46d105

• change to dj_install

  BeachWang

  committed
  a0da444

• developer doc done

  BeachWang

  committed
  02f8dda

• merge dj_install

  BeachWang

  committed
  635a8a9

• + add auto mode for analyzer: load all filters that produce stats to analyze the target dataset

  HYLcool

  committed
  083b665

Commits on Dec 13, 2024

• + add default mem_required for those model-based OPs

  HYLcool

  committed
  662df5e

• query sent_int mapper

  BeachWang

  committed
  3b04908

• query sentiment test done

  BeachWang

  committed
  6b4d525

• - support wordcloud drawing for str or str list fields in stats

  Show description for 926c3da
  HYLcool

  committed
  926c3da

• - take the minimum one of dataset length and auto num

  HYLcool

  committed
  27347c0

• * update default export path

  HYLcool

  committed
  d19f92f

• * set version limit for wandb to avoid exception

  HYLcool

  committed
  fbd6726

• change meta pass

  BeachWang

  committed
  58288f7

• + add docs for auto mode

  HYLcool

  committed
  9f9f85b

• doc done

  BeachWang

  committed
  b665c10

• merge main

  BeachWang

  committed
  07be552

Commits on Dec 16, 2024

• sentiment detection

  BeachWang

  committed
  8ba4156

• diff label

  BeachWang

  committed
  48b1761

• sentiment

  BeachWang

  committed
  8160725

• test done

  BeachWang

  committed
  01846d1

• + support t-test for Measure

  HYLcool

  committed
  566eb5b

• * fix some bugs

  HYLcool

  committed
  7b8ee5c

Commits on Dec 17, 2024

• dialog intent label

  BeachWang

  committed
  a76d975

• fix typo

  BeachWang

  committed
  2fb9fe4

• prompt adjust

  BeachWang

  committed
  324467f

• add more test

  BeachWang

  committed
  4a3ad39

• query intent detection

  BeachWang

  committed
  937b3f1

• for test

  BeachWang

  committed
  d4ca87b

• for test

  BeachWang

  committed
  8109c71

• change model

  BeachWang

  committed
  c749dcd

• fix typo

  BeachWang

  committed
  c7df0bc

• fix typo

  BeachWang

  committed
  c7662cb

• for test

  BeachWang

  committed
  6f44ec0

• for test

  BeachWang

  committed
  9b6652d

• doc done

  BeachWang

  committed
  fa306dc

• - support analyze a dataset object

  Show description for 601d9a2
  HYLcool

  committed
  601d9a2

• - support analysis on tags in meta

  HYLcool

  committed
  34f2ab6

• - support analysis with tagging OPs

  HYLcool

  committed
  8531a01

Commits on Dec 18, 2024

• - move tags into the meta field

  HYLcool

  committed
  4d6b701

• dialog topic detection

  BeachWang

  committed
  767b2f0

• dialog topic detection

  BeachWang

  committed
  c088cb1

• dialog topic detection

  BeachWang

  committed
  12351db

• dialog topic detection

  BeachWang

  committed
  4b4e946

• dialog topic detection

  BeachWang

  committed
  4506a8e

• dialog topic detection

  BeachWang

  committed
  d21db85

• query topic detection

  BeachWang

  committed
  6f394ee

• query topic detection

  BeachWang

  committed
  abee815

• query topic detection

  BeachWang

  committed
  0494741

• query topic detection

  BeachWang

  committed
  38523a1

• query topic detection

  BeachWang

  committed
  b03a33a

• - do not tell tags using their suffix

  Show description for 35aa6bd
  HYLcool

  committed
  35aa6bd

• doc done

  BeachWang

  committed
  ad226b1

• - add insight mining

  HYLcool

  committed
  85e1392

Commits on Dec 19, 2024

• meta tags aggregator

  BeachWang

  committed
  b02745b

• meta tags aggregator

  BeachWang

  committed
  f2654f1

• meta tags aggregator

  BeachWang

  committed
  23e5d6f

• meta tags aggregator

  BeachWang

  committed
  1c74709

• meta tags aggregator

  BeachWang

  committed
  a997726

• meta tags aggregator

  BeachWang

  committed
  2642847

• meta tags aggregator

  BeachWang

  committed
  2dae3b8

• meta tags aggregator

  BeachWang

  committed
  8bb2509

• meta tags aggregator

  BeachWang

  committed
  90303ee

• meta tags aggregator

  BeachWang

  committed
  e4c6ff1

• meta tags aggregator

  BeachWang

  committed
  12f8946

• meta tags aggregator

  BeachWang

  committed
  09b1599

• naive reverse grouper

  BeachWang

  committed
  203bc64

• naive reverse grouper

  BeachWang

  committed
  cf01e7e

• * resolve the bugs when running insight mining in multiprocessing mode

  HYLcool

  committed
  e3d7b8b

• Merge branch 'main' into feat/insight_mining

  Show description for 3ca9994
  HYLcool

  committed
  3ca9994

Commits on Dec 20, 2024

• * update unittests

  HYLcool

  committed
  16ca358

• * update unittests

  HYLcool

  committed
  dfb0bca

• * update unittests

  HYLcool

  committed
  f8b9539

• tags specified field

  BeachWang

  committed
  0ba6459

• * update readme for analyzer

  HYLcool

  committed
  45259e5

• Merge branch 'main' into feat/insight_mining

  Show description for 174ee05
  HYLcool

  committed
  174ee05

• merge main

  BeachWang

  committed
  4ad8b8d

• doc done

  BeachWang

  committed
  9f098bd

• * use more detailed key

  HYLcool

  committed
  51f53dc

• + add reference

  HYLcool

  committed
  58001ca

• Merge branch 'feat/insight_mining' of github.com:alibaba/data-juicer into dev/manage_meta

  BeachWang

  committed
  892cb48

• move mm tags

  BeachWang

  committed
  19fd15b

Commits on Dec 24, 2024

• move meta key

  BeachWang

  committed
  8fec0f7

Commits on Dec 30, 2024

• done

  BeachWang

  committed
  6fdc95b

• merge main

  BeachWang

  committed
  8e01f7e

Commits on Dec 31, 2024

• test done

  BeachWang

  committed
  af9e14d

• rm nested set

  BeachWang

  committed
  f57f454

Commits on Jan 3, 2025

• Update constant.py

  Show description for 7a7e3df
  yxdyc

  authored
  7a7e3df

• rename agg to batch meta

  BeachWang

  committed
  250ecbd

• Merge branch 'dev/manage_meta' of github.com:alibaba/data-juicer into dev/manage_meta

  BeachWang

  committed
  57fab1c

• export in naive reverse grouper

  BeachWang

  committed
  fcfced2

• fix bug

  BeachWang

  committed
  0a02163

• fix bug

  BeachWang

  committed
  855c4a7

• fix bug

  BeachWang

  committed
  fb8b11b