OSPP: Implementation of Domain-Specific Large Model Benchmarking Based on KubeEdge-Ianvs #144

IcyFeather233 · 2024-09-09T16:59:56Z

This PR is the implementation of #95

hsj576

The "print" in the code should be replaced by "LOGGER.info" (from core.common.log import LOGGER ").

IcyFeather233 · 2024-09-11T14:23:44Z

The "print" in the code should be replaced by "LOGGER.info" (from core.common.log import LOGGER ").

@hsj576 Thank you for your advice! Now they are all replaced.

MooreZheng

Rebases are needed.

There is a file with conflicts like docs/proposals/scenarios/llm-benchmarks/llm-benchmarks.md that needs to take care of. e.g., using rebase https://liwt31.github.io/2019/12/25/merge_rebase/.
all commits need sign-off, which can also be solved by using rebase

Commit sha: 9b7f9c5, Author: IcyFeather, Committer: IcyFeather; The sign-off is missing.
Commit sha: ce9f470, Author: IcyFeather, Committer: IcyFeather; The sign-off is missing.

To add your Signed-off-by line to every commit in this branch:

Ensure you have a local copy of your branch by checking out the pull request locally via command line.
In your local branch, run: git rebase HEAD~11 --signoff
Force push your changes to overwrite the branch: git push --force-with-lease origin dev

MooreZheng

Rebases are needed.

There is a file with conflicts like docs/proposals/scenarios/llm-benchmarks/llm-benchmarks.md that needs to take care of. e.g., using rebase https://liwt31.github.io/2019/12/25/merge_rebase/.

all commits need sign-off, which can also be solved by using rebase

Commit sha: 9b7f9c5, Author: IcyFeather, Committer: IcyFeather; The sign-off is missing.
Commit sha: ce9f470, Author: IcyFeather, Committer: IcyFeather; The sign-off is missing.

To add your Signed-off-by line to every commit in this branch:
Ensure you have a local copy of your branch by checking out the pull request locally via command line.
In your local branch, run: git rebase HEAD~11 --signoff
Force push your changes to overwrite the branch: git push --force-with-lease origin dev

Reminder: There are three CI issues that remain to be resolved, mainly for Pylint, see https://github.com/kubeedge/ianvs/actions/runs/11268242806

IcyFeather233 · 2024-10-14T10:48:16Z

@MooreZheng The similar pylint issues are described in #152

To solve this, sedna and ianvs need to corporate. We discussed this problem in the last week zoom meeting, and it seems that it should be discussed more to find a final solution.

MooreZheng · 2024-10-22T16:12:09Z

/lgtm

MooreZheng · 2024-10-22T16:12:35Z

@hsj576 might also want to review this project

MooreZheng

Looks fine, but we need to refine the Chinese words as in English.

examples/government/singletask_learning_bench/objective/testenv/acc.py

examples/government/singletask_learning_bench/subjective/testenv/llm_judgement.py

examples/llm_simple_qa/README.md

examples/llm_simple_qa/testenv/acc.py

IcyFeather233 · 2024-10-27T13:27:41Z

@MooreZheng I have changed Chinese words to English

MooreZheng · 2024-10-28T02:58:23Z

Overall it looks fine. Any comment from @hsj576 before merging?

MooreZheng

Good. There is only one Chinese statement right now and we are very close to the final version.

Currently, this pull request contains 25 commits, which can make maintenance difficult, considering the number of contributors, pull requests, and their commits in KubeEdge Ianvs. In the final stage, @IcyFeather233 can squash the commits into one using rebase techniques.

MooreZheng · 2024-10-28T02:50:34Z

examples/government/singletask_learning_bench/subjective/testenv/llm_judgement.py

+
+def extract_comprehensive_score(input_str):
+    # extract overall points
+    match = re.search(r"'综合得分': (\d+)", input_str)


In the re.search of llm_judgement.py, we need to replace Chinese characters with English.

Signed-off-by: IcyFeather <[email protected]> add opencompass and llm singletask learning bench Signed-off-by: IcyFeather <[email protected]> update llm single task learning bench readme Signed-off-by: IcyFeather <[email protected]> add government benchmark Signed-off-by: IcyFeather <[email protected]> update government benchmark Signed-off-by: IcyFeather <[email protected]> update llm government benchmark implementation Signed-off-by: IcyFeather <[email protected]> update llm government benchmark implementation Signed-off-by: IcyFeather <[email protected]> update government README Signed-off-by: IcyFeather <[email protected]> update llm benchmark format Signed-off-by: IcyFeather <[email protected]> update government benchmark Signed-off-by: IcyFeather <[email protected]> update government benchmark dataset Signed-off-by: IcyFeather <[email protected]> add llm-benchmarks proposal Signed-off-by: IcyFeather <[email protected]> update llm benchmark proposal Signed-off-by: IcyFeather <[email protected]> update llm benchmark proposal Signed-off-by: IcyFeather <[email protected]> update llm benchmark proposal Signed-off-by: IcyFeather <[email protected]> translate llm-benchmark proposal Signed-off-by: IcyFeather <[email protected]> update proposal, add opencompass tutorial Signed-off-by: IcyFeather <[email protected]> update government benchmark sedna package Signed-off-by: IcyFeather <[email protected]> update government benchmark Signed-off-by: IcyFeather <[email protected]> update llm benchmark format Signed-off-by: IcyFeather <[email protected]> update llm benchmark format Signed-off-by: IcyFeather <[email protected]> fix pylint check problem Signed-off-by: IcyFeather <[email protected]> fix pylint check problem Signed-off-by: IcyFeather <[email protected]> fix pylint check problem Signed-off-by: IcyFeather <[email protected]> trans Chinese comments to English Signed-off-by: IcyFeather <[email protected]> add government llm benchmark Signed-off-by: IcyFeather <[email protected]> update government llm benchmark Signed-off-by: IcyFeather <[email protected]>

IcyFeather233 · 2024-10-28T04:42:13Z

@MooreZheng All commits merged into one, and all Chinese words changed to English

MooreZheng · 2024-10-28T07:03:15Z

/lgtm

MooreZheng

This branch has conflicts that must be resolved
Use the web editor or the to resolve conflicts.
Conflicting files
core/testenvmanager/dataset/dataset.py

IcyFeather233 · 2024-10-30T05:23:24Z

@MooreZheng conflicts are resolved

MooreZheng

Conflicts still exists

This branch has conflicts that must be resolved
Use the web editor or the to resolve conflicts.
Conflicting files
core/testenvmanager/dataset/dataset.py

MooreZheng · 2024-10-30T14:02:33Z

Now conflicts have been solved after merging #162

MooreZheng · 2024-10-30T14:02:39Z

/lgtm

hsj576 · 2024-10-30T14:10:21Z

/lgtm

MooreZheng · 2024-10-30T14:11:39Z

/approve

MooreZheng

All concerns are fixed. Well done!

kubeedge-bot · 2024-10-30T14:12:20Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: MooreZheng

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [MooreZheng]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kubeedge-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 9, 2024

kubeedge-bot requested review from jaypume and MooreZheng September 9, 2024 17:00

kubeedge-bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Sep 9, 2024

IcyFeather233 force-pushed the dev branch from d3b604a to c6b4094 Compare September 9, 2024 17:05

MooreZheng requested a review from hsj576 September 10, 2024 08:12

hsj576 suggested changes Sep 11, 2024

View reviewed changes

kubeedge-bot assigned hsj576 Sep 11, 2024

MooreZheng requested changes Sep 20, 2024

View reviewed changes

kubeedge-bot assigned MooreZheng Sep 20, 2024

IcyFeather233 force-pushed the dev branch from ce9f470 to de1c7d0 Compare September 23, 2024 02:06

kubeedge-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 23, 2024

This was referenced Sep 26, 2024

OSPP: Cloud-edge collaborative inference for LLM based on KubeEdge-Ianvs #149

Merged

Pylint Errors Due to Compatibility Issues with Updated Sedna in CI Environment #152

Open

MooreZheng requested changes Oct 14, 2024

View reviewed changes

IcyFeather233 force-pushed the dev branch from 463998a to 59fba19 Compare October 20, 2024 06:54

kubeedge-bot added the lgtm Indicates that a PR is ready to be merged. label Oct 22, 2024

MooreZheng requested changes Oct 24, 2024

View reviewed changes

kubeedge-bot removed the lgtm Indicates that a PR is ready to be merged. label Oct 24, 2024

MooreZheng added the kind/feature Categorizes issue or PR as related to a new feature. label Oct 26, 2024

MooreZheng requested changes Oct 28, 2024

View reviewed changes

MooreZheng assigned IcyFeather233 Oct 28, 2024

IcyFeather233 force-pushed the dev branch from c60e187 to 0abcb9a Compare October 28, 2024 04:32

IcyFeather233 force-pushed the dev branch from 0abcb9a to 179d39d Compare October 28, 2024 04:37

IcyFeather233 requested a review from MooreZheng October 28, 2024 04:42

kubeedge-bot added the lgtm Indicates that a PR is ready to be merged. label Oct 28, 2024

MooreZheng requested changes Oct 29, 2024

View reviewed changes

kubeedge-bot removed the lgtm Indicates that a PR is ready to be merged. label Oct 29, 2024

Merge branch 'main' into dev

fa4d8c6

MooreZheng requested changes Oct 30, 2024

View reviewed changes

kubeedge-bot added the lgtm Indicates that a PR is ready to be merged. label Oct 30, 2024

kubeedge-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 30, 2024

kubeedge-bot merged commit 432d321 into kubeedge:main Oct 30, 2024
13 checks passed

MooreZheng approved these changes Oct 30, 2024

View reviewed changes

IcyFeather233 deleted the dev branch October 30, 2024 15:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OSPP: Implementation of Domain-Specific Large Model Benchmarking Based on KubeEdge-Ianvs #144

OSPP: Implementation of Domain-Specific Large Model Benchmarking Based on KubeEdge-Ianvs #144

IcyFeather233 commented Sep 9, 2024

hsj576 left a comment

IcyFeather233 commented Sep 11, 2024

MooreZheng left a comment

MooreZheng left a comment

IcyFeather233 commented Oct 14, 2024

MooreZheng commented Oct 22, 2024

MooreZheng commented Oct 22, 2024

MooreZheng left a comment •

edited

Loading

IcyFeather233 commented Oct 27, 2024

MooreZheng commented Oct 28, 2024

MooreZheng left a comment •

edited

Loading

MooreZheng Oct 28, 2024

IcyFeather233 commented Oct 28, 2024

MooreZheng commented Oct 28, 2024

MooreZheng left a comment

IcyFeather233 commented Oct 30, 2024

MooreZheng left a comment

MooreZheng commented Oct 30, 2024 •

edited

Loading

MooreZheng commented Oct 30, 2024

hsj576 commented Oct 30, 2024

MooreZheng commented Oct 30, 2024

MooreZheng left a comment

kubeedge-bot commented Oct 30, 2024

OSPP: Implementation of Domain-Specific Large Model Benchmarking Based on KubeEdge-Ianvs #144

OSPP: Implementation of Domain-Specific Large Model Benchmarking Based on KubeEdge-Ianvs #144

Conversation

IcyFeather233 commented Sep 9, 2024

hsj576 left a comment

Choose a reason for hiding this comment

IcyFeather233 commented Sep 11, 2024

MooreZheng left a comment

Choose a reason for hiding this comment

MooreZheng left a comment

Choose a reason for hiding this comment

IcyFeather233 commented Oct 14, 2024

MooreZheng commented Oct 22, 2024

MooreZheng commented Oct 22, 2024

MooreZheng left a comment • edited Loading

Choose a reason for hiding this comment

IcyFeather233 commented Oct 27, 2024

MooreZheng commented Oct 28, 2024

MooreZheng left a comment • edited Loading

Choose a reason for hiding this comment

MooreZheng Oct 28, 2024

Choose a reason for hiding this comment

IcyFeather233 commented Oct 28, 2024

MooreZheng commented Oct 28, 2024

MooreZheng left a comment

Choose a reason for hiding this comment

IcyFeather233 commented Oct 30, 2024

MooreZheng left a comment

Choose a reason for hiding this comment

MooreZheng commented Oct 30, 2024 • edited Loading

MooreZheng commented Oct 30, 2024

hsj576 commented Oct 30, 2024

MooreZheng commented Oct 30, 2024

MooreZheng left a comment

Choose a reason for hiding this comment

kubeedge-bot commented Oct 30, 2024

MooreZheng left a comment •

edited

Loading

MooreZheng left a comment •

edited

Loading

MooreZheng commented Oct 30, 2024 •

edited

Loading