Fix merging of stream datasets #1609
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…penvinotoolkit#1582) I ran into an issue where plugins loaded into the `Environment` are not retained after merging. Turns out a generator is being passed to [Environment.merge](https://github.com/openvinotoolkit/datumaro/blob/30b1add52d6fe458dba5e32e1f63ffeea999ad7b/src/datumaro/components/environment.py#L271) instead of the expected Sequence. This meant that custom plugins were never registered into the merged environment. Reproducible example: ```python from datumaro.components.dataset import Dataset from datumaro.components.environment import Environment from datumaro.components.exporter import Exporter from datumaro.components.hl_ops import HLOps from datumaro.components.media import Image class MyPlugin(Exporter): pass environment = Environment() environment.exporters.register('my_plugin', MyPlugin) dataset1 = Dataset(media_type=Image, env=environment) datasets = [dataset1, dataset1.clone()] merged = HLOps.merge(*datasets) assert 'my_plugin' in dataset1.env.exporters # Passes assert 'my_plugin' in merged.env.exporters # Fails ``` --------- Co-authored-by: Sooah Lee <[email protected]>
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.26.2 to 3.26.4. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/github/codeql-action/blob/main/CHANGELOG.md">github/codeql-action's changelog</a>.</em></p> <blockquote> <h1>CodeQL Action Changelog</h1> <p>See the <a href="https://github.com/github/codeql-action/releases">releases page</a> for the relevant changes to the CodeQL CLI and language packs.</p> <p>Note that the only difference between <code>v2</code> and <code>v3</code> of the CodeQL Action is the node version they support, with <code>v3</code> running on node 20 while we continue to release <code>v2</code> to support running on node 16. For example <code>3.22.11</code> was the first <code>v3</code> release and is functionally identical to <code>2.22.11</code>. This approach ensures an easy way to track exactly which features are included in different versions, indicated by the minor and patch version numbers.</p> <h2>[UNRELEASED]</h2> <p>No user facing changes.</p> <h2>3.26.4 - 21 Aug 2024</h2> <ul> <li><em>Deprecation:</em> The <code>add-snippets</code> input on the <code>analyze</code> Action is deprecated and will be removed in the first release in August 2025. <a href="https://redirect.github.com/github/codeql-action/pull/2436">#2436</a></li> <li>Fix an issue where the disk usage system call used for telemetry would fail on MacOS ARM machines with System Integrity Protection disabled, and then surface a warning. The system call is now disabled for these machines. <a href="https://redirect.github.com/github/codeql-action/pull/2434">#2434</a></li> </ul> <h2>3.26.3 - 19 Aug 2024</h2> <ul> <li>Fix an issue where the CodeQL Action could not write diagnostic messages on Windows. This issue did not impact analysis quality. <a href="https://redirect.github.com/github/codeql-action/pull/2430">#2430</a></li> </ul> <h2>3.26.2 - 14 Aug 2024</h2> <ul> <li>Update default CodeQL bundle version to 2.18.2. <a href="https://redirect.github.com/github/codeql-action/pull/2417">#2417</a></li> </ul> <h2>3.26.1 - 13 Aug 2024</h2> <p>No user facing changes.</p> <h2>3.26.0 - 06 Aug 2024</h2> <ul> <li><em>Deprecation:</em> Swift analysis on Ubuntu runner images is no longer supported. Please migrate to a macOS runner if this affects you. <a href="https://redirect.github.com/github/codeql-action/pull/2403">#2403</a></li> <li>Bump the minimum CodeQL bundle version to 2.13.5. <a href="https://redirect.github.com/github/codeql-action/pull/2408">#2408</a></li> </ul> <h2>3.25.15 - 26 Jul 2024</h2> <ul> <li>Update default CodeQL bundle version to 2.18.1. <a href="https://redirect.github.com/github/codeql-action/pull/2385">#2385</a></li> </ul> <h2>3.25.14 - 25 Jul 2024</h2> <ul> <li>Experimental: add a new <code>start-proxy</code> action which starts the same HTTP proxy as used by <a href="https://github.com/github/dependabot-action"><code>github/dependabot-action</code></a>. Do not use this in production as it is part of an internal experiment and subject to change at any time. <a href="https://redirect.github.com/github/codeql-action/pull/2376">#2376</a></li> </ul> <h2>3.25.13 - 19 Jul 2024</h2> <ul> <li>Add <code>codeql-version</code> to outputs. <a href="https://redirect.github.com/github/codeql-action/pull/2368">#2368</a></li> <li>Add a deprecation warning for customers using CodeQL version 2.13.4 and earlier. These versions of CodeQL were discontinued on 9 July 2024 alongside GitHub Enterprise Server 3.9, and will be unsupported by CodeQL Action versions 3.26.0 and later and versions 2.26.0 and later. <a href="https://redirect.github.com/github/codeql-action/pull/2375">#2375</a> <ul> <li>If you are using one of these versions, please update to CodeQL CLI version 2.13.5 or later. For instance, if you have specified a custom version of the CLI using the 'tools' input to the 'init' Action, you can remove this input to use the default version.</li> <li>Alternatively, if you want to continue using a version of the CodeQL CLI between 2.12.6 and 2.13.4, you can replace <code>github/codeql-action/*@V3</code> by <code>github/codeql-action/*@v3.25.13</code> and <code>github/codeql-action/*@v2</code> by <code>github/codeql-action/*@v2.25.13</code> in your code scanning workflow to ensure you continue using this version of the CodeQL Action.</li> </ul> </li> </ul> <h2>3.25.12 - 12 Jul 2024</h2> <ul> <li>Improve the reliability and performance of analyzing code when analyzing a compiled language with the <code>autobuild</code> <a href="https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages#codeql-build-modes">build mode</a> on GitHub Enterprise Server. This feature is already available to GitHub.com users. <a href="https://redirect.github.com/github/codeql-action/pull/2353">#2353</a></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/github/codeql-action/commit/f0f3afee809481da311ca3a6ff1ff51d81dbeb24"><code>f0f3afe</code></a> Merge main into releases/v3 (<a href="https://redirect.github.com/github/codeql-action/issues/2437">#2437</a>)</li> <li><a href="https://github.com/github/codeql-action/commit/e3543591a523b40efe5604333b75986a211c9d50"><code>e354359</code></a> Update changelog for v3.26.4</li> <li><a href="https://github.com/github/codeql-action/commit/ae01f807ca7dae87a61d9c5e36c06676ade69b1b"><code>ae01f80</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/2436">#2436</a> from rvermeulen/rvermeulen/deprecate-add-snippets</li> <li><a href="https://github.com/github/codeql-action/commit/72bc3f7f619fdbada5d8d589b47c52c32f9ad097"><code>72bc3f7</code></a> Address incorrect changelog location</li> <li><a href="https://github.com/github/codeql-action/commit/7388c476ae17d48d06bd893197f2865e72233827"><code>7388c47</code></a> Merge branch 'main' into rvermeulen/deprecate-add-snippets</li> <li><a href="https://github.com/github/codeql-action/commit/d7c48ef5a8182e33f305da9abd0f642eb687244d"><code>d7c48ef</code></a> Add link to PR deprecating <code>add-snippets</code> to CHANGELOG.md</li> <li><a href="https://github.com/github/codeql-action/commit/ec21b8f8a46388ea7527dbba9f40d839d52aabaf"><code>ec21b8f</code></a> Update changelog with deprecation.</li> <li><a href="https://github.com/github/codeql-action/commit/4067cdab784c667cf1b7fa95169f3a0e0a381d63"><code>4067cda</code></a> Add deprecation message to <code>add-snippets</code> input.</li> <li><a href="https://github.com/github/codeql-action/commit/202b3b97bf2ab4efa04bf0fde6e14313688b956b"><code>202b3b9</code></a> Stop checking disk usage for MacOS ARM with SIP disabled (<a href="https://redirect.github.com/github/codeql-action/issues/2434">#2434</a>)</li> <li><a href="https://github.com/github/codeql-action/commit/512e3066dd919110d2039970efdc5443f96d2eb5"><code>512e306</code></a> Merge pull request <a href="https://redirect.github.com/github/codeql-action/issues/2404">#2404</a> from github/marcogario/proxy_64</li> <li>Additional commits viewable in <a href="https://github.com/github/codeql-action/compare/429e1977040da7a23b6822b13c129cd1ba93dbb2...f0f3afee809481da311ca3a6ff1ff51d81dbeb24">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
- Ticket no. 149048 - Support tabular type dataset for `DmTorchDataset` - Convert transformed dataset of tabular data into torch dataset - Add new unit tests for framework converter
…t#1600) Bumps [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) from 1.9.0 to 1.10.1. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/pypa/gh-action-pypi-publish/releases">pypa/gh-action-pypi-publish's releases</a>.</em></p> <blockquote> <h2>v1.10.1</h2> <h2>🚑🔏 Oopsie... We missed a tiny bug in the attestations feature the other day</h2> <p>The problem was that the distribution file validity check was failing on any valid distribution being present and ready to be signed. What a silly mistake! It's now been fixed via <a href="https://github.com/pypa/gh-action-pypi-publish/commit/0ab0b79">https://github.com/pypa/gh-action-pypi-publish/commit/0ab0b79</a>, though. So everything's good!</p> <p>-- <a href="https://github.com/webknjaz"><code>@webknjaz</code></a><a href="https://github.com/sponsors/webknjaz">💰</a></p> <blockquote> <p>[!IMPORTANT] ✨ Despite this minor hiccup, we invite you to still opt into trying this feature out early. <a href="https://github.com/marketplace/actions/pypi-publish#generating-and-uploading-attestations">It can be enabled</a> like this:</p> <pre lang="yml"><code> with: attestations: true </code></pre> <p>Leave feedback in <a href="https://github.com/pypa/gh-action-pypi-publish/discussions/255">the v1.10.0 release discussion</a> or <a href="https://redirect.github.com/pypa/gh-action-pypi-publish/pull/236">the PR</a>.</p> </blockquote> <p><strong>🪞 Full Diff</strong>: <a href="https://github.com/pypa/gh-action-pypi-publish/compare/v1.10.0...v1.10.1">https://github.com/pypa/gh-action-pypi-publish/compare/v1.10.0...v1.10.1</a></p> <p><strong>🧔♂️ Release Manager:</strong> <a href="https://github.com/sponsors/webknjaz"><code>@webknjaz 🇺🇦</code></a></p> <p><strong>🙏 Special Thanks</strong> to <a href="https://github.com/hugovk"><code>@hugovk</code></a><a href="https://github.com/sponsors/hugovk">💰</a> for <a href="https://redirect.github.com/pypa/gh-action-pypi-publish/issues/256#issuecomment-2325925847">promptly validating the bug fix</a>, mere minutes after I pushed it — I even haven't finished writing this text by then!</p> <h2>v1.10.0</h2> <h2>🔏 Anything fancy, eh?</h2> <p>This time, <a href="https://github.com/woodruffw"><code>@woodruffw</code></a><a href="https://github.com/sponsors/woodruffw">💰</a> implemented support for <a href="https://peps.python.org/pep-0740/">PEP 740</a> attestations functionality in <a href="https://redirect.github.com/pypa/gh-action-pypi-publish/issues/236">#236</a> and <a href="https://redirect.github.com/pypa/gh-action-pypi-publish/issues/245">#245</a>. This is a big deal, as it is a huge step forward to replacing what the deprecated GPG signatures used to provide in a more meaningful way.</p> <blockquote> <p>[!IMPORTANT] ✨ Please, do opt into trying this feature out early. <a href="https://github.com/marketplace/actions/pypi-publish#generating-and-uploading-attestations">It can be enabled</a> as follows:</p> <pre lang="yml"><code> with: attestations: true </code></pre> <p>Leave any feedback on this in <a href="https://github.com/pypa/gh-action-pypi-publish/discussions/255">this release discussion</a> or <a href="https://redirect.github.com/pypa/gh-action-pypi-publish/pull/236">the PR</a>.</p> </blockquote> <p>🙏 And please, thank William for working on this amazing improvement for the ecosystem! The overall effort is tracked @ <a href="https://redirect.github.com/pypi/warehouse/issues/15871">pypi/warehouse#15871</a>, by the way.</p> <p><strong>🪞 Full Diff</strong>: <a href="https://github.com/pypa/gh-action-pypi-publish/compare/v1.9.0...v1.10.0">https://github.com/pypa/gh-action-pypi-publish/compare/v1.9.0...v1.10.0</a></p> <p><strong>🧔♂️ Release Manager:</strong> <a href="https://github.com/sponsors/webknjaz"><code>@webknjaz 🇺🇦</code></a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pypa/gh-action-pypi-publish/commit/0ab0b79471669eb3a4d647e625009c62f9f3b241"><code>0ab0b79</code></a> 🚑 Invert the dists-to-attest validity check</li> <li><a href="https://github.com/pypa/gh-action-pypi-publish/commit/8a08d616893759ef8e1aa1f2785787c0b97e20d6"><code>8a08d61</code></a> Expose PEP 740 attestations functionality</li> <li><a href="https://github.com/pypa/gh-action-pypi-publish/commit/fb9fc6a4e67ca27a7a76b17bbf90be83c2d3c716"><code>fb9fc6a</code></a> Merge pull request <a href="https://redirect.github.com/pypa/gh-action-pypi-publish/issues/245">#245</a> from trail-of-forks/ww/bump-twine</li> <li><a href="https://github.com/pypa/gh-action-pypi-publish/commit/4d020ff0a95865600010678657968f3f265f267d"><code>4d020ff</code></a> requirements: re-compile requirements with latest twine</li> <li>See full diff in <a href="https://github.com/pypa/gh-action-pypi-publish/compare/v1.9.0...v1.10.1">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
<!-- Contributing guide: https://github.com/openvinotoolkit/datumaro/blob/develop/CONTRIBUTING.md --> ### Summary <!-- Resolves openvinotoolkit#111 and openvinotoolkit#222. Depends on openvinotoolkit#1000 (for series of dependent commits). This PR introduces this capability to make the project better in this and that. - Added this feature - Removed that feature - Fixed the problem openvinotoolkit#1234 --> ### How to test <!-- Describe the testing procedure for reviewers, if changes are not fully covered by unit tests or manual testing can be complicated. --> ### Checklist <!-- Put an 'x' in all the boxes that apply --> - [ ] I have added unit tests to cover my changes. - [ ] I have added integration tests to cover my changes. - [ ] I have added the description of my changes into [CHANGELOG](https://github.com/openvinotoolkit/datumaro/blob/develop/CHANGELOG.md). - [ ] I have updated the [documentation](https://github.com/openvinotoolkit/datumaro/tree/develop/docs) accordingly ### License - [ ] I submit _my code changes_ under the same [MIT License](https://github.com/openvinotoolkit/datumaro/blob/develop/LICENSE) that covers the project. Feel free to contact the maintainers if that's a concern. - [ ] I have updated the license header for each file (see an example below). ```python # Copyright (C) 2024 Intel Corporation # # SPDX-License-Identifier: MIT ``` --------- Co-authored-by: Yunchu Lee <[email protected]> Co-authored-by: Wonju Lee <[email protected]>
<!-- Contributing guide: https://github.com/openvinotoolkit/datumaro/blob/develop/CONTRIBUTING.md --> ### Summary <!-- Resolves openvinotoolkit#111 and openvinotoolkit#222. Depends on openvinotoolkit#1000 (for series of dependent commits). This PR introduces this capability to make the project better in this and that. - Added this feature - Removed that feature - Fixed the problem openvinotoolkit#1234 --> ### How to test <!-- Describe the testing procedure for reviewers, if changes are not fully covered by unit tests or manual testing can be complicated. --> ### Checklist <!-- Put an 'x' in all the boxes that apply --> - [ ] I have added unit tests to cover my changes. - [ ] I have added integration tests to cover my changes. - [ ] I have added the description of my changes into [CHANGELOG](https://github.com/openvinotoolkit/datumaro/blob/develop/CHANGELOG.md). - [ ] I have updated the [documentation](https://github.com/openvinotoolkit/datumaro/tree/develop/docs) accordingly ### License - [ ] I submit _my code changes_ under the same [MIT License](https://github.com/openvinotoolkit/datumaro/blob/develop/LICENSE) that covers the project. Feel free to contact the maintainers if that's a concern. - [ ] I have updated the license header for each file (see an example below). ```python # Copyright (C) 2024 Intel Corporation # # SPDX-License-Identifier: MIT ```
Signed-off-by: Ilya Trushkin <[email protected]>
Signed-off-by: Ilya Trushkin <[email protected]>
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## releases/1.9.0 #1609 +/- ##
=================================================
Coverage ? 81.05%
=================================================
Files ? 277
Lines ? 32527
Branches ? 6619
=================================================
Hits ? 26366
Misses ? 4703
Partials ? 1458
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
|
@wonjuleee Retargeted this PR to 1.9 release branch. All Geti tests are green. Ready to merge! |
…e_fix Signed-off-by: Ilya Trushkin <[email protected]>
sooahleex
added a commit
to sooahleex/datumaro
that referenced
this pull request
Sep 25, 2024
<!-- Contributing guide: https://github.com/openvinotoolkit/datumaro/blob/develop/CONTRIBUTING.md --> When importing a stream dataset with multiple sources in eager mode by specifying `error_policy` or `progress_reporting`, an error occurs: ``` '_MergedStreamDataset' object has no attribute '_data' ``` <!-- Resolves openvinotoolkit#111 and openvinotoolkit#222. Depends on openvinotoolkit#1000 (for series of dependent commits). This PR introduces this capability to make the project better in this and that. - Added this feature - Removed that feature - Fixed the problem openvinotoolkit#1234 --> <!-- Describe the testing procedure for reviewers, if changes are not fully covered by unit tests or manual testing can be complicated. --> <!-- Put an 'x' in all the boxes that apply --> - [x] I have added unit tests to cover my changes. - [x] I have added integration tests to cover my changes. - [x] I have added the description of my changes into [CHANGELOG](https://github.com/openvinotoolkit/datumaro/blob/develop/CHANGELOG.md). - [x] I have updated the [documentation](https://github.com/openvinotoolkit/datumaro/tree/develop/docs) accordingly - [x] I submit _my code changes_ under the same [MIT License](https://github.com/openvinotoolkit/datumaro/blob/develop/LICENSE) that covers the project. Feel free to contact the maintainers if that's a concern. - [x] I have updated the license header for each file (see an example below). ```python ``` --------- Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Ilya Trushkin <[email protected]> Co-authored-by: williamcorsel <[email protected]> Co-authored-by: Sooah Lee <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Yunchu Lee <[email protected]> Co-authored-by: Wonju Lee <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
When importing a stream dataset with multiple sources in eager mode by specifying
error_policyorprogress_reporting, an error occurs:How to test
Checklist
License
Feel free to contact the maintainers if that's a concern.