Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

1.6.2 fail to build #187

Open
marillat opened this issue Dec 18, 2024 · 14 comments
Open

1.6.2 fail to build #187

marillat opened this issue Dec 18, 2024 · 14 comments
Labels
bug

Comments

@marillat
Copy link

Debian unstable amd64
python 3.12.8 and 3.13.1

1.6.1.1 build fine

dpkg-buildpackage: info: source package dvr-scan-dmo
dpkg-buildpackage: info: source version 1.6.2-dmo1
dpkg-buildpackage: info: source distribution unstable
dpkg-buildpackage: info: source changed by Christian Marillat <[email protected]>
 dpkg-source --before-build .
dpkg-buildpackage: info: host architecture amd64
dpkg-source: info: using options from dvr-scan-dmo-1.6.2/debian/source/options: --extend-diff-ignore=^[^/]*[.]egg-info/
 debian/rules clean
dh clean -Spybuild
   dh_auto_clean -O-Spybuild
   debian/rules override_dh_clean
make[1]: Entering directory '/build/dvr-scan-dmo-1.6.2'
dh_clean dvr_scan/docs
make[1]: Leaving directory '/build/dvr-scan-dmo-1.6.2'
   debian/rules execute_after_dh_clean
make[1]: Entering directory '/build/dvr-scan-dmo-1.6.2'
rm -f -r dist/installer/Prerequisites/Visual\ C++\ Redistributable\ for\ Visual\ Studio\ 2015-2019
make[1]: Leaving directory '/build/dvr-scan-dmo-1.6.2'
 dpkg-source -b .
dpkg-source: info: using options from dvr-scan-dmo-1.6.2/debian/source/options: --extend-diff-ignore=^[^/]*[.]egg-info/
dpkg-source: info: using source format '3.0 (quilt)'
dpkg-source: info: building dvr-scan-dmo using existing ./dvr-scan-dmo_1.6.2.orig.tar.gz
dpkg-source: warning: ignoring deletion of directory dist/installer/Prerequisites/Visual C++ Redistributable for Visual Studio 2015-2019
dpkg-source: warning: ignoring deletion of file dist/installer/Prerequisites/Visual C++ Redistributable for Visual Studio 2015-2019/VC_redist.x64.exe, use --include-removal to override
dpkg-source: info: building dvr-scan-dmo in dvr-scan-dmo_1.6.2-dmo1.debian.tar.xz
dpkg-source: info: building dvr-scan-dmo in dvr-scan-dmo_1.6.2-dmo1.dsc
 debian/rules binary
dh binary -Spybuild
   dh_update_autotools_config -O-Spybuild
   dh_autoreconf -O-Spybuild
   dh_auto_configure -O-Spybuild
   dh_auto_build -O-Spybuild
I: pybuild plugin_pyproject:129: Building wheel for python3.12 with "build" module
I: pybuild base:311: python3.12 -m build --skip-dependency-check --no-isolation --wheel --outdir /build/dvr-scan-dmo-1.6.2/.pybuild/cpython3_3.12_dvr-scan  
* Building wheel...
running bdist_wheel
running build
running build_py
creating build/lib/dvr_scan
copying dvr_scan/video_joiner.py -> build/lib/dvr_scan
copying dvr_scan/overlays.py -> build/lib/dvr_scan
copying dvr_scan/detector.py -> build/lib/dvr_scan
copying dvr_scan/config.py -> build/lib/dvr_scan
copying dvr_scan/__init__.py -> build/lib/dvr_scan
copying dvr_scan/subtractor.py -> build/lib/dvr_scan
copying dvr_scan/platform.py -> build/lib/dvr_scan
copying dvr_scan/scanner.py -> build/lib/dvr_scan
copying dvr_scan/region.py -> build/lib/dvr_scan
copying dvr_scan/opencv_loader.py -> build/lib/dvr_scan
copying dvr_scan/__main__.py -> build/lib/dvr_scan
creating build/lib/dvr_scan/app
copying dvr_scan/app/__init__.py -> build/lib/dvr_scan/app
copying dvr_scan/app/region_editor.py -> build/lib/dvr_scan/app
creating build/lib/dvr_scan/cli
copying dvr_scan/cli/__init__.py -> build/lib/dvr_scan/cli
copying dvr_scan/cli/controller.py -> build/lib/dvr_scan/cli
error: package directory 'dvr_scan/docs' does not exist

ERROR Backend subprocess exited when trying to invoke build_wheel
E: pybuild pybuild:389: build: plugin pyproject failed with: exit code=1: python3.12 -m build --skip-dependency-check --no-isolation --wheel --outdir /build/dvr-scan-dmo-1.6.2/.pybuild/cpython3_3.12_dvr-scan  
dh_auto_build: error: pybuild --build -i python{version} -p 3.12 returned exit code 13
make: *** [debian/rules:6: binary] Error 25
@Breakthrough
Copy link
Owner

Breakthrough commented Dec 18, 2024 via email

@marillat
Copy link
Author

marillat commented Dec 18, 2024 •
edited
Loading

Not related to DVR-Scan but to PySceneDetect

python3 dist/pre_release.py
Creating .version_info.
Traceback (most recent call last):
  File "/build/dvr-scan-dmo-1.6.2/dist/pre_release.py", line 181, in <module>
    write_version_info_for_windows_exe()
  File "/build/dvr-scan-dmo-1.6.2/dist/pre_release.py", line 24, in write_version_info_for_windows_exe
    import dvr_scan
  File "/build/dvr-scan-dmo-1.6.2/dvr_scan/__init__.py", line 40, in <module>
    from dvr_scan.platform import init_logger
  File "/build/dvr-scan-dmo-1.6.2/dvr_scan/platform.py", line 29, in <module>
    from scenedetect.platform import get_and_create_path, get_ffmpeg_version
  File "/usr/lib/python3/dist-packages/scenedetect/__init__.py", line 54, in <module>
    from scenedetect.scene_manager import SceneManager, save_images, SceneList, CutList, Interpolation
  File "/usr/lib/python3/dist-packages/scenedetect/scene_manager.py", line 413, in <module>
    image: cv2.Mat,
           ^^^^^^^
AttributeError: module 'cv2' has no attribute 'Mat'
make[1]: *** [debian/rules:9: override_dh_auto_build] Error 1

@Breakthrough
Copy link
Owner

Breakthrough commented Dec 18, 2024 via email

@Breakthrough
Copy link
Owner

Breakthrough commented Dec 19, 2024 •
edited
Loading

I'm going to hold off on making a new release until I can understand more about how the Debian packages are being created, and if it's possible for you to run the pre-release script there. You can see how I build the Python distribution here on Github:
https://github.com/Breakthrough/DVR-Scan/blob/main/.github/workflows/build.yml

You can also try doing this in a virtual environment if it would clobber your system packages.

In the meantime, you can still build DVR-Scan 1.6.2 if you copy the docs from the pre-built Python package. I zipped these files here so you can grab them quickly (dvr_scan_docs-1.6.2.zip). If you extract them into dvr_scan/docs the build should succeed. This folder and the contents are the result of running the pre-release script and include offline documentation for DVR-Scan.

To download them from a trusted build, you can also download the Python source package from PyPI or from Github Releases. If you extract dvr_scan-1.6.2.tar.gz it should contain an identical copy of the docs I uploaded above. Lastly, if you want to leave the docs out, you can remove these lines from setup.cfg: https://github.com/Breakthrough/DVR-Scan/blob/main/setup.cfg#L50-L53

Items I could use your response on @marillat:

  1. Depending on the result of the above, please let me know if you are able to integrate dist/pre_release.py into your build process, or if I need to check-in the built documentation into the git repo. Also let me know if I need to make a new tagged release, or if the workarounds are sufficient for you to update the Debian package.
  2. Is the offline documentation supposed to be in a different location?
  3. Could you advise on how to mark the python3-tk package as a dependency? My understanding is that this is typically a pre-built module not included on PyPI.
  4. Similarily, with OpenCV, do users typically use the python3-opencv Debian package? I usually test with the latest version, the one on packages.debian.org seems to be quite a bit older, so I may need to make some changes to both DVR-Scan and PySceneDetect to support that (since it seems to cause issues when you ran the pre release script, which means it also might fail when you try to run DVR-Scan)
  5. Let me know if you wish to include any artifacts in the source code to help with building the Debian package. I will leave distribution up to you, but I do not mind including build rules or automation to ensure DVR-Scan continues to build as expected (e.g. I would probably add a Linux builder that builds an amd64 .deb).

@marillat
Copy link
Author

I'm going to hold off on making a new release until I can understand more about how the Debian packages are being created, and if it's possible for you to run the pre-release script there. You can see how I build the Python distribution here on Github: https://github.com/Breakthrough/DVR-Scan/blob/main/.github/workflows/build.yml

You can also try doing this in a virtual environment if it would clobber your system packages.

We don't use venv to do python packages. Packages are build with pybuild under chroot.

https://wiki.debian.org/Python/Pybuild

3. Could you advise on how to mark the `python3-tk` package as a dependency?  My understanding is that this is typically a pre-built module not included on PyPI.

You must set python install_requires in setup.py and not setup.cfg with versioned dependency when needed.
Then the installation will fail if a package is missing or doesn't math the needed version
https://packaging.python.org/en/latest/discussions/install-requires-vs-requirements/

4. Similarily, with OpenCV, do users typically use the `python3-opencv` Debian package?  I usually test with the latest version, the one on packages.debian.org seems to be quite a bit older, so I may need to make some changes to both DVR-Scan and PySceneDetect to support that (since it seems to cause issues when you ran the pre release script, which means it also might fail when you try to run DVR-Scan)

See 3

5. Let me know if you wish to include any artifacts in the source code to help with building the Debian package.  I will leave distribution up to you, but I do not mind including build rules or automation to ensure DVR-Scan continues to build as expected (e.g. I would probably add a Linux builder that builds an amd64 .deb).

Probably a bad idea to add a debian directory. Let the Debian maintainer do the job.

Christian

@Breakthrough
Copy link
Owner

Breakthrough commented Dec 21, 2024 •
edited
Loading

You must set python install_requires in setup.py and not setup.cfg with versioned dependency when needed. Then the installation will fail if a package is missing or doesn't math the needed version https://packaging.python.org/en/latest/discussions/install-requires-vs-requirements/

This does not work for the tkinter module, as it is not managed the same way. Adding it to install_requires makes it impossible to install the package. Surely there must be another way to make the python3-tkinter Debian package a requirement for dvr-scan-dmo?

The same issue should happen for OpenCV, but somehow that is listed as a dependency of the package.

We cannot use install_requires for OpenCV either, since it is distributed as such with some Linux distributions under python3-opencv package, or on PyPI as opencv-python, both of which provide the cv2 module. As neither opencv-python nor cv2 are in the requirements, how does python3-opencv currently become a dependency of dvr-scan package when you create the package?

Can you try running the pre_release script after uninstalling your system's python3-opencv package, and pip installing the requirements from DVR-Scan's requirements.txt and docs/requirements.txt?

Probably a bad idea to add a debian directory. Let the Debian maintainer do the job.

Sounds good. Can you share how to build the package so I can reproduce the issues you are running into? I wasn't able to find the source code for how you are producing the .debs.

@Breakthrough
Copy link
Owner

According to the Debian Python Policy, the python3-tk package should be listed under Depends: of a debian/control file. Although I do my best to follow the Debian app/library style guide for Python, there's a few things about DVR-Scan which deviate:

  • you can install everything in a virtual env, but the python3-tk package must still be available and installed for the Python interpreter backing the venv
  • running dist/pre_release.py to generate the required documentation assets - this should be possible to do under a virtualenv if the python3-venv package is listed as a build requirement

For the last point, there are some workarounds I pointed out previously. Although here we use mkdocs instead of sphinx, it seems typical for documentation building to be integrated as part of the pybuild process, by overriding the dh_auto_build command. If python3-venv is a build requirement, is it not possible to run the pre_release script under a venv in a similar manner?

@Breakthrough
Copy link
Owner

Sorry for the reply spam, I've just updated the v1.6.2 release branch and tag, so you should be able to avoid running the pre-release script and get the Debian package updated now. Will leave this issue open until you can respond to the other open items.

Thank you!

@marillat
Copy link
Author

It's OK for me.

@marillat
Copy link
Author

According to the Debian Python Policy, the python3-tk package should be listed under Depends: of a debian/control file. Although I do my best to follow the Debian app/library style guide for Python, there's a few things about DVR-Scan which deviate:

* you can install everything in a virtual env, but the python3-tk package must still be available and installed for the Python interpreter backing the venv

As I'm saying before Debian don't use at all venv to build python package.

Christian

@Breakthrough
Copy link
Owner

As I'm saying before Debian don't use at all venv to build python package.

Gotcha. I think the OpenCV issue you were running into before was caused by Breakthrough/PySceneDetect#468 so that should be fixed now - when you get a chance, could you let me know if you are able to run the pre_release script successfully and integrate it into the Debian packaging workflow?

I am still unsure how to address the Tkinter issue, however I suspect you may need to add it to the control file.

@marillat
Copy link
Author

marillat commented Dec 24, 2024 •
edited
Loading

pre_release script still return an error and DeprecationWarning

I still don't understand why the website need to be generated by the package installer.

Otherwise, I'm happy to do a package without calling the pre_release script now that issue on PySceneDetect has bee, fixed.

python3 dist/pre_release.py
INFO    -  DeprecationWarning: 'materialx.emoji.twemoji' is deprecated.
Material emoji logic has been officially moved into mkdocs-material
version 9.4. Please use Material's 'material.extensions.emoji.twemoji'
instead of 'materialx.emoji.twemoji' in your 'mkdocs.yml' file.

markdown_extensions:
  - pymdownx.emoji:
      emoji_index: !!python/name:material.extensions.emoji.twemoji
      emoji_generator: !!python/name:material.extensions.emoji.to_svg

'mkdocs_material_extensions' is deprecated and will no longer be
supported moving forward. This is the last release.

  File "/usr/lib/python3/dist-packages/materialx/emoji.py", line 118, in twemoji
    return _patch_index(options)
  File "/usr/lib/python3/dist-packages/materialx/emoji.py", line 68, in _deprecated_func
    warnings.warn(

WARNING -  Material emoji logic has been officially moved into mkdocs-material
version 9.4. Please use Material's 'material.extensions.emoji.twemoji'
instead of 'materialx.emoji.twemoji' in your 'mkdocs.yml' file.

markdown_extensions:
  - pymdownx.emoji:
      emoji_index: !!python/name:material.extensions.emoji.twemoji
      emoji_generator: !!python/name:material.extensions.emoji.to_svg

'mkdocs_material_extensions' is deprecated and will no longer be
supported moving forward. This is the last release.

INFO    -  Cleaning site directory
INFO    -  Building documentation to directory: /build/dvr-scan-dmo-1.6.2/dvr_scan/docs
INFO    -  The following pages exist in the docs directory, but are not included in the "nav" configuration:
  - index.md
INFO    -  Documentation built in 0.29 seconds
Creating .version_info.
Building docs.
Postprocessing docs.
Traceback (most recent call last):
  File "/build/dvr-scan-dmo-1.6.2/dist/pre_release.py", line 182, in <module>
    build_docs(use_local_images=bool(len(sys.argv) == 2 and sys.argv[1] == "--use-local-images"))
  File "/build/dvr-scan-dmo-1.6.2/dist/pre_release.py", line 162, in build_docs
    ) if to_remove.is_dir() else to_remove.unlink()
                                 ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/pathlib.py", line 1344, in unlink
    os.unlink(self)
FileNotFoundError: [Errno 2] No such file or directory: 'dvr_scan/docs/assets/javascripts/bundle.220ee61c.min.js.map'
make[1]: *** [debian/rules:9: override_dh_auto_build] Error 1
make[1]: Leaving directory '/build/dvr-scan-dmo-1.6.2'
make: *** [debian/rules:6: binary] Error 2
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2

@marillat
Copy link
Author

In my last comment I'm talking about the 1.6.2 source from https://github.com/Breakthrough/DVR-Scan/releases/tag/v1.6.2-release

@Breakthrough
Copy link
Owner

Breakthrough commented Dec 25, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Breakthrough @marillat