Version 3.27.0rc0

mborsetti · Oct 29, 2024 · 7c3dae4 · 7c3dae4
1 parent af24a2d
commit 7c3dae4
Show file tree

Hide file tree

Showing 23 changed files with 582 additions and 395 deletions.
diff --git a/.github/workflows/ci-cd.yaml b/.github/workflows/ci-cd.yaml
@@ -65,9 +65,12 @@ jobs:
       matrix:
         # Python versions at https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json
         # RCs need to be specified fully, e.g. '3.13.0-rc.3'
-        python-version: ['3.12', '3.11', '3.10']
+        python-version: ['3.13', '3.12', '3.11', '3.10']
 #        python-version: ['3.12']
         os: [ubuntu-latest]
+        include:
+          # Free-threaded
+          - { python-version: '3.13', disable-gil: true }
 
     # Set up Redis per https://docs.github.com/en/actions/guides/creating-redis-service-containers
     # If you are using GitHub-hosted runners, you must use an Ubuntu runner
@@ -99,13 +102,26 @@ jobs:
       # Build Python and packages per https://github.com/actions/setup-python
       - name: Set up Python ${{ matrix.python-version }}
         uses: actions/setup-python@main
+        if: '${{ !matrix.disable-gil }}'
         with:
           python-version: ${{ matrix.python-version }}
           cache: 'pip'
           cache-dependency-path: |
             requirements.txt
             tests/requirements_pytest.txt
 
+      # from https://github.com/actions/setup-python/issues/771
+      - name: Set up Python ${{ matrix.python-version }} (free-threaded)
+        uses: deadsnakes/action@main
+        if: '${{ matrix.disable-gil }}'
+        with:
+          python-version: ${{ matrix.python-version }}
+          nogil: ${{ matrix.disable-gil }}
+      - name: Set PYTHON_GIL
+        if: '${{ matrix.disable-gil }}'
+        run: |
+          echo "PYTHON_GIL=0" >> $GITHUB_ENV
+
       - name: Upgrade pip etc.
         run: |
           pip install --upgrade pip setuptools wheel
@@ -331,7 +347,8 @@ jobs:
   deploy:
     name: Deploy release
     permissions:
-      id-token: write  # IMPORTANT: this permission is mandatory for trusted publishing
+      contents: write  # IMPORTANT: this permission is mandatory for trusted publishing
+      id-token: write  # IMPORTANT: this permission is mandatory for create GitHub release
 #    needs: [test_ubuntu, test_macos, test_windows]
     needs: [test_ubuntu, test_macos]
     # ref https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#github-context
@@ -365,13 +382,14 @@ jobs:
 #          password: ${{ secrets.pypi_password }}
 
       - name: Create GitHub release
-        id: create_release
-        uses: actions/create-release@main
+#        uses: actions/create-release@main
+        uses: softprops/action-gh-release@main
         with:
           tag_name: ${{ github.ref }}
           release_name: ${{ github.ref }}
           body_path: RELEASE.rst
           draft: false
           prerelease: false
+          make_latest: true
         env:
           GITHUB_TOKEN: ${{ secrets.github_token }}
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
@@ -33,14 +33,34 @@ can check out the `wish list <https://github.com/mborsetti/webchanges/blob/main/
    Internals, for changes that don't affect users. [triggers a minor patch]
 
 
+Version 3.27.0rc0
+==================
+Unreleased
+
+Added
+-----
+* New Sub-directive in ``pypdf`` Filter: Added ``extraction_mode`` sub-directive.
+* Python 3.13 Testing: **webchanges** now is tested on Python 3.13 before releasing. However, the `aioxmpp
+  <https://pypi.org/project/aioxmpp/>`__ library required by the ``xmpp`` reporter will not install in Python 3.13 (at
+  least on Windows), and the development of the `library <https://codeberg.org/jssfr/aioxmpp>`__ has been
+  halted.
+
+Internals
+---------
+* Added ``ai_google`` directive to the ``image`` differ to test Generative AI summarization of differences between two
+  images, but the results are still fairly bad and practically. This feature is in ALPHA and undocumented, and will
+  not be developed further until the models improve to produce useful summaries.
+
+
+
 Version 3.26.0
 ===================
 2024-10-13
 
 Added
 -----
 * Python 3.13 Support: **webchanges** now supports Python 3.13, but complete testing is pending due to dependencies
-  such has ``lxml`` not having yet published installation packages ("wheels") for 3.13.
+  such as ``lxml`` not having yet published installation packages ("wheels") for 3.13.
 * Glob Pattern Support for Hooks Files: The ``--hooks`` command-line argument now accepts glob patterns for flexible
   hook file selection.
 * Multiple Hook Specifications: Specify multiple hook files or glob patterns by repeating the ``--hooks`` argument.
@@ -50,7 +70,7 @@ Added
   improved efficiency (requires ``pip install -U webchanges[zstd]``).
 * ``ai_google`` Differ Enhancements (BETA):
 
-  * New ``additions_only`` Subdirective: When set to true, generates AI-powered summaries of only the added text. This
+  * New ``additions_only`` Sub-directive: When set to true, generates AI-powered summaries of only the added text. This
     is particularly helpful for monitoring pages with regularly added content (e.g., press releases).
   * New ``unified_diff_new`` Field: Added to the ``prompt`` directive.
 
@@ -64,7 +84,7 @@ Changed
      ``prompt`` directive have been renamed to ``old_text`` and ``new_text``, respectively.
   *  Improved Output Quality: Significantly enhanced output quality by revising the default values for
      ``system_instructions`` and ``prompt``.
-  *  Updated Documentation.
+  *  Updated documentation.
 
 Fixed
 -----

diff --git a/LICENSE b/LICENSE
@@ -28,7 +28,7 @@ CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
 SOURCE CODE REDISTRIBUTION NOTICE
 (urlwatch by Thomas Perl)
 
-This software redistributes source code of release 2.21 dated 30 July 2020 of
+This software redistributes source code of release 2.21, dated 30 July 2020, of
 urlwatch
 https://github.com/thp/urlwatch/tree/346b25914b0418342ffe2fb0529bed702fddc01f,
 which is subject to the following copyright notice and license (from

diff --git a/README.rst b/README.rst
@@ -9,22 +9,18 @@
 webchanges |downloads|
 ======================
 
-**webchanges** checks web content including images, and notifies you via email (or one of many other `supported
-services <https://webchanges.readthedocs.io/en/stable/introduction.html#reporters-list>`__) if a change is detected.
-**webchanges** can also check the output of local commands. The notification includes a detail of what has changed
-("diff"), with an optional summary generated by AI (BETA).
-
-**webchanges** *anonymously* alerts you of web changes.
-
+**webchanges** *anonymously* checks web content (including images) and commands for changes, delivering instant
+notifications and AI-powered summaries to your favorite `platform
+<https://webchanges.readthedocs.io/en/stable/introduction.html#reporters-list>`__.
 
 
 Requirements
 ============
 **webchanges** requires |support|.
 
-You should use the latest version of `Python <https://www.python.org/downloads/>`__ if possible, but older
-Python versions are supported for 3 years after being obsoleted by a new major release (3.x). For each major release,
-only the latest bug and security fix version (3.x.y) is supported.
+For the best experience, use the current version of `Python <https://www.python.org/downloads/>`__. We also support
+older Python versions for 3 years after they're replaced by a newer one; we just ask that you use the most up-to-date
+bug and security fix release from that older version.
 
 For Generative AI summaries (BETA), you need a free `API Key from Google Cloud AI Studio
 <https://aistudio.google.com/app/apikey>`__ (see `here
@@ -39,11 +35,10 @@ Install **webchanges** |pypi_version| |format| |status| |security| with:
 
    pip install webchanges
 
-
 Running in Docker
-=================
-**webchanges** can be run in a `Docker <https://www.docker.com/>`__ container; please see `here
-<https://github.com/yubiuser/webchanges-docker>`__ for one such implementation.
+-----------------
+**webchanges** can easily run in a container; you can find a `Docker <https://www.docker.com/>`__ implementation
+`here <https://github.com/yubiuser/webchanges-docker>`__.
 
 
 Documentation |readthedocs|
@@ -63,7 +58,6 @@ Initialize
 
       webchanges --edit
 
-
 #. Run the following command to change the default `configuration
    <https://webchanges.readthedocs.io/en/stable/configuration.html>`__, e.g. to receive change notifications
    ("`reports <https://webchanges.readthedocs.io/en/stable/reporters.html>`__")
@@ -73,26 +67,27 @@ Initialize
 
       webchanges --edit-config
 
-
 Run
 ---
-To check the sources in your jobs and report on (e.g. display or via email) any changes found from the previous
-execution, just run:
+To check the sources in your jobs and report on (e.g. display or via email) any changes found from the last time the
+program ran, just run:
 
 .. code-block:: bash
 
    webchanges
 
-**webchanges** does not include a scheduler. We recommend using a system scheduler to automatically run it
-periodically:
 
-- On Linux or macOS, you can use cron (if you have never used cron before, see
-  `here <https://www.computerhope.com/unix/ucrontab.htm>`__); `crontab.guru <https://crontab.guru>`__ will build a
-  schedule expression for you;
-- On macOS, you can use `launchd <https://developer.apple
-  .com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html>`__;
-- On Windows, you can use the built-in `Windows Task Scheduler
-  <https://en.wikipedia.org/wiki/Windows_Task_Scheduler>`__.
+Schedule
+--------
+**webchanges** leverages the power of a system scheduler:
+
+- On Linux you can use cron, and a tool like `crontab.guru <https://crontab.guru>`__ can build a
+  schedule expression for you (note: see `here <https://www.computerhope.com/unix/ucrontab.htm>`__ if you have never
+  used cron before);
+- On Windows you can use `Windows Task Scheduler <https://en.wikipedia.org/wiki/Windows_Task_Scheduler>`__;
+- On macOS you can use `launchd <https://developer.apple
+  .com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html>`__ (note: see `here
+  <https://launchd.info/>`__ if you have never used launchd before).
 
 
 Code
@@ -116,42 +111,50 @@ License
 =======
 |license|
 
-Released under the `MIT License <https://opensource.org/licenses/MIT>`__ but redistributing modified source code from
-`urlwatch 2.21 <https://github.com/thp/urlwatch/tree/346b25914b0418342ffe2fb0529bed702fddc01f>`__ dated 30 July 2020
+See the `complete licenses <https://raw.githubusercontent.com/mborsetti/webchanges/refs/heads/main/LICENSE>`__ (released
+under the `MIT License <https://opensource.org/licenses/MIT>`__ but redistributing modified source code, dated 30
+July 2020, from `urlwatch 2.21 <https://github.com/thp/urlwatch/tree/346b25914b0418342ffe2fb0529bed702fddc01f>`__
 licensed under a `BSD 3-Clause License
-<https://raw.githubusercontent.com/thp/urlwatch/346b25914b0418342ffe2fb0529bed702fddc01f/COPYING>`__. See the complete
-license `here <https://github.com/mborsetti/webchanges/blob/main/LICENSE>`__.
+<https://raw.githubusercontent.com/thp/urlwatch/346b25914b0418342ffe2fb0529bed702fddc01f/COPYING>`__).
 
 
 Compatibility with and improvements from **urlwatch**
 =====================================================
 
 This project is based on code from `urlwatch 2.21
-<https://github.com/thp/urlwatch/tree/346b25914b0418342ffe2fb0529bed702fddc01f>`__ dated 30 July 2020. You can
-easily upgrade to **webchanges** from the current version of **urlwatch** using the same job and configuration files
-(see `here <https://webchanges.readthedocs.io/en/stable/migration.html>`__) and benefit from many improvements,
-including:
-
-* Summary of changes in plain text using Generative AI, useful for long documents (e.g. legal);
-* Depicting changes to an image;
-* Element-by-element changes of JSON or XML data;
-* Much better `documentation <https://webchanges.readthedocs.io/>`__, easing implementation;
-* Much clearer to HTML reports, which include:
-
-  * Links that are `clickable <https://pypi.org/project/webchanges/>`__!
-  * Retaining of original formatting such as **bolding / headers**, *italics*, :underline:`underlining`, list bullets
-    (•) and indentation;
-  * Lines that are :additions:`added` and :deletions:`deleted` are clearly highlighted by color and strikethrough,
-    and long lines wrap around;
-  * Correct rendering by email clients who override stylesheets (e.g. Gmail);
-  * Other legibility improvements;
-
-* New filters such as `additions_only <https://webchanges.readthedocs.io/en/stable/diff_filters.html#additions-only>`__,
-  which makes it easier to track content that was added without the distractions of the content that was deleted;
-* New command line arguments, such as ``--errors`` (to catch jobs that no longer work);
-* More reliability and stability, including a ~30 percentage point increase in testing coverage;
-* Many other additions, refinements and fixes (see `detailed information
-  <https://webchanges.readthedocs.io/en/stable/migration.html#upgrade-details>`__).
+<https://github.com/thp/urlwatch/tree/346b25914b0418342ffe2fb0529bed702fddc01f>`__ dated 30 July 2020.
+
+You can easily upgrade to **webchanges** from the current version of **urlwatch** using the same job and
+configuration files (see `here <https://webchanges.readthedocs.io/en/stable/migration.html>`__) and benefit from many
+improvements, including:
+
+* :underline:`AI-Powered Summaries`: Summary of changes in plain text using generative AI, useful for long documents
+  (e.g. legal);
+* :underline:`Image Change Detection`: Monitor changes to images and receive notifications with an image highlighting
+  the differences;
+* :underline:`Structured Data Monitoring`: Track changes in JSON or XML data on an element-by-element basis;
+* :underline:`Improved Documentation`: We've revamped the `documentation <https://webchanges.readthedocs.io/>`__ to make
+  implementation easier;
+* :underline:`Enhanced HTML Reports`: HTML reports are now much clearer and include:
+
+  * Clickable links!
+  * Retention of most original formatting (**bolding / headers**, *italics*, :underline:`underlining`, lists with
+    bullets (•), and indentation;
+  * :additions:`added` and :deletions:`deleted` lines clearly highlighted with color and strikethrough;
+  * Wrapping of long lines (instead of truncation);
+  * Improved compatibility with a wider range of HTML email clients, including those that override stylesheets (e.g.,
+    Gmail);
+  * General legibility improvements.
+
+* :underline:`New Filtering Options`: New filters, like `additions_only
+  <https://webchanges.readthedocs.io/en/stable/diff_filters.html#additions-only>`__, which allows you to focus on
+  added content without the distraction of deletions;
+* :underline:`New Command Line Arguments`: New command-line arguments such as ``--errors``, which helps you identify
+  jobs that are no longer functioning correctly;
+* :underline:`Increased Reliability and Stability`: Testing coverage has increased by approximately 30 percentage
+  points;
+* :underline:`Additional Enhancements`: Numerous other additions, refinements, and bug fixes have been implemented.
+  For more information, see `here <https://webchanges.readthedocs.io/en/stable/migration.html#upgrade-details>`__.
 
 Example enhancements to HTML reporting:
 

diff --git a/RELEASE.rst b/RELEASE.rst
@@ -1,38 +1,12 @@
 Added
 -----
-* Python 3.13 Support: **webchanges** now supports Python 3.13, but complete testing is pending due to dependencies
-  such has ``lxml`` not having yet published installation packages ("wheels") for 3.13.
-* Glob Pattern Support for Hooks Files: The ``--hooks`` command-line argument now accepts glob patterns for flexible
-  hook file selection.
-* Multiple Hook Specifications: Specify multiple hook files or glob patterns by repeating the ``--hooks`` argument.
-* Enhanced Version Information: ``--detailed-versions`` now displays the system's default value for
-  ``--max-threads``.
-* Optional ``zstd`` Compression: URL jobs without ``browser: true`` can now utilize ``zstd`` compression for
-  improved efficiency (requires ``pip install -U webchanges[zstd]``).
-* ``ai_google`` Differ Enhancements (BETA):
-
-  * New ``additions_only`` Subdirective: When set to true, generates AI-powered summaries of only the added text. This
-    is particularly helpful for monitoring pages with regularly added content (e.g., press releases).
-  * New ``unified_diff_new`` Field: Added to the ``prompt`` directive.
-
-Changed
--------
-* Relaxed Security for Job and Hook Files: The ownership requirement for files containing ``command`` jobs,
-  ``shellpipe`` filters, or hook files has been expanded to include root ownership, in addition to the current user.
-* ``ai_google`` Differ Refinements (BETA):
-
-  *  Renamed Prompt Fields (⚠ BETA breaking change):  For clarity, ``old_data`` and ``new_data`` fields in the
-     ``prompt`` directive have been renamed to ``old_text`` and ``new_text``, respectively.
-  *  Improved Output Quality: Significantly enhanced output quality by revising the default values for
-     ``system_instructions`` and ``prompt``.
-  *  Updated Documentation.
-
-Fixed
------
-* Markdown Handling: Improved handling of links with empty text in the Markdown to HTML converter.
-* ``image`` Differ Formatting: Fixed HTML formatting issues within the ``image`` differ.
-
-Removed
--------
-* Python 3.9 Support: Support for Python 3.9 has been dropped. As a reminder, older Python versions are supported for 3
-  years after being superseded by a new major release (i.e. approximately 4 years after their initial release).
+* Python 3.13 Testing: **webchanges** now is tested on Python 3.13 before releasing. However, the `aioxmpp
+  <https://pypi.org/project/aioxmpp/>`__ library required by the ``xmpp`` reporter will not install in Python 3.13 (at
+  least on Windows), and the development of the `library <https://codeberg.org/jssfr/aioxmpp>`__ has been
+  halted.
+
+Internals
+---------
+* Added ``ai_google`` directive to the ``image`` differ to test Generative AI summarization of differences between two
+  images, but the results are still fairly bad and practically. This feature is in ALPHA and undocumented, and will
+  not be developed further until the models improve to produce useful summaries.
diff --git a/docs/differs.rst b/docs/differs.rst
@@ -243,7 +243,7 @@ by using a prompt similar to the one here:
 More information about writing input prompts for these models can be found `here
 <https://ai.google.dev/gemini-api/docs/prompting-intro>`__. You may also ask the model itself (in `AI Studio
 <https://aistudio.google.com/>`__) to suggest prompts that are appropriate to your use case, or use the "Help me write"
-function in `AI Studio Prompts <https://console.cloud.google.com/vertex-ai/studio/freeform>`__.
+function in `AI Vertex Vertex Prompt <https://console.cloud.google.com/vertex-ai/studio/freeform>`__.
 
 Example
 ```````
@@ -307,7 +307,6 @@ This differ is currently in BETA and these directives MAY change in the future.
 * ``top_p`` (float between 0.0 and 1.0): The model's TopP parameter, or the cumulative probability cutoff for token
   selection; lower p means sampling from a smaller, more top-weighted nucleus and reduces diversity (see note below)
   (default: model-dependent, but typically 0.95 or 1.0, see Google documentation)
-* ``token_limit`` (int): An override of the maximum size of the model's context window (used for internal testing).
 * ``unified`` (dict): directives passed to :ref:`unified differ <unified_diff>`, which prepares the unified diff
   attached to this report.