v0.8.4 - New Scanner + Browse Authors/Artists + Tons of Bugfixes

[majora2007]

After rewriting 50K lines of code last release, I decided it was best to take a break and focus on the bugs that resulted and a few that had piled up. This release is bug focused, but of course I stuck a few big features in with it.

Scanner

One issue that has been cropping up as we attract more users with larger libraries or libraries laid out in the Publisher/Series style is the scanner. Users with large libraries modify their library quite a bit and the old scanner had issues with change detection. v0.8.3 had quite a nasty bug which required forced library scans to ingest new data. There was also the notorious FK issue that was more common for larger libraries.

To fix these issues, I rewrote the scanner from the ground up to be more reliable. This was a (painful) 3 week effort working closely with a dedicated set of testers on a canary release to be nightly and stable ready. The new scanner will be slower than the old one. As of now, I feel that once the main slow first scan is done, it's actually performing much better. With this new scanner, there were some other major changes.

People Re-architecture

In order to reduce the FK issue in the scanner, I also needed to tackle the 80+ hour re-architecture of how people work within Kavita. When I started, with my limited knowledge of the scene or what Kavita would grow to become, I implemented people as tags, much like how ComicInfo does. This is constrained when you want to make people more entity-focused, like having a dedicated page for them.

With this release, you can now view individual details of a person, set a cover and description about the person and see their works and individual issues they worked on. You can also browse authors and artists, like other software has. This is just the start, please leave feedback for me via Feature Requests, so I can build this feature out more and make it a richer experience. Note: Any Smart Filters that relied on people will need manual intervention to correct. A force scan will be required to regenerate writer information.

Special Keyword Removal

The last thing with the scanner was removing Kavita's ability to detect and classify certain keywords as Special. Like Omake or Annual would classify a file as a special. While this is extremely handy in the Manga scene, there were numerous reports over the years that required manual workarounds by users. In order to behave more reliably, I opened a discussion this year for removal and made the final call. Specials going forward will need the SP marker on the filename (or tagged internally).

CoversDB

Kavita has had the ability to download and display favicon and publisher images with a fallback repository being hosted on the main Kavita website. In this release, I've moved all that functionality and those images over to a new CoversDB repo. This repo will be the home for anything related to pulling images for Kavita instances. I plan to enhance this further as new features come to Kavita.

Small QoL Features

Last Read Filter is a new feature that allows you to pass a number that will be subtracted from TODAY, upping the dynamic function and allowing users to build Smart Filters like "Haven't read in a month".

Colorscape Polish The colorscape system has been polished even further. I spent a good amount of time adding some randomness, fixed a bug with colorscape resetting between navigation, and tried to tweak where colors go to avoid harsh contrasts. It's not perfect, but it's a step in the right direction. If you're passionate about the colorscape system, I welcome collaboration.

Bulk Library Actions This is something @DieselTech has been pushing me to add and the time is finally here. For users that have a lot of libraries (like Diesel and his 200+), sometimes it's quite a pain to update the settings of them all or perform some actions. Now in Kavita there are some limited bulk options, like refreshing covers or copying settings from one library to another (Scan currently disabled).

Recap

This release has been a ton of work on my part, but I felt it was important to squash the bugs first and ensure Kavita remains stable while I continue to add more features and bring a richer experience to users. My next release is still aimed at finishing the UX Overhaul with attention on Kavita+. With the holidays coming up, it is expected that this will take longer than most releases.

For those that have seen, we have hit over 65K active installs. It's absolutely amazing that a small project I started due to Ubooquity not having metadata has turned into such a massive project with a huge userbase. I want to give my thanks again to those in the community that have raised bugs, feature requests, give support in the discord/reddit, those who give me donations via OC/Kavita+, the translators, and of course my wife that helps support me pouring so much time into the project. I'm excited to continue building Kavita and bring about more of our visions into the software and see what is possible.

People metadata will be removed on update. A force scan is required to repopulate in the new system. A force scan/first scan will be much slower than before.

Added

• Added: Manage Libraries now has bulk actions. You can trigger multiple scans (not file scans, but refresh covers or colorscapes, etc) or copy settings from one library to multiple. Shift+select works as well.
• Added: Added a new page for all People within Kavita. From this page, you can see their works across Kavita.
• Added: Ability to see individual chapters/issues (limited to 20) a person has.
• Added: Ability to add images to the person via UI/API. A more automated method is planned in the future.
• Added: Added the ability to browse all Writers and Artists (cover artist) in one page. This should help book users that like to browse by author. (Feature Request: [Kavita] Browse by Author ( & penciller, letterer, editor, etc.?) #2556 - 23 votes) (Thanks @MrRobotjs for some css on the Person detail page)
• Added: Added the ability to trigger the general cleanup (which normally runs at midnight) manually in case you need to clear unlinked people, covers, etc.
• Added: Reading List and Collection cards now show a badge showing the number of series/items underneath.
• Added: New Filter option called Last Read which allows a number to be passed in and will dynamically calculate a date from TODAY() matched against reading time. This allows for users to build even more dynamic smart filters to find things not read in a month or read within the past 7 days.
• Added: Added back a lot of the metadata that had to be removed from the top of the series detail page in the UX overhaul, like Language, Release Year, Format, and Reading Time
• Added: Added Details tab to volume details page
• Added: (Kavita+) New Sync Details button on smart collections that shows an easier to digest view of Series and missing series.
• Added: Bulk send to is now available for chapters/volumes
• Added: Added bulk deletion for chapters/volumes

Changed

• Changed: Optimized a number of methods within the Scanner to reduce memory and CPU time
• Changed: Moved a lot of the chapter scanner logging to Trace.
• Changed: Scanner can now choose to parallel parse files when there are over 100 in a directory
• Changed: Changed how detection and scanning of dirty directories works. The scanner will now parse bottom-up to reduce any potential misses and avoid different layouts working differently. This has extra I/O checks but much greater reliability and should reduce the amount of work needed to ingest changes after the first scan.
• Changed: LocalizedSeries merging with Series is now done at a higher level and performs much better (in terms of reliability)
• Changed: People are now entity based which allows for a richer experience in Kavita.
• Changed: Clicking on any person in Kavita will now open their dedicated page which showcases their different works.
• Changed: (Performance) Some large refactors to the scanner to help with the FK bug. The scanner should be slower but consume much less memory, especially for well tagged systems.
• Changed: Kavita no longer parses keywords from filenames and groups as a Special. The way to mark a special going forward will be explicitly with SP01, SP02, etc. Specials can be flat in the series folder or ideally in Specials/ folder.
• Changed: Changed logging level on some of the loggers that could become noisy
• Changed: Refactored some of the post-series work to a separate thread.
• Changed: Ensure mobile actionable modal is fullscreen and centered vertically
• Changed: Update to Angular 18 and other backend dependencies
• Changed: When a setting is invalid, clicking outside the setting item will not close and edit is disabled.
• Changed: Changed the smart filters page to open in the same window (but still available to open in new tab)
• Changed: (Performance) Made the pipe that turns the language codes into full names use a dedicated API which is faster
• Changed: Improved the wording for Single Volume chapters on Series detail page (and person detail page)
• Changed: Kavita now uses a different repo for publisher images and favicons (https://github.com/Kareadita/CoversDB). Any changes should be made here.
• Changed: You can now edit the person name in the UI. If not corrected in the metadata, a duplicate person will be created. This will be tweaked in the future once Alias support is added.
• Changed: Significantly improved the feel of the colorscape experience. When transitioning between colorscapes, locations of accent points will randomize. Lighter colors will try to avoid the bottom right where there could be bad contrast.
• Changed: Changed Colorscape to use non-linear animation for transitioning between 2 colorscapes.
• Changed: If the user somehow wrote bad cron notation for custom jobs, on startup, Kavita will detect and default to daily
• Changed: Changed the overall look and feel of the preference side nav and main side nav (Thanks @MrRobotjs)
• Changed: Moved Test email settings to the top to align with the UX patterns
• Changed: Moved Reset to Defaults to the top to align with UX patterns and removed a few instances since it resets everything.
• Changed: Changed a lot of modals to be more consistent in how they open and their size to be slightly larger.
• Changed: When failing to get a page count on an archive, report it in media issues
• Changed: Improved memory consumption on some filtering conditions
• Changed: Tweaked some logging messages to be more clear when using logs to debug

Fixed

• Fixed: Fixed a bug where the UnauthorizedAccessException is not correctly ignored when using the docker container as non-root. (Thanks @Fesaa)
• Fixed: Fixed a bug where any media that was less than an hour in length wouldn't be included in stats. This requires Analyze Files to be ran on all libraries to retroactively fix
• Fixed: Fixed a bug where chapter order wasn't properly sorting on Volume Detail page
• Fixed: Fixed an issue where bookmarking wouldn't check if the source file was already in the target encoding and thus had issue when trying to convert. Also added another check to not do anything if target encoding is PNG (aka Kavita doesn't convert webp -> png).
• Fixed: Fixed a bug where Series Cover image could choose a Volume 0 instead of Volume 1
• Fixed: Fixed a bug where teams and locations wouldn't trigger updating people on update series metadata api
• Fixed: Fixed a bug where calculating the Lowest Folder for a Series could be incorrect when there are nested folders. This includes a migration to clear out existing entries to avoid scan series not seeing files.
  Example:
  / <- Lib root assume
  /love hina/love hina/v01.cbz
  /love hina/specials/sp01.cbz
  The lowest series folder was /love hina/love hina/ for some reason, meaning series scan wasn't getting the full series (all folders) scanned in.
• Fixed: Fixed up a check for using lowest folder path to lookup from a Folder Watcher task. This should find series much better from a lower file path.
• Fixed: Fixed a bug where teams and locations wouldn't trigger updating people on update series metadata api
• Fixed: Fixed a bug where users could change their age rating when they didn't have the permission
• Fixed: Fixed a bug where download wasn't functioning from the on deck cards
• Fixed: Fixed a typo on the localization string for Create Multiple Reading List/Collection pages
• Fixed: Fixed Promote/Unpromote not working from reading list detail page and collection detail page
• Fixed: Ensure we show details page when weblinks are present, including chapter detail page
• Fixed: Fixed related tab not having enough bottom padding
• Fixed: Fixed a few different combinations where IsEmpty was not working
• Fixed: Fixed broken Send To on mobile and Send To not actually functioning for chapter/volume cards.
• Fixed: Tweaked read-time-left pipe to have better control on how we present the now decimal number.
• Fixed: Fixed a bug where continue from was choosing ch instead of vol
• Fixed: Fixed read-more text not displaying newlines correctly in some situations (Thanks @Fesaa)
• Fixed: Fixed the search bar closing when toggling show chapters and files toggle (Thanks @Fesaa)
• Fixed: Fixed a bug where validation wasn't showing on Hostname
• Fixed: Next expected card had layout issues and missing localization
• Fixed: Don't close a modal when a typeahead is open and pressing Escape key
• Fixed: Fixed tooltips not rendering on some cards, like reading list/collection cards
• Fixed: Fixed a missing case to show storyline tab on Light Novel libraries when users use chapters (not common)
• Fixed: Ensure the edit series modal is showing the localized publication status
• Fixed: Updated a lot of missing localization keys
• Fixed: Fixed the fade between colorscape so there isn't a reset to default anymore
• Fixed: Fixed an issue with how specials render on reading list detail page
• Fixed: Fixed the duplicate Dark theme issue once and for all
• Fixed: Fixed custom tasks not being possible
• Fixed: Fixed Irish not working when selected as locale
• Fixed: Fixed edit chapter modal failing to save when setting Age Rating
• Fixed: Fixed a bug where when a card doesn't have a format, 22px was still beating taken up.
• Fixed: Ensure More In Genre doesn't choose from a Genre that's not exposed to the dashboard
• Fixed: Fixed a bug where bulk mode on customize side nav wasn't laying out the bulk update well.
• Fixed: Fixed delete chapter not being hooked up from Volume details page
• Fixed: Fixed user rating filter not working at all. It was accepting a percentage, but should have been a number 0-5 for the star rating.
• Fixed: Fixed the bulk actions bar laying out in the wrong position
• Fixed: In the rare occasion someone turns off all the file types for a library, the scanner will now respect that.
• Fixed: Fixed Delete and Analyze Files on the library detail actionables not being hooked up
• Fixed: Fixed an issue with doujin filenames causing series to all parse as (
• Fixed: (Kavita+) Fixed a bug where old Book libraries could be stuck in Scrobbling On state
• Fixed: Fixed age rating icon not updating upon series metadata changes
• Fixed: Ensure Jumpkeys are grouped non-case sensitive
• Fixed: Fixed a bug where want to read page was using a custom query passed through the filtering pipeline, thus breaking some combos.
• Fixed: Fixed a bug where manage email settings was spamming the backend due to an infinite loop
• Fixed: Fixed select all on edit user modal affecting the role/library control as well.
• Fixed: On Age Rating filter, the comparisons could be duplicated
• Fixed: Fixed a bug with bulk selection not pruning submenu actions when there are no children left. (aka empty send to)

API

• Search now returns a list of PersonDto instead of strings for People results.
• Removed metadata/chapter-summary api since it hasn't been used in years

Theme

• Side nav selected tab now uses --side-nav-item-active-text-color on the text instead of just being white
• Added --side-nav-width variable to control side nav bar width
• Added --navbar-header-margin, set it to 8px and --nav-offset to 56px to get the Plex floating navbar.
• Added --navbar-border-radius to fix the navbar but let users customize for that Plex look.
• Added --navbar-header-mobile-x-margin and --navbar-header-mobile-y-margin to allow for the Plex navbar on mobile

Known Issues

• (Kavita+) MAL Stack imports/Smart Collection total series count is not correct. This will be fixed with v0.8.5.
• Uploading some dimensions for person cover can cause the image to be squished. This will be fixed with v0.8.5.
• When using multiple reading progress filters, some series with 100% reading progress will show (clicking On Deck). After 14 hours, I can't solve it. I will loop around in v0.8.5.

---

This discussion was created from the release v0.8.4 - New Scanner + Browse Authors/Artists + Tons of Bugfixes.

[Aloe-recite]

Hi there,
I just updated to 0.8.4 and did a rescan of my only library but am facing a few big issues!
The authors page is not populated at all and all my authors disappeared from the "edit" → "people" mask, despite the line having the green lock. If I add them in again, they are not saved if I save, close and reopen the edit screen, not to mention that a re-scan doesn't pick them up. However, I noticed that if I add them to a comic that didn't have any people before, it works correctly.
Also, I'm missing some strings (in Italian - especially the left side bar on the edit screen) and the "bust cache" option didn't do anything, had to switch to English for everything to work (which is not a big deal, I prefer English anyway but still felt like reporting). I tried deleting browser cache and all to no avail. Is there anything I am missing? Thanks and keep up the good work!

[majora2007]

Do you have any metadata in your files or did you manually tag them prior?

[Aloe-recite]

@majora2007 previously manually tagged in Kavita under the previous version, up to 50% or so of my library. They're a mixture of CBZ (downloaded with HakuNeko if that matters) and PDF, not sure about files metadata though?

[majora2007]

Okay so anything that was manually tagged in the UI won't come back. A user wrote a script that might work for you. Grab a backup db (config/backups) and run it on your new db. This will attempt to port over the metadata.

https://pastebin.com/RPmfSSYq

For the future, I highly recommend using something like MangaManager or Komf with Archive mode to write metadata IN the files.

[Aloe-recite]

@majora2007 thanks for the clarification. Let me make constructive criticism though. I'm a bit disappointed that all my tagging efforts are put to waste, honestly. The only warning in the changelog is that "People metadata will be removed on update. A force scan is required to repopulate in the new system", which implies that a migration system of sorts would happen automatically. But since this is not the case, a clearer explanation and possibly a guide to circumvent this would have been really helpful (and save my time trying, and your time for questions like mine 😄 ).
I chose Kavita as my sole library manager also for its usability, now I'm not enthusiastic about having to rely on third party tools to get back to where I was. Komf never worked for me (I tried sometime ago to install it, but it never did anything despite following closely the instructions, I probably don't understand something important) and using the script is above my current technical level, needing to install node, sql3, etc. so I'll try anyway but with low expectations. Now I'm manually re-adding all authors and drawers, but at 200+ comics it will take days, and most of all there is still the bug that prevents me from re-adding the same author to two comics (the second time I add it I get an "400 Error Could not update metadata" and cannot save the edits, thus breaking the usefulness of tracking authors since I cannot benefit from seeing all series from the same one).
I hope I didn't come through as demanding. It's not at all my scope - just wanted to give honest feedback from a heavy (but casual - not too technical) user perspective (I use kavita to read mangas A LOT).
keep up the good work! 💪🏻

[majora2007]

@Aloe-recite I totally understand your point. It's frustrating to me as well because for me, this people feature (which was also heavily requested and some something I have as my goals for this year), never crossed my mind as a big data loss. I never thought about users that tag manually through the UI. The testers that we have in the discord that are active, all tag within the files. I tag within the files, so when I was doing testing and validation with the users, it was an annoying setback, but not a deal breaker.

Only after release did I see how people are using Kavita. I'm only one person building a massive piece of software outside of work, so I can only do so much and have so much insight into how people use it. It's really heartbreaking for me as well to see so many people put in a tough spot, as in the last few releases, especially with this last one (rebuilding the scanner is extremely time consuming and not fun), I put a ton of effort in and it feels diminished as people are hitting so many issues.

I have a hotfix planned to be released today which will fix the metadata update UI. I appreciate the criticism and I can say confidently that this was an eye opening release for me. I will continue to try my best to ensure the future releases minimize data loss.

On another note, you should drop by the discord and ask for support in the komf channel. Komf is pretty technical with little documentation, but we have a great community of users that are active and helpful. If you also feel like you want to help prevent issues like this, please join our nightly testers (usually a separate container). These people are responsible to help test features before the go to stable and shape a lot of the features.