Loading dumped search indexes doesn't seems to work (immediately)

[PeterCodar]

After executing the command
sudo docker-compose run --rm search load-search-indexes.sh
which fails, because

Thu May 20 17:04:25 UTC 2021: Load annotation search index...
load-search-indexes.sh: 'annotation' has data already
To delete it first, add the option '--force'.

and followed by the same command with additional --force
sudo docker-compose run --rm search load-search-indexes.sh --force
it doesn't seem to load search indexes.

Executing the command:
admin/check-search-indexes all
shows:

CORE           STATUS  INDEX  DB
editor         OK      0      /0
instrument     --      0      /1007
series         --      0      /13298
place          --      0      /47270
event          --      0      /51087
tag            --      0      /108281
area           --      0      /118518
label          --      0      /202151
cdstub         --      0      /289554
annotation     --      0      /423226
work           --      0      /1503944
artist         --      0      /1824160
release-group  --      0      /2270818
release        --      0      /2882942
url            --      0      /7763345
recording      --      0      /24819420

What kind of command or additional step to setup search indexes
https://github.com/metabrainz/musicbrainz-docker#set-up-search-indexes
is necessary to get the downloaded dump search indexes loaded into the database?

I know that I can rebuild the indexes with
sudo docker-compose exec indexer python -m sir reindex
but this takes at least 5 time longer then downloading the dumped search indexes and their import.

[PeterCodar]

Wait...
After a reboot of my Ubuntu Virtual machine and executing the above
admin/check-search-indexes all
again, I get this:

CORE           STATUS  INDEX     DB
editor         OK      0         /0
instrument     OK      1007      /1007
series         OK      13298     /13298
place          --      47266     /47270
event          --      51082     /51087
tag            --      97684     /108281
area           OK      118518    /118518
label          --      202141    /202151
cdstub         --      289555    /289554
annotation     --      458639    /423226
work           --      1503977   /1503944
artist         --      1824134   /1824160
release-group  --      2270743   /2270818
release        --      2882901   /2882942
url            --      7763159   /7763345
recording      --      24819875  /24819420

My test search for Madonna now return results as expected.

So what do we have to stop/start/restart to get this without reboot of the Virtual Machine?

And why do we don't get every entity with an OK and the same values for INDEX and DB?

[rhorber]

As I'm not a contributor, I can only share my own experiences, hoping they are useful for your case.

I needed to restart the services:
docker-compose restart

In my case, no entity returns OK:

CORE           STATUS  INDEX     DB
editor         OK      0         /0
instrument     --      1007      /1008
series         --      13617     /13861
place          --      47904     /48313
event          --      52185     /53549
tag            --      98414     /110432
area           --      118536    /118548
label          --      205008    /206550
cdstub         --      288872    /288579
annotation     --      470988    /436484
work           --      1524020   /1537932
artist         --      1846263   /1858676
release-group  --      2303505   /2322392
release        --      2926280   /2950239
url            --      7904823   /7971721
recording      --      25104906  /25250664

The database and search indexes dumps are a month apart, so obviously that's the reason in my case.

[PeterCodar]

This are the differences for the two newest search and data dumps from 15.08.2021:

CORE           STATUS  INDEX     DB
editor         OK      0         /0
instrument     --      1007      /1008
series         --      13617     /13882
place          --      47904     /48360
event          --      52185     /53622
tag            --      98414     /110502
area           --      118536    /118561
label          --      205008    /206659
cdstub         --      288872    /288559
annotation     --      470988    /436911
work           --      1524020   /1539899
artist         --      1846263   /1859608
release-group  --      2303505   /2324264
release        --      2926280   /2952698
url            --      7904823   /7977586
recording      --      25104906  /25266439

As you can see, there is not one OK status at all.
I assume this is because the search dump and the data dump are not created/exported at the exact same time.

And you are right:
data full export: latest-is-20210814-001651
search index export: latest-is-20210714-041021

[PeterCodar]

The reason for the outdated search index export was explained from @yvanzo (MetaBrainz Software Engineer) as follows:
https://community.metabrainz.org/t/no-newer-search-index-data-dump-since-2021-07-14/546678/3

Sorry, exporter’s configuration has not been correctly updated after server migration. Search indexes were correctly built but not exported. This has been fixed and dumps are available from main EU server already. Mirrors just started to synchronize, should be done soon.

[orkenstein]

The reason for the outdated search index export was explained from @yvanzo (MetaBrainz Software Engineer) as follows: https://community.metabrainz.org/t/no-newer-search-index-data-dump-since-2021-07-14/546678/3

Sorry, exporter’s configuration has not been correctly updated after server migration. Search indexes were correctly built but not exported. This has been fixed and dumps are available from main EU server already. Mirrors just started to synchronize, should be done soon.

Probably didn't happened yet? I still get non-sync indexes

[yvanzo]

Hi, sorry for not having answered here earlier. The initial issue has been resolved in 2021 at least.

Probably didn't happened yet? I still get non-sync indexes

@orkenstein: What makes you think so?

I just checked that fresh search index dumps are available.

Note that you also need replication with live indexing to be up-to-date before checking it again.