Massive lag/TPS drops when generating structures

[beltxd]

https://spark.lucko.me/c4ue0a6JEU

Tested with the latest version of RLD, looking into it. Structure generation completely freezes world for minutes at a time. (NOT CAUSED BY GEOGRAPHICRAFT)

[al-wei]

In my modpack, it takes over 70 seconds to generate a dungeon.

If the world is created at the same time as the dungeon.
Trying to spawn dungeon at chunkX 0 and chunkZ 0...
It only takes 2 minutes to create the world.
But if there is a dungeon in the process, it will take longer than 5 minutes.

[al-wei]

In my modpack, it takes over 70 seconds to generate a dungeon. Trying to spawn dungeon at chunkX 0 and chunkZ 0... If the world is created at the same time as the dungeon. It only takes 2 minutes to create the world. But if there is a dungeon in the process, it will take longer than 5 minutes.

This problem may itself exist in the Greymerk version.
Dungeon generation has to be optimized,

[KameiB]

In my modpack, it takes over 70 seconds to generate a dungeon. Trying to spawn dungeon at chunkX 0 and chunkZ 0... If the world is created at the same time as the dungeon. It only takes 2 minutes to create the world. But if there is a dungeon in the process, it will take longer than 5 minutes.

This problem may itself exist in the Greymerk version. Dungeon generation has to be optimized,

Hmmm
Generating a dungeon by command usually takes around 1 second (on vanilla) and sometimes up to 3 seconds in RLCraft.
While there's always a room for improvement, it would help more if you share the mods included in your pack, so fnar can narrow the problem more easily.

@fnar how do you feel about sharing a list of mods that currently clash with RLD?
I think that would help both you (so you don't get duplicate reports) and users (so they are warned of current incompatibilities).

[al-wei]

Roguelike Dungeons -- Fnar's Edition
mpver:7.0
time:5
https://www.curseforge.com/minecraft/modpacks/al-nl/files/5832456/changelog
Roguelike Dungeons
mpver:7.1
time:2
https://www.curseforge.com/minecraft/modpacks/al-nl/files/5843891

I deleted version f in version 7.1.
Older version used

Old dungeon, much smoother. There's not that much lag.

[al-wei]

This is a very sure thing.
The F version does cause a lot of performance problems.

[al-wei]

If the old Roguelike Dungeons still has problems. Then this integration package can only delete all content related to Roguelike Dungeons.

There are already a lot of people complaining about slow loading.
Including non-alnl modpack players.

[beltxd]

The greymark ver of roguelikedungeons have no issues of generating for me. I think there is an incompatibility with Geographicraft.

[al-wei]

Geographicraft may be in part. But alnl modpack only recently joined Geographicraft.
I think there are other unknowns.

[al-wei]

The greymark ver of roguelikedungeons have no issues of generating for me. I think there is an incompatibility with Geographicraft.

Yeah, actually, the greymark version has little serious impact. The impact is small enough.

[Deepfriedyag]

This issue is very real. Can confirm:

I don't use Geographicraft btw.

[beltxd]

I've just realized that isn't geographicraft while trying to reproduce it, it must be another mod influencing it. For my spark profiler it did say geographicraft. I will look more into it myself.

[beltxd]

Nvm, problem still persisted with ksyxis, world gen seems to be slightly better wiithout it, but freezes for upwards for minutes still occur. Deleted my initial reply

[fnar]

Thanks for the attention to detail on this.

There are some adjustments that can certainly impact the generation cost.

For starters, this mod is super un-optimized. Structure generations are not contained to only the chunks that are generating. Instead, it has the very real problem of cascading world-gen, where generating a chunk that spawns a dungeon may cause additional chunks to generate and, you guessed it, possibly cause additional dungeons to spawn, recursively.

As a user, you can ensure that the distance between dungeons is great, by increasing the spawnFrequency value in the roguelike.cfg file. This configuration option essentially means "generate 1 dungeon per NxN chunks". If the value is 5, then expect to have 1 dungeon every 5x5 chunks. The greater the value, the more rare dungeons are, and the less likely that one dungeon spawning will cascade into spawning another.

Limiting the size of dungeons can also reduce the number of chunks that generate and by consequence reduce cascading world gen. This can be accomplished by designing dungeons with shorter hallways and fewer rooms. In general, the smaller the dungeon, the fewer chunks it affects during generation, the fewer chunks it cascades into generating.

Nextly, the cost of dungeon generation is expensive in general. This is because of the recursive nature of resolving "dungeon settings." Think of it like how code looks up functions by name in a function look-up table. Dungeons settings reference each other by name, so each generation requires building the entire context and compiling it to ensure that the dungeon generates correctly using the latest setting. It isn't cheap. The more dungeon settings files you have, the more expensive it can be. If you're using the mod out-of-the-box, it's generally cheaper than using a large set of dungeon-settings files.

Nextly, of course, dungeon/chunk generation can be dramatically sped up by having a super powerful processor. This isn't a realistic option for most folks, but if you have the option to choose between a server with a poor CPU or a mega CPU know that this will impact chunk generation time, especially while using this mod.

What are our paths forward?

1. Cascading world gen is a serious problem in this mod but not an easy fix.
   Here's a resource on the issue: https://www.reddit.com/r/feedthebeast/comments/5x0twz/investigating_extreme_worldgen_lag/

2. Additional optimization. This would require additional profiling and targeted improvements.

As always, the pull requests are open.

[fnar]

@KameiB :

@fnar how do you feel about sharing a list of mods that currently clash with RLD?
I think that would help both you (so you don't get duplicate reports) and users (so they are warned of current incompatibilities).

I don't have a list on hand as most conflicts tend to be false-reports.

[Deepfriedyag]

Thanks for the attention to detail on this.

There are some adjustments that can certainly impact the generation cost.

For starters, this mod is super un-optimized. Structure generations are not contained to only the chunks that are generating. Instead, it has the very real problem of cascading world-gen, where generating a chunk that spawns a dungeon may cause additional chunks to generate and, you guessed it, possibly cause additional dungeons to spawn, recursively.

As a user, you can ensure that the distance between dungeons is great, by increasing the spawnFrequency value in the roguelike.cfg file. This configuration option essentially means "generate 1 dungeon per NxN chunks". If the value is 5, then expect to have 1 dungeon every 5x5 chunks. The greater the value, the more rare dungeons are, and the less likely that one dungeon spawning will cascade into spawning another.

Limiting the size of dungeons can also reduce the number of chunks that generate and by consequence reduce cascading world gen. This can be accomplished by designing dungeons with shorter hallways and fewer rooms. In general, the smaller the dungeon, the fewer chunks it affects during generation, the fewer chunks it cascades into generating.

Nextly, the cost of dungeon generation is expensive in general. This is because of the recursive nature of resolving "dungeon settings." Think of it like how code looks up functions by name in a function look-up table. Dungeons settings reference each other by name, so each generation requires building the entire context and compiling it to ensure that the dungeon generates correctly using the latest setting. It isn't cheap. The more dungeon settings files you have, the more expensive it can be. If you're using the mod out-of-the-box, it's generally cheaper than using a large set of dungeon-settings files.

Nextly, of course, dungeon/chunk generation can be dramatically sped up by having a super powerful processor. This isn't a realistic option for most folks, but if you have the option to choose between a server with a poor CPU or a mega CPU know that this will impact chunk generation time, especially while using this mod.

What are our paths forward?

1. Cascading world gen is a serious problem in this mod but not an easy fix.
   Here's a resource on the issue: https://www.reddit.com/r/feedthebeast/comments/5x0twz/investigating_extreme_worldgen_lag/

2. Additional optimization. This would require additional profiling and targeted improvements.

As always, the pull requests are open.

I don't get this massive lag when I switch to the Greymerk version instead of Fnar. When using the Fnar version, the chunks that the dungeon is trying to spawn in can take up to 5 minutes to load and while this happens you can't do anything or load any other chunks because of the immense tick lag this introduces. Just to let you know in case it helps.

[fnar]

In my modpack, it takes over 70 seconds to generate a dungeon.

If the world is created at the same time as the dungeon. Trying to spawn dungeon at chunkX 0 and chunkZ 0... It only takes 2 minutes to create the world. But if there is a dungeon in the process, it will take longer than 5 minutes.

@al-wei I've been thinking about this for a while. In your logs, you should see a message like the following:

[23:03:43] [Server thread/INFO] [roguelike]: Trying to spawn dungeon with id dungeon:generic at {"x": 11214, "y": 0, "z": 10068}...
[23:03:45] [Server thread/INFO] [roguelike]: Successfully generated dungeon with id dungeon:generic at {"x": 11214, "y": 0, "z": 10068}.
[23:03:45] [Server thread/INFO] [roguelike]: Task Dungeon.generate() completed in 2107 milliseconds.

You mention your world startup takes 5 minutes -- how long does RLD:FE believe it's taking to generate dungeons? Can you clear your logs, generate a new world, and then send me the logs? (in case there are multiple dungeons generating)

[al-wei]

In my modpack, it takes over 70 seconds to generate a dungeon.
If the world is created at the same time as the dungeon. Trying to spawn dungeon at chunkX 0 and chunkZ 0... It only takes 2 minutes to create the world. But if there is a dungeon in the process, it will take longer than 5 minutes.

@al-wei I've been thinking about this for a while. In your logs, you should see a message like the following:

[23:03:43] [Server thread/INFO] [roguelike]: Trying to spawn dungeon with id dungeon:generic at {"x": 11214, "y": 0, "z": 10068}...
[23:03:45] [Server thread/INFO] [roguelike]: Successfully generated dungeon with id dungeon:generic at {"x": 11214, "y": 0, "z": 10068}.
[23:03:45] [Server thread/INFO] [roguelike]: Task Dungeon.generate() completed in 2107 milliseconds.

You mention your world startup takes 5 minutes -- how long does RLD:FE believe it's taking to generate dungeons? Can you clear your logs, generate a new world, and then send me the logs? (in case there are multiple dungeons generating)

You can try the 7.0 modpack link above for yourself
You can feel it for yourself. I don't have the right environment to test now.

With the mod pack update i removed the content about rld: fe and used rld

[fnar]

This issue is very real. Can confirm:

I don't use Geographicraft btw.

https://spark.lucko.me/c4ue0a6JEU

Tested with the latest version of RLD, looking into it, there are some issues with Geographicraft (also latest version). Might wanna look into this. Structure generation completely freezes world for minutes at a time.

Still digging into this. Exploring @beltxd's profiling tool, I observe that while RLD is at the top of the call stack that's at 45%, it quickly hands off 44% to Climate Control. What determined that RLD was the culprit here? (It certainly has flaws, just trying to find the evidence in this case.)

[al-wei]

This issue is very real. Can confirm:

I don't use Geographicraft btw.

https://spark.lucko.me/c4ue0a6JEU
Tested with the latest version of RLD, looking into it, there are some issues with Geographicraft (also latest version). Might wanna look into this. Structure generation completely freezes world for minutes at a time.

Still digging into this. Exploring @beltxd's profiling tool, I observe that while RLD is at the top of the call stack that's at 45%, it quickly hands off 44% to Climate Control. What determined that RLD was the culprit here? (It certainly has flaws, just trying to find the evidence in this case.)

I'm pretty sure it's not CC/CG, because my modpack just joined CC/CG a while ago, but the lag problem did exist before.

You can delete cc/cg or even rtg, etc., in the test.

[Deepfriedyag]

This issue is very real. Can confirm:

I don't use Geographicraft btw.

https://spark.lucko.me/c4ue0a6JEU
Tested with the latest version of RLD, looking into it, there are some issues with Geographicraft (also latest version). Might wanna look into this. Structure generation completely freezes world for minutes at a time.

Still digging into this. Exploring @beltxd's profiling tool, I observe that while RLD is at the top of the call stack that's at 45%, it quickly hands off 44% to Climate Control. What determined that RLD was the culprit here? (It certainly has flaws, just trying to find the evidence in this case.)

You can see in my profiling I don't use that mod either and there is nothing that causes any tick lag anything close to RLD: fnar (RLD at %32 while the closest mod to that at %4 of the whole %42 caused by world gen in total.) Also it is the DungeonGenerator_1.12.Generate() method. I don't understand much about modding minecraft and java but I do have some kind of programming knowledge and this seems to be some kinda fault in your code (somehow idk). Even though none of the mods mentioned by others in this thread were in it, I was still using a relatively big modpack so this might very well be a mod incompatibility.

[fnar]

@al-wei , I don't have the answer, but I have a strong lead. Adding more logging to RLD:FE yields this using your modpack (during world creation):

[00:13:46] [Server thread/INFO]: Preparing spawn area: 22%
[00:13:48] [Server thread/INFO]: Trying to spawn dungeon at chunkX 0 and chunkZ 0...
[00:18:24] [Server thread/INFO]: Task Dungeon.isTooCloseToStructures() completed in 275458 milliseconds.
[00:18:25] [Server thread/INFO]: Task Dungeon.isTooCloseToStructures() completed in 751 milliseconds.
[00:18:26] [Server thread/INFO]: Task Dungeon.isTooCloseToStructures() completed in 581 milliseconds.
[00:18:26] [Server thread/INFO]: Task Dungeon.isTooCloseToStructures() completed in 640 milliseconds.
[00:18:27] [Server thread/INFO]: Task Dungeon.isTooCloseToStructures() completed in 612 milliseconds.
[00:18:27] [Server thread/INFO]: Task Dungeon.isTooCloseToStructures() completed in 607 milliseconds.
[00:18:28] [Server thread/INFO]: Task Dungeon.isTooCloseToStructures() completed in 566 milliseconds.
[00:18:29] [Server thread/INFO]: Task Dungeon.isTooCloseToStructures() completed in 563 milliseconds.
[00:18:29] [Server thread/INFO]: Task Dungeon.isTooCloseToStructures() completed in 559 milliseconds.
[00:18:30] [Server thread/INFO]: Task Dungeon.isTooCloseToStructures() completed in 600 milliseconds.
[00:18:30] [Server thread/INFO]: Task Dungeon.selectLocation() completed in 281149 milliseconds.
[00:18:30] [Server thread/INFO]: Task Dungeon.generateInChunk() completed in 282198 milliseconds.
[00:18:30] [Server thread/INFO]: Preparing spawn area: 25%

The function isTooCloseToStructures, when given a chunk/pos checks if the location is a good candidate for dungeon generation, specifically whether the position is sufficiently far enough away from the structures enumerated in rogue.cfg in the config field vanillaStructuresToCheckMinimumDistanceFrom.

This function works by delegating to Forge's function ChunkProvider.getNearestStructurePos() (rld reference). From RLD's perspective, this is just a hand-off. The unoptimized code exists in Forge. The best we can do is simply not invoke it.

In your modpack, you have configured the following value in rogue.cfg:
vanillaStructuresToCheckMinimumDistanceFrom=Stronghold,Mansion,Monument,Village,Mineshaft,Temple

These should be checked in order, so it's likely the case that when checking for Stronghold during world creation/dungeon generation is taking too long.

The simplest answer is to not use this feature and clear the value. This is something you would do in your configuration by setting the value to nothing (the empty string) any other text that isn't a vanilla structure (e.g. "nothing"). I'd wager this would also work for @Deepfriedyag and @beltxd .

On my end, I can release an updated version which exposes the durations of common operations so that we can dig into optimization more.

Analogously, I could also modify the default value for which structures are avoided so that others don't face this problem with their packs.

Mind you, using this mod in isolation has no performance problem with this check, so there is likely still another mod contributing to each of your problems, such as an additional worldgen mod.

---

I tested this approach using your pack and this was the result:

[00:55:06] [Server thread/INFO]: Preparing spawn area: 33%
[00:55:07] [Server thread/INFO]: Trying to spawn dungeon at chunkX 0 and chunkZ 0...
[00:55:08] [Server thread/ERROR]: Invalid value for config `vanillaStructuresToCheckMinimumDistanceFrom`: "NTHX". Skipping...
[00:55:08] [Server thread/INFO]: Task Dungeon.isTooCloseToStructures() completed in 4 milliseconds.
[00:55:08] [Server thread/ERROR]: Invalid value for config `vanillaStructuresToCheckMinimumDistanceFrom`: "NTHX". Skipping...
[00:55:08] [Server thread/INFO]: Task Dungeon.isTooCloseToStructures() completed in 0 milliseconds.
[00:55:08] [Server thread/ERROR]: Invalid value for config `vanillaStructuresToCheckMinimumDistanceFrom`: "NTHX". Skipping...
[00:55:08] [Server thread/INFO]: Task Dungeon.isTooCloseToStructures() completed in 0 milliseconds.
[00:55:08] [Server thread/INFO]: Task Dungeon.selectLocation() completed in 323 milliseconds.
[00:55:08] [Server thread/INFO]: Trying to spawn dungeon with id fnar:dungeon_cave at {"x": -262, "y": 0, "z": 62}...
[00:55:14] [Server thread/INFO]: Successfully generated dungeon with id fnar:dungeon_cave at {"x": -262, "y": 0, "z": 62}.
[00:55:14] [Server thread/INFO]: Task Dungeon.generate() completed in 6092 milliseconds.
[00:55:14] [Server thread/INFO]: Task Dungeon.generateInChunk() completed in 7533 milliseconds.
[00:55:15] [Server thread/INFO]: Preparing spawn area: 36%

From this, I reason the issue lies within a configuration / mod conflict issue.

[Deepfriedyag]

The function isTooCloseToStructures, when given a chunk/pos checks if the location is a good candidate for dungeon generation, specifically whether the position is sufficiently far enough away from the structures enumerated in rogue.cfg in the config field vanillaStructuresToCheckMinimumDistanceFrom.

This function works by delegating to Forge's function ChunkProvider.getNearestStructurePos() (rld reference). From RLD's perspective, this is just a hand-off. The unoptimized code exists in Forge. The best we can do is simply not invoke it.

In your modpack, you have configured the following value in rogue.cfg:
vanillaStructuresToCheckMinimumDistanceFrom=Stronghold,Mansion,Monument,Village,Mineshaft,Temple

These should be checked in order, so it's likely the case that when checking for Stronghold during world creation/dungeon generation is taking too long.

The simplest answer is to not use this feature and clear the value. This is something you would do in your configuration by setting the value to nothing (the empty string) any other text that isn't a vanilla structure (e.g. "nothing"). I'd wager this would also work for @Deepfriedyag and @beltxd .

How exactly do I turn this feature off? When I delete the "Stronghold,Mansion,Monument,Village,Mineshaft,Temple" thing it comes back after i launch the game. @fnar

[fnar]

How exactly do I turn this feature off? When I delete the "Stronghold,Mansion,Monument,Village,Mineshaft,Temple" thing it comes back after i launch the game. @fnar

Yeah, sorry about that. That's a quirk of how the original mod handles configuration to ensure that default values are present.

If it's blank, the value will reset to the default.

To "clear it out" just put any noise into it, such as the text string "nothing" (without quotes). With this, a value will be present and it won't reset to the default (even though the value is invalid).

[Deepfriedyag]

The function isTooCloseToStructures, when given a chunk/pos checks if the location is a good candidate for dungeon generation, specifically whether the position is sufficiently far enough away from the structures enumerated in rogue.cfg in the config field vanillaStructuresToCheckMinimumDistanceFrom.
This function works by delegating to Forge's function ChunkProvider.getNearestStructurePos() (rld reference). From RLD's perspective, this is just a hand-off. The unoptimized code exists in Forge. The best we can do is simply not invoke it.
In your modpack, you have configured the following value in rogue.cfg:
vanillaStructuresToCheckMinimumDistanceFrom=Stronghold,Mansion,Monument,Village,Mineshaft,Temple
These should be checked in order, so it's likely the case that when checking for Stronghold during world creation/dungeon generation is taking too long.
The simplest answer is to not use this feature and clear the value. This is something you would do in your configuration by setting the value to nothing (the empty string) any other text that isn't a vanilla structure (e.g. "nothing"). I'd wager this would also work for @Deepfriedyag and @beltxd .

How exactly do I turn this feature off? When I delete the "Stronghold,Mansion,Monument,Village,Mineshaft,Temple" thing it comes back after i launch the game. @fnar

Yeah, sorry about that. That's a quirk of how the original mod handles configuration to ensure that default values are present.

If it's blank, the value will reset to the default.

To "clear it out" just put any noise into it, such as the text string "nothing" (without quotes). With this, a value will be present and it won't reset to the default (even though the value is invalid).

It seems to have fixed the lag issue completely from my testing so far. I think it would just be better if you removed that feature with the distance check all together because so many people use large modpacks on 1.12.2 and the issue seems to stem from using this mod with big modpacks with perhaps a lot of structures. But that is my opinion anyways. It fixed the issue... @fnar

[fnar]

It seems to have fixed the lag issue completely from my testing so far. I think it would just be better if you removed that feature with the distance check all together because so many people use large modpacks on 1.12.2 and the issue seems to stem from using this mod with big modpacks with perhaps a lot of structures. But that is my opinion anyways. It fixed the issue... @fnar

Happy to hear it helped and thanks for the input.

I'll leave this issue open for a little while longer and then close it when traffic dies down.

[fnar]

@Deepfriedyag , @beltxd is there somewhere that I can get a copy of your modpacks?

Are you by chance using RTG and/or Yung's Better Mineshafts?

[beltxd]

@Deepfriedyag , @beltxd is there somewhere that I can get a copy of your modpacks?

Are you by chance using RTG and/or Yung's Better Mineshafts?

Yes I am using Yung's better mineshafts and RTG plus, I will try the advice you just suggested.

[beltxd]

!!!! it runs well! For the past days I was trying to get to the bottom of the issue, and it seems to have worked. I will do more testing, but this is extremely promising, great catch.

[Deepfriedyag]

@Deepfriedyag , @beltxd is there somewhere that I can get a copy of your modpacks?

Are you by chance using RTG and/or Yung's Better Mineshafts?

Not RTG but Yung's Better Mineshafts yeah. Why?

[fnar]

Not RTG but Yung's Better Mineshafts yeah. Why?

I suspect the intersection of these two mods is the source of this issue (just gathering data).

[al-wei]

extremely messy.
RTG can customize a lot of content. Custom generation.

Caves, valleys, lakes, streams...
Mineshafts, Dungeons, Temples, Strongholds...

RTG is bound to cause a variety of causes. But there may be no solution.

[beltxd]

Not RTG but Yung's Better Mineshafts yeah. Why?

I suspect the intersection of these two mods is the source of this issue (just gathering data).

Yung's better mineshaft + RTG work alongside with your ver of RLD, I've tested it the first time around. I think there may be a mod influencing world gen that somehow affects vanilla structure checks, that was what I was trying to boil it down to.

[Deepfriedyag]

Not RTG but Yung's Better Mineshafts yeah. Why?

I suspect the intersection of these two mods is the source of this issue (just gathering data).

Yung's better mineshaft + RTG work alongside with your ver of RLD, I've tested it the first time around. I think there may be a mod influencing world gen that somehow affects vanilla structure checks, that was what I was trying to boil it down to.

I use mods that add a lot of structures like recurrent complex and ancient warfare 2. Those might be worth investigating but sadly I don't have the time for it currently. (in my modpack witchery is the only mod that adds structures which show up when using the /locate command alongside mineshafts etc. This is a little weird as well)

[al-wei]

My modpack has a lot of structure mods.
However, deleting rld:fe will allow people to enter the map in 2 minutes.
If it's a conflict issue, it's really hard to resolve.

[fnar]

Possibly related:
Zeno410/Climate-Control#23

Given, that the stack trace mentioned here in the initial comment contains ClimateControl and findNearestStructure in the call stack:

https://spark.lucko.me/c4ue0a6JEU