[Fix] threthold tolerance error (#6550)

[chunibyo-wly]

Type

• Bug fix (non-breaking change which fixes an issue): Fixes #
• New feature (non-breaking change which adds functionality). Resolves #
• Breaking change (fix or feature that would cause existing functionality to not work as expected) Resolves #

Motivation and Context

fix #6550

Checklist:

• I have run python util/check_style.py --apply to apply Open3D code style
  to my code.
• This PR changes Open3D behavior or adds new functionality.
  • Both C++ (Doxygen) and Python (Sphinx / Google style) documentation is
    updated accordingly.
  • I have added or updated C++ and / or Python unit tests OR included test
    results (e.g. screenshots or numbers) here.
• I will follow up and update the code if CI fails.
• For fork PRs, I have selected Allow edits from maintainers.

Description

Here is the Merge result on this data without tolerance. The coplanar patches didn't be merged. The reason is normal deviate diff threshold is $1$, and the dot product of direcation normal (plane patches) would be $1$ too. So the robust planar test will fail on this.

And also the box size need to be large than zero, but for ideal points, it maybe calculated to zero. So we need to set a tolerance on this too.

the final result on this data

[update-docs]

Thanks for submitting this pull request! The maintainers of this repository would appreciate if you could update the CHANGELOG.md based on your changes.

[ssheorey]

Thanks @chunibyo-wly for this bug fix! Looks good.