Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lmdb: Platform-specific default map size #111

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

lmdb: Platform-specific default map size #111

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 13 additions & 4 deletions zict/lmdb.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

import pathlib
import sys
import platform
from collections.abc import ItemsView, Iterable, Iterator, ValuesView

from zict.common import ZictBase
Expand All @@ -25,7 +26,8 @@ class LMDB(ZictBase[str, bytes]):
directory: str
map_size: int
On Linux and MacOS, maximum size of the database file on disk.
Defaults to 1 TiB on 64 bit systems and 1 GiB on 32 bit ones.
Defaults to 128 GiB on aarch64 and riscv64, 1 TiB on other 64 bit systems like
x86-64 and 1 GiB on 32 bit ones.

On Windows, preallocated total size of the database file on disk. Defaults to
10 MiB to encourage explicitly setting it.
Expand All @@ -48,10 +50,17 @@ def __init__(self, directory: str | pathlib.Path, map_size: int | None = None):

super().__init__()
if map_size is None:
if sys.platform != "win32":
map_size = min(2**40, sys.maxsize // 4)
else:
machine = platform.machine()
if sys.platform == "win32":
map_size = 10 * 2**20
elif machine in ["x86_64", "x64"]:
map_size = 2**40
elif machine in ["i386", "i686", "x86"]:
map_size = 2**30
Comment on lines +56 to +59
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
elif machine in ["x86_64", "x64"]:
map_size = 2**40
elif machine in ["i386", "i686", "x86"]:
map_size = 2**30

Wasn't this already covered by sys.maxsize // 4?

elif machine.startswith("aarch64") or machine.startswith("armv8") or machine.startswith("riscv64"):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
elif machine.startswith("aarch64") or machine.startswith("armv8") or machine.startswith("riscv64"):
elif (
machine.startswith("aarch64")
or machine.startswith("armv8")
or machine.startswith("riscv64")
):
# x86-64 uses at least 48-bit of virtual address space. On other 64-bit platforms like aarch64
# or riscv64, the minimum allowed virtual address is 39-bit so 2**40 allocation will fail.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about 32-bit armv8?

map_size = 2**37
else:
map_size = min(2**40, sys.maxsize // 4)

self.db = lmdb.open(
str(directory),
Expand Down
Loading