Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document that high db pool size requires increased shared memory for postgres #260

Open
Nutomic opened this issue Jul 26, 2023 · 4 comments
Labels
bug Something isn't working

Comments

@Nutomic
Copy link
Member

Nutomic commented Jul 26, 2023

Setting a high db pool size causes various sql query errors, most notably logins failing entirely. To fix it, shared memory for postgres needs to be increased with the following line in docker-compose.yml. This should be documented, but not sure where it fits.

postgres:
    image: postgres:15-alpine
    shm_size: 1g
    ...

Original Lemmy issue

Logins on lemmy.ml are consistently failing with known working user/password. Server logs show the following error:

lemmy_1     | 2023-07-26T12:52:38.922771Z  WARN lemmy_server::root_span_builder: incorrect_login: could not resize shared memory segment "/PostgreSQL.1513506408" to 220224 bytes: No space left on device
lemmy_1     |    0: lemmy_api::local_user::login::perform
lemmy_1     |            with self=Login { username_or_email: Sensitive, password: Sensitive, totp_2fa_token: None }
lemmy_1     |              at crates/api/src/local_user/login.rs:16
lemmy_1     |    1: lemmy_server::root_span_builder::HTTP request
lemmy_1     |            with http.method=POST http.scheme="http" http.host=lemmy.ml http.target=/api/v3/user/login otel.kind="server" request_id=1853cb5f-c9c7-4794-bf02-4bbef2ed5c41
lemmy_1     |              at src/root_span_builder.rs:16
lemmy_1     | LemmyError { message: Some("incorrect_login"), inner: could not resize shared memory segment "/PostgreSQL.1513506408" to 220224 bytes: No space left on device, context: SpanTrace [{ target: "lemmy_api::local_user::login", name: "perform", fields: "self=Login { username_or_email: Sensitive, password: Sensitive, totp_2fa_token: None }", file: "crates/api/src/local_user/login.rs", line: 16 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"http\" http.host=lemmy.ml http.target=/api/v3/user/login otel.kind=\"server\" request_id=1853cb5f-c9c7-4794-bf02-4bbef2ed5c41", file: "src/root_span_builder.rs", line: 16 }] }

A lemmy restart fixes the problem temporarily. There is plenty of free disk space on the server.

The problem might be related to the fact that I set the db pool size to 95 because of LemmyNet/lemmy#3661. I lowered it to the previous value of 32 now, lets see if the problem happens again.

Version: 0.18.2

@Nutomic Nutomic added the bug Something isn't working label Jul 26, 2023
@daviessm
Copy link

could not resize shared memory segment "/PostgreSQL.1513506408" is a problem with shared memory (/dev/shm) not disk space. Is postgres constrained in the amount of shared memory it can use, or is there only a small amount on the server?

@Nutomic
Copy link
Member Author

Nutomic commented Jul 26, 2023

df -h shows this (same if executed inside the postgres docker container):

tmpfs            16G     0   16G   0% /dev/shm

Also it looks like lowering db pool size really did fix the issue.

@Nutomic Nutomic changed the title [Bug]: All logins failing [Bug]: All logins failing with high db pool size Jul 26, 2023
@daviessm
Copy link

@Nutomic
Copy link
Member Author

Nutomic commented Jul 27, 2023

Thanks, that works! I added the following to docker-compose.yml as suggested in the stackoverflow link.

postgres:
    image: postgres:15-alpine
    shm_size: 1g
    ...

Gonna move this to the docs repo.

@Nutomic Nutomic transferred this issue from LemmyNet/lemmy Jul 27, 2023
@Nutomic Nutomic changed the title [Bug]: All logins failing with high db pool size Document that high db pool size requires increased shared memory for postgres Jul 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants