Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

avoid direct calls to ceph API for volume resize triggers #445

Open
balpert89 opened this issue Nov 27, 2024 · 2 comments
Open

avoid direct calls to ceph API for volume resize triggers #445

balpert89 opened this issue Nov 27, 2024 · 2 comments
Labels
enhancement New feature or request

Comments

@balpert89
Copy link

Summary

We have seen continuously increasing memory usage for the libvirt provider caused by left open connections to the ceph API (see #444).

The idea is to avoid direct calls to ceph API from libvirt-provider entirely. This provides some advantages, prominently a cleaner implementation and less external dependencies.

This issue is meant to pool ideas how to accomplish this. Current ideas for approaches:

  • using QEMU as an event trigger for resizes: QEMU uses an implementation for ceph volumes anyway, it may be possible to leverage this.
  • extending the machine runtime interface (IRI): extending the machine-iri protocol to get updates for volumes may serve as a source for resize triggers but has a bigger impact for current implementations.

Please feel free to add other ideas and to discuss the implications, problems, advantages for each approach.

@balpert89 balpert89 added the enhancement New feature or request label Nov 27, 2024
@github-project-automation github-project-automation bot moved this to Todo in Compute Nov 27, 2024
@lukas016
Copy link
Contributor

libvirt is using hypervisor for getting block device information.
Currently qemu doesn't support capacity and it has report current size: https://github.com/qemu/qemu/blob/7872e5fdf38ac0d8d0083aabb98d67da1f530ef4/block/rbd.c#L100
This size is set during connection and resize.

All QMP query-block... commands returned just current size but i wasn't able get full capacity of rbd image.
I couldn't find any suitable command for our purpose: https://qemu-project.gitlab.io/qemu/interop/qemu-qmp-ref.html#qapidoc-304

    {
      "io-status": "ok",
      "device": "",
      "locked": false,
      "removable": false,
      "inserted": {
        "iops_rd": 0,
        "detect_zeroes": "off",
        "image": {
          "virtual-size": 71940702208,
          "filename": "json:{\"driver\": \"raw\", \"file\": {\"pool\": \"ceph\", \"image\": \"...\'}}",
          "cluster-size": 4194304,
          "format": "raw",
          "dirty-flag": false
        },
        ...

Expected size is 75161927680 bytes

@lukas016
Copy link
Contributor

RBD image has to be resized first with libvirt command. After resize of rbd image OS inside VM can automatically detect change of volume size.

[ 1194.769976] virtio_blk virtio1: [vda] new size: 140509184 512-byte logical blocks (71.9 GB/67.0 GiB)
[ 1194.771600] vda: detected capacity change from 136314880 to 140509184
Nov 27 11:56:21 localhost kernel: virtio_blk virtio1: [vda] new size: 140509184 512-byte logical blocks (71.9 GB/67.0 GiB)
Nov 27 11:56:21 localhost kernel: vda: detected capacity change from 136314880 to 140509184

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Todo
Development

No branches or pull requests

2 participants