Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New buffer API (and support for reading extra bytes in LAS files) #22

Merged
merged 40 commits into from
Nov 2, 2023

Conversation

Mortano
Copy link
Collaborator

@Mortano Mortano commented Sep 7, 2023

This PR introduces a new, improved buffer API to pasture, which is effectively a rewrite of large parts of pasture-core. This will break a lot of existing code, which is unfortunate, but I believe the new API is significantly clearer and more extensible than the old buffer API. Using this new API, new features have been added to pasture-io, such as reading of extra bytes in LAS files. Here is a list of the new features that this PR introduces to pasture:

  • New, unified buffer API with:
    • Simplified names for the most common buffers. InterleavedVecPointStorage becomes VectorBuffer, PerAttributeVecPointStorage becomes HashMapBuffer. Also renamed the per-attribute memory layout into columnar, as it more closely matches e.g. columnar databases, which use a similar memory layout, so hopefully this term is clearer than per-attribute
    • A clear distinction between buffer traits that determine who owns the buffer memory, and the type of memory layout. There is now a hierarchy of three ownership traits (BorrowedBuffer, BorrowedMutBuffer, and OwningBuffer), as well as multiple memory layout traits (InterleavedBuffer, InterleavedBufferMut, ColumnarBuffer and ColumnarBufferMut). This allows e.g. mutable, but not resizable buffer types to be used as generic bounds and makes some code more efficient and/or ergonomic
    • More flexible slice API. There are two new traits SliceBuffer and SliceBufferMut that provide similar functionality to the Index and IndexMut traits in the Rust standard library. Many buffers support slicing, also recursively (i.e. slicing a slice) in a more coherent manner than the previous buffer API
    • Cleaner transition between runtime-typed buffers and statically typed buffers using the PointView and AttributeView types (with mutable variants). Where previously, pasture only supported typed iterators, these views implement accessor functions like at and set_at for strongly typed data. This makes it easier to work with strongly typed point data in pasture
    • A new buffer type that uses an external memory resource as its underlying storage: ExternalMemoryBuffer. Currently, this only supports interleaved data, but it allows interesting optimizations, such as mmap-ing an LAS file and viewing its point records through a pasture point buffer for very fast LAS parsing (see [pasture-io/examples/fast_las_parsing.rs])
    • Conversions between buffers of different memory layouts, using the BufferLayoutConverter type
  • Improvements to LAS parsing:
    • Support for reading extra bytes from LAS files. All extra bytes records are converted to pasture point attributes during parsing
    • Several bugfixes for LAS reading and writing
    • LAZ reader now supports all LAS 1.4 point formats except 9 and 10

Pascal added 30 commits August 14, 2023 11:46
…xed bug when reading LAS files that have leftover bytes between the end of VLRs and the beginning of the point records
@Mortano Mortano merged commit 2d06a87 into main Nov 2, 2023
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant