CommunityToolkit.Storage: A modern storage abstraction #229
Replies: 7 comments 19 replies
-
Those interfaces (and others) could be named - |
Beta Was this translation helpful? Give feedback.
-
It is an excellent initiative. But why not replace IAddressableFile and IAddressableFolder with IAddressableStorage? |
Beta Was this translation helpful? Give feedback.
-
Proposal Amendment 1Getting folder contentsSome people have been asking why we went with 2 methods ( After some discussion and brainstorming I was able to improve a bit on this (and have updated the proposal) What are we expecting to work with?When it comes to file systems, there are 2 ways to index things, and performance considerations for both Types: Ideally, our goals are:
Which flavor has best perf?Looking at our options, there's a major issue with both:
To solve both of these, we need to provide all 3 options: IAsyncEnumerable<IStorable> GetItemsAsync();
IAsyncEnumerable<IFile> GetFilesAsync();
IAsyncEnumerable<IFolder> GetFolderAsync(); As long as the underlying implementation is done correctly, this approach is as fast as possible in all scenarios, leaving absolute performance up to the caller. The "worst case" performance scenario becomes when the underlying filesystem only supports getting all files, and the caller decides to ask for both separately (double enumeration), but that's up to the caller to avoid doing. They have the option to ask for all items, which is almost certainly more performant. But now we've left a mess for the implementor, so let's clean it up. Consolidating methods[Flags]
public enum StorableKind
{
None = 0, // Required because it's a flag
File = 1, // Only get files
Folder = 2, // Only get folders
All = File | Folder, // Get all items.
}
public interface IFolder
{
// ...
IAsyncEnumerable<IStorable> GetItemsAsync(StorableKind kind = StorableKind.All)
{
// Call any underlying methods you need, and use the most performant option.
}
}
// Not pictured: extension methods for GetFoldersAsync and GetFilesAsync that cast the return type to File/Folder Change summary
|
Beta Was this translation helpful? Give feedback.
-
Accessing file contentsCurrently, there's no way to open a file and restrict other processes from reading/writing to it as well. That exposes the software to multiple unexpected error cases where one process reads and parses the data whilst the second one is writing to it. Two processes writing to the same file at the same time can also result in data corruption. To mitigate these cases, we can implement a FileShare flag. Using
|
Beta Was this translation helpful? Give feedback.
-
Traversing directoriesThe current Example APITask<IFile> GetFileAsync(string fileName, CancellationToken cancellationToken = default);
Task<IFolder> GetFolderAsync(string folderName, CancellationToken cancellationToken = default); We can simplify it further for the implementor by reducing it to one method: Task<IStorable> GetItemAsync(string itemName, StorableKind kind, CancellationToken cancellationToken = default); However with the above approach, StorableKind.All flag would have to be removed or ignored (ArgumentException could also be thrown). The remaining GetFileAsync and GetFolderAsync methods would be polyfilled with extensions for simplicity. |
Beta Was this translation helpful? Give feedback.
-
Proposal Amendment 2Standardizing storage propertiesOriginal post has been updated. Full explainer below. As stated above in the original post, we have 3 requirements for properties:
Originally, we had said:
Rather than leaving this completely in the air, we (in the UWP Community Discord) brainstormed what a "best approach" might look like, and made it generic enough for any set of properties to use. We now have a basic skeleton that serves as a guide for any future property set. // The recommended pattern for file properties.
public interface IStorageProperties<T> : IDisposable
{
public T Value { get; }
public event EventHandler<T> ValueUpdated;
}
// The container for individual music properties.
public class MusicData
{
// todo
}
public interface IMusicProperties
{
// Must be behind an async method
public Task<IStorageProperties<MusicData>> GetMusicPropertiesAsync();
}
// If the implementor is capable, they can implement modifying props as well.
public interface IModifiableMusicProperties
{
public Task UpdateMusicProperties(MusicData newValue);
} Doing it this way means
Names are not yet final, please offer suggestions if you have them! |
Beta Was this translation helpful? Give feedback.
-
@Arlodotexe I know there's been more discussion on Discord lately than what's been mirrored here. Maybe good for another update or a link to other places to stay in the loop? I know one question from the 8.0 release notes that came up for WinUI3/WASDK specifically was the packaged vs. unpackaged paradigm, so not sure if you want to touch on that anywhere as well, if that's been tested against any of the working implementation you have in OwlCore? |
Beta Was this translation helpful? Give feedback.
-
Motivation
Too many file systems. No standard abstractions.
Listing a few off the top of my head: (click to expand)
On-disk APIs
Cloud storage
Network protocol
Blob storage
Plus countless custom HTTP based file systems. These are usually created in-house for a specific project, but they still count.
The problem with this
History
This started out as an experiment called AbstractStorage. It was a 1:1 recreation of the most used parts of the StorageFile APIs, but in .NET Standard 2.0, and was created in Strix Music v2 before being moved into OwlCore, which enabled others to start using it and giving feedback.
Over time, we gathered feedback from those in the UWP Community who were using it: Fluent Store, Strix Music, Quarrel, ZuneModdingHelper, etc.
We learned a lot from the feedback we got, but the AbstractStorage experiment is now over. The next stage is in this proposal.
Proposal
We'll use Labs to build a significantly improved version of AbstractStorage under the name
CommunityToolkit.Storage
, and work to bring our experiment to the .NET Community Toolkit.While this seems easy on the surface, we can't just turn the Windows StorageFile API into a set of ns2.0 interfaces and call it a day. We learned from our first version that this approach has issues once we put them to purpose.
We can do better.
Minimum end goal
Without sacrificing potential functionality or performance, we should strive to:
Specification
if (instance is IMyThing thing)
is faster and more natural than catching thrownNotSupportException
s.The starting point
Not all file systems are created equal, but they do share some common traits. To maximize compatibility, this should be our starting point.
Even with just this, you can do a LOT. Keep in mind, these work with any local disk API, with any Cloud provider, with FTP, SCP, SMB, IPFS, HTTP and pretty much anything else. Plus -- you can mock them for unit tests.
Some examples of things I've made with just this:
Now that we have this, we can build on top of it.
Adding addressability
Sometimes, there is no folder
There are 3 ways to obtain a file.
A file, in its most basic form, must be a unique resource with a data stream and a name.
This description perfectly matches both in-memory files and some custom file types like blobs - none of which have a "containing" folder.
Examples of this:
(click to expand):
IFileAbstraction
. You can do this 100% in memory if you need to. No path, no parent folder.Even when there's no folder structure built around it, a file can exist standalone. In some edge cases, a file could even exist in multiple folders at once!
For full compatibility, we need to address this.
Proposed solution
These interface names are not final, please offer suggestions!
Current name candidates
IAddressableFile
ILocatableFile
Any APIs meant to be used in the context of an identifiable folder structure should be abstracted to a new interface:
Storage properties
"Storage properties" are the additional information about a resource provided by most file systems. It's everything from the name to the "Last modified" date to the Thumbnail, and the majority of them can change / be changed.
In AbstractStorage, we simply copied the c# properties and recreated
StorageFileContentProperties
, and even though we only addedGetMusicPropertiesAsync
and made the property nullable, it caused a lot of confusion for everyone else who implemented it.We'll continue using our strategy of "separate interfaces" for this. However, there are a LOT of storage properties we can add, so as long as we separate them into different interfaces, can safely leave "which" for later and instead figure out the "how".
There are 3 requirements
This gives us 2 options:
SetThingAsync
methodsGetThingAsync
Methods +SetThingAsync
methodsAfter some brainstorming, we have a basic skeleton that serves as a guide for any property set.
Doing it this way means
UpdateMusicPropertiesAsync()
vsChangeNameAsync()
).Names are not yet final, please offer suggestions if you have them!
Manipulating folder contents
Since folders are responsible for holding files, folders should also be responsible for manipulating their own contents. Also, as previously established, a File can exist without the context of a folder - an important detail for Copy and Move operations.
When designing file system APIs, most implementations don't do this, but given the above, we're 100% sure that we want this method to be implemented on the folder and not the file.
We'll use that for the interface, and have an extension method that swaps them around to something like this:
Wrapping up
Our original requirements aimed to:
The above proposal allows for:
Requirements have definitely been met! Now on to the next part.
Remaining work to do
File / Folder properties
Notice that we left out implementation details for properties entirely. This is a large, complex space, and now that we know the requirements and options, we can start to gather decent suggestions for a given property or set of properties.
We might skip this entirely for the first PR and just do the basics, since adding a new interface is not a breaking change.
Gather feedback & refine
(Make sure you've read the full proposal first)
Add your feedback, ask questions, make suggestions. If we can improve this, let's do it!
Beta Was this translation helpful? Give feedback.
All reactions