p1629 - Standard Text Encoding for C++29

[ThePhD]

Submitted for SG16, Köln, Germany 2019 for inclusion in C++23.

[ThePhD]

SG16 met and talked about the paper. They are okay with the design, but there is a crucial "middle" road for performance that also needs to be considered by this paper:

• Low-level, object-based: encoding objects (okay, maybe try to make it a bit simpler if possible)
• Mid-level, iterators: transcoding iterators / std::text / std::text_view (specifically transcoding, and how to make those faster and not round-trip through UTF32)
• High-level, fast: fast eager functions for encode/decode/transcode (also just fine the way they are)

Low-level and high-level are covered. The paper needs to explicitly talk more about the high-level functionality in detail with extension points.

The mid-level also needs to be explicitly addressed, to allow someone to make fast transcoding iterators and have that plug into the general framework without an insane amount of work. This might work by letting someone write (partial) specializations of std::text::transcoding_iterator<FromEncoding, ToEncoding, ErrorHandler>.

Flesh this out for the Cologne meeting...

[ThePhD]

Köln Feedback:

• Have Tony Tables
• std::text::encoding concept
• std::text::encoding_trait type to expose
• specify minimum range requirements
• think about transcode_view
• need to specify all of encode, decode, and transcode
• add simplification for error handling (too generic, too overgeneralized?)

[ThePhD]

Presentation done at Köln 2019 meeting: https://thephd.github.io/presentations/unicode/sg16/K%C3%B6ln/ThePhD%20-%20K%C3%B6ln%202019%20Standards%20C++%20Meeting%20-%20Catch%20Up.pdf

[meastp]

What's the status of this paper? I find it very valuable and the issue over at cplusplus/papers#420 mentions that it needs a revision...? :)

[ThePhD]

A library is going to be released soon:

https://twitter.com/__phantomderp/status/1355565481416159234/photo/1

I doubt it'll make it into C++23. But we don't really need the Standard to make the library work; it would just be nice if we could stop doing ad-hoc transformations in the Standard itself for dealing with this and just specified it in terms of this, but alas.

[meastp]

Great - I'll have a look when you publish it, especially if it's available through vcpkg ;) I hope this doesn't mean that your standardisation effort of p1629 is abandoned?

[ThePhD]

It just means it'll take more time. I don't think the Standards Committee will be ready to review a paper that large by C++23, honestly.

[npuichigo]

Kindly ask what's the current status. Deferred to C++26?

[ThePhD]

@npuichigo Deferred to C++26 or C++29. I have to rewrite P1629 still, but the library implementation (for both the C and C++-style versions of this) are alive and well:

https://github.com/soasis/text
https://github.com/soasis/cuneicode