Tracking Issue for `utf8_chunks`

[dylni]

Feature gate: #![feature(utf8_chunks)]

This is a tracking issue for an improved API for str::from_utf8.

Public API

// core::str

pub struct Utf8Chunks<'a> { ... }

impl<'a> Utf8Chunks<'a> {
    pub fn new(bytes: &'a [u8]) -> Self;
}

impl<'a> Iterator for Utf8Chunks<'a> {
    type Item = Utf8Chunk<'a>;
}

impl<'a> Clone for Utf8Chunks<'a>;
impl<'a> Debug for Utf8Chunks<'a>;
impl<'a> FusedIterator for Utf8Chunks<'a>;


pub struct Utf8Chunk<'a> { ... }

impl<'a> Utf8Chunk<'a> {
    pub fn valid(&self) -> &'a str;
    pub fn invalid(&self) -> &'a [u8];
}

impl<'a> Clone for Utf8Chunk<'a>;
impl<'a> Debug for Utf8Chunk<'a>;
impl<'a> PartialEq for Utf8Chunk<'a>;
impl<'a> Eq for Utf8Chunk<'a>;

Steps / History

• Proposal: Expose Utf8LossyChunksIter libs-team#54
• Implementation: Expose Utf8Lossy as Utf8Chunks #99544
• Final comment period (FCP)¹
• Stabilization PR

Unresolved Questions

• Should the constructor be Utf8Chunks::new or <[u8]>::utf8_chunks?
• Should Utf8Chunks::debug or a similar method be exposed?

Footnotes

1. https://std-dev-guide.rust-lang.org/feature-lifecycle/stabilization.html ↩

[dtolnay]

I'd be interested in using this to implement Display and Debug for CxxString in the cxx crate. Here is the current implementation without Utf8Chunks:

• https://github.com/dtolnay/cxx/blob/6432e9e5babd1207ecf1259262e023fd3a496bcc/src/cxx_string.rs#L210-L220
• https://github.com/dtolnay/cxx/blob/6432e9e5babd1207ecf1259262e023fd3a496bcc/src/lossy.rs

Here is what it looks like using Utf8Chunks as currently exists in nightly:

• https://github.com/dtolnay/cxx/pull/1289/files

Are there other known use cases so far that we could look at before an FCP? One thing I am interested in is how the current Utf8Chunks API compares with this alternative one, not based on Iterator, with just 1 type:

pub struct Utf8Chunks<'a>;

impl<'a> Utf8Chunks<'a> {
    pub fn next_valid(&mut self) -> &'a str;
    pub fn next_invalid(&mut self) -> &'a [u8];
}

• https://github.com/dtolnay/cxx/blob/5289ce332cde36d31c54faba42cd64a148c3f94f/src/lossy.rs

[dylni]

@dtolnay I am currently waiting on this ACP for stabilization.

Are there other known use cases so far that we could look at before an FCP?

I am aware of the following use cases.

• Lossy conversion (String::from_utf8_lossy)
• Debug formatting (as you mentioned)

I was originally going to use this feature in os_str_bytes for Debug formatting, but invalid returning individual "sequences" made this usage cumbersome. OsStr cannot be assumed to have the same invalid sequences. However, the individual sequences are required for lossy conversion to work with Utf8Chunks in its current form within libstd.

One thing I am interested in is how the current Utf8Chunks API compares with this alternative one, not based on Iterator, with just 1 type:

My concern is that the alternate API is easier to misuse (e.g., calling next_valid twice for two valid chunks). It also requires parsing each invalid sequence twice.

[Dylan-DPC]

@dylni generally, you don't need an ACP for this to stabilise (unless the team explicitly asked for it which I don't think happened in this case).
The next step is an FCP. In which case, you can submit a stabilisation pr for it linking this issue and preferably putting the report you shared here in that pr and then the team will run an fcp either in the pr or the issue.

[dylni]

@Dylan-DPC Right, but the problem is that the ACP would change the API. Stabilizing at this point would prevent the API change from landing.

[dtolnay]

@rust-lang/libs-api:
@rfcbot fcp merge

I propose stabilizing core::str::Utf8Chunks and core::str::Utf8Chunk ❗with the minor modification described in rust-lang/libs-team#190 ❗.

I recently wanted this API for C++ string Debug impls in cxx as described in #99543 (comment), and also in libproc_macro for synthesizing C-string literals in #123769.

[rfcbot]

Team member @dtolnay has proposed to merge this. The next step is review by the rest of the tagged team members:

• @Amanieu
• @BurntSushi
• @dtolnay
• @joshtriplett
• @m-ou-se

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[BurntSushi]

Note that this same API, under the same name, is also in bstr. See also Utf8Chunks and Utf8Chunk. I can only identify two differences:

• bstr doesn't have a Utf8Chunks::new constructor. This seems okay.
• Utf8Chunk in std lacks bstr's Utf8Chunk::incomplete method. This seems okay and something we can add later if the use cases arise.

With that said, does this mean we also have an itertools situation here? bstr works by defining methods on [u8] via an extension trait.