Exception using PlaceMarkersUsfmUpdateBlockHandler with Devanagari

[pmachapman]

Serval is generating an error when retrieving USFM for a back translation to Hindi with preserve paragraph markers enabled:

Exception thrown: 'SIL.Machine.Corpora.UsfmUpdateBlockHandlerException' in SIL.Machine.dll: 
'No token "ं" found in text ""कब च तेरु परमेश्वर?" इन सवालों के मेरे दूसरे ताना मि पर मरदिन, हर बगत इन लगदु जन व मेरे सर पे ते कटदिन नह ं। " at or beyond index 106. Is the versification correctly specified?'

This issue is affecting one project on Scripture Forge, but I have supplied a workaround to the user (select "Move to end" or "Remove" for paragraph breaks). The workaround is probably not ideal given the draft is from the Psalms.

This unit test recreates the bug:

Note: The update rows and alignment are from the back translation, and I have substituted the BSB for the source project.

    [Test]
    public void UpdateBlock_AnusvaraTokenization()
    {
        // The anusvara ("ं") is the third to last character in the update rows string
        string source =
            "Like the crushing of my bones, my enemies taunt me, while they say to me all day long, “Where is your God?”";
        string updateRows =
            "\"कब च तेरु परमेश्वर?\" इन सवालों के मेरे दूसरे ताना मि पर मरदिन, हर बगत इन लगदु जन व मेरे सर पे ते कटदिन नह ं। ";
        PlaceMarkersAlignmentInfo alignInfo = new PlaceMarkersAlignmentInfo(
            sourceTokens: [.. Tokenizer.Tokenize(source)],
            translationTokens: [.. Tokenizer.Tokenize(updateRows)],
            alignment: ToWordAlignmentMatrix(
                "0-0 1-1 2-2 3-3 4-4 5-5 6-6 7-7 8-8 9-9 10-10 11-11 12-12 13-13 14-14 15-15 16-16 17-17 18-18 19-19 20-20 21-21 22-22 23-23 24-24 24-25 25-26 26-27 26-28 26-29 27-30"
            ),
            paragraphBehavior: UpdateUsfmMarkerBehavior.Preserve,
            styleBehavior: UpdateUsfmMarkerBehavior.Strip
        );
        IReadOnlyList<UpdateUsfmRow> rows =
        [
            new UpdateUsfmRow(
                ScrRef("PSA 42:10"),
                updateRows,
                new Dictionary<string, object> { { "alignment_info", alignInfo } }
            ),
        ];
        string usfm =
            @"\id PSA
\c 42
\q1
\v 10 Like the crushing of my bones, my enemies taunt me,
\q2 while they say to me all day long, “Where is your God?”
";

        string target = UpdateUsfm(
            rows,
            usfm,
            paragraphBehavior: UpdateUsfmMarkerBehavior.Preserve,
            usfmUpdateBlockHandlers: [new PlaceMarkersUsfmUpdateBlockHandler()]
        );

        string result =
            @"\id PSA
\c 42
\q1
\v 10 ""कब च तेरु परमेश्वर?"" इन सवालों के मेरे दूसरे
\q2 ताना मि पर मरदिन, हर बगत इन लगदु जन व मेरे सर पे ते कटदिन नह ं ।
";

        AssertUsfmEquals(target, result);
    }

[benjaminking]

It looks like this is happening because one of the tokens is a combining character and does not appear in the unicode string once the characters are combined. I believe this mismatch occurs because the SentencePiece tokenizer uses NFKC normalization and the rest of the code uses NFC. Does anyone know how we have managed this mismatch in the past?

[benjaminking]

I was wrong about Sentence Piece being relevant here. This is what I think is actually going on:

At line 176, IndexOf is called without specifying a StringComparison parameter. This makes it default to StringComparison.CurrentCulture, which has some weird collation rules. I think this makes it unable to find a substring that consists exclusively of a combining character. I don't know if there would be unintended consequences from specifying StringComparison.Ordinal, but I think that would fix this issue.