Fnt4 font problem in psv

[lzhhzl]

First of all, thank you so much for making this ShinDataUtil project, it's really useful.
We are currently trying to make a Chinese translation patch for KonoSuba Kibou no Meikyuu to Tsudoishi Boukenshata(PCSG01258) on psvita, I try to use ShinDataUtil of konosuba branch to extract default.fnt in psv, it does work and extract those font pngs. But I noticed that some font pngs are slightly different from default.fnt in the KonoSuba Kibou no Meikyuu to Tsudoishi Boukenshata Plus(switch version), such as these pictures below:

I understand that it is normal between the font in psv and switch to have different resolution of their font pngs, but I noticed that some shape of the font pngs extract from the psv fnt file, such as Q and W, is strangely different. Obviously, the Q and W extracted from the psv file are rectangular. But the Q and W extracted from the switch fnt file are square(which I think this is the right shape). And also most other Japanese kanji pngs have same problem.

So could this be a bug? I would really appreciate it if you would look into this situation. Here are the default.fnt and eboot.bin.elf(extract by using vita-unmake-fself) in psv if you need them:
drive.google.com/drive/folders/1MtFqt8VdZrc8S5WdQtffmrNGH9OllduE?usp=sharing

[DCNick3]

Because font textures also use mip-maps, the texture sizes have to be aligned to a pretty extreme value (I am not sure of the exact requirement, but my guess is that it's 64 pixels). This is so that the lower mip levels (with each being 2x smaller) will have integer sizes. As far as I understand, this is a requirement of the graphics API.

Then, the switch and vita, having different pixel densities, probably end up using font sizes that end up different in number of pixels, and the vita one probably just ends up with an unfortunate number that often needs a lot of vertical padding added. In case of Q, 72 is padded up to 128.

The engine ends up loading the large texture as-is, but uses only the area that actually has glyph in there (by passing the corresponding texture coordinates to the render command). There is enough per-glyph info stored to calculate that (even though ShinDataUtil does not export it).

Now, including this padding when exporting the font might not be the best decision, but I didn't know all the details of the format yet, so I did what was the simplest. When I was re-doing this functionality in sdu, I actually made it trim the unused padding, so using that you'll be able to get nicer results:

P.S.

Since you are poking at fonts, you might be interested in Neurochitin's Kaleido functionality for generating font files: https://gitlab.com/Neurochitin/kaleido/-/tree/saku/fnt

[lzhhzl]

OMG. Thank you for your quick reply and give me detailed explanation and advise.

Since you are poking at fonts, you might be interested in Neurochitin's Kaleido functionality for generating font files:

At first I did want to replace some Japanese kanji in fnt with simplified Chinese character, so I wonder if I could replace some font pngs extracted from fnt and make a repack script for fnt based on ShinDataUtil to achieve the display font replacement. But after listening to your explanation, I might realize that making a replace-png with the exact same shape and size as every font pngs extracted by ShinDataUtil doesn't seem to be the right point. Thank you for your sdu CLI and sharing the Neurochitin's Kaleido project, they were really helpful for me to understand those glyph in fnt and make my repack script for fnt. I will spend some days learning these tools and their codes. Thank you again.

[lzhhzl]

Sorry for replying so late🙏. In March I finally completed some python scripts for extracting and regenerating fnt(FNT4) font file base on sdu in shin and the lz77 compress method in Kaleido——those great projects. For those in need, I wrote a variant of the lz77 compression algorithm that can re-compress the completely same compress-glyph-data in vanilla version 1 FNT4 (only test in vita game PCSG01258, but not very efficient because of my poor knowledge in algorithms, maybe others can make a better algorithm than mine).

By the way, I notices that in sdu relevant code font.rs line 207-218 :

let uncompressed_size = header.texture_width as usize * header.texture_height as usize;
let info = header.into();
let data = if compressed_size == 0 {
    GlyphData::Raw(Vec::<u8>::read_options(
        reader,
        endian,
        VecArgs {
            count: uncompressed_size,
            inner: (),
        },
    )?)

It seems that the size of uncompress-glyph is header.texture_width * header.texture_height. And the uncompress-size of compress-glyph is header.texture_width * header.texture_height + (header.texture_width * header.texture_height)/4 + (header.texture_width * header.texture_height)/16 + (header.texture_width * header.texture_height)/64, which I understood because one compress-glyph data contain 4 mip-maps uncompress data. So why is uncompressed_size not like:

let uncompressed_size = header.texture_width as usize * header.texture_height as usize;
let whole_uncompressed_size = uncompressed_size + uncompressed_size / 4 + uncompressed_size / 16 + uncompressed_size / 64;

instead using 1 mip-map size? Or probably just I misunderstood this part in the font.rs.

[lzhhzl]

Oh and also, my friend YuriSizuku also find me a FNT4 font file in vita game Hatsuyuki Sakura(PCSG00997, also ENTERGRAM made). But a little different is that the file structure and the file formats in this game are not like those common shin engine games:

As you can see, I extract those game file in data_00.psarc to data_00 dir. No SNR, TXA, PIC or other familiar files we know in common shin engine game. But I can sure that font_00.fnt data struct is very similar with those FNT4(version 1) we already know:

You can see that its header data is slightly different from the known FNT4(version 1). Which I guess this is a old FNT4 version 0 structure:

0x00-0x03  FNT4 magic
0x04-0x07  fnt file size (verified)
0x08-0x09  ascent
0x0A-0x0B  descent
0x0C-0x0F  guess mybe version, that means it is version 0

Then I followed this thought to look down the data block from 0x10 to 0xB2CF. Indeed, I could find that those 0x10-0xB2CF data block contain 11,440(Max) glyphs offset information(every 0x4), which is less than FNT4 version 1. Next, I did find some FNT4-glyph-like data by following those offset-info. Here I use offset 0xB2DA data fragment as an example:

But here comes the problem that I couldn't solve. For now, I only can guess that 0x00-0x07 is glyph header information:

0x00-0x05 don't know but some width height information must in it
0x06-0x07 glyph data compressed size

Then, no matter how I tried to decompress those compressed-glyph data block with different length offset_bits LZ77 or LZSS, I couldn't rebuild the glyph correctly and completely. These glyph data have been compressed too hard and too small than the situation in FNT4 version 1.

So I wonder if you have some interest in helping to have a look about this strange old FNT4 structure and solve this puzzle? I put this font_00.fnt file in here and I would be very grateful if you would like to give some help.

**Add: The reason why I thought this font_00.fnt is related with shin engine and FNT4 version 1 structure is because my friend YuriSizuku used to help other analyse the font_00.fnt in switch game 9-nine and also base your shin project to finish the analysis about this font. I also confirm that the font_00.fnt in switch game 9-nine is exactly the same FNT4 version 1 structure we know in common shin engine game. So I simply take a look at the game file structure in 9-nine:

What a coincidence! It is largely very similar to the game file structure in Hatsuyuki Sakura, many file formats are also similar, also made by ENTERGRAM. That's why I thought font_00.fnt in Hatsuyuki Sakura is a old FNT4 structure similar with the FNT4 we know in shin engine and decided to share in this project. Maybe this FNT4 font structure is some kind of ENTERGRAM-style game font, I guess. Or this engine in Hatsuyuki Sakura / 9-nine just a variant of shin engine, whatever >v< .

[DCNick3]

It seems that the size of uncompress-glyph is header.texture_width * header.texture_height. And the uncompress-size of compress-glyph is header.texture_width * header.texture_height + (header.texture_width * header.texture_height)/4 + (header.texture_width * header.texture_height)/16 + (header.texture_width * header.texture_height)/64, which I understood because one compress-glyph data contain 4 mip-maps uncompress data. So why is uncompressed_size not like:

let uncompressed_size = header.texture_width as usize * header.texture_height as usize;
let whole_uncompressed_size = uncompressed_size + uncompressed_size / 4 + uncompressed_size / 16 + uncompressed_size / 64;

instead using 1 mip-map size? Or probably just I misunderstood this part in the font.rs.

This is indeed a bug! I probably didn't notice it because umineko doesn't have any uncompressed glyphs.

What a coincidence! It is largely very similar to the game file structure in Hatsuyuki Sakura, many file formats are also similar, also made by ENTERGRAM. That's why I thought font_00.fnt in Hatsuyuki Sakura is a old FNT4 structure similar with the FNT4 we know in shin engine and decided to share in this project. Maybe this FNT4 font structure is some kind of ENTERGRAM-style game font, I guess. Or this engine in Hatsuyuki Sakura / 9-nine just a variant of shin engine, whatever >v< .

Indeed! This appears to be precisely an older version of the FNT format used by shin on PS Vita (and I also call it FNT v0)! Hatsuyuki Sakura is using another entegram's in-house engine which I know almost nothing about, and I don't even have a name for it besides not-shin-5. However, it appears they've re-used the font format.

The main differences from the switch-flavoured format are that it uses Shift-JIS instead of UTF-8 to encode codepoints, lack of mip-maps and a slightly different format of lz77 bitstream. I do not have a nice end-user tool for it yet, but I have actually implemented PoC glyph extraction in shin-translation-tools's codebase. You can use it as a reference.

Just for posterity, I know some other switch games that are also using the same engine besides 9-nine-:

title id	release date	name
01003A6013486000	2021-02-25	金色ラブリッチェ-Golden Time-
0100FE001327C000	2021-02-25	金色ラブリッチェ
0100D1A014A4A000	2021-08-26	かけぬけ★青春スパーキング！
01005CE015CA2000	2022-06-23	9-nine-

[nini22P]

To repacking the fnt files for AstralAir no Shiroki Towa -White Eternity-.

I ported konosuba_py to the Rust. As I am unfamiliar with the lz77 algorithm, this is merely a straightforward port. Nevertheless, its performance is already sufficiently good.

Update: I have implemented font rebuild and character replace, which is extremely useful for translating into other languages.

[nini22P]

I tried changing the fonts for some games, and it seems to be working fine.