Quantization is used to store floats using less space. An array of quantized data will be accompanied by a byte describing the quantization. The high nibble of the quantization byte tells you the data type, and the low nibble gives you a value to divide each value by.
format = quantization >> 4
divisor = 1 << (quantization & 0b1111)The observed formats in Sluggers data are:
| Format | Data type | Byte size |
|---|---|---|
| 0 | s16 | 2 |
| 3 | s16 | 2 |
| 4 | float | 4 |
| 7 | float | 4 |
| 0xa | float | 4 |
The theoretical GX table (1=float, 2=u16, 3=s16, 4=u8, 5=s8) does not match what's observed. In practice, format 4 is float (not u8), and formats 7 and 0xa also appear as floats. Any unrecognised format should be treated as float.
The byte is read per array at runtime, so every array can use its own format. Color arrays are an exception to this table: quantization 48 on a color array is 2-byte RGBA4444 (see Mesh Data Section).