Currently the library only supports: ```cpp typedef enum { TRANSCRIBE_KV_TYPE_AUTO = 0, TRANSCRIBE_KV_TYPE_F32 = 1, TRANSCRIBE_KV_TYPE_F16 = 2, } transcribe_kv_type; ``` Could be cool to allow for Q8 or Q4 quantization for KV cache in low memory scenarios.
Currently the library only supports:
Could be cool to allow for Q8 or Q4 quantization for KV cache in low memory scenarios.