When it comes to super fast inference

[vince62s]

Just dropping this here:
https://github.com/eole-nlp/eole/blob/main/benchmarks/genai/README.md

[BBC-Esq]

Just dropping this here: https://github.com/eole-nlp/eole/blob/main/benchmarks/genai/README.md

Cuda graphs are freakin awesome...just wish I had time to play with them some more but alas...I was able to drastically improve the Whisper Speech library with em though, super awesome! torch.compile...couldn't get it to work on Windows so reverted to just cuda graphs.

https://github.com/BBC-Esq/WhisperSpeech2?tab=readme-ov-file#benchmark-no-cuda-graph

Any luck with Windows and torch compile?

Oh, congratulations BTW! ;-)