I used 6 and that dropped the token time to 220ms.
For -ngl, I tried using 24, and then 30 and then 40, and never got to an out of memory error, and got exactly the same token timing, stuck at 220ms.
But, this is very helpful, thank you!
Also curious to know whether the wall clock time (just prepend your command with 'time ') is any different.
I used 6 and that dropped the token time to 220ms.
For -ngl, I tried using 24, and then 30 and then 40, and never got to an out of memory error, and got exactly the same token timing, stuck at 220ms.
But, this is very helpful, thank you!