hermes(stt): greedy single-temperature final decode for fast startup
Beam-2 with a temperature-fallback ladder made the final-model warmup run all three temperature retries under beam search before the server bound its port, so /health was refused for ~8 min and STT was down that whole time on every roll (and hinted at slow per-utterance decodes). Production now runs the accurate large-v3-turbo model greedily at a single temperature, keeping the proper-noun priming prompt that fixes names like Amy/Córdoba - fast startup, fast decodes, accuracy intact. Beam stays env-tunable (HERMES_STT_FINAL_BEAM_SIZE) for a future pass; serve-before-warmup is a recommended follow-up so cold start never blocks readiness. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvMSXH8VH2tMWXanb8SJdf
This commit is contained in:
parent
546e960185
commit
b99952f16c
@ -19,7 +19,7 @@ spec:
|
||||
app: hermes-stt
|
||||
annotations:
|
||||
ai.bstein.dev/role: private-chat-speech-to-text
|
||||
ai.bstein.dev/model: whisper-large-v3-turbo-multilingual (tiny rolling previews)
|
||||
ai.bstein.dev/model: whisper-large-v3-turbo greedy + proper-noun prompt (tiny rolling previews)
|
||||
ai.bstein.dev/gpu: titan-21 dedicated speech time-slice
|
||||
spec:
|
||||
runtimeClassName: nvidia
|
||||
@ -40,6 +40,13 @@ spec:
|
||||
- {name: XDG_CACHE_HOME, value: /tmp/cache}
|
||||
- {name: HERMES_STT_PORT, value: "9000"}
|
||||
- {name: HERMES_STT_MODEL, value: large-v3-turbo}
|
||||
# Fast-and-accurate production decode: the accurate turbo model and
|
||||
# the proper-noun priming prompt, but greedy single-temperature so
|
||||
# startup warmup and per-utterance latency stay low (beam search
|
||||
# made cold start ~8min on this Jetson). Beam is env-tunable if a
|
||||
# future accuracy pass wants it.
|
||||
- {name: HERMES_STT_FINAL_BEAM_SIZE, value: "1"}
|
||||
- {name: HERMES_STT_FINAL_TEMPERATURE, value: "0.0"}
|
||||
- {name: HERMES_STT_CACHE, value: /opt/models/whisper}
|
||||
- {name: NVIDIA_VISIBLE_DEVICES, value: all}
|
||||
- {name: NVIDIA_DRIVER_CAPABILITIES, value: "compute,utility"}
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user