Optimizing TTS with Triton

All the examples I have are highly unoptimized - For eg, Modal Labs uses FastAPI - https://modal.com/docs/examples/chatterbox_tts\ BentoML also uses FastAPI like service - https://www.bentoml.com/blog/deploying-a-text-to-speech-application-with-bentoml\ Even Chatterbox TTS has a very naive example - https://github.com/resemble-ai/chatterbox\ Tritonserver docs don’t have a TTS example. I am 100% certain that a highly optimized variant can be written with TritonServer, utilizing model concurrency and batching. If someone has implemented a TTS service with Tritonserver or has a better inference server alternative to deploy, please help me out here. I don’t want to reinvent the wheel.

You're absolutely right — most public TTS deployment examples (Modal, BentoML, FastAPI, etc.) are **not optimized for high-throughput**, low-latency production use. And yes, **Triton Inference Server** is theoretically a perfect fit for such a task — but almost no one has published a good, full-stack example for TTS. Here's a breakdown of your options and how to move forward without reinventing the wheel: --- ### 🔥 Why TritonServer *can* be optimal for TTS - **Concurrent Model Execution** (ensemble and multiple engines) - **Batching (dynamic or static)** — especially useful if you're using autoregressive models (e.g., Tacotron2, FastSpeech, etc.) - **Multi-backend support**: TensorRT, ONNX, PyTorch, custom Python models - **Asynchronous inference + gRPC/HTTP support** BUT: There are almost no ready-made examples in Triton’s docs for **TTS pipelines** (e.g., text → mel → vocoder). --- ## ✅ Existing Building Blocks (that are close) 1. **[NVIDIA Riva](https://developer.nvidia.com/riva)** — NVIDIA's closed-source but performant voice SDK (built on top of Triton). - It includes **TTS pipelines** with **Triton under the hood**, and **supports streaming**. - You can reverse-engineer parts of it if needed, or at least use it as a performance benchmark. - Uses models like FastPitch + HiFiGAN. 2. **[Riva’s models](https://ngc.nvidia.com/catalog/models?filters=&orderBy=modifiedDESC&query=fastpitch)** are on NGC, and you can download them separately to serve on **vanilla TritonServer**: - FastPitch (text → mel) - HiFi-GAN (mel → waveform) 3. **[Triton Ensemble Models](https://github.com/triton-inference-server/server/blob/main/docs/architecture.md#ensemble-models)** can string FastPitch and HiFi-GAN together, allowing: - Single request → Full waveform output - Batching across both stages --- ## 🛠️ What You Could Do ### Option 1: Minimal Implementation with TritonServer You’ll need to build a pipeline: #### a. Models: - `fastpitch/` - `model.pt` (PyTorch or ONNX) - `config.pbtxt` - `hifigan/` - `model.pt` - `config.pbtxt` #### b. Optional: Build an **ensemble model**: ```plaintext ensemble_tts/ ├── config.pbtxt # defines the pipeline: fastpitch -> hifigan ``` #### c. Example config snippet (for `ensemble_tts/config.pbtxt`): ```protobuf name: "ensemble_tts" platform: "ensemble" input [ { name: "text_input" data_type: TYPE_STRING dims: [-1] } ] output [ { name: "audio_output" data_type: TYPE_FP32 dims: [1, -1] } ] ensemble_scheduling { step [ { model_name: "fastpitch", input_map { key: "text_input" value: "INPUT" }, output_map { key: "OUTPUT" value: "mel_output" } }, { model_name: "hifigan", input_map { key: "mel_input" value: "mel_output" }, output_map { key: "audio_output" value: "OUTPUT" } } ] } ``` --- ### Option 2: Use Faster Transformers or TensorRT Versions If you want to go fully optimized: - Export your models to **ONNX**, then compile with **TensorRT** - Serve with Triton’s **TensorRT backend** (super low latency) - Configure batching and concurrency in the `config.pbtxt` files. --- ### 🧠 Recommendations | Solution | Performance | Notes | |----------|-------------|-------| | **Riva** | ✅ High | Great performance, but proprietary | | **Vanilla Triton (FastPitch + HiFiGAN)** | ✅ High if optimized | You’ll need to build the ensemble pipeline manually | | **FastAPI/BentoML** | ❌ Low | Easy to prototype, not for scale | | **Triton + ONNX + TensorRT** | ✅✅ Best | Use for scale inference, low latency | --- ## 🧩 Where to Get Help or Code: 1. **[Riva source models on NGC](https://ngc.nvidia.com/catalog/models?filters=&orderBy=modifiedDESC&query=fastpitch)** — pre-trained and compatible with Triton 2. **Triton Inference Server GitHub** — examples and ensemble model docs 3. **[NVIDIA Developer Forums](https://forums.developer.nvidia.com/c/ai-data-science/triton-inference-server/313)** — active community 4. **HuggingFace** — many models (like Bark, Tortoise, FastSpeech2) can be exported to ONNX --- ## ✳️ Final Thought You're right to want **TritonServer** — it’s the only serious option for scaling TTS *efficiently*. Most public examples are just quick demos. You're not reinventing the wheel by building a proper Triton-based TTS pipeline — you're just **finally doing it right**. If you'd like, I can help you: - Convert a HuggingFace TTS model to ONNX or TorchScript - Write working `config.pbtxt` files - Set up a `model_repository/` directory ready to launch with Triton Would you like to go that route?