Groq

Use Groq’s ultra-fast inference for chat completions, tool calling, and audio processing through Portkey.

Quick Start

Get started with Groq in under 2 minutes:

Python

from portkey_ai import Portkey

# 1. Install: pip install portkey-ai
# 2. Add @groq provider in model catalog
# 3. Use it:

portkey = Portkey(api_key="PORTKEY_API_KEY")

response = portkey.chat.completions.create(
    model="@groq/llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": "Hello!"}]
)

print(response.choices[0].message.content)

Add Provider in Model Catalog

Before making requests, add Groq to your Model Catalog:

1. Go to Model Catalog → Add Provider
2. Select Groq
3. Enter your Groq API key
4. Name your provider (e.g., groq)

Complete Setup Guide

See all setup options and detailed configuration instructions

---

Groq Capabilities

Tool Calling

Use Groq’s tool calling feature to trigger external functions:

Supported Models

View Groq models that support tool calling

Python

from portkey_ai import Portkey

portkey = Portkey(api_key="PORTKEY_API_KEY")

tools = [{
    "type": "function",
    "function": {
        "name": "getWeather",
        "description": "Get the current weather",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {"type": "string", "description": "City and state"},
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
            },
            "required": ["location"]
        }
    }
}]

response = portkey.chat.completions.create(
    model="@groq/llama-3.3-70b-versatile",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What's the weather like in Delhi?"}
    ],
    tools=tools,
    tool_choice="auto"
)

print(response.choices[0].finish_reason)

Speech to Text (Whisper)

Transcribe or translate audio using Groq’s Whisper model:

Python

from portkey_ai import Portkey

portkey = Portkey(api_key="PORTKEY_API_KEY", provider="@groq")

audio_file = open("/path/to/file.mp3", "rb")

# Transcription
transcription = portkey.audio.transcriptions.create(
  model="whisper-large-v3",
  file=audio_file
)
print(transcription.text)

# Translation
translation = portkey.audio.translations.create(
  model="whisper-large-v3",
  file=audio_file
)
print(translation.text)

Text to Speech

Convert text to natural-sounding audio:

Python

from portkey_ai import Portkey
from pathlib import Path

portkey = Portkey(api_key="PORTKEY_API_KEY", provider="@groq")

speech_file_path = Path(__file__).parent / "speech.mp3"
response = portkey.audio.speech.create(
  model="playai-tts",
  voice="Fritz-PlayAI",
  input="Today is a wonderful day to build something people love!"
)

with open(speech_file_path, "wb") as f:
    f.write(response.content)

---

Supported Models

Groq provides ultra-fast inference for various open-source models:

Model Family	Models Available
Llama	llama-3.3-70b-versatile, llama-3.1-70b-versatile, llama-3.1-8b-instant
Mixtral	mixtral-8x7b-32768
Gemma	gemma-7b-it, gemma2-9b-it
Whisper	whisper-large-v3, whisper-large-v3-turbo
TTS	playai-tts

Check Groq’s documentation for the complete list of available models.

---

Next Steps

Gateway Configs

Add fallbacks, load balancing, and more

Observability

Monitor and trace your Groq requests

Prompt Library

Manage and version your prompts

Metadata

Add custom metadata to requests

For complete SDK documentation:

SDK Reference

Complete Portkey SDK documentation