Documentation IndexFetch the complete documentation index at: /llms.txtUse this file to discover all available pages before exploring further.
Fetch the complete documentation index at: /llms.txt
Use this file to discover all available pages before exploring further.
import asyncio from agno.agent import Agent from agno.models.vllm import VLLM agent = Agent( model=VLLM(id="Qwen/Qwen2.5-7B-Instruct", top_k=20, enable_thinking=False), markdown=True, ) asyncio.run(agent.aprint_response("Share a 2 sentence horror story", stream=True))
Set up your virtual environment
uv venv --python 3.12 source .venv/bin/activate
uv venv --python 3.12 .venv\Scripts\activate
Install Libraries
uv pip install -U agno openai vllm
Start vLLM server
vllm serve Qwen/Qwen2.5-7B-Instruct \ --enable-auto-tool-choice \ --tool-call-parser hermes \ --dtype float16 \ --max-model-len 8192 \ --gpu-memory-utilization 0.9
Set your API key
export VLLM_API_KEY=xxx
Run Agent
python cookbook/90_models/vllm/basic.py
Was this page helpful?
