Ray Serve Deployment Guide¶

Ray Serve enables multi-node, scalable distributed model serving.

1. Launching Turing Engine on Ray Serve¶

pip install "ray[serve]"

serve run integrations/ray/ray_serve_turing:app

2. Testing Ray Serve Endpoint¶

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.1-70b",
    "messages": [{"role": "user", "content": "Hello from Ray Serve!"}]
  }'