Skip to content
UsecaseBoard
← All use cases
hashnodeLocal AI

Measure local model speed before tuning an email assistant

At a glance

Areef Reza separates prompt-reading speed from generation speed to understand delays in his local email assistant. This workflow is listed under Local AI, Performance, Email Assistants.

Agents / tools
Ollama, Mistral NeMo 12B, HTTP client

SourceBuilding a tool for these workflows? See weekly rates.

What they did

1) Install Ollama, download the model with ollama pull mistral-nemo:12b, and start the local service with ollama serve if it is not already running. 2) POST to http://localhost:11434/api/generate with this JSON body: {"model":"mistral-nemo:12b","prompt":"Write 200 words about transformers.","stream":false,"keep_alive":-1,"options":{"num_ctx":4096}}. 3) Read the completed response's timing fields. Calculate input tokens per second as prompt_eval_count divided by (prompt_eval_duration / 1e9), and output tokens per second as eval_count divided by (eval_duration / 1e9). 4) Use those separate rates to identify whether prompt size or answer length dominates latency before changing the assistant. The author's changes included trimming history and limiting generated responses.

How it went

The author measured roughly 127 input tokens and 17 output tokens per second on his MacBook Pro.

Worth knowing

Those speeds describe the author's hardware. This card reproduces the benchmark, not his unpublished complete email application.

Save ideas in this browser. Upvotes count across the board.

Prompt excerpt

No prompt excerpt was provided by the source.

Sponsored

CodeBuddy

Tencent Cloud AI coding assistant for IDE and CLI.

Try CodeBuddy ↗
Explore all →