Day 45: Cost Optimization Cho LLM/RAG Production

Request này tốn bao nhiêu tiền?
Stage nào đang làm cost phình lên?
Token budget của từng endpoint là gì?
Nếu vượt budget, hệ thống degrade như thế nào?
Nếu giảm context/model/output tokens, quality có còn đạt release gate không?
Dùng được trong production không, nếu có thì cần điều kiện gì?

User question
  -> normalize
  -> optional semantic cache lookup
  -> query embedding
  -> dense search + lexical search
  -> hybrid merge
  -> rerank
  -> context builder
  -> LLM generation
  -> citation validation
  -> trace/cost logger

1. Instrument
   Log token, model, cache, retry, latency và stage-level trace.

2. Baseline
   Tính cost/request, cost/day, cost/month theo tenant, feature và model.

3. Budget
   Đặt token budget, request quota, eval budget và retry budget.

4. Optimize
   Áp dụng cache, model routing, context pruning, batch hoặc distillation.

5. Verify
   Chạy regression eval: retrieval, answer quality, citation, no-answer, latency.

6. Roll out
   Canary theo tenant/traffic %, theo dõi cost và quality cùng lúc.

7. Degrade or rollback
   Nếu vượt budget hoặc quality giảm, bật degrade mode hoặc rollback config.

cost_per_request =
  llm_input_tokens * input_price_per_token
+ llm_cached_input_tokens * cached_input_price_per_token
+ llm_output_tokens * output_price_per_token
+ reasoning_tokens * reasoning_price_per_token
+ embedding_tokens * embedding_price_per_token
+ reranker_units * reranker_price_per_unit
+ tool_calls * tool_price_per_call
+ retry_count * retry_cost
+ infra_cost_allocated_per_request
+ observability_cost_allocated_per_request

cost_per_day =
  sum(cost_per_request_by_type * requests_per_day_by_type)
+ offline_jobs_cost_per_day
+ fixed_infra_cost_per_day

cost_per_month = cost_per_day * 30

{
  "trace_id": "tr_20260510_001",
  "timestamp": "2026-05-10T10:30:00Z",
  "tenant_id": "tenant_a",
  "user_tier": "paid",
  "feature": "rag_query",
  "request_type": "normal_rag",
  "pipeline_version": "rag-cost-v2",
  "corpus_version": "policy-2026-05",
  "prompt_version": "rag-answer.v4",
  "models": {
    "embedding": "embedding-small",
    "reranker": "reranker-base",
    "generator": "llm-medium"
  },
  "usage": {
    "prompt_tokens": 4200,
    "cached_prompt_tokens": 1800,
    "completion_tokens": 520,
    "reasoning_tokens": 0,
    "embedding_tokens": 32,
    "rerank_units": 24
  },
  "retrieval": {
    "dense_top_k": 50,
    "sparse_top_k": 50,
    "rerank_top_n": 24,
    "context_chunks": 5,
    "context_tokens": 3100
  },
  "cache": {
    "prompt_cache_hit_tokens": 1800,
    "semantic_cache_hit": false,
    "semantic_cache_similarity": null
  },
  "latency_ms": {
    "embedding": 42,
    "dense_search": 38,
    "sparse_search": 21,
    "rerank": 170,
    "generation": 1460,
    "total": 1810
  },
  "retry": {
    "count": 0,
    "reason": null
  },
  "estimated_cost_usd": 0.0031,
  "quality_signals": {
    "answer_status": "answered",
    "citation_valid": true,
    "user_feedback": null
  }
}

receive request
  -> classify request_type
  -> load tenant/user budget policy
  -> reject or trim query if too long
  -> cap retrieval top_k and rerank_top_n
  -> build candidate context
  -> prune until context_tokens <= budget
  -> set max_output_tokens on model call
  -> cap retry count
  -> log budget decision

def enforce_token_budget(request, retrieved_chunks, policy, tokenizer):
    query_tokens = tokenizer.count(request.question)
    if query_tokens > policy.max_query_tokens:
        raise BudgetError("question_too_long")

    selected = []
    total_context_tokens = 0

    for chunk in retrieved_chunks[: policy.max_context_chunks * 2]:
        chunk_tokens = tokenizer.count(chunk.text)
        if total_context_tokens + chunk_tokens > policy.max_context_tokens:
            continue
        selected.append(chunk)
        total_context_tokens += chunk_tokens
        if len(selected) >= policy.max_context_chunks:
            break

    return GenerationPlan(
        context_chunks=selected,
        max_output_tokens=policy.max_output_tokens,
        max_retries=policy.max_retries,
        budget_name=policy.name,
    )

system instruction
+ developer instruction
+ response schema
+ static examples
+ stable tool definitions
+ stable policy block
+ dynamic context
+ user question

question
  -> normalize
  -> classify cache eligibility
  -> embedding(question)
  -> vector search trong cache index
  -> kiểm tra similarity threshold
  -> kiểm tra tenant_id, permission_hash, corpus_version, prompt_version
  -> nếu hit: trả cached answer
  -> nếu miss: chạy full RAG pipeline rồi lưu cache entry

{
  "cache_id": "sc:tenant_a:policy-2026-05:rag-answer.v4:8f31",
  "tenant_id": "tenant_a",
  "permission_hash": "roles:employee",
  "corpus_version": "policy-2026-05",
  "prompt_version": "rag-answer.v4",
  "answer_schema_version": "v1",
  "embedding_model": "embedding-small",
  "generator_model": "llm-medium",
  "normalized_question": "ngay nghi phep nam full time",
  "answer": "Nhân viên full-time có 12 ngày nghỉ phép năm [S1].",
  "citations": ["doc_hr:v3:chunk_004"],
  "source_chunk_hashes": ["sha256:..."],
  "created_at": "2026-05-10T10:30:00Z",
  "expires_at": "2026-05-17T10:30:00Z"
}

def get_semantic_cache(question, auth, versions, embedding):
    if not is_cacheable(question, auth):
        return CacheMiss(reason="not_cacheable")

    normalized = normalize_question(question)
    query_vector = embedding.embed(normalized)
    candidates = cache_index.search(
        vector=query_vector,
        top_k=5,
        filters={
            "tenant_id": auth.tenant_id,
            "permission_hash": auth.permission_hash,
            "corpus_version": versions.corpus,
            "prompt_version": versions.prompt,
        },
    )

    best = candidates[0] if candidates else None
    if best and best.score >= 0.92 and still_valid(best.metadata):
        payload = redis.get(best.cache_id)
        return CacheHit(payload=payload, similarity=best.score)

    return CacheMiss(reason="low_similarity_or_not_found")

{
  "request_type": "normal_rag",
  "tenant_tier": "paid",
  "remaining_monthly_budget_usd": 42.5,
  "query_complexity": "medium",
  "risk_level": "policy",
  "latency_slo_ms": 2500,
  "cache_hit": false,
  "provider_health": "ok"
}

def route_model(features, budgets, eval_scores):
    if budgets.tenant_monthly_remaining_usd <= 0:
        return Route(model="llm-small", mode="degraded", reason="budget_exhausted")

    if features.cache_hit:
        return Route(model=None, mode="cache_hit", reason="semantic_cache")

    if features.request_type == "simple_faq":
        return Route(model="llm-small", mode="normal", reason="simple_faq")

    if features.risk_level in {"legal", "finance"} and budgets.user_tier == "paid":
        return Route(model="llm-strong", mode="normal", reason="high_risk_paid")

    if features.query_complexity == "complex":
        return Route(model="llm-strong", mode="normal", reason="complex_query")

    return Route(model="llm-medium", mode="normal", reason="default_rag")

retrieve top 50 dense + top 50 sparse
  -> RRF merge
  -> rerank top 24
  -> remove duplicate document/section
  -> keep top evidence spans
  -> apply max context token budget
  -> optional compression
  -> build prompt with citations

def prune_context(reranked_hits, budget, tokenizer):
    selected = []
    used_sections = set()
    used_tokens = 0

    for hit in reranked_hits:
        section_key = (hit.document_id, hit.heading)
        if section_key in used_sections and hit.score < budget.duplicate_section_score:
            continue

        span = extract_relevant_span(hit.text, hit.query_terms, max_tokens=budget.max_span_tokens)
        span_tokens = tokenizer.count(span)
        if used_tokens + span_tokens > budget.max_context_tokens:
            continue

        selected.append(ContextBlock(source_id=hit.source_id, text=span, metadata=hit.metadata))
        used_sections.add(section_key)
        used_tokens += span_tokens

        if len(selected) >= budget.max_context_chunks:
            break

    return selected

Realtime path
  -> strict latency SLO
  -> small per-request budget
  -> limited retry
  -> user-facing quota

Offline path
  -> queue/batch
  -> separate quota
  -> lower priority
  -> resumable job
  -> result audit table

collect stable task traces
  -> filter high-quality examples
  -> generate teacher labels bằng strong model
  -> human review hoặc rule validation
  -> train/fine-tune student model
  -> eval theo task
  -> canary route một phần traffic
  -> monitor quality/cost drift

1. Giảm max output tokens.
2. Bật concise answer mode.
3. Giảm context_top_k nhưng giữ citation validation.
4. Ưu tiên semantic cache cho FAQ.
5. Route simple task sang small model.
6. Tạm dừng eval/synthetic/offline job.
7. Reject complex/debug request không quan trọng.
8. Hard stop tenant hoặc user nếu vượt quota nghiêm trọng.

1. Mỗi request tốn bao nhiêu tiền?
2. Cost đến từ stage nào?
3. Giới hạn token/quota nằm ở đâu và enforce như thế nào?
4. Khi vượt budget, hệ thống degrade ra sao?
5. Quality có còn đạt release gate sau tối ưu không?

{
  "pricing_version": "provider-pricing-2026-05-10",
  "currency": "USD",
  "token_unit": 1000000,
  "llm_models": {
    "llm-small": {
      "input_per_1m": "0.1500",
      "cached_input_per_1m": "0.0150",
      "output_per_1m": "0.6000",
      "reasoning_per_1m": "0.6000"
    },
    "llm-medium": {
      "input_per_1m": "0.5000",
      "cached_input_per_1m": "0.0500",
      "output_per_1m": "2.0000",
      "reasoning_per_1m": "2.0000"
    },
    "llm-strong": {
      "input_per_1m": "2.0000",
      "cached_input_per_1m": "0.2000",
      "output_per_1m": "8.0000",
      "reasoning_per_1m": "8.0000"
    }
  },
  "embedding_models": {
    "embedding-small": {
      "input_per_1m": "0.0200"
    }
  },
  "rerankers": {
    "reranker-base": {
      "per_1000_units": "0.0800"
    }
  },
  "tool_calls": {
    "web_search": {
      "per_call": "0.0100"
    }
  },
  "infra_allocation": {
    "fixed_daily_usd": "15.0000",
    "expected_requests_per_day": 10000
  }
}

{
  "trace_id": "tr_001",
  "timestamp": "2026-05-10T10:30:00Z",
  "tenant_id": "tenant_a",
  "user_id_hash": "sha256:...",
  "feature": "rag_query",
  "request_type": "normal_rag",
  "pipeline_version": "rag-cost-v2",
  "pricing_version": "provider-pricing-2026-05-10",
  "prompt_version": "rag-answer.v4",
  "corpus_version": "policy-2026-05",
  "is_batch": false,
  "models": {
    "generator": "llm-medium",
    "embedding": "embedding-small",
    "reranker": "reranker-base"
  },
  "usage": {
    "prompt_tokens": 4200,
    "cached_prompt_tokens": 1800,
    "completion_tokens": 520,
    "reasoning_tokens": 0,
    "embedding_tokens": 32,
    "rerank_units": 24,
    "tool_calls": {
      "web_search": 0
    }
  },
  "cache": {
    "semantic_cache_hit": false,
    "prompt_cache_hit_tokens": 1800
  },
  "retry": {
    "count": 0,
    "reason": null
  },
  "latency_ms": {
    "total": 1810
  },
  "quality_signals": {
    "answer_status": "answered",
    "citation_valid": true
  }
}

request_type,requests_per_day,prompt_tokens,cached_prompt_tokens,completion_tokens,embedding_tokens,rerank_units,semantic_cache_hit_rate,retry_rate,model,cost_per_request_usd,cost_per_day_usd
simple_faq,700,1800,900,220,24,8,0.35,0.01,llm-small,,
normal_rag,250,4200,1800,520,32,24,0.05,0.02,llm-medium,,
complex_rag,50,7800,2500,900,48,36,0.00,0.03,llm-strong,,
eval,30,3800,1600,350,32,24,0.00,0.00,llm-medium,,

version: rag-token-budget-v2
defaults:
  max_retries: 1
  reject_when_question_tokens_over: true
  log_budget_decision: true

policies:
  simple_faq:
    max_query_tokens: 120
    max_context_chunks: 3
    max_context_tokens: 1800
    max_output_tokens: 250
    max_rerank_candidates: 12
    allowed_models: ["llm-small", "llm-medium"]

  normal_rag:
    max_query_tokens: 250
    max_context_chunks: 5
    max_context_tokens: 3500
    max_output_tokens: 600
    max_rerank_candidates: 24
    allowed_models: ["llm-medium", "llm-strong"]

  complex_rag:
    max_query_tokens: 500
    max_context_chunks: 8
    max_context_tokens: 7000
    max_output_tokens: 900
    max_rerank_candidates: 36
    allowed_models: ["llm-strong"]
    required_tiers: ["paid", "enterprise"]

  eval_run:
    max_query_tokens: 250
    max_context_chunks: 5
    max_context_tokens: 3500
    max_output_tokens: 400
    max_rerank_candidates: 24
    max_retries: 0
    force_batch_when_items_over: 100

{
  "trace_id": "tr_001",
  "budget_policy": "normal_rag",
  "budget_version": "rag-token-budget-v2",
  "query_tokens": 84,
  "context_tokens_before_pruning": 6900,
  "context_tokens_after_pruning": 3380,
  "context_chunks_before": 12,
  "context_chunks_after": 5,
  "max_output_tokens": 600,
  "decision": "allowed"
}

sc:{tenant_id}:{permission_hash}:{corpus_version}:{prompt_version}:{schema_version}:{embedding_model}:{cache_hash}

{
  "cache_id": "sc:tenant_a:roles_employee:policy_2026_05:rag_v4:v1:embedding_small:abc123",
  "normalized_question": "ngay nghi phep nam full time",
  "answer": "Nhân viên full-time có 12 ngày nghỉ phép năm [S1].",
  "citations": [
    {
      "document_id": "doc_hr",
      "chunk_id": "tenant_a:doc_hr:v3:004",
      "source_id": "S1"
    }
  ],
  "source_chunk_hashes": ["sha256:..."],
  "quality": {
    "citation_valid": true,
    "answer_status": "answered"
  },
  "created_at": "2026-05-10T10:30:00Z",
  "ttl_seconds": 604800
}

version: rag-routing-v2

routes:
  - name: semantic_cache_hit
    when:
      semantic_cache_hit: true
    action:
      mode: cache_hit
      model: null

  - name: simple_faq_low_risk
    when:
      request_type: simple_faq
      risk_level: low
    action:
      mode: normal
      model: llm-small
      max_output_tokens: 250

  - name: high_risk_paid
    when:
      risk_level_in: ["legal", "finance", "policy"]
      user_tier_in: ["paid", "enterprise"]
      query_complexity_in: ["medium", "complex"]
    action:
      mode: normal
      model: llm-strong
      max_output_tokens: 900

  - name: conserve_budget
    when:
      tenant_budget_state_in: ["conserve", "degraded"]
    action:
      mode: degraded
      model: llm-small
      max_output_tokens: 350
      context_top_k_delta: -2

default:
  mode: normal
  model: llm-medium
  max_output_tokens: 600

#!/usr/bin/env python3
import argparse
import csv
import json
from collections import defaultdict
from dataclasses import dataclass
from decimal import Decimal, ROUND_HALF_UP
from pathlib import Path


MILLION = Decimal("1000000")
THOUSAND = Decimal("1000")


def money(value: Decimal) -> Decimal:
    return value.quantize(Decimal("0.000001"), rounding=ROUND_HALF_UP)


def d(value, default="0") -> Decimal:
    if value is None:
        return Decimal(default)
    return Decimal(str(value))


@dataclass(frozen=True)
class CostBreakdown:
    llm_input: Decimal = Decimal("0")
    llm_cached_input: Decimal = Decimal("0")
    llm_output: Decimal = Decimal("0")
    llm_reasoning: Decimal = Decimal("0")
    embedding: Decimal = Decimal("0")
    rerank: Decimal = Decimal("0")
    tool: Decimal = Decimal("0")
    infra: Decimal = Decimal("0")

    @property
    def total(self) -> Decimal:
        return (
            self.llm_input
            + self.llm_cached_input
            + self.llm_output
            + self.llm_reasoning
            + self.embedding
            + self.rerank
            + self.tool
            + self.infra
        )


def load_json(path: Path) -> dict:
    with path.open("r", encoding="utf-8") as f:
        return json.load(f)


def iter_jsonl(path: Path):
    with path.open("r", encoding="utf-8") as f:
        for line_no, line in enumerate(f, start=1):
            line = line.strip()
            if not line:
                continue
            try:
                yield line_no, json.loads(line)
            except json.JSONDecodeError as exc:
                raise ValueError(f"Invalid JSON at line {line_no}: {exc}") from exc


def require(trace: dict, field: str):
    current = trace
    for part in field.split("."):
        if not isinstance(current, dict) or part not in current:
            raise ValueError(f"Missing required field {field} in trace {trace.get('trace_id')}")
        current = current[part]
    return current


def per_token_cost(tokens: Decimal, price_per_1m: Decimal, multiplier: Decimal) -> Decimal:
    return (tokens / MILLION) * price_per_1m * multiplier


def calculate_trace_cost(trace: dict, pricing: dict) -> CostBreakdown:
    usage = trace.get("usage", {})
    models = trace.get("models", {})

    semantic_cache_hit = bool(trace.get("cache", {}).get("semantic_cache_hit", False))
    batch_multiplier = Decimal("1")
    if trace.get("is_batch"):
        batch_multiplier = d(pricing.get("batch_multiplier", "1"))

    infra = pricing.get("infra_allocation", {})
    expected_requests = d(infra.get("expected_requests_per_day", "0"))
    fixed_daily = d(infra.get("fixed_daily_usd", "0"))
    infra_per_request = Decimal("0") if expected_requests <= 0 else fixed_daily / expected_requests

    llm_input = Decimal("0")
    llm_cached_input = Decimal("0")
    llm_output = Decimal("0")
    llm_reasoning = Decimal("0")

    if not semantic_cache_hit:
        generator = require(trace, "models.generator")
        model_price = pricing["llm_models"][generator]

        prompt_tokens = d(usage.get("prompt_tokens"))
        cached_tokens = d(usage.get("cached_prompt_tokens"))
        if cached_tokens > prompt_tokens:
            raise ValueError(f"cached_prompt_tokens > prompt_tokens in trace {trace.get('trace_id')}")

        non_cached_tokens = prompt_tokens - cached_tokens
        completion_tokens = d(usage.get("completion_tokens"))
        reasoning_tokens = d(usage.get("reasoning_tokens"))

        llm_input = per_token_cost(
            non_cached_tokens,
            d(model_price.get("input_per_1m")),
            batch_multiplier,
        )
        llm_cached_input = per_token_cost(
            cached_tokens,
            d(model_price.get("cached_input_per_1m", model_price.get("input_per_1m"))),
            batch_multiplier,
        )
        llm_output = per_token_cost(
            completion_tokens,
            d(model_price.get("output_per_1m")),
            batch_multiplier,
        )
        llm_reasoning = per_token_cost(
            reasoning_tokens,
            d(model_price.get("reasoning_per_1m", model_price.get("output_per_1m"))),
            batch_multiplier,
        )

    embedding_cost = Decimal("0")
    embedding_model = models.get("embedding")
    if embedding_model and d(usage.get("embedding_tokens")) > 0:
        embedding_price = pricing["embedding_models"][embedding_model]
        embedding_cost = per_token_cost(
            d(usage.get("embedding_tokens")),
            d(embedding_price.get("input_per_1m")),
            batch_multiplier,
        )

    rerank_cost = Decimal("0")
    reranker = models.get("reranker")
    if reranker and d(usage.get("rerank_units")) > 0:
        reranker_price = pricing["rerankers"][reranker]
        rerank_cost = (d(usage.get("rerank_units")) / THOUSAND) * d(reranker_price.get("per_1000_units"))

    tool_cost = Decimal("0")
    for tool_name, count in usage.get("tool_calls", {}).items():
        tool_price = pricing.get("tool_calls", {}).get(tool_name)
        if not tool_price:
            raise ValueError(f"Missing pricing for tool call {tool_name}")
        tool_cost += d(count) * d(tool_price.get("per_call"))

    return CostBreakdown(
        llm_input=money(llm_input),
        llm_cached_input=money(llm_cached_input),
        llm_output=money(llm_output),
        llm_reasoning=money(llm_reasoning),
        embedding=money(embedding_cost),
        rerank=money(rerank_cost),
        tool=money(tool_cost),
        infra=money(infra_per_request),
    )


def group_key(trace: dict) -> tuple[str, str, str, str, str]:
    models = trace.get("models", {})
    return (
        trace.get("tenant_id", "unknown"),
        trace.get("feature", "unknown"),
        trace.get("request_type", "unknown"),
        models.get("generator", "cache_or_unknown"),
        trace.get("pricing_version", "unknown"),
    )


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--traces", required=True, type=Path)
    parser.add_argument("--pricing", required=True, type=Path)
    parser.add_argument("--out", required=True, type=Path)
    args = parser.parse_args()

    pricing = load_json(args.pricing)
    summary = defaultdict(lambda: {
        "requests": 0,
        "prompt_tokens": Decimal("0"),
        "cached_prompt_tokens": Decimal("0"),
        "completion_tokens": Decimal("0"),
        "cost_total": Decimal("0"),
        "cost_llm_input": Decimal("0"),
        "cost_llm_cached_input": Decimal("0"),
        "cost_llm_output": Decimal("0"),
        "cost_embedding": Decimal("0"),
        "cost_rerank": Decimal("0"),
        "cost_tool": Decimal("0"),
        "cost_infra": Decimal("0"),
        "semantic_cache_hits": 0,
        "retries": 0,
    })

    for _, trace in iter_jsonl(args.traces):
        cost = calculate_trace_cost(trace, pricing)
        key = group_key(trace)
        row = summary[key]
        usage = trace.get("usage", {})

        row["requests"] += 1
        row["prompt_tokens"] += d(usage.get("prompt_tokens"))
        row["cached_prompt_tokens"] += d(usage.get("cached_prompt_tokens"))
        row["completion_tokens"] += d(usage.get("completion_tokens"))
        row["cost_total"] += cost.total
        row["cost_llm_input"] += cost.llm_input
        row["cost_llm_cached_input"] += cost.llm_cached_input
        row["cost_llm_output"] += cost.llm_output + cost.llm_reasoning
        row["cost_embedding"] += cost.embedding
        row["cost_rerank"] += cost.rerank
        row["cost_tool"] += cost.tool
        row["cost_infra"] += cost.infra
        row["semantic_cache_hits"] += int(bool(trace.get("cache", {}).get("semantic_cache_hit")))
        row["retries"] += int(trace.get("retry", {}).get("count", 0) or 0)

    args.out.parent.mkdir(parents=True, exist_ok=True)
    with args.out.open("w", encoding="utf-8", newline="") as f:
        writer = csv.writer(f)
        writer.writerow([
            "tenant_id",
            "feature",
            "request_type",
            "generator_model",
            "pricing_version",
            "requests",
            "avg_prompt_tokens",
            "cached_token_ratio",
            "avg_completion_tokens",
            "semantic_cache_hit_rate",
            "retry_rate",
            "cost_total_usd",
            "cost_per_request_usd",
            "cost_llm_input_usd",
            "cost_llm_cached_input_usd",
            "cost_llm_output_usd",
            "cost_embedding_usd",
            "cost_rerank_usd",
            "cost_tool_usd",
            "cost_infra_usd",
        ])

        for key, row in sorted(summary.items()):
            requests = Decimal(row["requests"])
            prompt_tokens = row["prompt_tokens"]
            cached_tokens = row["cached_prompt_tokens"]
            completion_tokens = row["completion_tokens"]
            cost_total = money(row["cost_total"])

            cached_ratio = Decimal("0") if prompt_tokens == 0 else cached_tokens / prompt_tokens
            writer.writerow([
                *key,
                row["requests"],
                money(prompt_tokens / requests),
                money(cached_ratio),
                money(completion_tokens / requests),
                money(Decimal(row["semantic_cache_hits"]) / requests),
                money(Decimal(row["retries"]) / requests),
                cost_total,
                money(cost_total / requests),
                money(row["cost_llm_input"]),
                money(row["cost_llm_cached_input"]),
                money(row["cost_llm_output"]),
                money(row["cost_embedding"]),
                money(row["cost_rerank"]),
                money(row["cost_tool"]),
                money(row["cost_infra"]),
            ])


if __name__ == "__main__":
    main()

python scripts/calc_cost_from_traces.py \
  --traces data/traces/day44-query-traces.jsonl \
  --pricing config/pricing.config.json \
  --out reports/cost-summary.csv

1. Xác nhận spike
   So sánh billing provider, internal cost summary và request volume.

2. Xác định phạm vi
   Group by tenant, feature, endpoint, model, prompt_version, route_name.

3. Kiểm tra nguyên nhân phổ biến
   - Traffic tăng thật.
   - Retry loop.
   - Prompt/context dài hơn sau deploy.
   - Cache hit giảm.
   - Router route nhiều request sang model mạnh.
   - Eval/batch job chạy nhầm realtime quota.

4. Bật mitigation
   - Disable job/offline workload.
   - Bật concise mode.
   - Giảm max output tokens.
   - Giảm context_top_k.
   - Route simple task sang small model.
   - Rate limit tenant/user gây spike.

5. Verify quality
   Chạy smoke eval cho citation/no-answer trước khi giữ mitigation lâu dài.

6. Postmortem
   Cập nhật budget policy, alert threshold, test và dashboard.

## Cost Optimization PR

### Problem
- Current cost/request:
- Current cost/day:
- Main driver:

### Change
- Technique:
- Affected endpoint/tenant:
- Config changed:

### Before/After estimate
| Metric | Before | After | Expected delta |
|---|---:|---:|---:|
| Cost/request | | | |
| p95 latency | | | |
| Prompt tokens/request | | | |
| Output tokens/request | | | |
| Cache hit rate | | | |

### Quality guardrail
- Golden set:
- Retrieval recall:
- Citation correctness:
- No-answer accuracy:
- Schema success:

### Risk
- Quality risk:
- Security/privacy risk:
- Operational risk:

### Rollout
- Canary scope:
- Monitoring window:
- Alert:

### Rollback
- Config flag:
- Owner:
- Time to rollback:

### Production readiness answer
Có/không dùng được trong production? Điều kiện còn thiếu là gì?

1,000 requests/day
70% simple FAQ
25% normal RAG
5% complex RAG
30 eval questions/day

llm-small:  input 0.15 / 1M, cached input 0.015 / 1M, output 0.60 / 1M
llm-medium: input 0.50 / 1M, cached input 0.050 / 1M, output 2.00 / 1M
llm-strong: input 2.00 / 1M, cached input 0.200 / 1M, output 8.00 / 1M
embedding-small: 0.02 / 1M tokens
reranker-base: 0.08 / 1000 units
fixed infra: 15 USD/day at 10k requests/day baseline

scenario,request_type,requests_per_day,prompt_tokens,cached_prompt_tokens,output_tokens,embedding_tokens,rerank_units,semantic_cache_hit_rate,retry_rate,model,cost_per_request,cost_per_day
1k,simple_faq,700,1800,900,220,24,8,0.35,0.01,llm-small,,
1k,normal_rag,250,4200,1800,520,32,24,0.05,0.02,llm-medium,,
1k,complex_rag,50,7800,2500,900,48,36,0.00,0.03,llm-strong,,
1k,eval,30,3800,1600,350,32,24,0.00,0.00,llm-medium,,

non_cached_input_tokens = prompt_tokens - cached_prompt_tokens

llm_cost =
  non_cached_input_tokens / 1_000_000 * input_price
+ cached_prompt_tokens / 1_000_000 * cached_input_price
+ output_tokens / 1_000_000 * output_price

embedding_cost = embedding_tokens / 1_000_000 * embedding_price
rerank_cost = rerank_units / 1000 * reranker_price
infra_cost = fixed_infra_daily / expected_requests_per_day

raw_cost_per_request = llm_cost + embedding_cost + rerank_cost + infra_cost
effective_cost_per_request =
  raw_cost_per_request * (1 - semantic_cache_hit_rate)
+ cache_lookup_cost_estimate
+ raw_cost_per_request * retry_rate

1. Request type nào tốn nhiều nhất mỗi ngày?
2. Stage nào tốn nhiều nhất: input, output, rerank, infra, retry?
3. Nếu traffic tăng 10 lần, cost có tăng tuyến tính không? Vì sao?
4. Cache hit rate cần đạt bao nhiêu để giảm 20% cost simple FAQ?
5. Retry rate tăng từ 2% lên 8% ảnh hưởng cost/month thế nào?

policies:
  simple_faq:
    max_query_tokens:
    max_context_chunks:
    max_context_tokens:
    max_output_tokens:
    max_retries:
    allowed_models:

  normal_rag:
    max_query_tokens:
    max_context_chunks:
    max_context_tokens:
    max_output_tokens:
    max_retries:
    allowed_models:

  documents_ingest:
    max_document_tokens_per_file:
    max_files_per_job:
    max_embedding_batch_tokens:
    max_retries:
    force_batch_when_tokens_over:

Static prefix:
- system instruction
- developer instruction
- response schema
- citation rules
- stable examples

Dynamic suffix:
- tenant-specific context
- retrieved chunks
- user question
- current timestamp nếu thật sự cần

Prompt caching có giảm output token cost không?
Nếu prompt version đổi mỗi deploy thì chuyện gì xảy ra?
Với dữ liệu multi-tenant, phần nào không nên xem là cache prefix dùng chung?

sc:{tenant_id}:{permission_hash}:{corpus_version}:{prompt_version}:{schema_version}:{embedding_model}:{cache_hash}

def should_use_semantic_cache(question, auth, request_type, risk_level):
    if request_type not in {"simple_faq", "normal_rag"}:
        return False
    if risk_level in {"legal", "finance", "medical", "personal_data"}:
        return False
    if auth.permission_hash is None:
        return False
    if len(question) > 500:
        return False
    return True

def build_cache_scope(auth, versions, embedding_model):
    return {
        "tenant_id": auth.tenant_id,
        "permission_hash": auth.permission_hash,
        "corpus_version": versions.corpus,
        "prompt_version": versions.prompt,
        "schema_version": versions.answer_schema,
        "embedding_model": embedding_model,
    }

def lookup_semantic_cache(question, scope):
    normalized = normalize_question(question)
    embedding = embed_query(normalized, model=scope["embedding_model"])
    hits = redis_vector_search(embedding, scope=scope, top_k=3)
    best = hits[0] if hits else None
    if best and best.score >= 0.92:
        return best.payload
    return None

def store_semantic_cache(question, answer, citations, scope):
    if not citations:
        return
    payload = {
        "question": normalize_question(question),
        "answer": answer,
        "citations": citations,
        "scope": scope,
        "ttl_seconds": 7 * 24 * 60 * 60,
    }
    redis_store_answer(payload)

routes:
  - name: simple_faq_small_model
    condition: request_type == "simple_faq" and risk_level == "low"
    model: llm-small

  - name: complex_policy_strong_model
    condition: request_type == "complex_rag" and user_tier in ["paid", "enterprise"]
    model: llm-strong

  - name: budget_conserve_mode
    condition: tenant_budget_state in ["conserve", "degraded"]
    model: llm-small

retrieve dense 50 + sparse 50
  -> merge RRF
  -> rerank top 24
  -> dedupe document/heading
  -> extract relevant spans
  -> enforce context budget
  -> build prompt

Giảm top_k từ 8 xuống 4 có thể làm cost giảm nhưng rủi ro gì?
Compression có thể làm citation sai như thế nào?
Với HR/legal/policy assistant, nên ưu tiên raw evidence span hay summary?

realtime_queue
  - user-facing query
  - strict latency SLO
  - small retry budget

offline_batch_queue
  - eval/reindex/synthetic
  - separate quota
  - resumable
  - lower priority

Task:
Teacher model:
Student model:
Training data source:
Label quality control:
Eval metrics:
Rollout plan:
Rollback plan:
Expected cost saving:
Risks:

tenant_budget:
  monthly_usd: 500
  states:
    normal:
      until_percent: 70
    watch:
      until_percent: 85
      actions: ["alert_owner"]
    conserve:
      until_percent: 95
      actions: ["concise_mode", "reduce_context_top_k"]
    degraded:
      until_percent: 100
      actions: ["small_model_for_simple_tasks", "pause_offline_jobs"]
    hard_stop:
      actions: ["reject_non_critical_requests", "admin_override_required"]

python scripts/calc_cost_from_traces.py \
  --traces data/traces/day44-query-traces.jsonl \
  --pricing config/pricing.config.json \
  --out reports/cost-summary.csv

{"trace_id":"tr_001","tenant_id":"tenant_a","feature":"rag_query","request_type":"normal_rag","pricing_version":"provider-pricing-2026-05-10","models":{"generator":"llm-medium","embedding":"embedding-small","reranker":"reranker-base"},"usage":{"prompt_tokens":4200,"cached_prompt_tokens":1800,"completion_tokens":520,"embedding_tokens":32,"rerank_units":24},"cache":{"semantic_cache_hit":false},"retry":{"count":0}}

tenant_id,feature,request_type,generator_model,requests,cost_total_usd,cost_per_request_usd,avg_prompt_tokens,cached_token_ratio,semantic_cache_hit_rate,retry_rate
tenant_a,rag_query,normal_rag,llm-medium,250,0.82,0.00328,4200,0.428,0.05,0.02

## Optimization 1

### Change

### Expected cost saving

### Quality risk

### Metrics to monitor

### Rollback

Dùng được trong production không?

Có, nếu:
- trace log đã có token/model/cache/retry/cost theo request;
- token budget được enforce ở backend;
- cache scoped theo tenant, ACL, corpus version và prompt version;
- model routing có eval per route;
- context pruning có regression eval cho citation/no-answer;
- offline batch có quota riêng;
- budget alert, degrade mode và rollback flag đã sẵn sàng.

Chưa nên production nếu:
- chỉ có estimate thủ công mà không có trace thật;
- semantic cache chưa có permission boundary;
- chưa có quality gate sau khi giảm context/model;
- không có hard stop khi retry hoặc batch job gây cost spike.