32.5 Measuring cost per successful task

The Right Metric

Don't measure cost per request. Measure cost per successful task.

// Bad metric: Cost per request = $0.01
// But 30% of requests fail and need retry...
// Actual cost per SUCCESS = $0.01 / 0.7 = $0.014

// Even worse: Some successes are wrong
// Accuracy = 85%
// Cost per CORRECT answer = $0.014 / 0.85 = $0.016

Tracking Implementation

interface TaskMetrics {
  taskId: string;
  attempts: number;
  totalTokens: number;
  totalCost: number;
  success: boolean;
  correct: boolean | null;  // null if not yet evaluated
}

async function trackTask(task: Task): Promise {
  const metrics: TaskMetrics = {
    taskId: task.id,
    attempts: 0,
    totalTokens: 0,
    totalCost: 0,
    success: false,
    correct: null
  };
  
  while (metrics.attempts < 3 && !metrics.success) {
    metrics.attempts++;
    const result = await executeTask(task);
    
    metrics.totalTokens += result.tokens;
    metrics.totalCost += result.cost;
    metrics.success = result.success;
  }
  
  return metrics;
}

// Aggregate metrics
function calculateCostPerSuccess(metrics: TaskMetrics[]) {
  const successful = metrics.filter(m => m.success);
  const totalCost = metrics.reduce((sum, m) => sum + m.totalCost, 0);
  
  return totalCost / successful.length;
}

Optimization Loop

// Weekly optimization review
1. Calculate cost per successful task by feature
2. Identify highest-cost features
3. For each high-cost feature:
   - Can we use a smaller model?
   - Can we reduce context size?
   - Can we improve first-attempt success rate?
   - Can we batch requests?

4. A/B test changes
5. Compare new cost per success
6. Repeat

Where to go next

• 33. Latency Optimization