vidtheque.

read-only demo

Videos Lc8zRh9muoY

Your Agent Failed in Prod. Good Luck Reproducing It. - Tisha Chawla & Susheem Koul, Microsoft

index_state ready data_status ok

AI Engineer· published 2026-06-29· 0:14:09· en-US· indexed 2026-08-11 05:54

Open on YouTube

Scene timeline

  1. Shot 0, 0:00 to 0:29, 1 of 1 keyframes kept
  2. Shot 1, 0:29 to 0:58, 0 of 1 keyframes kept
  3. Shot 2, 0:58 to 1:27, 0 of 1 keyframes kept
  4. Shot 3, 1:27 to 1:56, 1 of 1 keyframes kept
  5. Shot 4, 1:56 to 2:25, 1 of 1 keyframes kept
  6. Shot 5, 2:25 to 3:08, 1 of 1 keyframes kept
  7. Shot 6, 3:08 to 3:32, 1 of 1 keyframes kept
  8. Shot 7, 3:32 to 3:43, 1 of 1 keyframes kept
  9. Shot 8, 3:43 to 4:31, 1 of 1 keyframes kept
  10. Shot 9, 4:31 to 5:02, 1 of 1 keyframes kept
  11. Shot 10, 5:02 to 5:34, 0 of 1 keyframes kept
  12. Shot 11, 5:34 to 6:03, 1 of 1 keyframes kept
  13. Shot 12, 6:03 to 6:32, 0 of 1 keyframes kept
  14. Shot 13, 6:32 to 7:08, 1 of 1 keyframes kept
  15. Shot 14, 7:08 to 7:22, 1 of 1 keyframes kept
  16. Shot 15, 7:22 to 8:06, 1 of 1 keyframes kept
  17. Shot 16, 8:06 to 8:35, 1 of 1 keyframes kept
  18. Shot 17, 8:35 to 8:38, 1 of 1 keyframes kept
  19. Shot 18, 8:38 to 9:16, 1 of 1 keyframes kept
  20. Shot 19, 9:16 to 9:24, 1 of 1 keyframes kept
  21. Shot 20, 9:24 to 9:26, 1 of 1 keyframes kept
  22. Shot 21, 9:26 to 9:28, 1 of 1 keyframes kept
  23. Shot 22, 9:28 to 9:36, 1 of 1 keyframes kept
  24. Shot 23, 9:36 to 9:40, 1 of 1 keyframes kept
  25. Shot 24, 9:40 to 9:43, 1 of 1 keyframes kept
  26. Shot 25, 9:43 to 10:18, 1 of 1 keyframes kept
  27. Shot 26, 10:18 to 10:19, 1 of 1 keyframes kept
  28. Shot 27, 10:19 to 11:01, 1 of 1 keyframes kept
  29. Shot 28, 11:01 to 11:03, 1 of 1 keyframes kept
  30. Shot 29, 11:03 to 11:37, 1 of 1 keyframes kept
  31. Shot 30, 11:37 to 11:40, 1 of 1 keyframes kept
  32. Shot 31, 11:40 to 12:08, 1 of 1 keyframes kept
  33. Shot 32, 12:08 to 12:38, 1 of 1 keyframes kept
  34. Shot 33, 12:38 to 13:08, 0 of 1 keyframes kept
  35. Shot 34, 13:08 to 13:52, 1 of 1 keyframes kept
  36. Shot 35, 13:52 to 14:09, 1 of 1 keyframes kept

36 shot(s).

keyframes kept every frame deduplicated

What was stored

cues
182
whisperx 182
chunks
26
from 182 cues
keyframes
31
kept of 36 captured
frames with text
31
779 lines read
chapters
0
from the source metadata
keyframe bytes
3.0 MB
word timings on 182 cues

Provenance

Each pipeline stage, its state and the model that produced it
stage state model started took
fetch done — 2026-08-11 05:51 1m 19s
stt done — 2026-08-11 05:52 16s
chunk done — 2026-08-11 05:53 0s
text_embed done — 2026-08-11 05:53 1s
keyframe done — 2026-08-11 05:53 39s
ocr done — 2026-08-11 05:53 21s
frame_embed done — 2026-08-11 05:54 5s

Frames, and what the machine read

  • 0:09 #0 done5 line(s)

    shot 0·sharpness 1415.7

    1. AlEngineer1.00
    2. World's Fair0.98
    3. Your agent failed in prod.0.98
    4. good luck reproducing it.0.98
    5. Tisha Chawla· Susheem Koul · Microsoft0.96

  • 0:32 #1 skipped

    shot 1·duplicate of #0

  • 1:18 #2 skipped

    shot 2·duplicate of #0

  • 1:41 #3 done1 line(s)

    shot 3·sharpness 369.5

    1. asked: sell $1,0000.96

  • 2:13 #4 done6 line(s)

    shot 4·sharpness 910.5

    1. asked: sell $1,0000.95
    2. sold:1.00
    3. $190,0001.00
    4. broker·POST/orders1.00
    5. 200 OK0.96
    6. order.status FILLED · $190,0000.97

  • 2:34 #5 done1 line(s)

    shot 5·sharpness 428.7

    1. temperature = 01.00

  • 3:18 #6 done14 line(s)

    shot 6·sharpness 1977.8

    1. # unedited, from the threads0.96
    2. HARD DATA1.00
    3. 1,000 prompts1.00
    4. r/LocalLLaMA0.96
    5. "different outputs at temperature = 0,1.00
    6. → 80 answers0.98
    7. mostly the MoE architecture."1.00
    8. temp 0·vLLM·Qwen-3-8B0.99
    9. r/LocalLLaMA0.96
    10. "even at temp 0 you get different0.99
    11. answers, you're using GPUs."0.99
    12. Hacker News1.00
    13. "completely correct or completely wrong,0.99
    14. depending on minute numerical diffs."1.00

  • 3:38 #7 done1 line(s)

    shot 7·sharpness 707.1

    1. First principles first0.98

  • 4:20 #8 done9 line(s)

    shot 8·sharpness 1603.2

    1. 1 sampling determinism ≠ system determinism1.00
    2. temp 0 fixes the rule (argmax), not the logits you argmax over.0.99
    3. 2 float addition is NOT associative0.99
    4. (0.1 + 1e20) - 1e20 = 0 0.1 + (1e20 − 1e20) = 0.10.92
    5. reorder a reduction → a logit's last bits move → argmax flips.0.99
    6. 31.00
    7. the culprit is batch invariance1.00
    8. same matmul, same GPU, 1000× → bitwise identical.0.99
    9. prod batches you with strangers; the kernel depends on batch shape.0.99

  • 4:56 #9 done13 line(s)

    shot 9·sharpness 2300.8

    1. 1 sampling determinism ≠ system determinism1.00
    2. temp 0 fixes the rule (argmax), not the logits you argmax over.0.99
    3. 2 float addition is NOT associative0.99
    4. (0.1 + 1e20) - 1e20 = 0 0.1 + (1e20 - 1e20) = 0.10.91
    5. reorder a reduction → a logit's last bits move → argmax flips.0.98
    6. 31.00
    7. the culprit is batch invariance1.00
    8. same matmul, same GPU, 1000× → bitwise identical.0.97
    9. prod batches you with strangers; the kernel depends on batch shape.1.00
    10. 41.00
    11. MoE routing jitter: expert capacity ceiling, route depends on the batch.1.00
    12. same token? no. we need the SYSTEM to run the same0.99
    13. STATE TRANSITION.0.98

  • 5:06 #10 skipped

    shot 10·duplicate of #8

  • 5:51 #11 done16 line(s)

    shot 11·sharpness 2243.2

    1. X wrong question: can we make the model deterministic.0.98
    2. right question: can we debug & test a run we can't reproduce.1.00
    3. determinism was never the goal. record the run, replay the recording.1.00
    4. bitwise determinism1.00
    5. replayability0.98
    6. = controllability1.00
    7. = observability0.99
    8. same input → identical output.0.99
    9. reconstruct a run that happened,1.00
    10. you won't get it from a hosted API,1.00
    11. well enough to debug. you don't1.00
    12. and you don't want it: that0.99
    13. need determinism, you need0.99
    14. randomness makes the model good.0.97
    15. the run recorded.1.00
    16. you don't freeze the model. you capture what it did.0.99

  • 6:09 #12 skipped

    shot 12·duplicate of #11

  • 6:54 #13 done15 line(s)

    shot 13·sharpness 2394.7

    1. record above the wire, not on it.1.00
    2. Xat the network layer0.99
    3. at the boundary0.99
    4. half your agent never touches1.00
    5. capture what enters each node1.00
    6. the network: local retrieval,1.00
    7. and what leaves it, every I/O,1.00
    8. in-process tools, memory.1.00
    9. network or not.0.97
    10. the socket can't record0.99
    11. the meaning of each step,1.00
    12. what isn't on it.0.98
    13. not the packets.1.00
    14. tracing records it. replay re-runs it offline: stub the model, O calls.0.97
    15. Openlnference· Arize Phoenix· LangGraph checkpointers· framework-agnostic0.98

  • 7:17 #14 done43 line(s)

    shot 14·sharpness 597.9

    1. C0.57
    2. https://onedrive.live.com/:p:/g/personal/d51d325a0b92dae6/IQBAOZIWI0tiT5nXWmK4xswbAaikJAf3vkme9N9RrJAYvL0?rtime=xpaRVQvO3kg&rede...0.99
    3. ☆0.94
    4. 品0.96
    5. 口0.65
    6. Basics1.00
    7. Quick Lookups1.00
    8. Tools1.00
    9. Sheets0.94
    10. Reference Material1.00
    11. Personal1.00
    12. Save to Instapaper1.00
    13. agent-replay-whiteboard1.00
    14. Search1.00
    15. Sign in1.00
    16. File1.00
    17. Home1.00
    18. Insert1.00
    19. Draw1.00
    20. Design1.00
    21. Transitions1.00
    22. Animations1.00
    23. Slide Show1.00
    24. Review1.00
    25. View1.00
    26. Help1.00
    27. Comments1.00
    28. Present1.00
    29. Editing1.00
    30. 91.00
    31. 40.58
    32. 演0.51
    33. 4:551.00
    34. II0.54
    35. Live Demo1.00
    36. 演0.58
    37. 111.00
    38. two kinds of check.0.88
    39. 121.00
    40. tldr:0.78
    41. 清0.57
    42. 131.00
    43. code • writeup0.89

  • 7:53 #15 done23 line(s)

    shot 15·sharpness 961.8

    1. M40.98
    2. README.md1.00
    3. trade_notional.py1.00
    4. zsh1.00
    5. {}002-place_order-1.json M1.00
    6. {}001-agent-1.json M0.98
    7. Preview README1.00
    8. ①README.md0.96
    9. Preview1.00
    10. Markdown1.00
    11. @boundary wrapper (LIVE)1.00
    12. @boundary("place_order", kind="tool")0.99
    13. annotate once0.97
    14. INPUT captured0.99
    15. symbol=ACME0.99
    16. quantity=10001.00
    17. side=sell (args → InputState)0.98
    18. def place_order(symbol: str, quantity: int) -> dict:0.99
    19. notional = quantity * SHARE_PRICE1.00
    20. return {"status": "filled", "notional_cents": notional}0.99
    21. OUTPUT captured1.00
    22. {"status": "filled", "notional_cents": 19000000, ...}0.98
    23. Envelope → store → fixtures/traces/0.99

  • 8:12 #16 done36 line(s)

    shot 16·sharpness 808.5

    1. M+README.md0.98
    2. trade_notional.py1.00
    3. zsh1.00
    4. {}002-place_order-1.json M1.00
    5. {}001-agent-1.json M0.98
    6. Preview README1.00
    7. 日0.87
    8. examples>financial_incidentstrade_notional.py0.99
    9. 261.00
    10. USER_MESSAGE = "Sell about $1,000 of ACME from my portfolio to rebalance."1.00
    11. 271.00
    12. 281.00
    13. 29 > def set_mode(mode: str) -> None: -0.95
    14. 341.00
    15. 351.00
    16. 36 > def _order_input(*args, **kwargs) -> InputState:0.97
    17. 531.00
    18. 541.00
    19. 551.00
    20. @boundary(TopL, kind="tool", extract_input=_order_input)0.98
    21. 561.00
    22. > def place_order(symbol: str, quantity: int, *, side: str = "sell") -> dict[str, Any]:0.98
    23. 861.00
    24. 871.00
    25. 881.00
    26. @boundary("agent", kind="llm", extract_input=agent_input)0.99
    27. 89 > def agent_plan(state: dict[str, Any]) -> dict[str, Any]:0.97
    28. 1021.00
    29. 1031.00
    30. 1041.00
    31. @boundary("agent", kind="llm", extract_input=agent_input)0.99
    32. 105 > def agent_finalize(state: dict[str, Any], tool_result: dict[str, Any]) -> dict[str, Any]:0.99
    33. 1201.00
    34. 1211.00
    35. 122 > def run_agent(user_message: str = USER_MESSAGE) -> dict[str, Any]:0.99
    36. 1371.00

  • 8:38 #17 done7 line(s)

    shot 17·sharpness 234.9

    1. M+0.92
    2. README.md1.00
    3. trade_notional.py1.00
    4. python3.10 {}002-place_order-1.json M{}001-agent-1.json M0.99
    5. Pre+田0.85
    6. susheemkoul@Susheems-MacBook-Pro chronicle % python examples/financial_incidents/run.py trade record0.99
    7. X0.68

  • 8:43 #18 done36 line(s)

    shot 18·sharpness 1146.7

    1. M+README.md0.94
    2. trade_notional.py1.00
    3. zsh0.92
    4. {}002-place_order-1.json M0.99
    5. {}001-agent-1.json M0.99
    6. Preview RE+0.99
    7. 日0.92
    8. susheemkoul@Susheems-MacBook-Pro chronicle % python examples/financial_incidents/run.py trade record1.00
    9. RECORD1.00
    10. trade-notional1.00
    11. User request1.00
    12. Sell about,000of ACME from my portfolio to rebalance.0.99
    13. Boundary results0.99
    14. Node1.00
    15. Kind1.00
    16. Mode1.00
    17. Input1.00
    18. Output0.99
    19. agent@11.00
    20. llm0.80
    21. LIVE1.00
    22. Sell about $1,000 of ACME from my p...0.99
    23. → place_order(symbol=ACME, quantity=1000, side=sell)0.97
    24. place_order@11.00
    25. tool1.00
    26. LIVE1.00
    27. symbol=ACME, quantity=1000, side=se...0.97
    28. filled: Sold 1000 ACME at $190.00($190,000.00 total)0.99
    29. agent@20.99
    30. llm0.91
    31. LIVE1.00
    32. tool_result: filled0.98
    33. Done. Sold 1000 ACME at $190.00 ($190,000.00 total)0.98
    34. Trace exported0.99
    35. fixtures/traces/trade-notional/1.00
    36. susheemkoul@Susheems-MacBook-Pro chronicle %0.99

  • 9:21 #19 done51 line(s)

    shot 19·sharpness 757.1

    1. M+ README.md0.89
    2. trade_notional.py1.00
    3. zsh0.86
    4. {}002-place_order-1.json M {}001-agent-1.json M0.97
    5. Preview REAC0.96
    6. 日0.93
    7. fixtures >traces > trade-notional > {}002-place_order-1.json >{}input_state >{}graph_state > symbol0.95
    8. "schema_version": "1.0",0.99
    9. "envelope_id":"2a1f7bf0-d045-4d19-9ba2-67ce2247a849",1.00
    10. "trace_id": "trace-trade-notional-001",0.99
    11. "node_id": "place_order",0.99
    12. "boundary_kind": "tool",0.99
    13. "parent_envelope_id": "0527fba4-9311-4750-994c-fb5c109e84fc",0.99
    14. 81.00
    15. "sequence": 2,0.98
    16. "invocation_index": 1,0.99
    17. 101.00
    18. "timestamp": "2026-06-19T16:00:55.136822Z",0.99
    19. 111.00
    20. "metadata": {0.99
    21. 121.00
    22. "model_version": "demo-model",0.99
    23. 131.00
    24. "sampling_params": {0.98
    25. 141.00
    26. "temperature": null,0.99
    27. 151.00
    28. "top_p": null,0.97
    29. 161.00
    30. "max_tokens": null,0.99
    31. 171.00
    32. "seed": null,0.97
    33. 181.00
    34. "extra": {}0.97
    35. 191.00
    36. 201.00
    37. "build_id": "financial-demo-trade-notional",0.99
    38. 211.00
    39. "tool_schemas": [],0.97
    40. 221.00
    41. "framework": "chronicle.boundary",0.99
    42. 231.00
    43. "node_id": "place_order",0.99
    44. 241.00
    45. "trace_id": "trace-trade-notional-001",1.00
    46. 251.00
    47. "extra": {}0.95
    48. 261.00
    49. },0.90
    50. 271.00
    51. "input_state": {0.99

  • 9:25 #20 done57 line(s)

    shot 20·sharpness 717.9

    1. M README.md0.95
    2. trade_notional.py1.00
    3. zsh0.90
    4. {}002-place_order-1.json M {}001-agent-1.json M0.98
    5. Preview REAC0.98
    6. 日0.96
    7. fixtures>traces > trade-notional > {}002-place_order-1.json >{}input_state >{}graph_state > symbol0.96
    8. 51.00
    9. "node_id": "place_order",0.98
    10. "boundary_kind":"tool",1.00
    11. "parent_envelope_id":"0527fba4-9311-4750-994c-fb5c109e84fc",1.00
    12. 81.00
    13. "sequence": 2,0.98
    14. "invocation_index": 1,0.99
    15. 101.00
    16. "timestamp":"2026-06-19T16:00:55.136822Z",1.00
    17. 111.00
    18. "metadata": {0.99
    19. 121.00
    20. "model_version":"demo-model",1.00
    21. 131.00
    22. "sampling_params": {0.99
    23. 141.00
    24. "temperature": null,0.99
    25. 151.00
    26. "top_p": null,0.98
    27. 161.00
    28. "max_tokens": null,1.00
    29. 171.00
    30. "seed": null,0.99
    31. 181.00
    32. "extra": {}0.97
    33. 191.00
    34. 200.99
    35. "build_id": "financial-demo-trade-notional",0.99
    36. 211.00
    37. "tool_schemas": [],0.95
    38. 221.00
    39. "framework": "chronicle.boundary",0.99
    40. 231.00
    41. "node_id": "place_order",0.99
    42. 241.00
    43. "trace_id": "trace-trade-notional-001",0.99
    44. 251.00
    45. "extra": {}0.97
    46. 261.00
    47. },0.93
    48. 271.00
    49. "input_state": {1.00
    50. 281.00
    51. "messages": [],0.99
    52. 291.00
    53. "system_prompt": null,0.99
    54. 301.00
    55. "rag_chunks": [],0.99
    56. 311.00
    57. "graph_state": {0.99

  • 9:28 #21 done59 line(s)

    shot 21·sharpness 708.5

    1. M README.md0.94
    2. trade_notional.py1.00
    3. zsh0.83
    4. {}002-place_order-1.json M {}001-agent-1.json M0.97
    5. Preview REAC0.98
    6. 日0.96
    7. fixtures >traces > trade-notional > {}002-place_order-1.json >{}input_state >{}graph_state > symbol0.95
    8. "parent_envelope_id": "0527fba4-9311-4750-994c-fb5c109e84fc",0.99
    9. 81.00
    10. "sequence": 2,0.97
    11. 91.00
    12. "invocation_index": 1,0.98
    13. 100.92
    14. "timestamp":"2026-06-19T16:00:55.136822Z",1.00
    15. 111.00
    16. "metadata": {0.99
    17. 121.00
    18. "model_version":"demo-model",1.00
    19. 131.00
    20. "sampling_params": {0.99
    21. 141.00
    22. "temperature": null,0.99
    23. 151.00
    24. "top_p": null,0.99
    25. 161.00
    26. "max_tokens": null,1.00
    27. 171.00
    28. "seed": null,0.97
    29. 181.00
    30. "extra": {}0.96
    31. 191.00
    32. 201.00
    33. "build_id": "financial-demo-trade-notional",0.99
    34. 211.00
    35. "tool_schemas": [],0.98
    36. 221.00
    37. "framework": "chronicle.oundary",0.99
    38. 231.00
    39. "node_id": "place_order",0.99
    40. 241.00
    41. "trace_id": "trace-trade-notional-001",0.99
    42. 251.00
    43. "extra": {}0.95
    44. 261.00
    45. },0.95
    46. 271.00
    47. "input_state": {0.98
    48. 281.00
    49. "messages": [],0.99
    50. 291.00
    51. "system_prompt": null,0.99
    52. 300.99
    53. "rag_chunks": [],0.98
    54. 311.00
    55. "graph_state": {0.99
    56. 321.00
    57. "symbol": "ACME',0.96
    58. 330.81
    59. "quantity": 1000,1.00

  • 9:34 #22 done60 line(s)

    shot 22·sharpness 740.4

    1. M+ README.md0.91
    2. trade_notional.py1.00
    3. 日 zsh0.82
    4. {}002-place_order-1.json M {}001-agent-1.json M0.97
    5. Preview REAC0.94
    6. 日0.95
    7. fixtures > traces >trade-notional > {}002-place_order-1.json >{}input_state >{}graph_state > symbol0.95
    8. 271.00
    9. "input_state": {0.98
    10. 291.00
    11. "system_prompt": null,0.99
    12. 301.00
    13. "rag_chunks": [],0.99
    14. 311.00
    15. "graph_state": {0.99
    16. 321.00
    17. "symbol": "ACME"0.96
    18. 331.00
    19. "quantity": 1000,0.95
    20. 341.00
    21. "side": "sell",0.99
    22. 351.00
    23. "share_price_cents": 19000,0.99
    24. 361.00
    25. "intended_notional_cents":100000,1.00
    26. 371.00
    27. "implied_notional_cents": 19000000,0.99
    28. 381.00
    29. "max_order_notional_cents": 5000000.99
    30. 391.00
    31. 401.00
    32. },0.94
    33. 411.00
    34. "action_result": {0.99
    35. 421.00
    36. "tool_calls": [],0.99
    37. 431.00
    38. "completion":"filled",1.00
    39. 441.00
    40. "finish_reason": null,0.98
    41. 451.00
    42. "token_usage": {},0.99
    43. 461.00
    44. "raw_response": {0.98
    45. 471.00
    46. "status": "filled",1.00
    47. 481.00
    48. "blocked": false,1.00
    49. 491.00
    50. "symbol": "ACME",0.98
    51. 501.00
    52. "quantity": 1000,1.00
    53. 511.00
    54. "side": "sell",0.99
    55. 521.00
    56. "fill_price_cents": 19000,0.99
    57. 531.00
    58. "notional_cents": 19000000,0.98
    59. 541.00
    60. "message": "Sold 1000 ACME at $190.00 ($190,000.00 total)"0.99

  • 9:40 #23 done47 line(s)

    shot 23·sharpness 598.8

    1. M README.md0.93
    2. trade_notional.py1.00
    3. zsh0.88
    4. {}002-place_order-1.json M {}001-agent-1.json M0.98
    5. Preview REAC0.94
    6. 日0.95
    7. fixtures>traces >trade-notional>{}00~/Desktop/chronicle/fixtures/traces/trade-notional/001-agent-1.json·Modified0.99
    8. 271.00
    9. "input_state": {0.99
    10. 311.00
    11. "graph_state": {0.97
    12. 381.00
    13. "max_order_notional_cents": 5000000.99
    14. 391.00
    15. 401.00
    16. 411.00
    17. "action_result": {0.98
    18. 421.00
    19. "tool_calls": [],0.98
    20. 431.00
    21. "completion": "filled",0.99
    22. 441.00
    23. "finish_reason": null,0.99
    24. 451.00
    25. "token_usage": {},0.97
    26. 461.00
    27. "raw_response": {0.97
    28. T0.99
    29. 471.00
    30. "status": "filled",0.98
    31. 481.00
    32. "blocked": false,0.98
    33. 491.00
    34. "symbol":"ACME",1.00
    35. 501.00
    36. "quantity": 1000,1.00
    37. 511.00
    38. "side": "sell",0.99
    39. 521.00
    40. "fill_price_cents": 19000,0.99
    41. 531.00
    42. "notional_cents": 19000000,1.00
    43. 541.00
    44. "message": "Sold 1000 ACME at $190.00 ($190,000.00 total)"0.99
    45. 551.00
    46. 561.00
    47. 571.00

Transcript

182 cues· 2,368 words· 12,851 chars

  1. 0:00 Imagine something your agent didn't prod was wrong.
  2. 0:04 Coy the wrong tool, it wrote the wrong thing and now suddenly your team is on call rotation to figure out what actually went wrong.
  3. 0:14 Pretty common, right?
  4. 0:15 Now, as per the standard engineering response, your gut will tell you to pull the raw prompt from the telemetry logs, pass it to the same model, using the same prompt and run it locally to isolate the bug, which we'll all do.
  5. 0:31 And surprisingly, it will work as well.
  6. 0:33 Run it again, it will work again.
  7. 0:36 You run it 10 more times, it will be just perfect every time.
  8. 0:40 But
  9. 0:41 Now let's talk about that one run which costed you and that will be gone.
  10. 0:46 You can reproduce it.
  11. 0:48 And if you can reproduce it, you can debug it.
  12. 0:50 And if you can debug it, you can promise it won't happen to your next customer or user, right?
  13. 0:57 Now, I am Tisha.
  14. 0:58 I have Sushin with me as my co-presenter.
  15. 1:01 We both run agents against real production backends.
  16. 1:05 You know, the kind of place where a bad write isn't.
  17. 1:08 Oh well, done it again.
  18. 1:09 It's you on a call with a customer explaining where the data actually went.
  19. 1:15 This whole talk is going to be about that one thing, to lose the second an agent goes hay by in production, which is being able to reproduce it.
  20. 1:24 That will be a North Star for the next 10 minutes to follow.
  21. 1:29 Now let's look at how this actually blows up.
  22. 1:33 You've got an agent hooked to a broker API, which is the scenario I'm taking.
  23. 1:37 The user says, hey, sell $1,000 of stock.
  24. 1:42 Now comes the interesting part.
  25. 1:44 Instead of doing the math, the agent sells the raw number 1,000 and dumps it straight into the quantity field.
  26. 1:53 Guess what?
  27. 1:53 It says 1,000 shares instead.
  28. 1:56 Now, at 190 bucks a share, $1,000 in tech will become how much?
  29. 2:03 $190,000 disaster, right?
  30. 2:07 And the very side part is that the API on my infrared returned a clean 200 OK in 30 milliseconds.
  31. 2:14 We got zero exceptions, zero alerts.
  32. 2:17 If you see the trade is completely wrong, but your dashboards are sitting there perfectly green, perfectly flawless.
  33. 2:26 Then such a scenario as we last discussed comes up.
  34. 2:30 What's the first thing which you will do to try and fix this?
  35. 2:35 The reflex here is to
  36. 2:37 know just turn the model temperature down to absolute zero assuming greedy decoding will make everything deterministic right but that's a complete misconception setting the temperature to zero doesn't fix a broken reasoning path it just means the model is going to make the exact same logical error the exact same way at the exact same time and honestly even worse than that
  37. 3:05 To back up the scenario we just discussed, look at the engineering threads on Reddit and Hacker News.
  38. 3:11 The hard data shows that temperature zero isn't even truly deterministic on a hardware level.
  39. 3:17 Running the same prompt a thousand times can still return dozens of completely different responses just due to the underlying GPU non-determinism and the MOE architectures which are there.
  40. 3:32 So to understand why this actually happens, we'll have to look at it from first principles.
  41. 3:38 It comes down to four simple things.
  42. 3:41 One, sampling determinism is a system determinism.
  43. 3:46 Temperature zero just means always take the argmax, but it doesn't guarantee that the underlying scores stay identical run to run.
  44. 3:55 2.
  45. 3:56 Floating point math isn't associative.
  46. 3:59 The order you add your decimal matters, right?
  47. 4:03 But a tiny shift in matrix operation alters the final logic and which in turn will flip the winning token.
  48. 4:11 3.
  49. 4:13 It's not a concurrency issue.
  50. 4:15 Rank the same matrix multiplication alone on the GPU a thousand times and I'll guarantee you'll get this exact same bits.

Open at this second