vidtheque.

read-only demo

Videos bUJgirn4_yc

When Agents Meet Physical Data: The Other Physics of Agent Harnesses - Dmitry Petrov, DataChain

index_state ready data_status ok

AI Engineer· published 2026-07-20· 0:27:32· en-US· indexed 2026-08-10 19:47

Open on YouTube

Scene timeline

  1. Shot 0, 0:00 to 0:27, 1 of 1 keyframes kept
  2. Shot 1, 0:27 to 0:56, 1 of 1 keyframes kept
  3. Shot 2, 0:56 to 1:09, 1 of 1 keyframes kept
  4. Shot 3, 1:09 to 1:31, 1 of 1 keyframes kept
  5. Shot 4, 1:31 to 1:42, 1 of 1 keyframes kept
  6. Shot 5, 1:42 to 2:31, 1 of 1 keyframes kept
  7. Shot 6, 2:31 to 2:45, 1 of 1 keyframes kept
  8. Shot 7, 2:45 to 3:20, 1 of 1 keyframes kept
  9. Shot 8, 3:20 to 3:54, 1 of 1 keyframes kept
  10. Shot 9, 3:54 to 4:21, 1 of 1 keyframes kept
  11. Shot 10, 4:21 to 4:48, 1 of 1 keyframes kept
  12. Shot 11, 4:48 to 5:15, 1 of 1 keyframes kept
  13. Shot 12, 5:15 to 5:54, 1 of 1 keyframes kept
  14. Shot 13, 5:54 to 6:18, 1 of 1 keyframes kept
  15. Shot 14, 6:18 to 6:32, 1 of 1 keyframes kept
  16. Shot 15, 6:32 to 6:40, 1 of 1 keyframes kept
  17. Shot 16, 6:40 to 6:43, 1 of 1 keyframes kept
  18. Shot 17, 6:43 to 6:46, 1 of 1 keyframes kept
  19. Shot 18, 6:46 to 6:48, 1 of 1 keyframes kept
  20. Shot 19, 6:48 to 6:50, 1 of 1 keyframes kept
  21. Shot 20, 6:50 to 6:51, 1 of 1 keyframes kept
  22. Shot 21, 6:51 to 6:55, 1 of 1 keyframes kept
  23. Shot 22, 6:55 to 7:27, 1 of 1 keyframes kept
  24. Shot 23, 7:27 to 7:44, 1 of 1 keyframes kept
  25. Shot 24, 7:44 to 7:47, 1 of 1 keyframes kept
  26. Shot 25, 7:47 to 7:54, 1 of 1 keyframes kept
  27. Shot 26, 7:54 to 8:05, 1 of 1 keyframes kept
  28. Shot 27, 8:05 to 8:16, 1 of 1 keyframes kept
  29. Shot 28, 8:16 to 8:31, 1 of 1 keyframes kept
  30. Shot 29, 8:31 to 8:34, 1 of 1 keyframes kept
  31. Shot 30, 8:34 to 8:37, 1 of 1 keyframes kept
  32. Shot 31, 8:37 to 9:05, 1 of 1 keyframes kept
  33. Shot 32, 9:05 to 9:08, 1 of 1 keyframes kept
  34. Shot 33, 9:08 to 9:11, 1 of 1 keyframes kept
  35. Shot 34, 9:11 to 9:16, 1 of 1 keyframes kept
  36. Shot 35, 9:16 to 9:18, 1 of 1 keyframes kept
  37. Shot 36, 9:18 to 9:20, 1 of 1 keyframes kept
  38. Shot 37, 9:20 to 9:22, 1 of 1 keyframes kept
  39. Shot 38, 9:22 to 9:53, 1 of 1 keyframes kept
  40. Shot 39, 9:53 to 10:13, 1 of 1 keyframes kept
  41. Shot 40, 10:13 to 10:16, 1 of 1 keyframes kept
  42. Shot 41, 10:16 to 10:17, 1 of 1 keyframes kept
  43. Shot 42, 10:17 to 10:36, 1 of 1 keyframes kept
  44. Shot 43, 10:36 to 10:41, 1 of 1 keyframes kept
  45. Shot 44, 10:41 to 10:49, 1 of 1 keyframes kept
  46. Shot 45, 10:49 to 10:54, 1 of 1 keyframes kept
  47. Shot 46, 10:54 to 11:32, 1 of 1 keyframes kept
  48. Shot 47, 11:32 to 11:34, 1 of 1 keyframes kept
  49. Shot 48, 11:34 to 11:36, 1 of 1 keyframes kept
  50. Shot 49, 11:36 to 12:06, 1 of 1 keyframes kept
  51. Shot 50, 12:06 to 12:36, 1 of 1 keyframes kept
  52. Shot 51, 12:36 to 13:06, 1 of 1 keyframes kept
  53. Shot 52, 13:06 to 13:36, 0 of 1 keyframes kept
  54. Shot 53, 13:36 to 13:52, 1 of 1 keyframes kept
  55. Shot 54, 13:52 to 14:36, 1 of 1 keyframes kept
  56. Shot 55, 14:36 to 15:05, 1 of 1 keyframes kept
  57. Shot 56, 15:05 to 15:53, 1 of 1 keyframes kept
  58. Shot 57, 15:53 to 16:24, 1 of 1 keyframes kept
  59. Shot 58, 16:24 to 16:55, 1 of 1 keyframes kept
  60. Shot 59, 16:55 to 16:59, 1 of 1 keyframes kept
  61. Shot 60, 16:59 to 17:29, 1 of 1 keyframes kept
  62. Shot 61, 17:29 to 17:58, 0 of 1 keyframes kept
  63. Shot 62, 17:58 to 18:34, 1 of 1 keyframes kept
  64. Shot 63, 18:34 to 19:09, 1 of 1 keyframes kept
  65. Shot 64, 19:09 to 19:31, 1 of 1 keyframes kept
  66. Shot 65, 19:31 to 20:04, 1 of 1 keyframes kept
  67. Shot 66, 20:04 to 20:32, 1 of 1 keyframes kept
  68. Shot 67, 20:32 to 21:01, 1 of 1 keyframes kept
  69. Shot 68, 21:01 to 21:47, 1 of 1 keyframes kept
  70. Shot 69, 21:47 to 21:59, 1 of 1 keyframes kept
  71. Shot 70, 21:59 to 22:33, 1 of 1 keyframes kept
  72. Shot 71, 22:33 to 23:04, 1 of 1 keyframes kept
  73. Shot 72, 23:04 to 23:48, 1 of 1 keyframes kept
  74. Shot 73, 23:48 to 24:06, 1 of 1 keyframes kept
  75. Shot 74, 24:06 to 24:09, 1 of 1 keyframes kept
  76. Shot 75, 24:09 to 24:14, 1 of 1 keyframes kept
  77. Shot 76, 24:14 to 24:18, 1 of 1 keyframes kept
  78. Shot 77, 24:18 to 24:27, 1 of 1 keyframes kept
  79. Shot 78, 24:27 to 24:28, 1 of 1 keyframes kept
  80. Shot 79, 24:28 to 24:29, 1 of 1 keyframes kept
  81. Shot 80, 24:29 to 24:32, 0 of 1 keyframes kept
  82. Shot 81, 24:32 to 24:33, 1 of 1 keyframes kept
  83. Shot 82, 24:33 to 24:35, 1 of 1 keyframes kept
  84. Shot 83, 24:35 to 24:39, 1 of 1 keyframes kept
  85. Shot 84, 24:39 to 25:15, 1 of 1 keyframes kept
  86. Shot 85, 25:15 to 25:51, 0 of 1 keyframes kept
  87. Shot 86, 25:51 to 25:56, 1 of 1 keyframes kept
  88. Shot 87, 25:56 to 26:31, 1 of 1 keyframes kept
  89. Shot 88, 26:31 to 27:15, 1 of 1 keyframes kept
  90. Shot 89, 27:15 to 27:31, 1 of 1 keyframes kept

90 shot(s).

keyframes kept every frame deduplicated

What was stored

cues
209
whisperx 209
chunks
48
from 209 cues
keyframes
86
kept of 90 captured
frames with text
86
2,413 lines read
chapters
0
from the source metadata
keyframe bytes
10.8 MB
word timings on 209 cues

Provenance

Each pipeline stage, its state and the model that produced it
stage state model started took
fetch done — 2026-08-10 04:15 2m 45s
stt done — 2026-08-10 04:17 25s
chunk done — 2026-08-10 04:18 0s
text_embed done — 2026-08-10 19:47 0s
keyframe done — 2026-08-10 04:18 1m 30s
ocr done — 2026-08-10 04:19 1m 07s
frame_embed done — 2026-08-10 19:47 15s

Frames, and what the machine read

  • 0:21 #0 done8 line(s)

    shot 0·sharpness 945.6

    1. 吧0.79
    2. AI ENGINEER WORLD'S FAIR 20260.99
    3. When Agents Meet Physical Data1.00
    4. The Other Physics of Agent Harnesses1.00
    5. Dmitry Petrov1.00
    6. Co-Founder,DataChain1.00
    7. [email protected]1.00
    8. @FullStackML1.00

  • 0:45 #1 done13 line(s)

    shot 1·sharpness 1159.6

    1. 吧0.83
    2. Coding agents are great at code. Not data.1.00
    3. VANILLA HARNESS0.99
    4. DATA HARNESS0.98
    5. 21%1.00
    6. 95%0.89
    7. Same Claude model. Answer accuracy on data questions.1.00
    8. OpenAI1.00
    9. Anthropic1.00
    10. In-house data agent·0.99
    11. Data analytics with Clay0.98
    12. Jan 20260.96
    13. June 20260.98

  • 1:07 #2 done16 line(s)

    shot 2·sharpness 1339.3

    1. Layers of data context (OpenAI)0.97
    2. Data agent's layers of context1.00
    3. S0.93
    4. RUNTIME CONTEXT0.99
    5. MEMORY1.00
    6. 41.00
    7. INSTITUTIONAL KNOWLEDGE0.98
    8. 31.00
    9. CODEX ENRICHMENT0.99
    10. 21.00
    11. HUMAN ANNOTATIONS0.99
    12. 11.00
    13. TABLE USAGE1.00
    14. OpenAI1.00
    15. In-house data agent·0.99
    16. Jan 20260.99

  • 1:16 #3 done9 line(s)

    shot 3·sharpness 1436.0

    1. THE EXTREME END OF DATA0.98
    2. Physical Al:1.00
    3. cross into a black hole - the laws change0.99
    4. THE FORGIVING VERSION0.99
    5. Structured warehouse0.99
    6. Schemas, indexes, a query engine1.00
    7. MY WORLD0.97
    8. Petabytes of files in S3/GCS1.00
    9. Video·sensor data·robot telemetry1.00

  • 1:41 #4 done7 line(s)

    shot 4·sharpness 1152.0

    1. 吧0.84
    2. 10 years making Al work on messy data1.00
    3. Data Scientist at Microsoft1.00
    4. DC Built DVC — git for data0.92
    5. Co-Founder at DataChain0.99
    6. Dmitry Petrov0.99
    7. @FullStackML1.00

  • 2:20 #5 done22 line(s)

    shot 5·sharpness 983.8

    1. 已0.64
    2. ONEDATAHARNESS1.00
    3. The body, rebuilt0.98
    4. The model is the brain. In this world, every part of the body gets rebuilt.0.99
    5. W0.89
    6. EYES1.00
    7. LEGS1.00
    8. HANDS1.00
    9. MEMORY1.00
    10. SEE1.00
    11. RUN1.00
    12. VERIFY1.00
    13. REMEMPER0.94
    14. Dataset1.00
    15. Engine1.00
    16. The built slice1.00
    17. Knowle1.00
    18. See data you can't hold1.00
    19. Run what won't fit0.97
    20. Check what youan't0.97
    21. Never1.00
    22. eyeballL0.92

  • 2:39 #6 done8 line(s)

    shot 6·sharpness 478.0

    1. 吧0.63
    2. PART 1 OF 4· THE EYES0.96
    3. SEE1.00
    4. Give the agent eyes.1.00
    5. SEE1.00
    6. RUN1.00
    7. VERIFY1.00
    8. REMEMBER1.00

  • 3:15 #7 done11 line(s)

    shot 7·sharpness 1000.9

    1. PART 1·SEE– PROBLEM0.94
    2. 吧0.62
    3. 2,000 files is a neutron star.1.00
    4. RADIUS = FILES0.97
    5. MASS = OBJECTS0.97
    6. 2,0001.00
    7. 100M1.00
    8. size of a city1.00
    9. outweighs the sun1.00
    10. 100M objects never fit a context window.1.00
    11. The agent is blind.0.99

  • 3:33 #8 done10 line(s)

    shot 8·sharpness 1002.8

    1. PART 1·SEE– PROBLEM0.93
    2. 2,000 files is a neutron star.0.99
    3. RADIUS = FILES0.97
    4. MASS = OBJECTS0.96
    5. 2,0001.00
    6. 100M1.00
    7. size of a city1.00
    8. outweighs the sun1.00
    9. 100M objects never fit a context window.1.00
    10. The agent is blind.0.99

  • 4:18 #9 done12 line(s)

    shot 9·sharpness 1333.2

    1. LIVE · PART 1 - THE TRAP0.94
    2. 吧0.66
    3. JSON sprawl, then two languages.0.99
    4. OPTION A·A PILE OF JSON0.97
    5. OPTION B·A DATABASE0.98
    6. Millions of tiny files0.99
    7. Python + islands of SQL1.00
    8. Slow·no versioning1.00
    9. Two languages to maintain1.00
    10. Still not a dataset1.00
    11. Worse for agents1.00
    12. Two escapes from messy files — both break.0.98

  • 4:29 #10 done11 line(s)

    shot 10·sharpness 1341.4

    1. LIVE · PART 1 - THE TRAP0.95
    2. JSON sprawl, then two languages.0.99
    3. OPTION A·A PILE OF JSON0.96
    4. OPTION B·A DATABASE0.97
    5. Millions of tiny files0.99
    6. Python + islands of SQL0.99
    7. Slow·no versioning1.00
    8. Two languages to maintain1.00
    9. Still not a dataset1.00
    10. Worse for agents1.00
    11. Two escapes from messy files — both break.0.98

  • 5:12 #11 done24 line(s)

    shot 11·sharpness 2046.8

    1. LIVE· PART 1 - SOLUTION0.96
    2. One language. One schema: Pydantic.1.00
    3. Type the rows with Pydantic — one queryable, versioned table.0.99
    4. class File(BaseModel):1.00
    5. path: str0.97
    6. size: int = 01.00
    7. last_modified: datetime0.99
    8. etag: str = ""0.97
    9. typed rows1.00
    10. schema1.00
    11. version: str = ""0.99
    12. class BBox(BaseModel):1.00
    13. x: int y: int w: int h: int0.98
    14. versions1.00
    15. lineage1.00
    16. class Detection(BaseModel):0.99
    17. file: File0.96
    18. Rows point to S3 – bytes sten0.95
    19. the bucket.0.95
    20. frame: int label: str0.99
    21. bbox:BBox0.96
    22. confidence: float0.99
    23. velocity: float | None0.96
    24. No such thing as unstructured data — only a schema you hayen'0.97

  • 5:46 #12 done16 line(s)

    shot 12·sharpness 1653.1

    1. LIVE · PART 1 - SOLUTION0.96
    2. 吧0.63
    3. Now the agent can see.1.00
    4. time1.00
    5. class1.00
    6. speed1.00
    7. > objects.filter(1.00
    8. label=="pedestrian",0.99
    9. 00:14:21 pedestrian 1.4 m/s1.00
    10. velocity > 0) 非 moving0.95
    11. 00:51:07 pedestrian 0.9 m/s1.00
    12. scanning 100M objects..0.95
    13. 01:32:55 pedestrian 1.8 m/s0.99
    14. done in 12 ms · 0 bytes in0.97
    15. RAM1.00
    16. Milliseconds. Nothir0.99

  • 6:11 #13 done3 line(s)

    shot 13·sharpness 460.3

    1. ~/src/demo0.99
    2. demo/ $ claude --dangerously-skip-permissions0.99
    3. bck-i-search: ski_0.98

  • 6:30 #14 done8 line(s)

    shot 14·sharpness 1608.4

    1. ~/src/demo0.96
    2. demo/ $ datachain skill install --local --target claude0.99
    3. Installed skills (local, target=claude):0.98
    4. core → /Users/dmitry/src/demo/.claude/skills/core1.00
    5. knowledge → /Users/dmitry/src/demo/.claude/skills/knowledge1.00
    6. jobs → /Users/dmitry/src/demo/.claude/skills/jobs0.99
    7. demo/$1.00
    8. demo/ $ claude --dangerously-skip-permissions0.99

  • 6:33 #15 done15 line(s)

    shot 15·sharpness 1976.5

    1. *Claude Code0.92
    2. demo/ $ datachain skill install --local --target claude0.99
    3. Installed skills (local, target=claude):0.99
    4. core → /Users/dmitry/src/demo/.claude/skills/core1.00
    5. knowledge → /Users/dmitry/src/demo/.claude/skills/knowledge1.00
    6. jobs → /Users/dmitry/src/demo/.claude/skills/jobs0.99
    7. demo/$1.00
    8. demo/ $ claude --dangerously-skip-permissions1.00
    9. Claude Code v2.1.1911.00
    10. Opus 4.8 (1M context) · Claude Team0.95
    11. ~/src/demo1.00
    12. "fix lint errors"1.00
    13. demo/ 0pus 4.8 (1M context)0.97
    14. high·/effort1.00
    15. bypass permissions on (shift+tab to cycle). ← for agents0.98

  • 6:42 #16 done16 line(s)

    shot 16·sharpness 2723.6

    1. *Claude Code0.96
    2. demo/ $ datachain skill install --local --target claude1.00
    3. Installed skills (local, target=claude):0.99
    4. core → /Users/dmitry/src/demo/.claude/skills/core1.00
    5. knowledge → /Users/dmitry/src/demo/.claude/skills/knowledge1.00
    6. jobs → /Users/dmitry/src/demo/.claude/skills/jobs0.99
    7. demo/$1.00
    8. demo/ $ claude --dangerously-skip-permissions0.99
    9. Claude Code v2.1.1911.00
    10. Opus 4.8 (1M context) · Claude Team0.96
    11. ~/src/demo1.00
    12. Build fleet detections from the dashcam clips in s3://dc-readme/fleet-cameras/2026-01/front/1.00
    13. Detected objects with velocity. Save is as dashcam-jan dataset.1.00
    14. demo/ Opus 4.8 (1M context)0.98
    15. ctrl+g to edit in Vim1.00
    16. bypass permissions on (shift+tab to cycle)1.00

  • 6:44 #17 done17 line(s)

    shot 17·sharpness 2662.2

    1. ·Build fleet detections from dashcam clips0.99
    2. demo/ $ datachain skill install --local --target claude0.99
    3. Installed skills (local, target=claude):0.99
    4. core → /Users/dmitry/src/demo/.claude/skills/core1.00
    5. knowledge → /Users/dmitry/src/demo/.claude/skills/knowledge1.00
    6. jobs → /Users/dmitry/src/demo/.claude/skills/jobs0.99
    7. demo/$1.00
    8. demo/ $ claude --dangerously-skip-permissions0.99
    9. Claude Code v2.1.1910.98
    10. Opus 4.8 (1M context) · Claude Team0.96
    11. ~/src/demo1.00
    12. Build fleet detections from the dashcam clips in s3://dc-readme/fleet-cameras/2026-01/front/1.00
    13. Detected objects with velocity. Save is as dashcam-jan dataset.0.99
    14. Pollinating...0.98
    15. demo/ Opus 4.8 (1M context)0.98
    16. high·/effort1.00
    17. bypass permissions on (shift+tab to cycle)· ← for agents0.97

  • 6:47 #18 done16 line(s)

    shot 18·sharpness 2517.1

    1. Build fleet detections from dashcam clips1.00
    2. knowledge → /Users/dmitry/src/demo/.claude/skills/knowledge1.00
    3. jobs → /Users/dmitry/src/demo/.claude/skills/jobs1.00
    4. demo/$1.00
    5. demo/ $ claude --dangerously-skip-permissions1.00
    6. Claude Code v2.1.1911.00
    7. Opus 4.8 (1M context)· Claude Team0.97
    8. ~/src/demo1.00
    9. Build fleet detections from the dashcam clips in s3://dc-readme/fleet-cameras/2026-01/front/1.00
    10. Detected objects with velocity. Save is as dashcam-jan dataset.0.99
    11. I'll start by loading the datachain-knowledge skill since this involves creating a dataset from1.00
    12. an S3 bucket.1.00
    13. Pollinating… (4s · ↓ 146 tokens · thought for 1s)0.97
    14. demo/ Opus 4.8 (1M context) 2% ctx0.97
    15. high·/effort1.00
    16. bypass permissions on (shift+tab to cycle)· ← for agents0.97

  • 6:48 #19 done17 line(s)

    shot 19·sharpness 2263.2

    1. Build fleet detections from dashcam clips0.99
    2. demo/$1.00
    3. demo/ $ claude --dangerously-skip-permissions0.99
    4. Claude Code v2.1.1911.00
    5. Opus 4.8 (1M context) · Claude Team0.98
    6. ~/src/demo1.00
    7. Build fleet detections from the dashcam clips in s3://dc-readme/fleet-cameras/2026-01/front/0.99
    8. Detected objects with velocity. Save is as dashcam-jan dataset.0.99
    9. I'll start by loading the datachain-knowledge skill since this involves creating a dataset from0.99
    10. an S3 bucket.1.00
    11. Skill(/datachain-knowledge)1.00
    12. Successfully loaded skill1.00
    13. *0.90
    14. Pollinating… (5s · ↑ 207 tokens · thought for 1s)0.97
    15. demo/ 0pus 4.8 (1M context) 2% ctx0.98
    16. high·/effort1.00
    17. bypass permissions on (shift+tab to cycle). ← for agents0.97

  • 6:50 #20 done14 line(s)

    shot 20·sharpness 2464.7

    1. Build fleet detections from dashcam clips0.99
    2. demo/$1.00
    3. demo/ $ claude --dangerously-skip-permissions0.99
    4. Claude Code v2.1.1911.00
    5. Opus 4.8 (1M context)· Claude Team0.99
    6. ~/src/demo1.00
    7. Build fleet detections from the dashcam clips in s3://dc-readme/fleet-cameras/2026-01/front/1.00
    8. Detected objects with velocity. Save is as dashcam-jan dataset.1.00
    9. I'll start by loading the datachain-knowledge skill since this involves creating a dataset from0.99
    10. an S3 bucket.0.99
    11. Skill(/datachain-knowledge)1.00
    12. Successfully loaded skill1.00
    13. This is Mode B (Dataset Creation). Let me start with the required precondition reads.0.99
    14. * Pollinating.… (7s · ↓ 358 tokens · thinking)0.95

  • 6:52 #21 done16 line(s)

    shot 21·sharpness 2627.6

    1. Build fleet detections from dashcam clips0.99
    2. demo/$1.00
    3. demo/ $ claude --dangerously-skip-permissions0.99
    4. Claude Code v2.1.1911.00
    5. Opus 4.8 (1M context)· Claude Team0.99
    6. ~/src/demo1.00
    7. Build fleet detections from the dashcam clips in s3://dc-readme/fleet-cameras/2026-01/front/1.00
    8. Detected objects with velocity. Save is as dashcam-jan dataset.1.00
    9. I'll start by loading the datachain-knowledge skill since this involves creating a dataset from1.00
    10. an S3 bucket.1.00
    11. Skill(/datachain-knowledge)1.00
    12. Successfully loaded skill0.98
    13. This is Mode B (Dataset Creation). Let me start with the required precondition reads.0.99
    14. Reading 2 files… (ctrl+o to expand)0.98
    15. L $ cat /Users/dmitry/src/demo/dc-knowledge/index.md 2>/dev/null Il echo "INDEX_NOT_FOUND"0.98
    16. Pollinating.… (9s · ↓ 423 tokens · thought for 1s)0.97

  • 7:17 #22 done17 line(s)

    shot 22·sharpness 3381.2

    1. ·Build fleet detections from dashcam clips0.98
    2. knowledge → /Users/dmitry/src/demo/.claude/skills/knowledge1.00
    3. jobs → /Users/dmitry/src/demo/.claude/skills/jobs0.99
    4. demo/$1.00
    5. demo/ $ claude --dangerously-skip-permissions0.98
    6. Claude Code v2.1.1911.00
    7. Opus 4.8 (1M context)· Claude Team0.98
    8. ~/src/demo1.00
    9. Build fleet detections from the dashcam clips in s3://dc-readme/fleet-cameras/2026-01/front/1.00
    10. Detected objects with velocity. Save is as dashcam-jan dataset.0.99
    11. I'll start by loading the datachain-knowledge skill since this involves creating a dataset from1.00
    12. an S3 bucket.0.99
    13. Skill(/datachain-knowledge)1.00
    14. Successfully loaded skill1.00
    15. This is Mode B (Dataset Creation). Let me start with the required precondition reads.0.99
    16. Read 2 files (ctrl+o to expand)0.98
    17. Methodology loaded. This is a Sense (L3) build - object detection + velocity on dashcam video.0.98

  • 7:32 #23 done18 line(s)

    shot 23·sharpness 2531.9

    1. * Build fleet detections from dashcam clips0.98
    2. Model1.00
    3. Velocity Granularity □ Scope √ Submit →0.94
    4. Which detection model? (method is ultralytics YOLO + ByteTrack tracker - the one sensible CPU0.98
    5. option here at 28.6 ms/frame; this picks the weight class within it. Confidence threshold defaults0.99
    6. to 0.25, overridable.)0.97
    7. )1. yolov8n(~8 min)✓0.89
    8. Nano. Fastest, ~28.6 ms/frame measured. ~8 min for 91 clips. Lower accuracy on small/distant1.00
    9. objects. Good default for a first fleet pass. $0 (local CPU).0.99
    10. 2. yolov8s (~12 min)0.99
    11. Small. ~3x inference cost, better recall on small objects. ~12 min for 91 clips. $0 (local1.00
    12. CPU).0.97
    13. 3. yolov8m (~20 min)1.00
    14. Medium. ~7x inference cost, best accuracy of the three on CPU. ~20 min for 91 clips. $00.99
    15. (local CPU).0.97
    16. 4. Type something.0.95
    17. 5. Chat about this0.97
    18. Enter to select · Tab/Arrow keys to navigate · Esc to cancel0.98

Transcript

209 cues· 3,197 words· 17,536 chars

  1. 0:02 How coding agent work with physical data?
  2. 0:06 Video recordings, sensor data, robot's telemetry, and sometimes all of those combine in a single multi-model project.
  3. 0:15 If you try this, you probably have seen how badly it fails.
  4. 0:18 And today we'll discuss the reasons and all the physics laws behind the problems and how to fix it.
  5. 0:30 This year, two frontier labs published very interesting results and surprising results that agents in general are not good at data.
  6. 0:43 Anthropic published that accuracy for data projects on their agents is only 21% until you add specific data harnesses to them and provide context.
  7. 0:57 OpenAI published a whole layers of context, six layers of context in order to make the data agent work.
  8. 1:07 And all of those are unstructured data, which lives in a very houses with tables, execution engine, and all this like luxury.
  9. 1:20 In my life, I don't have this luxury, unfortunately, because I live in a very extreme side of the data universe, messy, unstructured data.
  10. 1:32 I worked with data for about 10 years.
  11. 1:35 I built a data version control project, Git for Data, and now work on data chain.
  12. 1:45 In order to make
  13. 1:47 agent work for unstructured data, for physical data, we need to build not only the brain, which
  14. 1:57 we already have, right?
  15. 1:58 It's LLM.
  16. 2:00 But we need to make data harness for agents to understand this physical world.
  17. 2:08 It should see the data properly.
  18. 2:10 It should be able to run, kind of giving him a leg.
  19. 2:15 It should be able to touch data, verify the result, run tests, and also remember
  20. 2:21 the crucial important data sets important result so this information could be reused in the future project and the first we start from the how you see data what does it take to understand all these complicated binary file that you have in your object storages
  21. 2:48 In reality, how it usually looks, there are just several files, sometimes several thousand of files, right?
  22. 2:55 It doesn't seem like a big deal, right?
  23. 2:57 But what usually happens, those files are very complex inside.
  24. 3:04 That's what makes unstructured multimodal data complicated.
  25. 3:10 Because video recordings might have
  26. 3:13 Clips inside.
  27. 3:14 Clips might have frames, frames, objects, objects, confident, type, class, label.
  28. 3:23 There's connection between those pieces together.
  29. 3:27 It makes kind of like explosion.
  30. 3:29 It looks like a neutron star.
  31. 3:34 It's small size on the surface, right?
  32. 3:37 It's just a size of one city.
  33. 3:40 But the mass of this object is tremendous.
  34. 3:43 It's more than the mass of our sun.
  35. 3:46 And 2,000 objects, 2,000 files of videos could easily generate you millions of objects inside the videos.
  36. 3:56 And how people usually do deal with these problems?
  37. 3:59 They usually go through like two major steps.
  38. 4:02 First step.
  39. 4:03 Let's put this meta information to JSON files and put it on S3 next to the images, right?
  40. 4:09 And they end up with millions of Jasons.
  41. 4:13 Crazy latency, not efficiency, not consistency.
  42. 4:18 And the next idea, why don't we use database?
  43. 4:23 Brilliant.
  44. 4:23 And the most advanced team do exactly this.
  45. 4:26 Let's put a centralized database when all the metadata is in there.
  46. 4:33 great but this way you end up with a two system with a two programming languages and all the mess around two different stacks and this text useless for most of the researchers because they don't want to deal with this complexity
  47. 4:51 We found that the easiest way for researchers and developers to deal with the schema is Pydentic.
  48. 4:58 So you use the same language for the data, for the schemas, as well as code.
  49. 5:04 There are no SQL islands in your code base.
  50. 5:07 So in that way, you kind of transition from the massive world of unstructured data to the structure.

Open at this second