vidtheque.

read-only demo

Videos iKQ78wyJEXU

We Vetted 2000 AI Skills Before They Reached Developers — Lucas Palma, Nubank

index_state ready data_status ok

AI Engineer· published 2026-07-29· 0:16:24· en-US· indexed 2026-08-10 19:42

Open on YouTube

Scene timeline

  1. Shot 0, 0:00 to 0:03, 1 of 1 keyframes kept
  2. Shot 1, 0:03 to 0:05, 1 of 1 keyframes kept
  3. Shot 2, 0:05 to 0:12, 1 of 1 keyframes kept
  4. Shot 3, 0:12 to 0:27, 1 of 1 keyframes kept
  5. Shot 4, 0:27 to 1:02, 1 of 1 keyframes kept
  6. Shot 5, 1:02 to 1:36, 1 of 1 keyframes kept
  7. Shot 6, 1:36 to 2:10, 0 of 1 keyframes kept
  8. Shot 7, 2:10 to 2:40, 1 of 1 keyframes kept
  9. Shot 8, 2:40 to 3:13, 1 of 1 keyframes kept
  10. Shot 9, 3:13 to 3:46, 0 of 1 keyframes kept
  11. Shot 10, 3:46 to 4:26, 1 of 1 keyframes kept
  12. Shot 11, 4:26 to 4:52, 0 of 1 keyframes kept
  13. Shot 12, 4:52 to 5:19, 0 of 1 keyframes kept
  14. Shot 13, 5:19 to 5:46, 0 of 1 keyframes kept
  15. Shot 14, 5:46 to 6:21, 1 of 1 keyframes kept
  16. Shot 15, 6:21 to 6:57, 0 of 1 keyframes kept
  17. Shot 16, 6:57 to 7:25, 0 of 1 keyframes kept
  18. Shot 17, 7:25 to 7:52, 1 of 1 keyframes kept
  19. Shot 18, 7:52 to 8:20, 0 of 1 keyframes kept
  20. Shot 19, 8:20 to 8:48, 0 of 1 keyframes kept
  21. Shot 20, 8:48 to 9:16, 0 of 1 keyframes kept
  22. Shot 21, 9:16 to 9:56, 1 of 1 keyframes kept
  23. Shot 22, 9:56 to 10:22, 1 of 1 keyframes kept
  24. Shot 23, 10:22 to 10:49, 0 of 1 keyframes kept
  25. Shot 24, 10:49 to 11:15, 0 of 1 keyframes kept
  26. Shot 25, 11:15 to 11:48, 1 of 1 keyframes kept
  27. Shot 26, 11:48 to 12:21, 0 of 1 keyframes kept
  28. Shot 27, 12:21 to 12:54, 0 of 1 keyframes kept
  29. Shot 28, 12:54 to 13:34, 1 of 1 keyframes kept
  30. Shot 29, 13:34 to 14:21, 0 of 1 keyframes kept
  31. Shot 30, 14:21 to 14:49, 0 of 1 keyframes kept
  32. Shot 31, 14:49 to 15:16, 0 of 1 keyframes kept
  33. Shot 32, 15:16 to 15:43, 0 of 1 keyframes kept
  34. Shot 33, 15:43 to 15:55, 1 of 1 keyframes kept
  35. Shot 34, 15:55 to 16:07, 1 of 1 keyframes kept
  36. Shot 35, 16:07 to 16:23, 0 of 1 keyframes kept

36 shot(s).

keyframes kept every frame deduplicated

What was stored

cues
128
whisperx 128
chunks
28
from 128 cues
keyframes
17
kept of 36 captured
frames with text
17
348 lines read
chapters
11
from the source metadata
keyframe bytes
4.5 MB
word timings on 128 cues

Provenance

Each pipeline stage, its state and the model that produced it
stage state model started took
fetch done — 2026-08-09 23:29 0s
stt done — 2026-08-09 14:26 16s
chunk done — 2026-08-09 14:27 0s
text_embed done — 2026-08-10 19:42 1s
keyframe done — 2026-08-09 14:27 3m 26s
ocr done — 2026-08-09 14:30 9s
frame_embed done — 2026-08-10 19:42 3s

Frames, and what the machine read

  • 0:02 #0 done2 line(s)

    shot 0·sharpness 455.2

    1. AlEngineer0.95
    2. World's Fair0.97

  • 0:03 #1 done2 line(s)

    shot 1·sharpness 670.0

    1. AlEngineer0.96
    2. World's Fair0.99

  • 0:11 #2 done24 line(s)

    shot 2·sharpness 2742.1

    1. LAB & PLATINUM SPONSORS0.97
    2. Amazon AGI Lab0.99
    3. ANTHROP\C1.00
    4. Google DeepMind1.00
    5. MINIMAX0.98
    6. OpenAl0.92
    7. Akamai1.00
    8. arize1.00
    9. aws1.00
    10. Braintrust bright data0.99
    11. B1.00
    12. Browserbase1.00
    13. docker1.00
    14. :neo4j0.91
    15. ORACLE1.00
    16. PayPal1.00
    17. qodo0.99
    18. reducto1.00
    19. Sonar1.00
    20. Makers of1.00
    21. together.ai0.98
    22. Unblocked1.00
    23. WorkOS1.00
    24. SonarQube1.00

  • 0:18 #3 done2 line(s)

    shot 3·sharpness 48.1

    1. AlEngineer0.97
    2. World's Fair0.98

  • 0:38 #4 done14 line(s)

    shot 4·sharpness 1760.4

    1. Al Engineer World's Fair, July 20261.00
    2. AlEngineer0.99
    3. World's Fair0.97
    4. Who am I0.92
    5. LP (Lucas Palma)1.00
    6. PRESENTED BY1.00
    7. Microsoft1.00
    8. Product Security Manager at1.00
    9. Nubank1.00
    10. Financial Services,1.00
    11. Engineering, Security,0.99
    12. Innovation1.00
    13. Engineering the future of Al0.98
    14. World's Fair0.96

  • 1:13 #5 done25 line(s)

    shot 5·sharpness 2538.4

    1. AlEngineer0.98
    2. The story1.00
    3. World's Fair0.98
    4. THE SHIFT1.00
    5. THE RISK0.94
    6. THE SYSTEM0.99
    7. THE LESSON1.00
    8. PRESENTED BY1.00
    9. Al skills became part0.98
    10. of the developer0.99
    11. configuration, but0.99
    12. They looked like1.00
    13. review before skills0.99
    14. We built security1.00
    15. workflow, not only the0.99
    16. Protect the whole1.00
    17. workflow.1.00
    18. behaved like supply0.98
    19. reached developers.1.00
    20. generated code.1.00
    21. Microsoft1.00
    22. chain dependencies.1.00
    23. TRACK 3· JULY 2, 20260.96
    24. Al in Finance0.99
    25. World's Fair1.00

  • 2:03 #6 skipped

    shot 6·duplicate of #5

  • 2:22 #7 done20 line(s)

    shot 7·sharpness 2729.7

    1. AlEngineer0.99
    2. The new supply chain does not always look like code0.99
    3. World's Fair0.98
    4. TRADITIONAL SUPPLY CHAIN1.00
    5. AI-ERA SUPPLY CHAIN0.99
    6. Packages1.00
    7. Traditional supply chain1.00
    8. Skills1.00
    9. Containers1.00
    10. Plugins1.00
    11. CI/CD dependencies0.99
    12. MCP servers1.00
    13. Agent rules1.00
    14. Infrastructure modules1.00
    15. Tool manifests1.00
    16. Prompt instructions1.00
    17. Local automation scripts0.99
    18. TRACK 3· JULY 2, 20260.93
    19. Al in Finance0.99
    20. World'sFair1.00

  • 2:57 #8 done21 line(s)

    shot 8·sharpness 3118.8

    1. AlEngineer0.99
    2. What is an Al skill?1.00
    3. World'sFair1.00
    4. Developer1.00
    5. AI Tool0.97
    6. System Output0.93
    7. AI Skill0.99
    8. CORE CAPABILITY (WHAT A SKILL IS)0.98
    9. IMPACT & SCALE (WHAT CHANGES / HOW FAR REACHES)0.98
    10. A packaged capability for a model or agent0.99
    11. Shapes how developers write, review, or modify code1.00
    12. Bundles instructions, context, workflows, and0.98
    13. Reusable and shareable across many people and0.97
    14. supporting files0.99
    15. projects1.00
    16. May describe tools, commands, or how to interact0.98
    17. Turns guidance into concrete output0.99
    18. with systems1.00
    19. TRACK 3· JULY 2, 20260.94
    20. Al in Finance0.95
    21. World'sFair1.00

  • 3:36 #9 skipped

    shot 9·duplicate of #8

  • 4:02 #10 done28 line(s)

    shot 10·sharpness 3098.0

    1. In regulated environments, Al adoption needs speed and control1.00
    2. AlEngineer0.99
    3. World's Fair0.96
    4. DEVELOPERS NEED0.99
    5. Developer1.00
    6. Customer &1.00
    7. TRUST & SAFETY NEEDS1.00
    8. Velocity1.00
    9. Production1.00
    10. Faster coding1.00
    11. safety1.00
    12. Customer protection1.00
    13. Better context0.97
    14. Auditability1.00
    15. Less repetitive work1.00
    16. Credential safety1.00
    17. VELOCITY1.00
    18. SAFETY1.00
    19. Production safety1.00
    20. Reusable workflows1.00
    21. Clear ownership1.00
    22. Regulated Balance: Controls must0.98
    23. Safety by default1.00
    24. run transparently within the0.98
    25. existing workflow.1.00
    26. TRACK 3· JULY 2,20260.95
    27. Al in Finance1.00
    28. World's Fair0.97

  • 4:49 #11 skipped

    shot 11·duplicate of #7

  • 5:01 #12 skipped

    shot 12·duplicate of #7

  • 5:32 #13 skipped

    shot 13·duplicate of #5

  • 5:53 #14 done33 line(s)

    shot 14·sharpness 4021.4

    1. AlEngineer0.99
    2. Review before distribution, not after adoption1.00
    3. World's Fair0.98
    4. THE MARKETPLACE PROBLEM1.00
    5. VETTING PIPELINE WORKFLOW0.98
    6. 011.00
    7. Skill Creator1.00
    8. Internal marketplace makes skills discoverable1.00
    9. Author packages the capability1.00
    10. PRESENTED BY1.00
    11. Microsoft1.00
    12. Discoverability creates rapid organizational scale0.99
    13. 021.00
    14. Pull Request1.00
    15. Code submission triggers automation1.00
    16. Scale turns small mistakes into widely repeated1.00
    17. behavior1.00
    18. 031.00
    19. Skill Vetter GATE0.97
    20. Security review & risk classification1.00
    21. Therefore, review must happen before skills ever1.00
    22. reach the marketplace0.99
    23. 041.00
    24. Marketplace1.00
    25. Approved & discoverable directory0.99
    26. "The marketplace boundary became0.99
    27. our security boundary."1.00
    28. 051.00
    29. Developers1.00
    30. Safe adoption in developer workflows0.98
    31. TRACK 3· JULY 2, 20260.94
    32. Al in Finance1.00
    33. World's Fair0.98

  • 6:26 #15 skipped

    shot 15·duplicate of #14

  • 7:21 #16 skipped

    shot 16·duplicate of #7

  • 7:39 #17 done38 line(s)

    shot 17·sharpness 3586.2

    1. AlEngineer0.99
    2. Skill Vetter: local checks, Cl checks, marketplace gate1.00
    3. World'sFair1.00
    4. HIGH-LEVEL FLOW0.97
    5. VETTING PIPELINE WORKFLOW1.00
    6. 1.1.00
    7. A skill is created or changed0.99
    8. Local scan1.00
    9. 2.1.00
    10. Engineers can run the scanner locally1.00
    11. Pull request1.00
    12. 3.1.00
    13. Cl runs automatically when a skill is added or modified0.98
    14. 031.00
    15. Deterministic scanners0.99
    16. 4.1.00
    17. Deterministic checks flag known risk patterns1.00
    18. 041.00
    19. LLM review1.00
    20. 5.1.00
    21. LLM-based analysis reviews higher-context behavior0.99
    22. 6.1.00
    23. Findings are reported in PR0.99
    24. 051.00
    25. PR feedback1.00
    26. 7.1.00
    27. Findings are converted to SARIF / code scanning1.00
    28. 061.00
    29. SARIF1.00
    30. 8.1.00
    31. Depending on severity and policy, the skill can require0.99
    32. 071.00
    33. Marketplace decision1.00
    34. remediation or be blocked before marketplace distribution.0.99
    35. TRACK 3· JULY 2, 20260.95
    36. Al in Finance0.97
    37. AlEngineer1.00
    38. World'sFair1.00

  • 8:14 #18 skipped

    shot 18·duplicate of #7

  • 8:45 #19 skipped

    shot 19·duplicate of #17

  • 8:51 #20 skipped

    shot 20·duplicate of #7

  • 9:24 #21 done43 line(s)

    shot 21·sharpness 3344.5

    1. AlEngineer0.99
    2. What we scanned1.00
    3. World's Fair0.98
    4. INSTRUCTIONS1.00
    5. COMMANDS1.00
    6. DATA & CREDENTIALS1.00
    7. TOOLS1.00
    8. Unsafe instructions1.00
    9. Destructive shell1.00
    10. Credential1.00
    11. Risky MCP/tool1.00
    12. commands1.00
    13. requests1.00
    14. use1.00
    15. Review1.00
    16. manipulation1.00
    17. Dynamic command1.00
    18. Credential files1.00
    19. Over-broad1.00
    20. execution1.00
    21. permissions1.00
    22. Agent behavior1.00
    23. Sensitive paths0.97
    24. drift1.00
    25. Production-impacti1.00
    26. Shared service0.98
    27. ng CLl usage0.97
    28. principals1.00
    29. Sensitive data1.00
    30. Prompt-level1.00
    31. exposure1.00
    32. approvals1.00
    33. Host/file1.00
    34. Lack of host-level1.00
    35. pretending to be1.00
    36. modifications1.00
    37. approval1.00
    38. controls1.00
    39. External calls or1.00
    40. exfiltration paths1.00
    41. TRACK 3· JULY 2, 20260.96
    42. Al in Finance0.99
    43. World's Fair0.96

  • 10:02 #22 done26 line(s)

    shot 22·sharpness 4019.7

    1. AlEngineer0.99
    2. What we found after reviewing 2,000+ skills1.00
    3. World's Fair0.98
    4. SCOPE1.00
    5. RISKS1.00
    6. RESOLVED1.00
    7. RED FLAGS1.00
    8. 2K+1.00
    9. 1.6K1.00
    10. ~1K1.00
    11. ~901.00
    12. Skills1.00
    13. Analyzed1.00
    14. Potential1.00
    15. Risks Found1.00
    16. Issues1.00
    17. Remediated1.00
    18. Priority Review Cases1.00
    19. ADDITIONAL INSIGHTS & HISTORICAL CONTEXT1.00
    20. • ~650 previously unvetted skills0.98
    21. • ~220 skills escalated beyond initial deterministic checks0.99
    22. The retroactive pass emitted ~450 findings across previously unvetted skills, turning a blind backlog into a prioritized0.99
    23. remediation queue.1.00
    24. TRACK 3·JULY 2,20260.97
    25. AlinFinance0.99
    26. World's Fair0.96

  • 10:36 #23 skipped

    shot 23·duplicate of #5

Transcript

128 cues· 2,081 words· 11,372 chars

  1. 0:12 Hello, everyone.
  2. 0:14 Good afternoon.
  3. 0:15 Today, I'm going to talk about how we vetted 2,000 AI skills before they reach the developers.
  4. 0:23 But before that, I'm Lucas Palma, but many people call me LP.
  5. 0:30 I'm the product security manager at NewBank, the product security structure that's within security, looking upon how we make code safe, and supporting engineers, product managers, and everybody to making our product safer.
  6. 0:50 over a decade of experiencing financial services engineering background also a lot of years working here at security and a close relationship with the part that i love which is innovation
  7. 1:04 So before beginning, I believe I want to bring to you why are we here.
  8. 1:11 So one thing that's important for all of you to understand is that now that we are using AI everywhere, even with coding, one thing that is important is that the AI skills are being part of the developer workflow.
  9. 1:32 And this might bring some risks because
  10. 1:37 Although they look like configuration, they behave like supply chain dependence, like, for example, libraries and others.
  11. 1:46 So what we made here was to build a security review system in order to check if these skills were safe or not to be used before deploying them.
  12. 1:58 So the lesson that I want to bring you here by the end of this presentation is that
  13. 2:03 we should be protecting the whole workflow, not only the code that's being generated.
  14. 2:11 So what I mean about the supply chain part is that traditionally, the supply chain has packets, containers, models, and so on.
  15. 2:23 But now in the AI era, it doesn't have only that.
  16. 2:26 It still have the traditional part, but it also includes skills, plugins, MCP servers, agent rules, and much more things to be acting as supply chain.
  17. 2:42 And where AI skill fits into this?
  18. 2:46 I believe that before I go into that, it's important for everybody to be on the same page on what is an AI skill.
  19. 2:53 So an AI skill has, normally the developer is using AI tools in order to generate an output, which will be code, most of the case.
  20. 3:06 And within this AI tool, there are a bunch of things that can be embedded.
  21. 3:10 One of them are the AI skill.
  22. 3:13 So with this skill, we can
  23. 3:16 have a capability to a model or to an agent, bundling some instructions, some context in order to have better guidance over what it can be done.
  24. 3:28 But there is also an impact over that because somebody can create their own skill and share with others.
  25. 3:35 So when we do that, this first person is guiding over the code that's being generated by the other person.
  26. 3:43 And then that can be dangerous.
  27. 3:48 And since we are here talking the AI in finance track, it's also important for us to understand that we are in a regulated environment.
  28. 3:58 So from one side, there are developers wanting better, faster coding, more context to have less repetitive work.
  29. 4:09 But from the other side,
  30. 4:12 Even more because of the regulate part, we need to be aware of the auditability, of looking upon credentials, safety by default, and many other security aspects.
  31. 4:26 And keeping that balance is hard, right?
  32. 4:30 So some people might say, are AI skills dangerous?
  33. 4:36 So I brought here a few examples of what do I mean by AI skills being dangerous.
  34. 4:44 So first,
  35. 4:45 uh one thing that can happen is that when people are describing what they skill can or cannot do it can it can ask for it to retrieve a token or something and it will begin using that token hard coded which will go to logs and so on and it can generate a data leak in the future
  36. 5:08 Another thing that can happen is also the person to instruct the AI to use shell commands, and then this skill will be used by another person.
  37. 5:18 And when they use on their shell, a lot of dangerous things that can happen and a lot of files being modified and so on.
  38. 5:26 And there is also permissions.
  39. 5:28 So depending on how the skill was configured, it might have excessive permissions, much more than what was needed.
  40. 5:37 And even a typo can make some dangerous stuff depending on who is using that search skill.
  41. 5:47 So first thing first, what we did initially is that, how do we share skills among ourselves?
  42. 5:56 How the engineers will be sharing those skills?
  43. 6:00 So we went through the marketplace solution.
  44. 6:04 So the skills are being canonically shared among marketplace with the plugins, including the skills among them.
  45. 6:13 So it's an internal marketplace where people can discover new skills.
  46. 6:18 And that's our boundary where we are trying to make it safer.
  47. 6:22 So what happens is that when someone creates a new skill,
  48. 6:28 will open the pull request, and normally it will go to the marketplace.
  49. 6:32 But we made a step before that, like a CI step, where we created a tool that's called Skill Vector.
  50. 6:40 And what this tool does is to check if this skill is safe or not to be used, using a lot of assessments that I will bring in here.

Chapters

  1. 0:00 Introduction: making code safe at a bank
  2. 1:32 AI skills as a supply chain risk
  3. 2:50 What counts as an AI skill
  4. 3:57 The extra weight of a regulated environment
  5. 6:07 From plugins to a vetted marketplace
  6. 6:58 What Skill Vector does
  7. 7:37 Deterministic checks, then the LLM
  8. 10:00 Scanning over two thousand skills
  9. 11:22 What worked and what needed improvement
  10. 13:30 Approval gates and human confirmation
  11. 14:23 Next steps and policies

Open at this second