Add /selftest.extension core extension to test other extensions by dhilipkumars · Pull Request #1758 · github/spec-kit

dhilipkumars · 2026-03-04T21:17:01Z

This PR adds an experimental testing sandbox under tests/extension-commands to evaluate how LLMs/Agents parse extension specifications and execute their mapped commands.

Sample Output from Gemini LLM

In my Gemini CLI if i entered below prompt

/speckit.selftest.run retrospective

It outputs the following.

 
    1 ============================= test session starts ==============================
    2 collected 3 items
    3
    4 test_selftest_discovery.py::test_catalog_search [PASS]
    5   Details: Found 1 extension(s): Retrospective Extension (v1.0.0).
    6
    7 test_selftest_installation.py::test_extension_add [PASS]
    8   Details: Extension installed successfully! Provided commands: speckit.retrospective.analyze.
    9
   10 test_selftest_registration.py::test_config_verification [PASS]
   11   Details: .specify/extensions.yml contains an enabled hook for the 'retrospective' extension.
   12
   13 ============================== [3] passed in 2.3s ==============================

AI Usage disclosure: Used Aniti-gravity to build this out.

Future Goal: Currently its manual to trigger these tests but at some point we can build a workflow for co-pilot to run these /selftests

Copilot

Pull request overview

Adds an experimental sandbox under tests/extension-commands/ intended to evaluate whether LLM/agent workflows can discover extension command definitions and “execute” their mapped markdown instructions.

Changes:

Introduces a mock Python entrypoint (main.py) that prints timestamped outputs for --lint and --deploy.
Adds a .specify/ sandbox containing command markdown files (lint.md, deploy.md) and an extensions.yml mapping.
Adds TESTING.md with a copy/paste prompt for running the exercise in an LLM chat.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/extension-commands/main.py	Mock CLI script for lint/deploy command execution output
tests/extension-commands/TESTING.md	Manual LLM evaluation instructions and expected terminal-style output
tests/extension-commands/.specify/extensions.yml	Declares command mappings for the sandbox
tests/extension-commands/.specify/lint.md	Markdown “command file” instructing how to run the mock linter
tests/extension-commands/.specify/deploy.md	Markdown “command file” instructing how to run the mock deploy

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tests/extension-commands/.specify/extensions.yml

tests/extension-commands/main.py

tests/extension-commands/.specify/lint.md

tests/extension-commands/.specify/deploy.md

tests/extension-commands/TESTING.md

dhilipkumars · 2026-03-04T21:40:19Z

@mnriem toying with some ideas for adding tests for extension. I'm starting with 'extension' commands. If you like it this pattern i can add more tests. this is manual for now but we can think of wiriting it up with the @copilot code review[agent] workflow at somepoint.

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tests/extension-commands/extension.yml

tests/extension-commands/.specify/deploy.md

dhilipkumars · 2026-03-04T21:42:57Z

@copilot code review[agent]
follow the instructions here tests/extension-commands/TESTING.md

mnriem · 2026-03-04T21:51:25Z

Wild idea, maybe as an extension with its own command /selftest "extension" ?

dhilipkumars · 2026-03-04T21:56:53Z

Interesting. Where do you think such a test extension should live? in this repo or outside as a community?

mnriem · 2026-03-05T15:28:21Z

Probably in the core repo under extensions/selftest ?

dhilipkumars · 2026-03-05T19:39:31Z

perfect let me re-work my PR

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

extensions/catalog.json

extensions/selftest/commands/selftest.md

extensions/selftest/extension.yml

extensions/catalog.json

…ploy explicitly

…culation

…relative paths

…ructure and argparse mutually exclusive groups

…ual tests sandbox

… schema

…on names

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

extensions/catalog.json

extensions/selftest/commands/selftest.md

extensions/catalog.json

mnriem

Can you address the Copilot feedback? Note this will be an official extension so it should be in the regular catalog.json and then refer to the full URL it will land on once we merge it in (extensions/selftest directory) as the download URL? Do we support referring to a directory instead of a tgz file? If we do NOT we should ;)

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

extensions/catalog.json

extensions/selftest/commands/selftest.md

extensions/selftest/extension.yml

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

extensions/selftest/commands/selftest.md

mnriem

Can you address the Copilot feedback where applicable. If not applicable, please explain why.

dhilipkumars · 2026-03-10T14:41:45Z

Can you address the Copilot feedback where applicable. If not applicable, please explain why.

@mnriem sure, on it. Have made some changes to address your previous comment was testing them locally last night. i should be get them changes in tonight or tomorrow.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-10T21:27:13Z

extensions/selftest/extension.yml

+  commands:
+    - name: speckit.selftest.extension
+      file: commands/selftest.md
+      description: Validate the lifecycle of an extension from the catalog.


The PR description/sample output uses /speckit.selftest.run ..., but the extension manifest registers the command as speckit.selftest.extension. Either update the docs/sample invocation to match the registered command name, or rename the command here so users can run the command shown in the PR description.

extensions/catalog.json

dhilipkumars · 2026-03-10T21:27:23Z

Can you address the Copilot feedback? Note this will be an official extension so it should be in the regular catalog.json and then refer to the full URL it will land on once we merge it in (extensions/selftest directory) as the download URL? Do we support referring to a directory instead of a tgz file? If we do NOT we should ;)

yes we do have that already as dev

specify extension add --help
                                                                                    
 Usage: specify extension add [OPTIONS] EXTENSION                                   
                                                                                    
 Install an extension.                                                              
                                                                                    
╭─ Arguments ──────────────────────────────────────────────────────────────────────╮
│ *    extension      TEXT  Extension name or path [required]                      │
╰──────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────────╮
│ --dev               Install from local directory                                 │
│ --from        TEXT  Install from custom URL                                      │
│ --help              Show this message and exit.                                  │
╰──────────────────────────────────────────────────────────────────────────────────╯

apart from that i have addressed all copilots review comments and updated catalog.json (not the community one). PTAL @mnriem

dhilipkumars requested a review from mnriem as a code owner March 4, 2026 21:17

Copilot AI review requested due to automatic review settings March 4, 2026 21:17

Copilot started reviewing on behalf of dhilipkumars March 4, 2026 21:17 View session

Copilot AI reviewed Mar 4, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings March 4, 2026 21:35

Copilot started reviewing on behalf of dhilipkumars March 4, 2026 21:35 View session

Copilot AI reviewed Mar 4, 2026

View reviewed changes

tests/extension-commands/extension.yml Outdated Show resolved Hide resolved

tests/extension-commands/.specify/deploy.md Outdated Show resolved Hide resolved

Copilot AI review requested due to automatic review settings March 5, 2026 21:15

Copilot started reviewing on behalf of dhilipkumars March 5, 2026 21:15 View session

Copilot AI reviewed Mar 5, 2026

View reviewed changes

extensions/catalog.json Outdated Show resolved Hide resolved

extensions/selftest/commands/selftest.md Show resolved Hide resolved

extensions/selftest/extension.yml Show resolved Hide resolved

extensions/catalog.json Show resolved Hide resolved

dhilipkumars marked this pull request as draft March 5, 2026 21:39

dhilipkumars mentioned this pull request Mar 6, 2026

fix: core and community extension catalogs should be searchable without override envvar #1769

Closed

5 tasks

dhilipkumars added 12 commits March 9, 2026 17:52

test(commands): create extension-commands LLM playground sandbox

f42af45

update(tests): format LLM evaluation as an automated test runner

1dc000b

test(commands): map extension-commands python script with timestamps

c1f47b3

test(commands): map extension-commands python script with timestamps

4d09e9b

test(commands): update TESTING.md to evaluate discovery, lint, and de…

e1e3011

…ploy explicitly

test(commands): simplify execution expectations and add timestamp cal…

e6316fa

…culation

fix(tests): address copilot review comments on prompt formatting and …

93d9af0

…relative paths

fix(tests): resolve copilot PR feedback regarding extension schema st…

9fe81a9

…ructure and argparse mutually exclusive groups

feat(extensions): add core selftest utility and migrate away from man…

1777448

…ual tests sandbox

fix(selftest): update command name array to match spec-kit validation…

48a0ea0

… schema

fix(selftest): wrap arguments in quotes to support multi-word extensi…

7dc74c7

…on names

update the command to be more meaningful

5048406

Copilot AI review requested due to automatic review settings March 9, 2026 22:47

dhilipkumars force-pushed the add-extensions-tests branch from 9d5649a to 5048406 Compare March 9, 2026 22:47

Copilot started reviewing on behalf of dhilipkumars March 9, 2026 22:48 View session

dhilipkumars marked this pull request as ready for review March 9, 2026 22:48

dhilipkumars changed the title ~~test: experimental tests for extension commands~~ Add /selftest.extension core extension to test other extensions Mar 9, 2026

Copilot AI reviewed Mar 9, 2026

View reviewed changes

extensions/catalog.json Show resolved Hide resolved

extensions/selftest/commands/selftest.md Outdated Show resolved Hide resolved

extensions/selftest/commands/selftest.md Outdated Show resolved Hide resolved

extensions/catalog.json Show resolved Hide resolved

mnriem requested a review from Copilot March 9, 2026 23:10

Copilot started reviewing on behalf of mnriem March 9, 2026 23:10 View session

mnriem requested changes Mar 9, 2026

View reviewed changes

Copilot AI reviewed Mar 9, 2026

View reviewed changes

extensions/catalog.json Show resolved Hide resolved

extensions/catalog.json Outdated Show resolved Hide resolved

extensions/selftest/commands/selftest.md Show resolved Hide resolved

extensions/selftest/extension.yml Show resolved Hide resolved

fix: if the extension is discovery only, it should not be installable

c421117

mnriem requested a review from Copilot March 10, 2026 13:05

Copilot started reviewing on behalf of mnriem March 10, 2026 13:05 View session

Copilot AI reviewed Mar 10, 2026

View reviewed changes

extensions/selftest/commands/selftest.md Show resolved Hide resolved

extensions/selftest/commands/selftest.md Show resolved Hide resolved

mnriem self-requested a review March 10, 2026 13:18

mnriem requested changes Mar 10, 2026

View reviewed changes

dhilipkumars added 2 commits March 10, 2026 17:09

Address review comments for selftest documentation

11b07a5

address review comments

32b95e5

Copilot AI review requested due to automatic review settings March 10, 2026 21:24

Copilot started reviewing on behalf of dhilipkumars March 10, 2026 21:24 View session

Copilot AI reviewed Mar 10, 2026

View reviewed changes

address review comments

9e718ea

Conversation

dhilipkumars commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Sample Output from Gemini LLM

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dhilipkumars commented Mar 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

dhilipkumars commented Mar 4, 2026

Uh oh!

mnriem commented Mar 4, 2026

Uh oh!

dhilipkumars commented Mar 4, 2026

Uh oh!

mnriem commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dhilipkumars commented Mar 5, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mnriem left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

mnriem left a comment

Choose a reason for hiding this comment

Uh oh!

dhilipkumars commented Mar 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

dhilipkumars commented Mar 4, 2026 •

edited

Loading

mnriem commented Mar 5, 2026 •

edited

Loading