[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-04-12

[github-actions[bot]]

Summary

• Date: 2026-04-12
• Pages visited: Home (/gh-aw/), Quick Start (/gh-aw/setup/quick-start/), CLI Commands (/gh-aw/setup/cli/)
• Overall impression: The docs have a solid structure and the Quick Start is well laid out, but there are several terminology and UX friction points that would confuse a true beginner. The security guardrails explanation on the homepage is excellent. The biggest pain point is the token/authentication story — it requires too much prior knowledge.

⚠️ Note: Playwright could not connect to the docs server due to container network isolation. Page analysis was done via curl + Python HTML parsing. Visual screenshots were unavailable.

---

🔴 Critical Issues

1. Search is broken in dev mode — on every page

Every page shows this banner in the search bar:

"Search is only available in production builds. Try building and previewing the site to test it out locally."

As a new user visiting the docs for the first time (e.g. via a deployed docs site that's accidentally running in dev mode), this looks like a site bug. It should either be hidden or replaced with a helpful message.

2. Video fallback text visible in HTML source

The Quick Start page includes <video> HTML with the fallback text "Your browser doesn't support HTML5 video. Download the video here." This is a static HTML fallback — it renders in environments that strip JavaScript or render raw HTML (e.g. curl, some screen readers, accessibility tools). The video caption/context should be available without relying on video playback.

---

🟡 Confusing Areas

View all 7 confusing areas

1. Two GitHub tokens: COPILOT_GITHUB_TOKEN vs GITHUB_TOKEN

Quick Start Step 2 mentions:

"COPILOT_GITHUB_TOKEN (a separate GitHub token with Copilot access — distinct from the default GITHUB_TOKEN)"

For a new user, the concept of two different GitHub tokens is baffling. Why isn't the existing GITHUB_TOKEN enough? This needs a one-sentence explanation inline (e.g., "The default GITHUB_TOKEN doesn't have Copilot permissions, so a separate PAT is needed.")

2. "lock file" term used without explanation

Step 2 says: "Adds the workflow and lock file to .github/workflows/". A new user won't know what a "lock file" means in this context. It should link to the Compilation Process reference or add a brief parenthetical.

3. "frontmatter" and "YAML" jargon in Step 4

Step 4 says "If you have changed the frontmatter (the YAML configuration block between --- markers)". While it explains frontmatter inline, it immediately follows with YAML jargon. Someone who doesn't know YAML won't understand what "YAML configuration block" means. A link to the Frontmatter reference page would help.

4. Prerequisite "AI Account" is vague

The prerequisites list: "AI Account - GitHub Copilot, Anthropic Claude or OpenAI Codex" but don't specify:

• Which tier of Copilot works? (free vs Individual vs Business vs Enterprise)
• Is a paid account required?

A new user would likely pause here not knowing if they qualify.

5. CLI Commands page: installation section comes after the command table

The page structure is:

1. Most Common Commands ← shown first
2. Installation ← shown second

A beginner would naturally want to install first, then learn the commands. The page ordering is confusing.

6. "Peli's Agent Factory" unexplained

Multiple pages reference "Peli's Agent Factory" without explaining who "Peli" is or what the Agent Factory is. This reads like an internal nickname that leaked into the public docs. A one-line description would help (e.g., "a curated collection of example workflows by the project author").

7. Sidebar navigation overwhelm

The sidebar contains 80+ links organized into 6 categories: Introduction, Setup, Guides, Design Patterns, Reference, and Troubleshooting. For a first-time visitor there's no visual hierarchy pointing them to "start here." The current Quick Start, Creating Workflows, and CLI Commands items under "Setup" don't stand out from the rest.

---

✅ What Worked Well

• Estimated time on Quick Start — "Estimated time: 10 minutes" is a great UX touch
• Numbered step structure — Steps 1–4 in Quick Start are easy to follow
• Security architecture on homepage — The 5-layer security explanation in plain English is excellent; builds trust immediately
• Authentication page "Which secret do I need?" table — The table at the top of /reference/auth/ is perfectly scannable
• CLI "Most Common Commands" table — 7-command quick reference is very effective
• Embedded video tutorial — The Quick Start has a video walkthrough which is great for visual learners
• In-context tip boxes — The "Tip" callouts for auth troubleshooting and add-wizard alternatives are helpful

---

📋 Recommendations

Quick wins (high impact, low effort):

1. Add a one-sentence explanation of why COPILOT_GITHUB_TOKEN is separate from GITHUB_TOKEN in Quick Start Step 2
2. Add parenthetical/link explanation for "lock file" in Quick Start Step 2
3. Add a brief description of "Peli's Agent Factory" wherever it's first mentioned
4. Clarify which Copilot tier is required in the Prerequisites section
5. Move the "Installation" section above "Most Common Commands" on the CLI page

Medium effort:
6. Add links from Step 4 "frontmatter" mention to the Glossary and Frontmatter reference
7. Consider adding a "New? Start here →" visual callout in the sidebar or homepage hero
8. Add a text transcript or description alongside the embedded video for accessibility

Longer term:
9. Evaluate sidebar information architecture — consider collapsing Reference and Patterns by default for new users
10. Add a "Prerequisites check" script or tool to help new users verify they have everything ready before Step 1

---

References:

• Workflow run §24307551557

Generated by Documentation Noob Tester · ● 3.2M · ◷

• expires on Apr 13, 2026, 1:27 PM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #26040.