Performance Test Cases

This page is the source of truth for benchmark case definitions. The performance summary should link here instead of embedding rulesets in the results table.

Assura reports four kinds of evidence:

• current checked cases: generated LS-Lint-equivalent repositories in the realistic-equivalent cohort
• native checked cases: generated Assura-only content and agent workflow repositories in the assura-native cohort
• optional external cases: pinned real repositories in the real-repo-headline cohort
• support cases: generated fixtures in generated-fixtures-v1 and generated-fixtures-v2
• diagnostic stress cases: synthetic cases used for regression detection

Every report row includes fixture_acceptance. Rows marked accepted-ls-lint-equivalent are part of the fixture-floor gate: the matching assura-cli row must be no slower than the native ls-lint-cli row for that fixture. Rows marked diagnostic, experimental, retired, or assura-native-diagnostic can explain behavior, but they do not prove the public no-slower claim.

Current Checked Cohort

The checked-in report rendered on the public performance page uses the realistic-equivalent cohort. These fixtures are generated in-repo so the baseline can be refreshed without downloading third-party source trees:

• simple_library
• web_app
• monorepo_packages
• monorepo_policy
• rule_heavy_repo
• ignored_generated_heavy_repo
• multipart_extension_regression
• many_configured_scopes_regression

Refresh the current checked artifacts with:

cargo build --release --bin assura --no-default-features --features json-output,yaml-config
cargo build --release --bin assura-full
cargo build --release -p assura-check-cli
target/release/assura performance-report \
  --output benches/history/current.json \
  --history benches/history/ls-lint-comparison-history.jsonl \
  --website-dir website/public/data/performance \
  --iterations 5

Assura-Native Checked Cohort

The native report is generated with assura performance-report --suite native. It writes benches/history/native-current.json, benches/history/native-history.jsonl, website/public/data/performance/native-current.json, and website/public/data/performance/native-history.jsonl.

Native rows use fixture_acceptance=assura-native-diagnostic because they do not have LS-Lint equivalents. They are still merge-gated by cargo xtask native-performance-no-regression, which checks the full fixture matrix, pass status, sample presence, expected Assura exit status, and native report metadata.

Current native fixtures:

• native_small
• native_medium
• native_large
• native_reference_heavy
• native_adapter_mix
• native_real_project

Current native row families:

• native:content-check-cli
• native:content-collections-cli
• native:content-instances-cli
• native:content-show-cli
• native:content-expand-cli
• native:content-search-cli
• native:content-missing-relations-cli
• native:content-references-cli
• native:context-pack-cli
• native:agent-query-keyword-search-cli
• native:agent-query-missing-relations-cli
• native:markdown-safe-fix-dry-run-cli
• native:session-agent-context-cli
• native:daemon-status-cli
• native:phase:config-model-load
• native:phase:schema-compile
• native:phase:file-index
• native:phase:object-load-validate
• native:phase:edge-collect
• native:phase:reference-validate
• native:phase:repository-validate-total
• native:phase:fact-ingest-load
• native:phase:incremental-replace-generation
• native:phase:warm-keyword-query
• native:phase:factset-serialize-json

The native:phase:* rows are attribution rows. They separate configuration and model loading, schema compilation, file indexing, object validation, relation edge collection, reference validation, project-intelligence fact ingestion, incremental generation replacement, warm query latency, and JSON serialization.

Refresh the native artifacts with:

cargo build --release --bin assura --no-default-features --features json-output,yaml-config
cargo build --release --bin assura-full
cargo build --release -p assura-check-cli
target/release/assura performance-report \
  --suite native \
  --output benches/history/native-current.json \
  --history benches/history/native-history.jsonl \
  --website-dir website/public/data/performance \
  --iterations 5
cargo xtask native-performance-no-regression benches/history/native-current.json

Optional Pinned Repository Cases

Pinned external repositories are optional extended evidence. They are cloned on demand so Assura can measure real project trees without vendoring third-party source, but they are only part of a checked report when the command includes --include-external-fixtures.

Run them with:

cargo build --release --bin assura --no-default-features --features json-output,yaml-config
cargo build --release --bin assura-full
cargo build --release -p assura-check-cli
target/release/assura performance-report \
  --include-external-fixtures \
  --output benches/history/current.json \
  --history benches/history/ls-lint-comparison-history.jsonl \
  --website-dir website/public/data/performance \
  --iterations 5

Set ASSURA_PERF_EXTERNAL_FIXTURE_CACHE=/path/to/cache to share the Git cache across runs. Without it, Assura uses target/performance/external-fixture-cache.

The exact LS-Lint configs live in src/cli/performance_report/external_fixture_catalog.rs. The report writes each LS-Lint config into the pinned checkout, then generates the matching Assura config through the LS-Lint migration layer so the comparison exercises the same policy surface.

Next.js Monorepo

Fixture: pinned_nextjs.

Field	Value
Repository	vercel/next.js
Resolved commit	51bfe3c1863b191f4b039bc230e8ed5c57b0baf3
Comparable project	large frontend framework monorepo
Metrics	9,216 checked files; 11,877 ignored paths; 9,812 directories; 22 LS-Lint rules
What is unique	very large framework tree with packages, examples, route-like filenames, intentionally arbitrary test fixtures, Turbopack fixtures, and nested package caches
Rule surface	package/example scopes, route-friendly regex naming, source-family rules, foreign-language bans, generated-output ignores
Policy protection	keeps package and example names inside framework-safe patterns, applies JS/JSX/TS/TSX naming inside scoped workspaces, bans accidental Rust/Go/Python/Ruby files in those scopes, and prunes known generated or intentionally arbitrary fixture output such as node_modules, .next, dist, coverage, .turbo, .vercel, test, and turbopack

mdBook Docs Project

Fixture: pinned_mdbook.

Field	Value
Repository	rust-lang/mdBook
Resolved commit	b7a27d2759e80d804a33a4bc9c31b2b6863a5cb2
Comparable project	Rust documentation tool
Metrics	229 checked files; 5 ignored paths; 57 directories; 14 LS-Lint rules
What is unique	compact Rust project where the product shape is source plus book/docs content instead of a large package workspace
Rule surface	required Rust source, required book Markdown, TOML/Markdown naming, JS/TS bans, target pruning
Policy protection	requires direct Rust source in src and Markdown content in book, keeps Rust/Markdown/TOML names inside the accepted policy, bans accidental JS/TS files in source/docs scopes, and prunes target output so generated build files do not become part of the source contract

Vite Tooling Monorepo

Fixture: pinned_vite.

Field	Value
Repository	vitejs/vite
Resolved commit	cf50e548d4ae7f6ed9ade507d70acbd39b4e0b93
Comparable project	frontend tooling monorepo
Metrics	2,040 checked files; 6 ignored paths; 642 directories; 22 LS-Lint rules
What is unique	frontend tooling repo with package workspaces, playground apps, Vue examples, docs, and VitePress build/cache output
Rule surface	package and playground scopes, JS/JSX/TS/TSX/Vue/Svelte naming, foreign-language bans, docs/build-output pruning
Policy protection	validates package and playground directory scopes, applies the file families Vite templates actually use, bans accidental Rust/Go/Python files in package/playground scopes, and ignores node_modules, package dist, and VitePress cache/dist output

Tailwind CSS Package

Fixture: pinned_tailwindcss.

Field	Value
Repository	tailwindlabs/tailwindcss
Resolved commit	7361468f77500105b0559e879e121f34306e8da2
Comparable project	CSS tooling package
Metrics	961 checked files; 2 ignored paths; 99 directories; 12 LS-Lint rules
What is unique	CSS tooling package with source, tests, HTML fixtures, and generated distribution/coverage paths
Rule surface	source/test naming, HTML fixtures, foreign-language bans, generated-output pruning
Policy protection	constrains top-level directories and dotfiles, applies JS/TS naming in src, applies JS/HTML naming in tests, bans accidental Rust/Go/Python files in those scopes, and keeps node_modules, dist, and coverage out of the checked tree

Prettier Formatter Repo

Fixture: pinned_prettier.

Field	Value
Repository	prettier/prettier
Resolved commit	c4ab460357478d2b847c60a1efb40098b1181931
Comparable project	JavaScript formatting tool
Metrics	8,638 checked files; 23 ignored paths; 2,879 directories; 19 LS-Lint rules
What is unique	large formatter repo with product source, thousands of tests and fixtures, website content, and intentionally unusual fixture names
Rule surface	source, tests, website, TypeScript/JSX fixtures, fixture ignores, generated-output pruning
Policy protection	validates source, tests, JSON, TypeScript/JSX fixtures, website JavaScript, and website Markdown, bans accidental Rust/Go/Python files in product scopes, and ignores known arbitrary-name fixture directories plus generated caches so the policy protects product structure rather than test corpus quirks

pnpm Monorepo

Fixture: pinned_pnpm.

Field	Value
Repository	pnpm/pnpm
Resolved commit	dd6b0b62d80340655ac1cf4c4365fcfffe1e3f2c
Comparable project	package-manager monorepo
Metrics	2,314 checked files; 2 ignored paths; 827 directories; 14 LS-Lint rules
What is unique	package-manager monorepo with many package scopes, generated outputs, and dedicated TypeScript declaration areas
Rule surface	package scopes, TypeScript declarations, frontend-source bans, dotfile allowances, generated-output pruning
Policy protection	constrains package workspace directory names, applies JS/TS and .d.ts naming inside packages, handles declarations in __typings__, bans accidental JSX/TSX/Rust/Go/Python files in package scopes, and ignores package-local node_modules, dist, and coverage output

Rustlings Exercises

Fixture: pinned_rustlings.

Field	Value
Repository	rust-lang/rustlings
Resolved commit	28d2bb04326d7036514245d73f10fb72b9ed108c
Comparable project	Rust education repository
Metrics	272 checked files; 2 ignored paths; 70 directories; 11 LS-Lint rules
What is unique	education repo where the important structure is exercise groups and solution files rather than normal library crates
Rule surface	exercise/solution group naming, required Rust files, JS/TS bans, target pruning
Policy protection	keeps exercise and solution groups in the numbered topic pattern, requires Rust files inside each group, bans accidental JS/TS files there, and ignores target output

clap Rust Workspace

Fixture: pinned_clap.

Field	Value
Repository	clap-rs/clap
Resolved commit	a751c5fe65cd33cb09e85ff3039b4fd0182cdb6e
Comparable project	Rust CLI workspace
Metrics	582 checked files; 2 ignored paths; 89 directories; 13 LS-Lint rules
What is unique	Rust CLI workspace organized around crate-specific directories plus examples and tests
Rule surface	explicit crate source scopes, examples/tests Rust policy, JS/TS bans, target pruning
Policy protection	requires Rust files under the major crate src directories, applies Rust naming to examples and tests, bans accidental JS/TS/TSX files at the workspace root, and excludes target output from source-policy checks

ripgrep CLI Repo

Fixture: pinned_ripgrep.

Field	Value
Repository	BurntSushi/ripgrep
Resolved commit	4649aa9700619f94cf9c66876e9549d83420e16c
Comparable project	Rust CLI repository
Metrics	213 checked files; 2 ignored paths; 56 directories; 13 LS-Lint rules
What is unique	Rust CLI repo with a crates workspace layout, tests, docs, scripts, and nested build output possibilities
Rule surface	crate source scopes, examples/tests Rust policy, JS/TS bans, target pruning
Policy protection	constrains crate directory names, requires Rust files under crate src directories, applies Rust naming to examples/tests, bans accidental JS/TS files in crate scopes and JS/TS/TSX at the root, and ignores both root target and crate-local target output

Tokio Async Workspace

Fixture: pinned_tokio.

Field	Value
Repository	tokio-rs/tokio
Resolved commit	14c17fc09656a30230177b600bacceb9db33e942
Comparable project	Rust async workspace
Metrics	775 checked files; 2 ignored paths; 110 directories; 12 LS-Lint rules
What is unique	multi-crate async Rust workspace with core crates, utility crates, examples, benches, and crate-local build outputs
Rule surface	explicit crate source scopes, examples/tests Rust policy, JS/TS bans, target pruning
Policy protection	requires Rust files under core crate src directories, applies Rust naming to examples/tests, bans accidental JS/TS/TSX files at the workspace root, and ignores root and crate-local target output

Generated Support Cases

Generated fixtures are still useful for deterministic local regression checks, but they are not the public aggregate when real-repo rows are present.

Simple library

Fixture: simple_library. Small library with source files, docs, tests, and ignored build output.

Field	Value
Comparable project	Rust or package library
Scale focus	Startup and config loading overhead
Current rule surface	directory naming, source naming, markdown count checks, test naming, ignored target output

Web app

Fixture: web_app. Frontend app with components, tests, CSS modules, assets, and ignored dist output.

Field	Value
Comparable project	small web app
Scale focus	component/test/asset naming plus generated-output pruning
Current rule surface	TSX/component naming, test naming, CSS module naming, asset naming, dist pruning

Package monorepo

Fixture: monorepo_packages. Two-package workspace with package-local source, tests, docs, and ignored dist output.

Field	Value
Comparable project	small package monorepo
Scale focus	nested package scopes
Current rule surface	package-scoped TypeScript/TSX rules, test naming, markdown count checks, dist pruning

Rule-heavy repo

Fixture: rule_heavy_repo. Repo-shaped generated tree with many extension-specific rules.

Field	Value
Comparable project	broad multi-extension repository
Scale focus	rule matching cost
Current rule surface	many extension and subextension rules over a modest tree

This case should be replaced or supplemented by a more meaningful policy-style case. The current extension list is mechanically useful, but it does not explain real project structure well.

Monorepo policy

Fixture: monorepo_policy. Generated TypeScript/Next-style workspace with app, package, documentation, scripts, infrastructure, config, tests, and CI scopes.

Field	Value
Comparable project	strict frontend/package monorepo
Scale focus	policy-shaped rule matching and ignored generated output
Current rule surface	root directory/file whitelisting, app/package overrides, config-file exceptions, JavaScript/JSX source bans, docs/scripts/infra policies, generated-output pruning

This is the generated fixture intended to replace the old extension-heavy rule_heavy_repo as the more meaningful policy case.

Generated-heavy repo

Fixture: ignored_generated_heavy_repo. Project with large generated and coverage folders inside the tree.

Field	Value
Comparable project	project with generated artifacts inside the repo
Scale focus	ignored-path pruning
Current rule surface	TypeScript naming and generated/coverage pruning

Generated Monorepo Policy Excerpt

The generated support suite includes a detailed LS-Lint policy that constrains the shape of a TypeScript/Next-style monorepo:

• root directory whitelist
• special root docs such as README, AGENTS, CONTRIBUTING, and LICENSE
• app and package directory overrides
• JavaScript disallowed in source subtrees except explicit config files
• docs directory restricted to docs-like file types
• infrastructure, specs, tests, and workflow-specific subtrees
• generated output, package-manager state, caches, and node_modules ignored

Representative excerpt:

ls:
  .dir: regex:(apps|packages|config|scripts|docs|specs|tests|infra|src|\.github)
  .md: regex:(README|AGENTS|CONTRIBUTING|CHANGELOG|LICENSE) | kebab-case

  apps:
    .dir: kebab-case
    "*":
      .dir: kebab-case
      .tsx: kebab-case
      .mjs: regex:(eslint\.config|postcss\.config|tailwind\.config)
      .js: regex:(next\.config|postcss\.config|tailwind\.config)
      "*":
        .tsx: kebab-case
        .js: regex:^$
        .jsx: regex:^$

  packages:
    .dir: kebab-case
    "*":
      .ts: kebab-case
      .tsx: kebab-case
      .js: regex:^$
      "*":
        .ts: kebab-case
        .tsx: kebab-case
        .js: regex:^$

  docs:
    .md: regex:(README|AGENTS) | kebab-case
    .js: regex:^$
    .ts: regex:^$

ignore:
  - node_modules
  - apps/*/node_modules
  - packages/*/node_modules
  - dist
  - apps/*/dist
  - packages/*/dist
  - coverage
  - .next
  - apps/*/.next
  - .turbo

This case is tracked as monorepo_policy.

LS-Lint Features Covered By Parity Fixtures

The upstream LS-Lint docs include several features that are represented by parity tests and should remain part of benchmark fixture coverage:

• wildcard extensions and subextensions
• multiple rules with the | operator
• directory-specific overrides
• glob and alternative directory patterns
• regex negation and alternation
• directory substitutions such as ${0} and ${1}
• exists counts and ranges for files or directories

Useful references:

• LS-Lint basics: directory patterns, wildcard extensions, and overrides
• LS-Lint rules: regex, directory substitutions, and exists