98.2% false-positive reduction on Mastodon

The problem

When we first scanned Mastodon with shadowaudit v0.8.0, we got 505 CRITICAL findings — routes flagged as "undocumented + no auth." After manually tracing every single one, we found they were ALL false positives. Mastodon's auth is solid; our scanner just couldn't detect it.

This is the #1 problem with static analysis tools: false positives destroy trust. A tool that cries wolf 505 times is worse than no tool at all.

The journey: 505 → 9

Round 1: Controller inheritance (v0.8.1)

Fix: Walk the controller inheritance chain — Api::V1::UsersController → Api::BaseController → ApplicationController. Auth declared in ANY ancestor applies to the child.

Result: 505 → 264 CRITICAL (48% reduction)

Round 2: Routing precision (v0.8.2)

Fix: with_options to: mapping, controller: modifier (symbol + string), improved pluralization (history → histories).

Result: 264 → 216 CRITICAL

Round 3: Module-nested parent class (v1.0.4)

Fix: module Admin; class DashboardController < BaseController — the scanner was looking for base_controller.rb at the root instead of admin/base_controller.rb. Added extractEnclosingModule() to resolve module-nested parents.

Result: 223 → 36 CRITICAL (84% reduction from v0.8.0)

Round 4: resources :name do regex fix (v1.0.4)

Fix: resources :webhooks do (no comma before do) was completely skipped — no REST routes, no context push, and end was popping the wrong namespace context. Fixed the regex to accept both forms.

Result: ~30+ false positives eliminated

Round 5: New auth patterns (v1.0.5)

Fix: Added 9 new patterns:

• require_moderator_or_admin_permissions (Mastodon's Admin::BaseController)
• requires_login (Discourse)
• requires_staff (Discourse)
• after_action :verify_authorized (Pundit safety net)
• AdminConstraint (Rails routing constraints)
• authenticate (Devise constraint)
• require_staff!, require_manager!

Result: 36 → 34 CRITICAL (without allowlist)

Round 6: Scope bare path + redirect detection (v1.0.5)

Fix: scope '/tag' do was not prepending /tag to inner routes. with_options to: redirect('/path') do was not marking inner routes as redirects.

Round 7: --allowlist flag (v1.0.4)

Fix: Configurable JSON file with public route patterns. Routes matching allowlist are downgraded from CRITICAL → INFO.

Result: 34 → 9 CRITICAL (with allowlist)

The numbers

Auth detection improvement: 96 → 736 routes (7.7× more routes correctly identified as auth-protected)

The remaining 9

The 9 remaining CRITICAL findings (with --allowlist) fall into 3 categories:

1. Devise auth routes (3) — DELETE/PATCH/PUT /auth are Devise session management routes. They're public by design (you need to be able to log in without being authenticated). Can be added to the allowlist.
2. Deep module-nested controllers (3) — Controllers like Admin::Reports::ActionsController use a 2-level namespace + module: modifier. The controller path resolution doesn't handle this depth yet.
3. Concern-based routes (3) — Routes from concern :approvable inside nested with_options + namespace blocks. Complex concern expansion with multiple nesting levels.

All 9 are false positives — but they require deeper controller path resolution or more complex concern expansion, which are v1.1.0 goals.

What I learned

1. Module nesting is the #1 source of false positives in Rails auth detection. Ruby's module Admin; class X < BaseController form means BaseController resolves to Admin::BaseController, not the root BaseController. Every static analysis tool that doesn't handle this will produce false positives on any Rails app with namespaced controllers.
2. Regex-based parsers are fragile. resources :name do (no comma) vs resources :name, only: [] do (with comma) required two different regex patterns. The fix was to make the regex accept both forms — but this kind of fragility is inherent in regex-based parsing.
3. Allowlists are essential. No static analysis tool can know that /health is intentionally public. Giving users a configurable allowlist (rather than hardcoding patterns) is the right approach.
4. Adversarial testing on real codebases is the only way to find these bugs. Toy fixtures never have module Admin; class X < BaseController or with_options to: redirect('/path') do. Only real-world code exposes these patterns.

Try it

npm install -g shadowaudit@1.0.5

# Scan with allowlist
shadowaudit --dir ./myapp --spec openapi.json --allowlist allowlist.json

# Scan-only mode (no spec needed)
shadowaudit --dir ./myapp --format json

# Find dead spec entries
shadowaudit --dir ./myapp --spec openapi.json --reverse