Skip to Content

System QA Checks

The System QA Checks module is the cross-cutting backstop of Amal’s safety architecture. It maintains a versioned registry of 32 named checks, routes runtime failures deterministically, and exposes three read endpoints for monitoring, without duplicating a single line of enforcement logic from the engine modules that already own it.

What this module does and does not do

This module registers and routes. It does not re-implement. Every check’s enforcement lives in the module that owns the rule (the skill-status engine, the cluster module, the bundle catalog, the language-safety layer, etc.). The QA module provides a single place to discover what checks exist, what their severity is, and when they fired at runtime.

The three responsibilities:

  1. Register: a versioned, publish-then-freeze config table (qa_check_definition, 32 rows, global with no tenant scoping) describes every check: its ID (QA_001..QA_032), the CI lint command that enforces it, a human-readable failure message in Arabic, and its severity.
  2. Route: when an engine module detects a violation, it calls routeQaFailure({ qaCheckId }). The router looks up the check’s severity and applies the deterministic routing matrix.
  3. Read: three admin/eng-only HTTP endpoints expose the registry and the runtime alert log.

The registry (32 checks)

The 32 checks cover the major cross-cutting categories:

CategoryExamples
Data sufficiencyQA_001 Tie Rule / do_not_decide_yet guard
Acute regressionQA_002 ≥20% drop triggers review, never auto-fail
Bulk activation safetyQA_003 DATA_INCOMPLETE exclusion, QA_004 acute-regression exclusion
Cluster consistencyQA_005 5-key cluster identity
Scaffold-tier integrityQA_006 per-(student × skill × bundle × task) uniqueness
Teacher confirmationQA_007 explicit confirmation required for bulk-activate
Sensitive languageQA_022 language filter applied, QA_023 parent-report safety
Arabic agreementQA_008 grammar check at import
Comparable dataQA_009 only comparable probes compared
Anchor preservationQA_020 anchor weight must remain the largest share
Overload preventionQA_021 max 2 supporting threads
Content validityQA_010 inference items need an evidence_sentence
Arabic feature scopeQA_011 no Arabic-feature sub-flag alone may trigger escalation
Runtime auditQA_012 rule-version pinning on all decisions
Parent-report safetyQA_023 no internal labels in parent PDF
Over-assessment guardQA_032 alert when probes exceed threshold in a window (new)

QA_027 (RTI fidelity gate) and QA_030 (fidelity-tracker cross-check) are registered and routing-ready but are not yet wired to their consumer WPs (WP-RTI-BE / fidelity-tracker). They will fire when those WPs bind routeQaFailure.

Severity routing matrix

The routing is deterministic by the registry severity field; no call-site can override it:

SeverityWhat happens
criticalwriteQaAlert (alert logged, surfacedToTeacher=true) + notifier.notifyCritical
highwriteQaAlert + notifier.notifyEngineering (not teacher-surfaced)
mediumwriteQaAlert only
lowLog to the row’s runtimeLogTarget only; no alert written, no notifier

In Wave 1, the notifier is a log-only seam (logOnlyQaNotifier). No network call is made. The Slack #amal-alerts integration is queued for post-launch (Q-QAC-1).

Runtime alerts land in the existing platform_alert_log table (no new table). QA-routed rows set alertKind = "qa_NNN" (lowercase) and a severity column, making filtered reads a covered index scan.

The over-assessment guard (QA_032)

QA_032 is a new check added by this work package (not from the partner v5 spec). It fires when a student accumulates more than over_testing_threshold progress-monitoring probes within over_testing_window days.

Key design points:

  • Never a block. The probe is always recorded. The guard fires after writing, as a soft advisory.
  • Working defaults: threshold = 8 probes, window = 14 days (to be confirmed with partner June-25).
  • The guard is injected into monitoring/repository/evidence-writer.ts at startup. There is no import cycle between modules/qa and modules/monitoring.

Registry versioning

Admins publish a new check-set with POST /api/admin/qa/checks-set. The endpoint enforces:

  • Partner approval must be supplied (partnerApprovedBy + partnerApprovedAt).
  • The diff must be explicitly acknowledged (acknowledgeDiff: true).
  • All 32 checks must be present in the payload.
  • Every row’s ciLintCommand must be a runnable workspace command.
  • Every row’s failureMessageAr must pass the language-safety filter (no banned substring).

A 422 is returned on any gap. The seed (v1) is the path used at initial deploy; this endpoint governs subsequent publishes.

API endpoints

All three read routes and the version-publish route require elevated permissions; they are not accessible by teacher or student tokens (returns 403):

EndpointPermissionWhat it returns
GET /api/qa/checksVIEW_QA_REGISTRY or RUN_BACKOFFICE or SUPER_ADMINThe full registry (global, no tenant filter, but requires requireTenantContext to derive permissions)
GET /api/qa/runtime-alertssameOrg-scoped alerts where alertKind LIKE 'qa_%'; supports ?severity=, ?from=, ?to=, ?qaCheckId=
GET /api/admin/qa/audit-summaryRUN_BACKOFFICE or SUPER_ADMINBy-check and by-severity counts for a date range (the partner pre-launch walkthrough feed)
POST /api/admin/qa/checks-setRUN_BACKOFFICE or SUPER_ADMINPublish a new registry version

VIEW_QA_REGISTRY is the 25th permission code in the closed 25-value enum.

test:qa:NNN convention

Each registry row’s ciLintCommand is the real workspace command that enforces it. package.json adds a thin test:qa:NNN alias per row delegating to that native command (e.g. test:qa:003 → test:cluster:exclusions), plus the deduplicated aggregate pnpm test:qa. The CI qa-registry job runs pnpm test:qa and pnpm lint:qa and blocks the PR on failure.

Last updated on
Getting Started