znxzsy/AlignLedger

English · 中文

AlignLedger is the human-feedback and data-governance layer for RSI research and iterative model development. It turns model outputs, human corrections and second-pass decisions into traceable records and frozen exports, supporting downstream SFT, RLHF, DPO/KTO preparation and multimodal evaluation.

The project uses the Python standard library and SQLite. There is no frontend build step. A local demo starts with one command and creates synthetic records that cover completed, partial, corrected, and unresolved cases.

Here, RSI means recursive self-improvement: evidence from one iteration informs the next. AlignLedger prepares reviewable feedback and versioned data snapshots. VeriInk supplies complementary synthetic practice; trainers and independent evaluators remain external.

Frozen exports are dataset candidates, not automatic training approval. Human labels require an appropriate review protocol; model-generated labels are not human ground truth. DPO/KTO candidate files still require downstream validation and format adaptation.

RSI data workflow · 中文流程 · Illustrative round record

Multi-person annotation fails in predictable ways: a stale browser overwrites a newer decision, a network retry creates duplicate writes, a batch looks complete while individual records remain empty, or the exported label can no longer be traced to the person who changed it. AlignLedger handles those cases directly.

Every write carries a version and an idempotency key. Completed records can be edited without losing history. Second-pass review does not overwrite the original annotation. An unresolved correction stays out of the frozen export.

The server separates initial annotation, review, and final adjudication. Reviewers can sample records at random or select a batch. Quality tables report both accuracy and work volume, with every count derived from stored decisions rather than browser state.

• Per-record annotation with correction fields and explicit unknown states
• Automatic draft saving, optimistic version checks, and idempotent writes
• Invite-based identity binding and reviewer attribution
• Separate pools for accepted, corrected, and unresolved records
• Random review, batch review, and final-label adjudication
• Workload and accuracy tables for annotators and reviewers
• Frozen exports with a manifest, SHA256, and FROZEN_OK

git clone https://github.com/znxzsy/AlignLedger.git
cd AlignLedger
./scripts/run_demo.sh

Open http://127.0.0.1:18068. The first run creates 12 synthetic groups with a mix of complete, partial, corrected, and unknown records.

Docker is also supported:

docker compose up --build

The importer accepts details_*.json shards. Each request supplies stable item indices, image references, and the raw model output.

python -m annotation_platform.importer \
  --details-dir ./my-details \
  --output-dir ./runtime/imported

python -m annotation_platform.server \
  --registry ./runtime/imported/source_groups.jsonl \
  --db ./runtime/review.sqlite3 \
  --audit ./runtime/audit.jsonl

The importer rejects unsafe local image paths, validates record structure, hashes source files, and writes a reviewable manifest.

The local demo disables authentication. A shared deployment should use a random session secret and one-time invite codes:

mkdir -p secrets runtime
python -c 'import secrets; print(secrets.token_hex(32))' > secrets/session-secret
python scripts/generate_invites.py --count 30

export ANNOTATION_AUTH_REQUIRED=1
export ANNOTATION_SESSION_SECRET_FILE=secrets/session-secret
export ANNOTATION_COOKIE_SECURE=1
./scripts/run_demo.sh

An invite binds to a name on first use; the server stores only the invite hash. runtime/invites.txt contains plaintext codes and must stay outside Git.

annotation_platform/
├── annotation_platform/   # HTTP server, SQLite store, import and export
│   └── static/            # framework-free frontend
├── scripts/               # demo data and invite tools
├── tests/                 # concurrency, safety, and page smoke tests
├── assets/                # English-only project figures
└── runtime/               # local database and audit logs, ignored by Git

python -m unittest discover -s tests -v
python -m compileall -q annotation_platform scripts tests

The tests cover partial-record filtering, placeholder safety, correction handling, the demo database, HTTP APIs, and the main pages.

The repository contains synthetic fixtures only. Do not commit real annotation databases, audit logs, invite codes, student images, identity mappings, private screenshots, credentials, internal hosts, or exported datasets. See SECURITY.md.

AlignLedger is intended as a readable base for teams that need stronger review logic than a temporary labeling page. Domain adapters, private deployment, and training-export integration are available through WeChat znxzsy.

MIT