mg/FIELDNOTES

Build history / Pageweft

Building Pageweft: extraction to a hosted workbench

From a Modal MinerU worker to source-aware exports, credit accounting and payment readiness.

1. Establish actual extraction

The initial requirement was private PDF-to-Markdown preparation using MinerU on the owner’s Modal account. The design moved from customer-provided Modal credentials to owner-hosted processing; customers do not provide cloud tokens. MinerU 2.7.6 runs in a cloud GPU image and retains original engine output. The laptop’s 2 GB GPU is not required.

A bounded one-page public trial was verified before describing full-book processing. The initial deployed web startup failed because modal_fn.py was missing from its uploaded source. Explicit inclusion fixed the image, followed by a fresh deployment and public workflow checks. Original verification record .

2. Make books and partial failure manageable

Book inspection/ranges, explicit quotes, 20-page sections, partial outputs, selected retries, force OCR, cheaper CPU text extraction and cancellation were implemented. Source page indices were mapped back to the original PDF. Corrections/revision checks remained separate from engine output; exports preserved relative images, manifests, bookmarks and optional source PDF.

Actual GPU extraction checked selected pages from an original four-page fixture. A separate complete 1,000-page synthetic PDF passed the CPU text route across 50 sections with ordered pages and correct fixture-credit settlement. This proves that workflow on those sources; it does not prove scanned-book fidelity. Book release .

3. Account for usage and recovery

The book ledger has one serialized writer, distinct from the older prototype ledger. Credit holds are reserved, settled or released transactionally; duplicate confirmation does not dispatch twice. Processing and file retention are separate from persistent financial records. Owner reports, admission envelopes, pause and encrypted snapshots were added. Admission estimates are not a whole-account cloud bill cap.

One-time $5/$10 packs replaced the earlier monthly access proposal. Rename commit 1372062 introduced Pageweft while retaining cloud/storage identities. Paddle integration commit e7a70ee superseded the Lemon Squeezy activation path. Signed events enter a durable inbox; fresh provider state and server-owned intents control grants/refunds. Current source has 78 automated checks; payment HTTP in those checks is mocked.

Successful commands and prerequisites

The established owner deployment uses Python 3.13, authenticated Modal SDK 1.5.3, existing permanent session/payment secrets and separate ledger/results/model volumes. GPU dependencies run in the cloud image.

python3.13 -m venv .venv
.venv/bin/pip install -r requirements-dev.txt
.venv/bin/python -m unittest discover -s tests -q
node --check static/app.js
.venv/bin/python -m modal deploy app.py

For the recorded local browser fixture, with explicitly synthetic funds and real CPU text extraction:

.venv/bin/python -m uvicorn scripts.preview_books:app --host 127.0.0.1 --port 8768

Owner verification uses scripts.probe_deployed_books with an authorized original fixture and an explicit --mode text or --mode auto. Those probes seed internal verification credit; they are not purchases. scripts/test_e2e.py targets an older interface and is not the active release test.

Owner commands which were actually used for operations include:

.venv/bin/python -m scripts.operations report
.venv/bin/python -m scripts.operations backup

Reports/backups remain private. Never publish a source book, recovery key, provider credential or decrypted ledger while reproducing them.

Recorded problems and current boundaries

FindingResolution or honest limit
Missing deployed Python module prevented web startupInclude the module explicitly and redeploy; check actual HTTPS rather than build output alone
A whole-book job can partly failPreserve successful sections and expose selected retry/cancellation; verify credit settlement and output order
A budget estimate looked like a maximum invoiceSeparate admission envelopes, actual workspace billing, customer credit and cash contribution in operations
Earlier monthly/BYO/Lemon assumptions no longer matched the productOwner-hosted processing, one-time packs and current Paddle integration supersede them without resetting balances
Payment mocks passed but account approval was incompleteKeep checkout closed; complete identity/domain, real isolated provider tests and controlled live acceptance

The dedicated payment checklist and Paddle release distinguish configured, tested and still-required steps. Representative difficult-document comparison and measured customer margin remain separate work.

← All Pageweft documentsBack to top ↑