Changelog

What changed for you

• A page the reading machine gets stuck on is now cut off, and says it was cut off. A page is not a conversation: the machine reads what is on it and stops. Sometimes it does not stop — it falls into repeating itself, and carried on until it ran out of room. A typical page produces around 1,100 to 2,000 characters; the stuck ones produced 52,497 characters and took half an hour, and there was nothing to tell them apart from a very full page. One of the two reading tools had always had a limit; the other had none at all. Both now share one, and a page that hits it is marked as cut off rather than filed as though it had been read through — an incomplete reading recorded as a whole one is the kind of quiet wrong answer this toolkit exists to avoid (#118).

What changed for you

• The reading machine is given longer on a dense page. The time limit for one page was set from one set of books, where the slowest page ever seen took 108 seconds. The denser books are not like that: measured on 180 pages of three of them, the slowest took 332 seconds and one page in a hundred took over 165 — against a limit of 180. Since one page over the limit fails the whole book, a 500-page book was almost certain to fail. The limit is now 480 seconds, comfortably past the slowest page yet seen on the harder books (#118).

What changed for you

• A page the reading machine is still working on is no longer asked for twice. When a page took too long, the toolkit gave up and asked again — but giving up did not stop the machine working on the first one. It carried on, and the second request was *added* to it rather than replacing it. So asking again made the machine slower, which made more pages take too long, which caused more asking. Measured on the reading machine: it was working on 9–12 pages at once when it had been asked for 4, and its output fell from about 340 words a second to 5. Every book failed for five hours. Two changes stop it: a page that times out is no longer retried at all — the machine is already doing that work, and asking again only adds to it — and the reading machine now stops work the moment the caller hangs up, instead of finishing a page nobody is waiting for. A small self-test that took 3.2 seconds during the jam now takes 0.1 (#118).

What changed for you

• The development site now says, on every page, that it is not the live one. There are two copies of this toolkit: the live one, and a development one the team tries changes on. They looked identical, and the development one's page numbers can be wrong — on one collection, every single one of its 69,160 pages currently claims a printed page number it does not have. Nothing on screen said so. Now the development site carries an amber DEVELOPMENT badge beside the name, a tinted bar, the line *"This is not the live site. Nothing here is real or citable."*, [DEV] at the front of the tab title so bookmarks and history say it too, and a tab icon with an amber corner so two open tabs can be told apart at a glance. An assistant connected to it is told the same thing in words, because an assistant never sees a banner. 🔴 And anything we cannot name gets the warning too. The old check asked whether the address began with "dev-", which quietly called an unset address, a local one, a misspelling and a lookalike *the live site*. Now only the exact live address counts as live. Wearing the warning by mistake is harmless; missing it is how a wrong page number ends up in somebody's notes (#120).
• The menu on a phone is a single button. The header used to take 26% of a phone screen before any content, with the top bar stacking into four rows — and the Admin menu opened 78 pixels off the side of the screen, so five of its entries could not be reached at all and nothing indicated they were there. The bar is now one row, with a ☰ button holding the menu, your name and Sign out. Measured at 360 pixels: 205 → 63 pixels, 89 with the development notice, and every destination reachable. It needs no JavaScript. Wide screens are unchanged (#121).
• The Revoke button on the Tokens page keeps its word. When a token had a long name, the button was squeezed until *Revoke* broke mid-word — "Revo" on one line and "ke" on the next. Measured on the rendered page with a long unbroken name: it broke across two lines on a desktop screen and six on a phone, in a button 35 pixels wide. The button now takes exactly the width it needs and a long name wraps in its own column instead. And the page no longer scrolls sideways on a phone — it used to run 386 pixels wider than the screen, more than the screen itself; the table now scrolls inside its own frame, as the other tables on the site already did. The same fix applies to the People page, which has the same table (#119).
• On a phone, each token is now a card, and Revoke is the first thing on it. The Tokens page was a seven-column table. The earlier fix stopped the page itself sliding sideways, but in a table that scrolls the column that disappears off the right is the last one — and that column is Revoke. Revoking is exactly what you do from a phone: a laptop left somewhere, a key pasted where it should not have been. Each token is now a card with its name and its Revoke button side by side on the top line, and every value below carries its own caption, so a date no longer needs you to scroll back to a header row to know what it means. Wide screens keep the table. The column once headed *name* now reads *assistant*, which is what it holds (#120).
• The health page no longer says "not checked" about the reading machine when it simply has not looked yet. The line describing the second machine that reads page images is refreshed by a background check that only starts when something first asks for it — so the very first answer came back before that check had run, reading as though the check had failed. It said so about a machine that was perfectly healthy, and it misled this project's own diagnosis for a day. It now starts the check when the service starts, and where it genuinely has not looked yet it says that in words, rather than leaving a blank that reads like bad news (#118).

What changed for you

• A reading run now asks the machine whether it has gone, instead of guessing from a tally. Yesterday's change stopped a reading run after two books failed in a row, on the reasoning that the machine reading them must have gone away. Overnight it turned out that reasoning was wrong. The real pattern was fail, fail, *succeed*, *succeed*, fail — the machine was answering the whole time with its graphics card at full load, and individual pages were timing out rather than the machine being down. Yesterday's rule would have stopped before the two that worked and thrown away 1,084 pages that read perfectly well. So the run now asks the machine directly whether it is reachable, and only stops when the answer is no. Where the reader is a program on the same computer and cannot be asked, the old behaviour stands — there, a failure means something is not installed, which waiting does not fix. Nothing that was read is ever lost, which was not clear yesterday and is worth saying: a book that fails keeps every page already read, and starting it again carries on from there (#118).

What changed for you

• A page the reading machine fails to answer for now costs ten minutes, not an hour. When the machine that reads page images stopped answering, the toolkit kept asking — six times, waiting ten minutes each time — so a single unanswered page held a book for an hour before giving up, and the message blamed the network for an hour it had spent waiting. Five books died that way on 23–24 September, one after another, spending about three and a half hours of a graphics card proving the same thing over and over. Measured: a page that is merely slow takes 46 seconds typically and has never once taken more than 108; the toolkit was waiting 600. It now waits 180 seconds for any one attempt and gives up on the page after ten minutes total, and it says which of the two failures happened — *refused the connection* and *took the connection and never answered* need different people to look at them. And a reading run now stops after two books fail in a row for the same reason, instead of working through the queue burning an hour on each. Books it did not reach are left untried rather than marked failed, so starting it again picks up where it stopped (#118).

What changed for you

• The reader check now tests a letter pair that one reader gets backwards. Two Arabic letters that are printed as a single joined shape were being pulled apart in the wrong order by one of the PDF readers, turning *dalālat* into a non-word on 1,108 pages — it got that word wrong 98% of the time it appeared — and turning *islām* into a different word altogether. None of the phrases used to check a reader contained that pair, so a book could be passed as readable by tests that never touched the fault. One that does has been added. It stays quiet where it has nothing to say: of 30 books sampled at random, 29 are unaffected and one gets a better reader. Where it does speak, it either picks a reader that gets the word right, or reports that no reader can read the book — which is the honest answer for a handful of Persian volumes where each available reader loses something different, and sends them to be read from the page images instead (#112). ⚠️ This takes effect as books are re-read; nothing already on the shelf changes until then.
• The name in Arabic at the top of every page now takes you home, as the English one does. It sat beside the English name as plain text -- the pointer passed over it and nothing happened -- and the same was true in both sibling apps. Measured in a browser before and after: the bar held one link home and the Arabic name was not clickable at any point on it; it now holds two links to the same place, and a click on the Arabic name lands home. The keyboard and a screen reader are unchanged: both names are one destination, so it is still announced once and still stopped at once. It looks exactly as it did -- same position, same size, same colour -- and only gains the highlight the English name has always had when you point at it. The large name on the Home and sign-in pages stays plain text, because neither name is a link there (#117).
• diwan is live, so the footer links it. The footer lists the other IlmFlow projects, and showed diwan as greyed-out text marked "not live yet" because its public site was not up yet. It has been answering since 24 Sep 2026, so it now reads as a link like the others. One setting decided that, which was the point of having the setting (#117).

What changed for you

• The health page can now tell you that text recognition is broken. It used to report the OCR tool as working whenever the program was installed — which it was, on a machine where every single page failed for months because the Arabic language model was missing. It now asks the program which languages it can actually read, compares that against the languages this library is written in, and says which ones are missing and where it looked. The same check runs against the same folder the real work uses, so the two cannot disagree (#92).
• A missing language model is reported to you instead of crashing. The code has always held the right message — *"no ara.traineddata under … ; fetch_tessdata.py downloads the models"* — and never showed it, because the check was switched off by its own default setting. It now runs whichever models are chosen, and every result names which model set read the page rather than just saying "the system's": the two available sets differ by a factor of seven in size and disagree about which pages they read better, so a result that cannot name one is not a measurement (#92).

What changed for you

• **A phrase containing the word *Allah* now finds pages where the reader dropped its last letter. On more than a quarter of the pages read by one of the PDF readers, `الله` is stored as `الل` — the final letter simply missing. Searching for any phrase containing it returned nothing, while every safety check reported the search was sound: the worst shape a wrong answer can take, because there was no sign anything was amiss. The last letter is now treated as optional when you type that word**, so the damaged page is found alongside the intact ones, and such results are labelled *matched on a truncated word* — the missing letter is not evidence, and the page may not say what you typed. The stored text is not altered: `الل` is equally the start of *al-layl* and *al-lugha*, so filling it in would be inventing a word the page does not contain. Searching for the word on its own is declined, with the reason given (#93).
• Written down: a deploy command that reports success and changes nothing. Copying the code to a server with the address held in a shell variable can silently copy it into a folder on the machine you are sitting at instead, because the shell quietly rewrites the address. The server keeps running the old version, which looks like a service that failed to restart. Recorded in the server runbook with the one-character fix (#111).

What changed for you

• The footer now says which family this tool belongs to, and links the others. After *A Truth Promoters Project* it reads *Part of the IlmFlow Project*, with IlmFlow linking to the project's own site, and then the sibling apps — daftar and diwan. Each of the three carries the same footer and leaves itself out, so wherever you are you can get to the other two. Which addresses it shows follows the site you are actually on: the development site links to the development ones. diwan's live site is not up yet, so its name is shown as plain text rather than a link that would go nowhere — it becomes a link the day it answers (#110).
• A result no longer offers a page number for a book that has no pages. Searching the Hawza library returned a "page" for almost every hit — 62,020 of its 69,160 — and that number was the record's position in a JSON file, not a page of any printing. It was the one collection whose text a human is allowed to quote, so a citation taken from it named a page that does not exist. Every hit now says what kind of locus it is carrying: a printed page a reader can open, a citation a scholar would actually write (*vol 1, ḥadīth 7*; *2:255*; *Genesis 1*; a speech's date), a position that is honestly a position, or nothing checked. The "Copy citation" button, "Add to notes" and the reading screen itself all follow it — the first two used to fall back to the position and label it "p", which is how a note could come to record "PDF p. 412" for a file that has no PDF, and the reading screen told you to "cite the record (412)" when 412 was the row's place in the file. It now names the citation instead. The Hawza text is unchanged and stays quotable: what was invented was the reference, not the words (#91). 📏 The ticket's stated cause was wrong, and the measurement is kept here rather than tidied away. #91 blamed an `n + 1` fallback in seven of the eight JSON handlers; instrumented over all 218 files and all 69,160 records, that fallback fires zero times — every record carries its key. Removing it would have changed nothing. The cause was one layer up: the source's own key is positional (thaqalayn's `id` is 1..N in 32 of 32 files; al-Mīzān's `page` restarts at 1 in all 115 sūra files and never exceeds 94, for a 20-volume tafsīr) and it was written into the field that means "the folio this printing prints" on the PDF path. Two further corrections: the Noor discs' 556,761 matching pages are correct and untouched — their `page` is a real folio (41% of files reach a page beyond their record count, 22% repeat one, 7% start mid-book), so the same handler was serving two opposite meanings and they are now told apart by the data; and Nahj al-Balāgha's `sermon-1`, `letter-1` and `saying-1` all carry `number` 1, so three different texts were citing as "1". What each shape can be cited by is now declared per schema in `rtk.core.locus`, with the measurement beside it, and the tests assert the declaration rather than the coincidence. Index schema v9; an index built before this change keeps being served, and reports "nothing has been checked" for a page whose number it cannot vouch for rather than guessing — which is the same rule as the fix itself, applied to the gap between a deploy and the next rebuild.
• 245 books from the Noor discs no longer show a title the publisher never gave them. Eight of them were listed as *"The Islamic Republic of Iran — Holy Qom"*, which is the address off the back of the title page; others carried a library subject heading, an editor's name, or a sentence from the middle of a page. The cause is exact rather than approximate: these are the files the discs' own catalogue has no entry for, so the packaging tool scraped the block off a rendered page — which is also why 236 of them read backwards. 47 of them print a proper colophon and now take their title from it; the other 198 keep the text, plainly marked as a scrape and needing a human to name them. The 5,334 books that do have a catalogue entry are untouched, and no reviewed name was changed (#49).
• A Qur'anic verse spelt with the hamza on a different letter is now found. Qur'an 17:36 failed to match between the Qur'an text in the corpus and the same verse quoted in a write-up, because the two place the hamza on different carriers. Reported by Cowork, who also found the same class of fault in the older tool (#94).
• A result about a bilingual book no longer claims it is in one language. Volumes with Arabic and a facing English translation were reported as English by one path and Arabic by another, depending on which answered — and a tool reading either would lose half the page. A result now carries both answers, says which is the measurement and which is a human's decision, and says plainly when they differ (#94).
• Uploads and the machine-read text are now kept off the local disks. Files you upload go straight to the network store rather than a server's own drive, and an hourly copy of the extracted text is pushed there too — so losing a machine no longer means losing the reading it had done. The copy never deletes: if something goes wrong locally, the copy that was meant to save you does not agree with the mistake (#108).
• The Library no longer says two things that were not true. "Resources by topic" reported 155 books searchable and nothing extracted, which read as work waiting — when every one of those books is the same book as one filed under Books, already extracted and searchable. It now says so. And three collections that hold nothing at all no longer get a tile inviting a click that could only disappoint (#106).

What changed for you

• 5,579 books from the Noor discs now have their real names. They appeared in the Library under filenames like `noor-khlfyat-ktab-masah-alzhra-alyha-alslam-394`, which is unreadable unless you know it is a transliteration. Each one now carries its Arabic title, its author, its subject heading and the publisher's own catalogue number — taken from the discs' own metadata, not guessed. They are marked unreviewed, like every other suggestion, until you confirm them.
• 236 of those titles were stored back to front and have been turned the right way round, with the original kept in the entry's note so you can check the repair. Those are marked as less certain than the rest (#107).
• Understood and written down: why one book tripped the quality check. A reader program was placing the vowel marks of Arabic words as separate pieces of text with spaces around them. The toolkit removes vowel marks and spaces when it searches — both correct on their own — and together they pushed letters next to each other that were never next to each other on the page, which looked like nonsense matching. Recorded where it will be found again, with a test (#100).

What changed for you

• A book is no longer thrown out of the search because one of the four readers could not read it. Each book is read by four different programs and checked with a nonsense word that must never match. If it matched for any one of them, the whole book was excluded — even when the other three read it perfectly. One Persian book of jurisprudence was being held out of the index for exactly that reason, and its own report said it contained no Arabic while listing 138 occurrences of a word. The failing reader is now set aside and the book is judged on the rest; only if every reader fails is the book held back (#100).

What changed for you

• Asking for the state of the shelf no longer returns a twelve-megabyte answer. Called without naming a collection — which is the obvious first thing to ask — it returned every book's full record twice over, which was too large for the assistant to read at all. It now answers with the counts, the breakdown per collection, and the short list of books that need attention; ask for `detail` when you want every row. 99% smaller, and it now says how to get the rest (#102).

What changed for you

• Each page now carries a fingerprint, so another tool can tell you when a page you quoted has changed. Two fingerprints, in fact: one of the page as extracted and one of the searchable form, plus the version of the spelling rules that produced it. Read together they separate "this page was re-read and the words moved" — which means a quote needs checking — from "the spelling rules changed", which means nothing needs checking at all (#103).
• Searches examine each candidate page a little under four times faster. The two matching forms added earlier today were being recomputed for every page of every search; they are now stored with the page (#105).
• This needs a fresh index. Until one is built, searching keeps working exactly as before, just without the speed-up.

What changed for you

• Nothing changes in how it behaves. This release adds the agreed set of spelling traps as permanent tests, so a future change cannot quietly undo one. Seventeen cases from the Diwan work and thirteen from the majlis tools, including two controls that check the test itself can say both yes and no (#104, #105).

What changed for you

• A phrase broken by punctuation is found again, without loosening ordinary searches. The fix in 0.2.15 widened what may sit between words everywhere, which would have let a phrase match across a full stop into a different sentence. It is now a separate, last-resort way of matching — everything-but-the-letters removed — so ordinary searches stay exactly as strict as they were (#104).
• 🔴 0.2.16 should not be used. It shipped with the page-ranking count disagreeing with the search itself, so a result's position could not be trusted. Fixed here (#105).

What changed for you

• Qur'anic verses written in the two common spellings now find each other. The Qur'an text in the corpus uses one orthography and the verses quoted in write-ups use another — they spell the same word four different ways, so a correct verse could look absent. Searching now tries a third spelling as well, which can only find more, never less (#105).

What changed for you

• A phrase with a colon or a comma in the middle of it no longer comes back empty. Searching for words that the page prints with punctuation between them — `قال: الصلاة، الزكاة` against the phrase `قال الصلاة الزكاة` — returned nothing at all, on text that was there in full. Printed hadith is full of such punctuation, so this was a confident "not found" about pages that plainly had it. It now matches through punctuation, while still refusing to jump over a whole word, which would be a different sentence (#104).

What changed for you

• Another service can now use the toolkit's Arabic normaliser and sign-in check without installing the whole thing. Diwan will reuse both rather than writing its own — one set of spelling rules across the estate, so a search that works in one place works in the other. Installing just the sign-in check no longer drags in a web server with it (#103).

What changed for you

• The rendered-page directory no longer grows for ever. Nothing had ever deleted one, so it accumulated a file for every distinct book-page-dpi anyone had looked at. The nightly now prunes it to a ceiling — `RTK_RENDER_CACHE_MB`, 20 GB by default — oldest first, and `/healthz` reports the ceiling beside the size so a number has a scale. `rtk renders prune --dry-run` says what would go without deleting anything (#97).
• The nightly pass could not use the GPU OCR node, and re-running the service installer would have taken it away from the others too. The installer builds each service's environment from one list, and that list had never mentioned the OCR node's address — so the two services that had it had been edited by hand, the nightly one had never had it at all, and the next run of the installer would have regenerated all of them without it. The address is now in the list, and a test asserts every service carries it, the nightly one included (#98).
• The services can be installed on Linux, not only on the Mac. The script that generates the machine's service definitions now speaks systemd as well as launchd, from one description of each job, so the two platforms cannot drift apart. It carries the awkward differences rather than hiding them: the nightly pass becomes a timer and a job, a missed nightly still runs at the next boot, a crash-looping service is retried for ever rather than given up on after five tries, and the files are readable only by root because they carry the sign-in key (#96).
• The installer no longer prints the sign-in key to the screen. It lists each setting so you can check the machine's configuration, and the sign-in key was in that list in full — into terminal scrollback, a screen share, or any log that captured it. The name is still shown, so you can still see whether it is set, but the value is now a length and a short digest. On Linux it also stops warning about Homebrew, which does not exist there, and checks instead that the page-rendering and OCR programs are actually on the service's path (#96).
• Fihrist now runs on the two Proxmox VMs, and there is a runbook for them. Both machines serve the toolkit over HTTPS with a certificate from the estate's own authority, run the nightly pass on a timer, and reach the GPU OCR node. `docs/RUNBOOK-FIHRIST-VM.md` records what the hosts are, how to regenerate their services, and the eight traps that cost time setting them up — including the one where a package quietly installs a second web server that would fight ours for the same ports (#96).
• A machine can now serve several names over HTTPS from one block. The installer took a single site name, so a machine that answers both to its own name and to the service name — which is every machine in the middle of being moved — could only have a certificate for one of them. `--host` is now repeatable, the names share one site block, and the MCP transport admits all of them (#96).
• The notebook is copied every hour, and it copies the right thing. Nothing had ever made a frequent copy of the two things on the machine that cannot be rebuilt from anything else, so a whole-machine backup twice a day left up to twelve hours of notes at risk. There is now an hourly copy, keeping a day of them for about 36 MB. It takes the note files, not the search index built from them — copying only the index gives a machine where searching finds a note that listing says is not there, which reads exactly like losing it (#99).

What changed for you

• The OCR can run on the CUDA box, about nine times faster than the mini. `doldol` — a Ryzen 9 with an RTX 5060 Ti — now serves the same vision model the mini uses, and `rtk shelf ocr --engine doldol --concurrency 4` reads through it. 📏 The whole remaining backlog (~36,000 pages: 18,304 with no text layer at all, ~17,900 whose text layer is demonstrably lossy) drops from about 42 days to about 5. The box is a stateless inference appliance — no corpus, no sidecars, nothing to back up; the mini renders the pages, writes the sidecars and stays the only system of record (#89).
• 🔴 And the model had to earn its place before a single backlog page went through it. All four hand-typed fixtures were re-run on the new hardware and matched — *Kamāl al-dīn* character for character. 📏 The first attempt used `dots.ocr`, the model a published Arabic benchmark *prefers*, and it scored 95.5% against `dots.mocr`'s 98.5%. Had the backlog gone through on the leaderboard's pick, every one of those 36,000 pages would have been measurably worse and nothing in the output would have said so (#89, #77).
• `book_ocr` works on the server for the first time. It had never once succeeded there: no Arabic model was reachable, and `/healthz` reported `tesseract: true` regardless, because it asked whether a binary was on the PATH rather than whether the engine could run. A caller lost work to it and could not tell why (#92).
• An unavailable engine now comes back as an answer you can act on, naming the fix, instead of an opaque crash with no message at all (#92).
• A second Arabic page is hand-typed, and it settles what is left to win. Shabbir typed the tawqīʿ page of *Kamāl al-dīn* — PDF 512, printed ٥١١, twelve lines, 67 search terms — and it is now the third ground-truth fixture. 📏 On it, dots.mocr finds 98.5% of the search terms (66 of 67) at 8.93% CER; Tesseract finds 86.6%; and the PDF's own text layer — the text the index holds for this book today — finds 89.6% at 18.01% CER. 🔴 Take the page's five compound honorific glyphs out of both sides and dots.mocr's CER is 0.00% and recall 100%: on clean Naskh print the vision model reads perfectly, and every remaining error on this page is behind one ligature (#22, and the case for #23).
• 📏 The `clean_layer` verdict says *"the publisher's own text"*. For an Internet Archive `_text.pdf` partner it is that publisher's OCR, and this page measures it at 18% CER with three of its seven missed terms being honorifics. First time the claim has been checked against a typed page rather than assumed (#22).
• The fixture was typed to the conventions and passed the convention check on the first pass, and was still wrong in two places — a bidi ordering slip (`[في ت]` for `[ت في]`) and a typing slip (`متّي` for `منّي`) — both caught against two independent witnesses, corrected before installation and recorded in the sidecar. `typed/README.md` says how (#22).

What changed for you

• The Corpus search is on the API and MCP too. `GET /v1/corpus?q=…` and `corpus_works(q=…)` narrow the Corpus exactly as the page's search bar does — author, title, language, type, spelling forgiven — and say how many of how many they kept (#88).

What changed for you

• The Corpus page has a search bar. Across the top, like the one on truthpromoters.com: type and the list narrows as you go — across author, title (Arabic, transliteration, English), language and type; every word you type must match; a section with nothing left folds away; the count says *n of N works*; clear the box and everything is back. Spelling is forgiven both ways: *طوسی* from a Persian keyboard finds *الطوسي*, and *Mutahhari*, *Motahhari* and *Muṭahharī* find each other — as do *Seyed* and *Sayyid*, *Khomeini* and *Khumayni*, *Saduq* and *Sadooq* — because a transliteration is reduced to what every spelling of it shares before it is compared (#88).
• The Lantern of the Path is one work, in its own tier. Shabbir's Wednesday sessions — 68 files, sessions 1 to 68 — were 62 separate rows under lodestar's guesses; they are one work now, each session a volume named by its file, under a tier of their own between the contemporary Imams and fiqh. Volumes sort naturally everywhere (2 before 10). Sessions 69 to 75 are not on the shelf yet.
• Catalogue review complete for every collection. After the books, hawza, Mutahhari and published sheets: 795 of 797 entries vouched for by a named person; the Corpus page has no unnamed file.

What changed for you

• Searches are fast again on the big index, and the Noor CD-ROM is searched only when you name it. The first index that held the Noor tree (1.72M pages, 21 Sep) made a plain search slow: 📏 `index_bench` on the twin measured p50 1.9 s / p95 18 s *warm*, and 101 s cold for *تهذيب الأحكام* scoped to *books* — because a collection filter was a per-row check over every page the term matched anywhere in the index. Two changes. Schema v6 puts the collection into the index as a token, so *books* is a doclist FTS5 intersects before it touches a row; and a collection marked `search_default = false` in `corpus.toml` — the Noor tree, 5,583 documents of fabricated titles and lossy text, 🔴 throughout — is left out of a search that names no collection. Every payload says what it covered (`trees`) and what it left out (`note`); *Noor CD-ROM* in the collection picker, `--tree noor` or `trees=["noor"]` searches it. The first rebuild after this release swaps the v6 file in; until then searches run on the old file, filtered the old way (#58).
• Home shows the OCR while it runs. The vision model reading an image-only book — three days on Tahdhīb al-aḥkām so far — was invisible: it held no lock, so Home, `/healthz`, the assistant's `dashboard` and `job_status` all said *idle*. Each book it reads is now a recorded job with page-by-page progress (`reading tahdhib-7-8 · page 620 of 735`), shown on Home and Maintenance and carried on `/healthz` as `also_running` beside whatever holds the lock; `activity` itself is unchanged, so a monitor alerting on it is untouched. A build starting beside the reader no longer marks it interrupted. The queue's current run predates this and shows only from its next book (#86).

What changed for you

• The index builds in minutes, not days. 📏 A retraction first: 0.2.5 said the FTS5 merge "ran for days" on the production machine and blamed SQLite's page cache. It was assumed. A `sample` of the three-day-old build on 21 Sep showed it still *inserting*: the build ran three full scans of the pages table per file (an FTS5 table cannot index a document id), so 6,575 files cost days of random reads, and the merge was never reached. Schema v5 keeps an indexed table of which rows belong to which document and writes the trigram rows from the values in hand: the laptop's 196,820 pages now build in 104 s (inserts 98 s, merge 5 s), and `rtk index build` prints the phases so the next slow build names its phase. A v5 reader opens a v4 index, so nothing goes dark at deploy; the first rebuild swaps the new file in (#58).
• The Corpus is on the API and MCP. `GET /v1/corpus` and the `corpus_works` tool give your assistant the Corpus page as data — every named work by tier and author, its kind, the languages of the copies, volumes, files, pages, and whether a person has vouched for the name — so it can answer "do we have X?" before it searches (#85).

What changed for you

• The digital books read like books. A record of a JSON work is laid out by what it is: the Qurʾan's verses in a Qurʾan typeface with ﴿n﴾ markers, al-Mīzān's commentary under its own headings (بيان, بحث روائي, …), a hadith as its chapter, its Arabic beside its English, and the grading the source records; a sermon, a prayer, a Bible chapter each as themselves. Arabic is set in Amiri, Persian in Vazirmatn (both OFL, shipped with the toolkit). On a phone the columns stack. Nothing changed in what is indexed: the blocks join to exactly the text the search reads, and the API carries them too (#84).
• A signed-in member stays signed in on the public pages. About, Roles, Contact, Privacy & terms and the Changelog showed everyone as a stranger and offered *Sign in*; they now read the session as a courtesy, never as a guard (#80).

What changed for you

• The Corpus page is by author, and says what each thing is. One row per *work* — a title and an author, so al-Nuʿmānī's and al-Ṭūsī's *Kitāb al-ghayba* are two rows and the two copies of Strack–Stemberger are one; an English monograph is a row under its English title, not "(unnamed)". Within a tier the rows sit under their author, who is named once. A type column says book, article, index or course; the language column names every language we hold the work in — a Persian translation beside its Arabic, Indonesian, Hebrew, Greek — and volumes are told from copies. The catalogue gained `kind`; `review-work` gained `--kind` and `--author`; the review sheet no longer lists the image-only twin of an OCR'd file as a second, unnamed work; and the estate model is told to transliterate Persian as Persian and Western-language works not at all (#82).
• Shabbir's review of the books sheet is in. 232 decisions applied to 413 files: names approved as shown, 60 corrected — al-Ṭūsī's *Kitāb al-ghayba* separated from al-Nuʿmānī's and joined with its Persian and English copies, *Lisān al-ʿArab* in 17 volumes, *al-Tibyān* in 9, *Tahdhīb al-aḥkām* and al-Ghaffārī's sharh, 14 Persian journal papers named with their authors, Malter's *Taʿanit* no longer Maimonides, two damaged files recorded as such. The books collection has one unnamed file left (#83).

Indexing

• A build into its own file takes no writer lock. `rtk index build --out PATH` replaces nothing shared, so two of them can run side by side for an A/B — the twin refused the second while the first held the lock, though they wrote different files (#58).

What changed for you

• The API explorer is at `/api`. `/docs` is reserved for the documentation on how to use fihrist; until that exists it takes you to Help (#81).
• Five pages need no sign-in. About, Roles, Contact, Privacy & terms and the Changelog are readable by anyone — daftar's pages in fihrist's words, English only — linked from the foot of every page and from the sign-in page. They name no member, show no nav and no page of the library; Contact points at daftar's form, because fihrist sends no mail of its own. `/terms` lands on the same page as `/privacy` (#80).
• The team guide starts with the website. A new Part 0 in `docs/TEAM-GUIDE.md`: signing in with daftar, finding a page, reading it, the notebook, and connecting your AI assistant with a token — nothing to install. The Terminal parts remain for anyone with the drive (#53).
• Saving a note no longer re-embeds the whole note. Only the paragraphs whose text is new go to the encoder; an appended locus costs one chunk, an unchanged save costs nothing. Seen during the load test, when five note-writers kept the embeddings machine's GPU at full tilt and each write waited a second for it (#75).
• A refused token leaves a trace. When a call carries a token the door does not accept, the audit records why — `no-scheme` (the header lacked the word *Bearer*, the commonest slip in a connector's *Value* box), `unknown`, `revoked`, `member-disabled` — with the token's fingerprint, never its value, at most once a minute per token and reason. The Tokens page shows each token's refusals from the last seven days beside it (#70).
• A search for الصلاة now also finds الصلوة. The classical spelling of a word-final *-āh* with a wāw — الصلوة, الزكوة, الحيوة — is what the hawza texts keep, and it was unreachable from the spelling a person types today: measured, 72 pages carry الصلوة and not الصلاة, 44 الزكوة, 103 الحيوة. Every search now tries both forms, in either direction, and `variants_tried` lists what it tried (#68).
• The catalogue can be reviewed one work at a time. `rtk catalogue works` lists every multi-volume work with each spelling its volumes carry and which volumes carry it — 25 of the 34 works disagree with themselves today (Bihar's 98 volumes under 16 spellings, al-Dharīʿa's 28 under 21, half of them credited to their editors). `rtk catalogue review-work` applies one decision to every volume: each name field as given, or the form most volumes already carry, all marked reviewed by the named person (#69). The review itself is Shabbir's.
• The hawza library has names. Its 218 digital books — al-Kāfī, Man lā yaḥḍuruhu al-Faqīh, Nahj al-Balāgha, the Ṣaḥīfa, al-Mīzān, Ṣadūq's collections, the Qurʾan with translations, the Bible in nineteen versions — were filenames until now. The 213 whose files name them unmistakably are entered by hand, marked as such and unreviewed until a person vouches; the five that needed judgement (Imam Khamenei's collected words in three languages, the al-Ahed speech files) were asked of the estate model. `rtk catalogue sheet` writes the review sheet: every unreviewed work, what it is called and by whom, a 🔴 snippet from its first page so a person can recognise it, and a line for the decision (#79).
• A Corpus page. What we hold, by *work*: one row each, volumes counted, the Qurʾan first, then the primary collections of hadith and the words of the Imams, tafsir, other hadith and supplication, the contemporary Imams and the Resistance, fiqh and philosophy and history, other scriptures, and everything else — with each work's languages, pages, whether its pages are searched or only readable yet, its 🔴 grade, and whether a person has vouched for its name. A file the catalogue cannot name is not a row; the page counts them as the backlog. The order is one list, `rtk.core.labels.CORPUS_TIERS`, Shabbir's to reorder (#79).

Indexing

• The index build stops re-reading its own file. The last step of a build — merging the trigram index into one piece — was written when it took seconds over 75,000 pages; over today's 235,000 it ran for *days* on the production machine, doing 12,700 random reads a second on SQLite's default 2 MB page cache. The build now uses a build-sized cache (1 GiB, `RTK_BUILD_CACHE_MB`), keeps temporary work in memory, and writes the disposable `.building` file without a journal or per-transaction sync; the merge itself is a measured switch (`RTK_BUILD_OPTIMIZE`: `full` today's behaviour, `none` leaves the segments as they are), so the twin can measure both before the choice is made for production. `rtk index build --out PATH` builds into any file — for that A/B, and to build on one machine and ship the file to another (#58).

What changed for you

• The name is written the way daftar writes its own: `fihrist`, lowercase, with `فهرست` at the right-hand edge — in the header of every screen, on the sign-in page and on Home, the Latin hard left and the Arabic hard right, each at the edge its own script reads from. The Arabic is the same word and is hidden from screen readers so the name is announced once (#71).
• Two bugs the load-test rehearsal found before the team did. A page whose printed number is in Arabic or Persian digits — most of the library — failed to render as an image (a header could not carry the digits); it renders now, with the label kept as printed (#72). And under a handful of people searching at once, a valid token could be refused: the sign-in store shared one database connection between requests; it now keeps one per worker (#73).
• Twenty people searching at once no longer wait behind each other's page images. The service had one pool of four workers for everything, and every page render sat in it; the load test on the staging twin (twenty searchers, five note-writers, ten minutes) was correct to the last request — 1,538 searches, 144 renders, 665 loci, zero errors — and slow to the last person: a search waited 5 s typically, 9 s at the 95th percentile, on a machine that was two-thirds idle. Renders now have their own small bound (two at a time — the disk's limit) and searches get eight workers on these ten-core machines; both are configuration (`RTK_RENDER_THREADS`, `RTK_THREADS`) (#74).
• Searches no longer take turns. Doubling the workers changed nothing on the twin (2.5 → 2.4 searches a second), because most of a search was Python re-normalising every candidate page's text to count occurrences — text the index had already normalised when it was built. The count now runs on the stored text with patterns compiled once per query (proven identical over 3,200 real page comparisons), and the catalogue is parsed once per change instead of once per search (#76).

Measurements

• `scripts/load_test.py` — the §7 load test as a script: twenty searchers and five note-writers for ten minutes against a public name, renders every tenth search, the box sampled over ssh; pass is p95 search under 2 s, zero 5xx, zero lost notes. Rehearsed locally: the two bugs above, then a clean run (158/158 searches, 49/49 renders, 34/34 loci) (#53 §7, #66).

What changed for you

• Fihrist has a front door. Every screen now asks who you are: you sign in with Daftar (your passkey or magic link), Daftar hands the toolkit a sealed sixty-second note saying who you are, and the toolkit checks it against its own list of members before letting you in. Two roles — *researcher* and *admin* — and a People screen where an admin adds someone by email, changes their role, or switches them off (which ends their sessions at once). A person with a Daftar account but no Fihrist membership sees one polite page and nothing else; a stranger sees only the sign-in page. Nothing about the 🔴 rule changes for any role (#57, #53).
• Your assistant acts as you. A Tokens screen makes tokens for Cowork, Claude or any MCP client — shown once, revocable, scoped to *read*, *notes* or *admin* — and every call to the API or the MCP endpoint now carries one and is that person's. A call beyond a token's scope is refused with the scope it lacked named, so the assistant can ask for a wider one rather than guess. The shared `RTK_TOKEN` switch, never set anywhere, is gone (#61).
• Your notebook is yours. Every note now carries who wrote it and is private by default; a *Share with the team* button opens one to every member, where it appears under Shared notebooks by author — readable and searchable by all, changeable only by its author. Nobody, admins included, can open another member's private note through the toolkit. The git history behind the notebook records the real author of each change, and a glyph decision is signed by the person who made it, with nothing to type (#62).
• It is called Fihrist. The name is on the sign-in page, the header and the Home screen — *Fihrist (Research Toolkit)* — and the header is two rows now: the name and who is signed in, then the menu, with the admin screens under one Admin ▾ dropdown; the version lives in the footer only.

The webtop

• Sign-in tokens no longer land in the logs. The sign-in handoff arrives as `/auth/callback?token=…&state=…`; the token is spent on use and lives sixty seconds, but its payload is plain base64 — the id, email and name of everyone who signs in — and it was being written in full to the web server's access log and the toolkit's own. Both now write `token=REDACTED&state=REDACTED` (one filter fragment for both of the web server's access logs; a logging filter on the toolkit's access line, installed through the server's own log configuration so it cannot be dropped at start). The web server was already hiding cookie and bearer values; the query string was the gap (#64).
• The page screen shows its page again behind the front door. Its image and the *Download at 300 dpi* link pointed at the API, which since the Tokens work asks every call for a bearer token — and a browser's `<img>` cannot send one, so behind the sign-in every page was a broken image while the screen around it rendered fine. Found on the staging twin the first evening it was reachable from the internet; never in a release. The image is served by the webtop itself now, under the same session as the screen, from the one render routine the API uses, and the test that guards the reading surface follows the image as a browser would instead of checking that the page mentions it. The render is also marked `private` on both surfaces: it was `public`, which entitles any shared cache between you and the toolkit to hand the same page to the next person who asks (#67).

Testing

• The repository has CI: every push and pull request runs `rtk selftest`, the test suite and `ruff` on a Mac runner with the same poppler and tesseract as the machines that serve people. The runner has no corpus, so the tests that need the real shelf are marked `needs_corpus` and skipped there with the count printed as the run's last line — a green says the toolkit works, not that the shelf answers (#59). Also: the repository is `ruff`-clean, and the three copies of the "needs the corpus" guard became one that checks the index has documents rather than that a variable is set.

Indexing

• `rtk index build` runs under the writer lock as a recorded job, and refuses — visibly, exit 1 — while another writer holds it, instead of starting a second builder on the same file. The build also refuses a `.building` file touched within the last two minutes: that is another builder alive, not debris. Found on 18 September when a queue-triggered build and the nightly ran at once and the nightly would have swapped the other's half-finished index live (#56). 📏 The index is now 13+ GB with the hawza tree and the OCR'd pages in it, and a build takes hours, not the ten minutes the runbook used to say.

Installing

• `scripts/install_service.py` now derives the one command that needs root from which services are *actually* loaded (`launchctl print`, no root needed to ask), so a fresh machine gets all four plists in one chain and a machine adding HTTPS gets only the two it lacks. It used to assume the first machine's history, and handed a second machine a two-plist line for a four-plist install (#55).
• The toolkit refuses to start if it would trust the test key on a public name. The sign-in test key is committed to the repository, private half included, for the test suite and for development. Configured on an `https://` origin it would let anyone who can read the repository sign in as anyone; now `rtk serve` exits at start naming the key, matched by its bytes so renaming it does not help. Plain-`http://` development origins are unaffected (#65).
• The public name's site (`--public-host`) now tells the toolkit every request arrived over HTTPS. The gateway terminates TLS and reaches the mini over WireGuard as plain HTTP, and without this the toolkit believed it was serving `http://` — measured from outside on 19 September, a trailing-slash URL redirected to `http://dev-fihrist…`, which would have dropped a member's session cookie on the way. Also: a CSS comment that named a person on every page is gone, and the test that guards the public pages now reads the whole page, not just the body (#63).

• The changelog page hides the *Unreleased* section while it is empty; the file keeps it, because the rule needs somewhere for the next line to go.

Webtop

• The top menu no longer overflows: *API* and *Changelog* moved to the footer, beside a new Connect your AI page (`/connect`) that explains MCP and how to connect Claude Desktop, Cowork, Claude Code, Cursor, VS Code, Gemini CLI and any other MCP client — the address is derived from the one the reader arrived by, the tools are listed from the server's own list, and the one rule (🔴 nothing read through the toolkit is quotable) is stated for the assistant too.

What changed for you

• The toolkit has a name on the screen — Research Toolkit — a dark theme in the Reflections palette, a home page that says what is searchable and whether the machinery is healthy, and plain words for everything (a *collection*, not a tree; a *Library*, not a shelf dump).
• Search results are grouped by book, and a broad search answers in about a second instead of four.
• Every book carries a name you can read (Arabic title, transliteration, English), a language read off its own text, and where the edition prints its page numbers, the printed folio.
• There is a Notebook for session notes, searchable by meaning; a Help page on adding books and what every message means; and a Glyphs sheet for reviewing the decoder.
• Three shelves that were dark are lit: the Persian Mutahhari volumes set in TrueType, the Hawza JSON library (al-Kāfī and the rest, the Qur'ān with translations, al-Mīzān), and the Greek New Testament. Scanned books are being read by a vision model, one by one.
• The rule has not moved. Machine-read text is 🔴 not quotable on every screen; the tool finds the page, and a human reads it.

Search and the index (#38, #47)

• Results are capped per book inside the SQL (a window over a proxy score that orders and never drops a row), with per-book totals from the same matched set, so the counts cannot disagree. 📏 `السلام` matched 10,625 pages across 608 documents and the 25 shown were all from one book before this.
• Index schema v4: a trigram FTS5 table over the space-stripped text makes the scan path indexable. 📏 `السلام` 4.4 s → 0.9 s; the scan itself 1.0–1.4 s → 3–16 ms with identical counts; +856 MB, 45 s at build. Positive and negative controls are remembered per index and scope. A current index is opened without writing to it (every search was a write transaction, and two readers at once produced "database is locked").
• The positive control follows the query's script: a Greek query is vouched for by `καί`, English by `the`, not by `من`.

The shelf (#3, #4, #5, #28, #12, #50, #51)

• The Hawza JSON tree is searchable: 218 books, 69,160 records, each record a page, seven file shapes read by one extractor, provenance kept per record.
• The direction gate is wired into extraction (visual-order Arabic reported by default, repaired or quarantined per collection), and the detector's one-vote rule was retracted after it produced false positives on real prose.
• noor-hawza has a derived index of identity and metadata consistency — and no titles: 📏 ten of ten files whose metadata matched the CD-ROM's own index exactly held a different book. The standing rule is now conditional on a measured-clean, human-reviewed extract.
• Every sidecar carries a recorded extractor-fitness verdict (#28: 438 books re-read), the gate's control set speaks Persian as well as Arabic (13 of 14 "undecided" Persian volumes now certify), and Greek, Cyrillic and Hebrew text layers are clean, not broken.
• A book's language is read off its text — sixteen scripts, our own rules for Arabic / Persian / Urdu and six Latin languages, an offline second opinion for the rest, mixed printings named as both — shown and filterable in the Library. 📏 The collection called "Arabic books" held 176 English volumes and 79 Arabic ones; it is now *Books*.
• `rtk shelf ocr` reads image-only scans through a vision model into sidecars, resumable per page, one process at a time. 📏 52 scans, 44,071 pages; ~55 s a page on dense text.
• The catch-up of the working folder: 491 PDFs where the manifest said 190; 158 of the "missing" ones were the same bytes already indexed under another collection.

The decoder (#1, #2, #6, #7, #10, #20)

• Marks stop decoding as letters (#7): a small glyph on fewer than five observations reads as nothing. 📏 held-out fixture recall 94.8% → 97.4%.
• Digits are labelled by the page number (#6): one offset per volume, fitted from every running head; least-significant digit first in reading order; two guards found by their failures. 📏 41 volumes, 15,499 anchors, purity 1.00; both fixtures' folios decode; the one eye-read row agrees with the fit exactly. The fit is written into `editions.toml` as page maps.
• TrueType bodies decode (#2): glyphs keyed by outline hash. 📏 16 of the 17 refused volumes at 98.8–99.4% coverage, 3,421 pages; the census corrected to three architectures mixed within volumes.
• The agreement floor (#20): a disputed entry (winner under 35% of ≥20 votes) reads as nothing. 📏 the negative control's zayn runs went from 7 pages to 0; the mini reports `status: ok` for the first time since 4 September.
• Coverage → recall measured with clustered gaps (#10): 📏 91.6% coverage is 65–80% term recall, most likely ~75%.
• The Glyphs sheet: the queue, the pictures, every decision signed; demoted entries shown as ∅ with their reason.

The webtop (#34–#41, #43, #45, #46)

• Brand, dark-only Reflections theme, logo, sans-serif type (the fonts had never loaded: the CSS was HTML-escaped inside `<style>`), the footer's two credits.
• A home dashboard; search moved to `/search`; the Library (one book per row, by collection, with coverage and verdicts); breadcrumbs back to the Library; a pager below the page; the menu lights the current screen; a styled file picker; a Help page.
• The catalogue (#39): Arabic title, transliteration, English, author, about — suggested by the estate's own language model, every entry marked unreviewed until a named human reviews it. 📏 541 entries suggested.
• The Notebook (#41, #42): Markdown session notes, git-backed, full-text search, "add to notes" from a hit with the locus; semantic search over notes through the estate encoder.
• HTTPS on the Mac mini's own reverse proxy with a certificate from the estate's own authority, renewed automatically (#43); the MCP served over the network at `/mcp` with five tools for a majlis (#48); a Mac install script (#40).

Measurements and rulings recorded

• 📏 `docs/DECODE-PROBE.md` addenda 8–10; `docs/bench/COVERAGE-RECALL.md`.
• Rulings of 17 September: our own published material is ✅ clean by decision (#9); the ezafe stands as typed (#8); noor is citable when the extract is clean and the name is human-reviewed (#49).

The first release: Phase 0 and Phase 1.

• The corpus manifest and content-hash identity; the Arabic normaliser and its trap suite; sidecars, the FTS5 index and `rtk find` with its controls; `rtk page render`.
• The Type 3 decoder: a glyph map keyed by shape, bootstrapped by OCR voting over 71 volumes, validated on two hand-typed pages. 📏 A volume the map had never seen decoded at 94.8% term recall; the shelf decoded at 91.6% weighted coverage — and coverage is not quality.
• Visual-order Arabic: detection and repair. The extractor-fitness gate. Editions, the honorific lacuna, IA bundles, the bab map. The first API, MCP and webtop on the Mac mini.