HTML edition, converted from Markdown 2026-06-19. Legend: ✓ done ☐ open ▮ partial ✗ dropped.
Last updated: 2026-07-13. Done items marked [x]. Phases 0–4 are complete; Phase 4.5 is in progress.
2026-06-15 progress pass (git log since 2026-06-13 13:30, through
HEAD 7bfab72): item 13 (outbound mail construction dedup) and
item 47 (random logouts — database-backed session restore) closed;
item 17 (page-resource folder UI) substantially landed (remainder
noted under the item). New production/startup work folded in as
items 69–71 and logged in Phase 2.5. No items removed.
1. ☐ Use the new WebSocket support in WebSite.php for chat /
live updates. When Yioop is being served by atto's WebSite,
prefer WebSockets over SSE for the messaging refresh path. When
Yioop is running under Apache/nginx (no WebSocket support
guaranteed), fall back to SSE.
2. ☐ Replace the polling loop on text chat (messagesStatus →
type=status, no active call) with push-based delivery.
Currently SSE is used only when a call is active; plain text
messaging still polls.
3. ☐ Investigate and fix bugs in the initial video-chat
connection establishment. Symptom: the call sometimes fails to
establish on the first try. Suspect signaling order between SSE
and the first ICE/SDP exchange in messagesStatus /
addCallEvent / nextCallEvent, or the call-group state machine
in GroupModel. Need a reproducer first.
4. ☐ Stand up TurnSite.php (from atto) as a TURN/STUN server
for Yioop video chats. Wire the ICE-server config in
Config.php (~line 1493) to be served by TurnSite when enabled.
5. ☐ Add Manage-Machines controls for TurnSite: on/off, status, logging.
6. ☐ Modernise the chat input row (UsermessagesElement.php
lines 366–406): wrap emoji + text + attach into a single rounded
"pill"; replace single-line <input type="text"> with
auto-growing multi-line <textarea> (Enter-to-send, Shift+Enter
for newline); swap mic/send based on whether input has text;
move file-input inside the pill.
7. ☐ Move tertiary chat-header actions behind a kebab menu.
8. ☐ Rethink the translation feature. Per-conversation translation dropdown is confusing UX. Either drop entirely or replace with a per-message context-menu affordance (iMessage / WhatsApp pattern).
9. ☐ Bubble + spacing polish. Sender-coloured "me" vs neutral "them" via the new CSS custom properties (item 24); time labels only between groups (already done — keep it); quoted-reply visual.
10. ✓ Add Manage-Machines controls for MailSite.php: on/off,
status, logging. Wired in MachineController::statuses (the
MailServer case dispatches CrawlDaemon::start /
CrawlDaemon::stop based on the start / stop action) and
in MachineController::log (the MailServer case reads
LOG_DIR/MailServer.log; the Mail case reads
LOG_DIR/<SmtpClient::MAIL_LOG_NAME> for the per-send mail
transcript). Status visibility is by membership in the
$statuses["MailServer"][-1] map.
11. ✓ Webmail interface inside Yioop — built. A capability
Yioop did not have before v10: the userMail activity reads and
writes MailSite mail through the MailBackend abstraction —
folders, message list, read/compose/reply, flags, move/delete —
so the webmail surface is built and in use. MailSite exposes the
webmail-friendly direct-call API it uses. Further enhancements are
filed individually as concrete items rather than tracked as an
open-ended "polish" task (example: swipe-left-to-delete a message
on mobile).
12. ✓ Fix bulk-mail headers so Gmail/Yahoo/etc. stop rejecting
BulkEmailJob output. Add Message-ID, MIME-Version,
Content-Type, Content-Transfer-Encoding, Reply-To, Return-Path,
and for bulk paths List-Unsubscribe + List-Unsubscribe-Post
(RFC 8058). Also DKIM signing (RFC 6376). SPF and DMARC are
DNS-side but document them in the install guide.
13. ✓ Refactor outbound mail to share message-construction code
with MailSite where useful, but keep the SMTP-client role.
Plan: extract RFC 5322 / MIME building into a shared helper;
when local MailSite is enabled and the recipient is local,
route via MailSite::deliverMail(); shrink the remaining
outbound code to remote-relay only. Status: partial. The
shared helper library/MimeMessage.php exists and is the
builder used by both SmtpClient::sendImmediate (the
legacy outbound path) and userMailDispatchMailsiteSend
(the new MailSite-outbound path from item 55). Local-
recipient routing via MailSite::deliverMail() is wired in
userMailDispatchMailsiteSend. Remaining work: trim
SmtpClient down to the pure remote-relay role now that
message construction lives in MimeMessage and local
delivery bypasses SMTP entirely; in particular,
SmtpClient::sendImmediate's convenience overload that
still builds a multipart body from to/subject/body/attachments
args can collapse to a thin call into MimeMessage::build
plus the wire-protocol exchange.
Closed 2026-06-14 (commit 0e85328, "mail construction dedup
and docs"): SmtpClient::sendImmediate now builds the wire
message via MimeMessage::build(...) and SmtpClient is trimmed
to the wire-protocol exchange; ~600 lines of duplicated multipart
construction removed across MimeMessage.php/SmtpClient.php, and
docblocks rewritten to plain English. Both outbound paths
(sendImmediate and userMailDispatchMailsiteSend) now share one
builder; local-recipient routing via MailSite::deliverMail() was
already wired (item 55). Item 12's header work (RFC 5322 / DKIM /
List-Unsubscribe) is separate and still open.
14. ✓ Rename library/MailServer.php →
library/SmtpClient.php. Landed early as part of
phase 0 (the rename was inexpensive and removed a long-
standing misname -- the old name suggested a server-side
listener, but the class is purely a client of an upstream
SMTP server). Item 13 (refactor outbound to share
construction code with MailSite) still pending for phase
2.
63. ✓ Configurable mail Banner (MAIL_HOST_NAME). New
Server-Settings field labelled "Banner:" that sets the host name
MailSite announces in the SMTP/IMAP greeting and uses as the
EHLO/HELO identity on outbound delivery. Single source of truth
MailSiteFactory::mailHostName() (= MAIL_HOST_NAME, else first
MAIL_DOMAINS, else php_uname('n'), else localhost);
MailServer::serverName and SmtpClient's outbound HELO both
delegate to it (HELO is now one consistent server identity instead
of the per-message sender domain). SystemComponent folds a PTR
row (resolved sending IP → banner host) into the Suggested DNS
Records panel, and the field placeholder shows the computed
default. Wiki Mail Services page updated with a Banner section +
PTR note. Advances the item-12/36 deliverability story (the
HELO/PTR/FCrDNS half); RFC-5322/DKIM/List-Unsubscribe header work
in item 12 is still open.
15. ☐ Diagnose and fix the "crawl/index speed degrades over
time" bug. Suspect Scheduler tier round-robin, QueueBundle /
Bloom-filter growth, IndexDocumentBundle partition rotation,
fetcher↔queue-server schedule round-trip cost as bundles get
bigger. Need instrumentation first. Sub-items:
a. Synthetic crawl test driver. Build a test mode that
simulates large crawls under realistic queue-server /
fetcher / name-server pressure without actually fetching
from the web. Synthetic URLs are minted from a seeded
deterministic generator; Fetcher swaps in a mock
getPages() that returns synthetic pages with controllable
size, fan-out, and content distributions; the rest of the
pipeline (parsing, scheduling, indexing, distribution to
queue servers, summary upload) runs unmodified. Lets us
drive a multi-million-page crawl on a single workstation
in minutes and observe where time and memory accumulate.
Wire as a config flag (SYNTHETIC_CRAWL_MODE) plus a
test executable in executables/ that orchestrates the
synthetic name-server + N synthetic fetchers.
b. Instrumentation. Add per-stage timing and memory
counters to the suspect paths (Scheduler tier walk,
QueueBundle add/peek, dictionary update, partition flush,
fetcher-to-queue-server upload). Emit to a structured log
(CSV or JSON-lines) the synthetic flow can post-process.
c. Diagnose and fix. Run (a) with (b), look at where
per-page wall time starts climbing, fix what's found.
73. ☐ Crash-safe index writes and quick recovery from a
corrupt index. This is the root-cause fix behind the Phase 4
out-of-memory investigation: a corrupt on-disk index made
getPostingsString read garbage-length postings, which on the
live server pushed memory over the limit. Phase 4 shipped the
mitigations that keep a corrupt index from taking the running
server down (a maximum readable posting length that skips
oversized reads, memory-scaled index caches, and per-cache and
max-posting-read instrumentation), but those are guards, not
prevention. The prevention is here. Sub-items:
a. Audit the index-bundle write path. Walk how
IndexDocumentBundle is added to during a crawl (partition
append, dictionary merge, postings write) and find every point
where an interrupted or partial write could leave a half-written
entry that a later read interprets as a garbage length or offset.
b. Make the writes crash-safe. Ensure that if the queue
server or dictionary process crashes, is interrupted with Ctrl-C,
or is killed partway through, the index is never left corrupt:
write-then-rename or a small journal/marker so a partially applied
step is either completed or rolled back on restart, giving a quick,
safe recovery to a known-good point rather than a corrupt bundle.
c. Reproduce and verify with the synthetic crawl driver
(item 15a): drive a crawl, kill the queue server or dictionary
process at chosen points in the write path, and confirm the index
comes back clean and readable every time.
16. ✓ Audit and rewrite the wiki-page parsing regexes. Current regexes are fragile and have potential security flaws (XSS via crafted markup, ReDoS). Build an adversarial test corpus and either harden the regexes or move to a proper tokenizer/parser. Overlaps with the security audit (item 27). Decided on the tokenizer and parser, written in both PHP (WikiParser) and JS (help.js), covering the mediawiki-style and markdown engines, with a new markdown help page. A golden regression corpus is in place; the PHP parser and the JavaScript help.js port are both done, the port covered by per-case JavaScript unit tests, and the old regex engine is gone. The adversarial probing found the nesting failures real but the XSS and ReDoS not reproducible with direct probes. Completed in Phase 4, closed 2026-07-10; see the Phase 4 arc plan.
17. ☐ Modernise the wiki page-resource folder UI
(WikiElement::renderResources line 969 / renderResource
line 1296; drag-drop in basic.js line 973):
Replace the fixed-in-width table with CSS Grid using
repeat(auto-fill, minmax(min, 1fr)) and a container query.
Add direct delete (🗑️) per row with confirm dialog. Keep clipboard for cross-page moves.
Replace text action buttons with icons. Keep tl() strings
as aria-label + title.
Inline rename: click name to edit, Enter / blur to commit, Esc to cancel.
Drag-drop between same-page subfolders (folder rows + up-level breadcrumb become drop targets).
Visible drop-zone overlay when files are dragged from the OS.
8f43299…7bfab72).
Done: per-row direct delete with confirm; text action buttons
replaced by a Mail-style toolbar (✓ confirm-rename / ✕ delete /
⋯ more-menu with Clip Copy + Clip Cut, plus ⚙ wiki-code and
Extract), tl() strings kept as label/title; inline rename
(click name, Enter/blur commit, Esc cancel); drag-drop between
same-page subfolders with the folder rows and a pinned Parent
Folder row (incl. move-to-root) as drop targets; OS-file drop
onto a folder with drop handling; clipboard retained for
cross-page moves; multifile and whole-folder drag-drop upload
(multifile $_FILES in WebSite, single clean version per
folder via the file's own headPutContents), with an upload
progress readout (transferred size + moving-average speed) and
H2 inbound upload-window sizing; the VersionManager
stranded-lock root-cause bug fixed (copyFileToGroupPageResource
passed $timestamp into the $lock slot, leaving a lock that
silently failed every later rename/version op — commit bff69c9);
the edit controls render identically under Apache and the
built-in server (the v10-introduced is_writable() UI gate was
the offender; resolved via work_directory permissions, the patch
not needed). Remaining: multi-select (Shift/Ctrl-click) +
keyboard shortcuts (Delete/F2/Ctrl+X/C/V/Esc); the fixed-in-
width table → CSS-Grid auto-fill/container-query rewrite (the
grid layout exists but is not the auto-fill rewrite). Keep open
for those two. Logged in Phase 2.5.18. ☐ Flesh out the {{category|...}} / {{category-list|...}}
mechanism. Named parameters; shipped style presets
(style=list|cards|grid|hero|ticker); pagination; auto-stub
category index pages; "Filed under" affordance via show
flag; cross-group lookup; replace custom <x-c-list> tags
with semantic HTML.
19. ☐ Add a wiki editor live preview pane. Split the editor:
textarea on the left, server-rendered preview on the right,
AJAX refresh on debounce. Renderer is the existing
WikiParser. ~50 lines of JS, no deps.
20. ☐ Add a contenteditable WYSIWYG-lite layer over the textarea (incremental, after item 19). Toolbar buttons insert wiki markup at cursor — MediaWiki classic-toolbar approach scaled down. Textarea remains source of truth. ~500–1500 lines, no deps.
21. ☐ Asynchronous email-based witness flow for secret
ballots. The existing {{secret-ballot|w1|w2|...}} mechanism
(SocialComponent::initializeBallot line 5176;
SigninModel::createWitnessKeyPair line 385,
createBallotFile line 253, countBallotFile line 289)
requires all witnesses to authenticate on the same form to
initialise the ballot and again to count it. Allow each
witness to participate via emailed signed links instead, while
preserving the current security and anonymity properties — the
server must never see witness passwords or be able to
reconstruct the private key without witness collaboration.
Approach (no new deps; uses sodium + hash_hmac that are
already in use):
On ballot init, server emails each witness a one-shot link
with a token = HMAC-SHA256(page_id|ballot_id|witness_id|
stage|expiry|nonce) keyed by AUTH_KEY. Click + sign-in +
witness password generates that witness's $random_i and
$hash_i per the existing createWitnessKeyPair formula.
To preserve the "server never sees the hashes" property:
each witness derives a per-ballot keypair from their
password (e.g. sodium_crypto_box_seed_keypair of
hash($password ⋅ $witness ⋅ $ballot_id)) and encrypts
their $hash_i to themselves. Server stores opaque blobs.
Once all n witness blobs are present, server can publish
the ballot's public key only after one final witness
confirmation step that releases an aggregated hash chain
(avoiding needing every witness on a single live form, but
still requiring all to have acted). Several variants of
this last step are possible; the simplest: the initiator
witness collects the blobs in a final email-link round and
computes the polynomial.
Count time uses the same email-link mechanism in reverse: each witness clicks a second emailed link, signs in, submits their password to decrypt their own blob; once all contribute, server reconstructs the private key as today.
Threat model unchanged from the current synchronous flow: requires all named witnesses to act; server never sees passwords or pre-aggregation hashes; voter anonymity identical (votes still encrypted with sealed-box to the ballot public key).
Tests must cover: token expiry, replay rejection, witness
who never responds, witness password mismatch on second
click, key-mismatch detection (the existing
KEY_MISMATCH path).
22. ☐ Make more MediaJob subclasses distributable. Survey:
Distributable today: BulkEmailJob, FeedsUpdateJob,
VideoConvertJob (full), AudioTranscriptionJob (partial).
Name-server-only today: PodcastDownloadJob,
WikiThumbDetailJob, DescriptionUpdateJob,
AnalyticsJob, CullOldRawImpressionsJob,
RecommendationJob, TrendingHighlightsJob.
23. ☐ Migrate if ($_SERVER["MOBILE"]) branches into CSS
media queries. ~30 PHP files currently branch on this for
layout. Use @media (max-width: …) and @container (Baseline
2023). File-by-file refactor.
24. ✗ Replace the UA string sniff with UA Client
Hints. Dropped 2026-06-19. Client Hints
(Sec-CH-UA-Mobile) are Chrome-only — Safari and Firefox do not
send them, so the UA-string sniff would have stayed as the
fallback anyway, leaving two mechanisms where there was one. And
items 23 and 25 move the mobile decisions into CSS and emit the
viewport unconditionally, removing most of the server-side need.
A one-line UA fallback stays. (The earlier narrow sub-task of
dropping fennec from the UA sniff was already done — commit
c590f5d.)
25. ☐ Emit the viewport meta tag unconditionally — moved
from Phase 0 to Phase 5. Was reverted because the surrounding
CSS isn't yet flex-friendly: with width=device-width, the
desktop layout extends past phone viewports and content gets
cut off. Lands together with items 23, 26, 27, 29, 30, 31 so
the layout actually adapts when the meta tag is honoured.
26. ☐ CSS custom properties for color, spacing, typography.
:root { --color-fg, --color-bg, --color-accent, --space-1 …
--space-6, --font-size-base }. Foundational — items 9, 17,
18 consume these.
27. ☐ Adopt CSS logical properties (margin-inline-start,
padding-block, inset-inline-end, …) to replace the
duplicated .html-ltr / .html-rtl selector pattern (106
duplications in search.css alone). Baseline since 2023.
Note: the mail message-action row introduced this pattern as
a first foothold during the Phase 2 work — .mail-message-
actions-trash-group uses border-inline-start /
padding-inline-start / margin-inline-start instead of
paired .rtl overrides. Use as the reference for the wider
sweep.
28. ☐ Built-in dark mode via color-scheme: light dark,
light-dark(), prefers-color-scheme.
29. ☐ Drop physical-page units (in, pt) from screen CSS
in favour of rem / em / % / ch / vw and clamp().
search.css currently has 276 in, 127 pt, 784 px, 4
rem/em.
30. ☐ Accessibility-aware media features:
prefers-reduced-motion, prefers-contrast,
@media (hover: hover), :focus-visible. Verify WCAG 2.2 AA
contrast in both themes; add axe-core or Lighthouse alongside
the existing WAVE check.
31. ☐ Run a full security audit before v10 release.
Crypto. crawlHash (Utility.php line 1616) is md5-
with-halves-XOR'd → 8 bytes (or ~12-char base64). It's wired
into binary index formats — doc IDs, term IDs, posting
lists, B+-tree dictionary keys — and 164 callsites across
the tree, so a blanket replacement would break every
existing index and quadruple storage. Add a sibling
crawlAuthHash($string) returning
hash_hmac('sha256', $string, AUTH_KEY) for security uses;
migrate just the security-relevant callsites to it
(CSRF token in Controller::generateCSRFToken line 649 +
check at line 671; the crawlHash("A") length-test at line
914 needs the new function's length; audit the rest).
Indexing callsites stay on crawlHash unchanged.
Constant-time comparisons. checkCSRFToken
(Controller.php line 672) uses ==. Switch to
hash_equals.
crawlCrypt fallbacks. Remove mcrypt_create_iv and
mt_rand branches once PHP floor lands at ≥ 8.0.
Session cookies. Verify HttpOnly, Secure,
SameSite; add CSP and X-Content-Type-Options: nosniff.
I/O paths. Spot-check $_REQUEST for
Controller::clean(); view echoes for escaping; file
uploads for traversal / MIME / executable.
SQL. Verify all models use prepared statements.
CSRF coverage. Every state-changing action calls
checkCSRFToken.
Dependency surface. composer.lock and bundled JS/CSS
for known CVEs.
32. ✓ Retire the image captcha; gate signup with proof-of-work plus cheap invisible checks and an email confirmation. Image captchas are a declining defence — cheap machine solvers and human captcha-farms beat them — and a burden for blind users, so the IMAGE_CAPTCHA mode and CaptchaModel's image drawing are removed. In its place, all self-contained (no outside service), layered:
HASH_CAPTCHA mode: the browser runs hash_captcha.js / sha1.js to find a nonce that meets a difficulty level, so a real signup waits a fraction of a second while a bot pays that cost on every one of thousands of attempts. Tune the difficulty (HASH_CAPTCHA_LEVEL), make it the single captcha, and simplify the admin Captcha-Type control.tabindex="-1"), aria-hidden="true", and autocomplete="off", so a blind user never reaches it or is wrongly flagged for filling it.Touch points (verified 2026-06-19): the auth / suggest mode (global C\CAPTCHA_MODE) lives in RegisterController (the switch (C\CAPTCHA_MODE) branches in create / recover / resend / suggest), SystemComponent (the CAPTCHA_MODES options), Config.php (IMAGE_CAPTCHA constant), SecurityElement (Captcha-Type control), and the auth forms (RegisterView, RecoverView, ResendEmailView, SuggestView). Separately, the image captcha has a second life as the embeddable wiki [{image-captcha}] placeholder — setupGraphicalCaptchaViewData in the base Controller.php, the user_captcha_text / captcha_text verification in SocialComponent, plus StaticController, WikiElement, StaticView — which is not gated by the mode and needs its own decision (drop / replace with proof-of-work / keep image only there). CaptchaModel is removed only if that placeholder is dropped or replaced. Repointed 2026-06-19: was “modernise the image captcha”; Chris approved retiring the image method for proof-of-work + invisible layers + email gate, all self-contained.
33. ✓ Replace the recovery-question system. (a) Default new
installs to EMAIL_RECOVERY (done). (b) Add email-based
one-time login codes: a short, time-limited, single-use code (or
signed link) mailed to the member's registered address, so they
can sign in or recover without a saved password and without any
external authenticator app. Yioop already sends mail, so this
adds no outside dependency — it is the same kind of email
one-time login Chris uses with other services. (c) For an
offline / airgapped install that keeps a question mode, let users
type their own question and answer; drop the
register_view_recoveryN_* translation infrastructure and remove
the existing EMAIL_AND_QUESTIONS_RECOVERY mode.
(Dropped 2026-06-19: TOTP and printed single-use recovery codes.
TOTP forces members to install a separate authenticator app,
which cuts against Yioop staying self-contained; the printed
codes were its sidekick. The email one-time code in (b) is the
self-contained replacement.)
47. ✓ Stop random logouts at the environment level — deeper
session-lifetime fix. The phase-0 ini_set hot-fix in
src/index.php raises session.gc_maxlifetime /
session.cookie_lifetime per request, but does not reach
Debian / Ubuntu's /etc/cron.d/php* job. That cron sweeps
session.save_path (default /var/lib/php/sessionsXY or
/tmp) every 30 minutes using the value of gc_maxlifetime
from /etc/php/X.Y/cli/php.ini, which is whatever the
distro shipped — independent of what PHP-FPM / Apache sets
per process. On at least one Yioop deployment this still
causes randomly-timed logouts; the symptom is "I came back
after lunch and was logged out" even though AUTOLOGOUT
is much longer than the lunch.
Pick one of the following, in increasing order of robustness:
(a) Move session storage out of /tmp / /var/lib/php.
Set session.save_path to WORK_DIRECTORY/data/sessions
(chmod 700) before session_start(). The Debian cron job
only sweeps its configured save_path; a Yioop-owned
directory is invisible to it. This is the smallest change
and probably fixes the observed deployment immediately.
Document the directory in the INSTALL guide. ~10 lines in
src/index.php + a mkdir -p in the installer.
(b) Custom file-based session handler
(SessionHandlerInterface) that stores in
WORK_DIRECTORY/data/sessions and respects AUTOLOGOUT for
its own GC. Strictly stronger than (a) — Yioop owns the entire
lifecycle and no external process can delete files
prematurely — but adds ~60 lines and a unit test.
(c) Database-backed session handler storing into the
existing Yioop SQL store (new USER_SESSIONS table:
SESSION_ID PK, USER_ID, DATA TEXT, LAST_ACCESS INT,
EXPIRES INT). Necessary for multi-machine / load-balanced
deployments where local-disk sessions don't follow the user
across servers. Adds row-level invalidation on logout, an
admin "log out everyone" button, and a clean place to track
sessions per user for re-authentication prompts and suspicious-activity
alerts.
Recommended sequencing: do (a) first (small change, likely fixes the reported deployment), then (c) as part of the broader auth work in items 32-35 (item 33's email-one-time-login work will want a per-session store anyway). Skip (b) unless (a) doesn't resolve it and (c) is too big for the v10 window.
Closed 2026-06-14 via approach (c), the robust
database-backed path (commits 9b4c27d, 492ae95, ac1b7a1,
ff39001→e1f41a1, fcc1dd5, 4d0f71d, d79e387). New
SESSION_USER table (SESSION_ID PK / USER_ID / EXPIRES),
SESSION_ID_LEN=64, DATABASE_VERSION 104→105 with
upgradeDatabaseVersion105. The long-running atto WebSite
keeps live sessions in an in-memory LRU heap (SplHeap,
rebuilt on restart) and rehydrates a dropped/expired-from-memory
session from the SESSION_USER row, so a person stays signed in
across a server restart or memory eviction — the actual cause of
the "logged out after lunch" reports under the long-lived server.
Rehydrate is wired lazily in index.php / Controller /
AdminController, persistence via new UserModel session methods
+ ProfileModel; sequence-restore is tolerant of a missing/stale
row. This also gives the per-user session store items 32–35 want
(re-authentication prompts, "log out everyone"). The lighter (a)
session.save_path move is unnecessary now that Yioop owns the
session lifecycle through the DB store, but is still worth a line
in the INSTALL guide for Apache/php-fpm deployments where the
distro cron sweeps /var/lib/php.
34. ☐ Update the online wiki documentation (last full update
was for v9.3, Dec 2022). Source PDFs to refresh:
yioop_docs.pdf, Ranking_Mechanisms.pdf,
Yioop_Install_Guides.pdf, Yioop_Wiki_Syntax.pdf,
Coding.pdf.
35. ☐ Document the new auth flows (email one-time login codes; the recovery modes and user-typed recovery question; and the hardened account-registration / activation-mail flow — opaque self-expiring activation token with no address in the link, generic anti-enumeration replies on the register and resend forms, per-IP and per-email resend throttling, a Reply-To from MAIL_REPLY_TO, and the multipart text + HTML activation message). Wiki source is delivered as a .txt for the wiki UI, never edited into the autogenerated help pages directly. Moved out of the Phase 3 arc 2026-06-25 into this documentation push; the behaviour has shipped, only the write-up remains.
36. ☐ Document mail deliverability requirements (DKIM,
SPF, DMARC, and now the Banner/MAIL_HOST_NAME host identity
+ PTR / forward-confirmed reverse DNS — see item 63). The Mail
Services wiki page already gained a Banner section and a PTR note
in Suggested DNS Records this session; the PDF deliverability docs
still need the same. Moved 2026-06-19 into the final v10
documentation push (Phase 10); it documents standards already
shipped.
37. ☐ Document the wiki resource-folder UI, the new category-list parameters and styles, the live preview / WYSIWYG modes, and the asynchronous secret-ballot witness flow.
38. ✓ Determined the exact current Yioop runtime requirements
for v10 release. PHP minimum pinned at 8.1. Reconciled
composer.json (now >=8.1.0 with ext-curl, ext-intl,
ext-fileinfo, ext-libxml required; pdo_mysql / pdo_pgsql
moved to suggest), INSTALL.txt (copyright bumped, dedicated
Requirements section added), SystemComponent.php runtime
check (PHP_VERSION_ID < 80100), and 25 locale
system_component_php_version strings. 6 pre-existing locale
translation bugs (de, el_GR, he, ko, pt, th) discovered and
fixed in passing. Commit a1cd8f5. (Update 2026-06-10:
ext-openssl re-added to composer.json and INSTALL.txt —
the mail/TLS work, STARTTLS, DKIM signing, and ACME all depend
on it again, so it is a hard runtime requirement once more.)
39. ✓ Updated the Coding-guidelines wiki page for v10. Surgical scope per the agreed plan; details in Phase 1 block below.
48. ✓ Mail import CLI tool: migrate existing user mailboxes
into MailSite storage. Use case: an existing Postfix +
maildrop + dovecot stack delivers mail to per-user
filesystem trees (Maildir under
/var/mail/<user>/Maildir/{cur,new,tmp} or mbox under
/var/mail/<user>). When a deployment switches to MailSite
for IMAP service, existing mail needs to migrate without
loss. Build src/executables/MailImport.php with sources
Maildir (priority), mbox, and optional live-IMAP. Common
pipeline: parse RFC 5322 bytes + flags + internal date, call
MailStorage::appendMessage($user, $folder, $bytes, $flags,
$internal_date). Add --dry-run and --resume (Message-ID
dedup) flags. Default destination: FileMailStorage rooted
at WORK_DIRECTORY/mail. Unit-test against a sample Maildir
tree under tests/test_files/ and an mbox sample. No longer
blocking step 5; can be sequenced
independently now that the webmail surface works against
delivered mail.
*Status (updated 2026-06-07): substantially advanced; used in
production to migrate the pollett.org cpollett mailbox.
The tool is src/executables/MailTool.php (subcommand
dispatch: import/verify/reuid/cleanup, alongside the
earlier migrate). Maildir source landed earlier (see the
historical note below). This session's import work:
Streaming, memory-flat import. An earlier version built
an array of all message descriptors up front, which OOM'd on
the real ~49k-message run. Replaced with a streaming
eachMessage generator + lightweight countMessages; the
appendMessage loop is itself memory-flat (verified a 64M
limit holds across thousands of appends). The earlier partial
crash could not auto-resume (it had written no journal), so
the chosen recovery was wipe-and-reimport; the streaming
path completed it cleanly.
Resumable + idempotent. Per-folder
import-<folder>.journal records done source basenames,
written before the mutation and kept after success, so a
re-run skips already-imported messages rather than
duplicating them, and a crash mid-run resumes where it left
off. reportImport distinguishes copied vs skipped.
Per-folder subscribe + flags-snapshot priming. Import now
subscribes each created folder (storage-layer createFolder
does not subscribe the way IMAP CREATE does, and LSUB filters
to subscribed folders, so Apple Mail would not show
unsubscribed folders) and primes the flags snapshot via a new
public refreshFlagsSnapshot. Verified a from-empty rebuild:
an emptied user is fully reconstructed by import per folder.
Per-folder UIDs from the start. Fresh imports allocate
per-folder UIDs (clients render UIDNEXT-1 as an estimated
message count, so a shared global UID space made Apple show
~143,931 messages); no separate reuid pass is needed after
a fresh import.
Progress + timestamps. A "Scanning..." line plus
"copied N/total" every 1000, and an outAt() helper that
prefixes [HH:MM:SS] on progress/milestone lines across
import/reuid/cleanup/verify/migrate.
New reuid subcommand. Renumbers a folder's UIDs into a
fresh per-folder space with a new UIDVALIDITY; crash-safe and
resumable via a reuid.journal. For repairing mailboxes that
predate per-folder UID allocation.
New cleanup subcommand. Removes orphaned
<digits>.flags/<digits>.date companion files left by the
pre-consolidation storage format (one .eml per message now;
flags live in the index, with flags.snapshot as the
compaction-time recovery fallback). Streams readdir, leaves
.eml/metadata/subdirs, idempotent.
Still open: mbox source (next slice) and optional live-IMAP
source (deferred). Possible nicety: a one-command "import whole
dovecot tree" wrapper, and extending cleanup to sweep stale
import journals.*
Status (updated 2026-06-10): closed. mbox source landed
on the import side (a file source is read as an mbox, split on
its "From " postmark lines, >From-unquoted, with flags from
Status/X-Status and the internal date from the Date header or
postmark), and the tool was trimmed to its durable command set.
The pre-production-only subcommands migrate, reuid, and
cleanup were removed — migrate converted a legacy
percent-encoded folder layout, reuid repaired the old global
UID space, and cleanup swept the orphaned .flags/.date
companion files; all three addressed one-time pre-production
states that no current or future install reaches, so they were
deleted along with their moveDirFiles/cleanupFolder helpers
rather than carried as dead maintenance surface. A new
export <user> <folder> <out_file> subcommand writes a
user's local mail folder to an mbox: each message gets a "From "
postmark dated from its IMAP internal date, Status/X-Status
headers carrying its stored flags (which live in the folder
index, not the message bytes, so they must be written back for a
later import to recover them), and ">From " quoting of body
lines beginning "From "; the folder is streamed a message at a
time so a large folder stays memory-flat. The postmark is
written in the configured local timezone so it round-trips
through the import side's strtotime parse (a UTC-written
postmark was read back as local, shifting dates). Round-trip
verified end to end: export then re-import preserves message
count, flags, internal dates, and >From body lines exactly.
The command set is now import/export/verify. Delivered as
yioop-mailtool-export-remove-commands.patch.
Earlier status (Maildir slice): New
src/library/MaildirSource.php parser, MailImport.php CLI
skeleton with subcommand dispatch (maildir <path> <user>
[--dry-run] [--resume]), and tests/MaildirSourceTest.php
covering parseFolderName / parseFlags / extractMessageId /
messageMetaFromPath / skipFile / isValid / folders /
messageFiles against a deterministic sample tree under
tests/test_files/ (frozen unix-timestamp
mtimes set in setUp() since git does not preserve mtime through
a patch). Subfolder hierarchy via Maildir++ dot convention
(".Sent" -> "Sent", ".Archive.2025" -> "Archive/2025"). Flag
mapping per Maildir++ section 4 (R/S/T/D/F ->
\Answered/\Seen/\Deleted/\Draft/\Flagged; P/Passed dropped;
"new/" messages get \Recent added). Dovecot metadata files
skipped. tmp/ not enumerated. --resume dedupes by Message-ID
against the destination folder.
49. ✓ Mail Services fieldset in Server Settings (commit
76bdec2). 8 new MAIL_* Profile.php constants, fieldset
with dropdown + sub-config visible only when MailSite is
selected. No listener spawned; intent recorded only.
50. ✓ Mail button in SocialControlsElement + userMail
activity stub (commit e34e230). Button gated on
C\MAIL_MODE !== 'disabled'. Routes through routeUserMail
to SocialComponent::userMail(), registered as a social
activity in both AdminController and GroupController.
51. ✓ src/library/ImapClient.php: minimal RFC 3501 client
for use by the Mail activity (commit 510d4b4). Ported
the ~200-line ImapClient class from atto's Anonymous WebMail
demo, brought up to Yioop conventions.
52. ✓ Mail activity: external IMAP accounts UI (commit
f53ba8b). Add/Edit/Delete account forms, account list,
inbox listing, single-message view; libsodium credential
encryption with master key in private DB; CRUD model with
user-scoping. 13 test cases / 36 assertions, all pass.
53. ✓ Mail activity: MailSite-stored mail. Decision made in favor of direct in-process MailStorage calls (no socket-loopback layer) wrapped by a backend abstraction so IMAP and MailSite share the same handler surface. Implementation arc (12 patches over ~3 sessions):
a. Backend abstraction. New
src/library/MailBackend.php — abstract base with 21
methods: connection lifecycle (close), account identity
(accountId, displayName, senderEmail, isSyntheticAccount),
folder ops (listFolders, annotateUnreadCounts,
createFolder, renameFolder, deleteFolder, purgeFolder),
messages (listMessages, fetchMessage, setFlag, moveMessage,
deleteMessage, appendMessage), and bulk variants
(setFlagBulk, moveMessagesBulk, deleteMessagesBulk). Static
factory MailBackend::forAccountId($account_id, $data,
$component) routes by MAILSITE_ACCOUNT_ID (=0,
synthetic) vs IMAP account row. Errors throw
MailBackendException with localized messages.
b. Concrete backends.
src/library/ImapMailBackend.php wraps the existing
ImapClient with lazy connect, cached selected-folder
state, and translateException() that maps ImapClient
ERROR_* tokens to localized social_component_mail_err_*
strings. src/library/MailSiteMailBackend.php wraps the
MailSiteFactory::build() direct-API path: in-process,
no sockets. Both share the abstract interface contract.
c. Handler migrations. Each user-visible Mail handler
in SocialComponent migrated to the backend dispatch
pattern: MailBackend::forAccountId(…) + $backend->method(…)
+ $backend->close(). Migrated:
userMailListFolders, userMailListMessages,
userMailViewMessage, userMailFolderAction,
userMailToggleFlag, userMailBulkAction,
userMailArchiveToSent, userMailDownloadAttachment,
userMailDownloadAllAttachments,
userMailBuildReplyPrefill, userMailLoadMessages.
Each migration also dropped the corresponding inline-IMAP
worker (userMailFindSentFolder, userMailFindTrashFolder,
userMailListAll, userMailBulkApply, userMailFolderApply,
userMailFetchBody, userMailImapErrorMessage).
d. Send-archive bug fix. ImapMailBackend::appendMessage
had two latent bugs (wrong arg order on the
ImapClient::appendMessage call, wrong response-shape check)
that didn't manifest until the first migrated handler
actually exercised it; fixed when the migration of
userMailArchiveToSent surfaced them.
e. Connection sharing. Discovered that
imapListMessagesAdvanced (the IMAP-specific sort/filter
branch of userMailListMessages) opened a second
connection per request, tripping Gmail's per-account
concurrent-session limit. Added public-with-@internal
imapClient() and replaceImapClient($new_client)
accessors on ImapMailBackend so the advanced flow shares
the backend's already-open client. Same pattern then applied
to userMailLoadMessages's IMAP branch.
f. Attachment download Apache-vs-WebSite fix.
userMailStreamAttachment and userMailStreamAttachmentsZip
used bare PHP header() and exit(). Under Apache this
works; under WebSite mode the response stream has already
started by the time the handler runs, so header() warns
and exit() kills the long-running server process.
Switched to $parent->web_site->header(…) and
\seekquarry\atto\webExit(). Same fix-pattern applied
proactively to all three userMessages JSON
echo + exit() handlers (load-more-messages,
invalid-thread bailout, success-emit).
g. UI affordances added during the migration.
Per-folder unread counts in the sidebar (visible for
both MailSite and IMAP); Junk folder in MailSite's default
folder set (special-use \Junk); user-created MailSite
folders surface in the sidebar (merge against
defaultFolders() in MailSiteMailBackend::listFolders);
Move action on the message view (dropdown self-submits
onchange, ordered as
[↩] [↩«] [➡] [Move...▾] | [🗑], RTL-correct via
border-inline-start); sidebar account-list scroll
position persisted via sessionStorage so the active
account stays in view after action redirects.
h. Test coverage. Three new test files in
tests/: MailHeaderParserTest.php (17 cases covering
parse/decodeMimeWord/splitAddresses — pure-function
parser); MailSiteMailBackendTest.php (13 cases against
a RamMailStorage-backed MailSite installed via the new
MailSiteFactory::setTestOverride() test seam; covers
synthetic account shape, folder catalog and merge,
append/fetch round-trip, flag toggling, move/delete soft-
vs-hard, bulk variants). ImapMailBackendTest.php is
deferred (needs an ImapClient socket-pair pattern like
ImapClientTest; the IMAP backend wraps ImapClient
which is already covered, so the marginal value vs the
work to script wire-level responses is moderate). New
tests/ManyEmailExperiment.php injects N synthetic
messages (default 500) into FileMailStorage for a target
user so the webmail UI can be exercised under realistic
counts (scrolling, pagination, sort, filter, bulk select);
knobs for date span, attachment ratio, seen/flagged ratios.
i. Post-migration cleanup (deferred). A patch 12
pass would revert the public-@internal visibility leaks
introduced by patches 5, 6, 18 (userMailLog,
userMailProbeFolderUnread, imapClient,
replaceImapClient) back to protected once the advanced
inbox + load-more flows fold into the backend. Estimated
~-100 lines. Not blocking webmail use. Separately, the
MAIL_CLONE_JOB.LAST_ERROR column drop that this arc made
possible (per-message errors moved to MAIL_CLONE_ERROR) is
fully shipped and verified 2026-06-07 — see the Step 5 note
in the phased ordering.
j. Remaining migration patch.
mail-backend-search — search UI + handler migration —
is the last handler-migration patch. Currently
userMailSearch (when wired up) still uses inline IMAP
SEARCH; will move through a new $backend->search($folder,
$query) abstract once the search UI is in place.
Estimated ~150 lines.
54. ✓ Cross-account move. Move a message from one mail account to another (e.g. IMAP → MailSite, or Gmail → Pollett-IMAP). Fetch-from-A + append-to-B + delete-from-A, with rollback on any step's failure. Done: the move UI exposes "Move to another account" alongside the existing within-account move dropdown, and an external IMAP account can be cloned wholesale into local MailSite storage.
55. ✓ MailSite outbound sending. Landed during phase 2
(commits 3d9967c, 7c36dfb, f6a2826 on May 24-25)
ahead of the post-v10 schedule.
userMailDispatchMailsiteSend in SocialComponent builds
the wire message once via MimeMessage::build, then
classifies recipients by domain: local recipients (whose
domain is in MAIL_DOMAINS) are delivered directly through
MailSite::deliverMail (in-process, no SMTP loopback);
external recipients are grouped by lowercase domain, each
domain's MX records are resolved, and per-MX-host the
outbound SmtpClient tries port 25 first with a port 587
fallback for networks that block 25. The session uses
opportunistic STARTTLS (upgrades when the peer advertises
it, plaintext otherwise) and no AUTH (peer-MTA mode rather
than smarthost submission). After delivery the same bytes
are archived to the sender's Sent folder via the MailSite
backend's appendMessage. Implementation chose option (a)
of the original two-options sketch (local MTA relay) but
used peer-MTA outbound rather than handing off to an
upstream relay; this avoids needing a configured smarthost
on each install but means the deployment server must have
outbound port 25 (or 587) reachable from its network. See
item 60 (SmtpTunnel) for the workaround when a deployment's
network egress blocks both.
56. ☐ MailSite spam classification. The Junk folder exists
(added during the backend-arc work) but no classifier writes
to it. Needs an SCL hook in MailSite::deliverMail, Mark/
Unmark Junk buttons in the message view, and a classifier
(Bayesian against the user's existing Junk + non-Junk corpus,
or DNSBL lookups for the upstream IP). Sequencing: ship
Mark/Unmark Junk first (manual user training data), then
Bayesian on top of the accumulated training set.
57. ☐ MailSite unread-count session caching. Per-folder
unread probe currently re-runs every listFolders call (so
the badges are always live). For users with many folders +
large message counts the probe walks each folder's
metadata; a short-TTL session cache (mirroring the IMAP
cache pattern) would amortise the cost.
58. ☐ MailSite sort / filter / unread-only / flagged-only.
The IMAP imapListMessagesAdvanced flow supports
server-side SORT, SEARCH-based filters, and the unread-
only / flagged-only toggles. MailSite currently runs
listMessages newest-first only, no filters. The "honest
branch" in userMailListMessages exists because of this
capability gap. Implement equivalents in
MailSiteMailBackend::listMessages (the storage already
has the indices; just needs to expose them).
59. ✓ Test Mail Mode + Test Mode Fallback Port. Two
related Server Settings toggles, both surfaced inside the
Mail Services fieldset. C\MAIL_TEST_MODE (boolean,
default false) controls two behaviors at once: (1) the
compose-form address validation in
SocialComponent::userMailIsValidAddress relaxes from
PHP's strict filter_var FILTER_VALIDATE_EMAIL to a looser
"non-empty-local@non-empty-domain" check, so test@127.0.0.1
and user@localhost are accepted in the To/Cc/Bcc fields; (2)
MailSiteFactory::build skips the onMailFrom/onRcptTo
FQDN-reject hook so MailSite's SMTP listener stops refusing
non-FQDN senders/recipients at MAIL FROM and RCPT TO time.
C\MAIL_TEST_MODE_FALLBACK_PORT (integer, default 0) names
a fallback outbound port the MailSite-to-peer-MTA path tries
after a TCP-level failure on port 25; consulted only when
test mode is on and the port is nonzero. The port-input
field in Server Settings is shown/hidden via the same
setDisplay pattern the MAIL_MODE-dependent blocks use:
mail-test-mode.onchange flips mail-test-mode-port-row.
Replaces an earlier unconditional 25→587 fallback (the
speculative path Postfix didn't actually accept
unauthenticated submission on). Companion: the standalone
smtp_tunnel.php (item 60) — set the fallback port to the
tunnel's listen port to reach an upstream Postfix from a
network that blocks 25. The null reverse-path <> is
always accepted on MAIL FROM regardless of this setting.
Persisted through ProfileModel::$profile_fields and the
SystemComponent::updateProfileFields boolean-fields list.
60. ✓ smtp_tunnel.php: standalone byte-level TCP forwarder
for port-25 egress workarounds. Single-file PHP CLI
script, no Yioop dependencies and no namespace; lives
outside the Yioop tree so it can be deployed on the
upstream-SMTP host independently of any Yioop release.
Single-process stream_select-driven TCP forwarder.
Accepts plain TCP connections on a configurable listen
port and bridges each one to a single fixed upstream
host:port, copying bytes bidirectionally with no
protocol awareness. Run on the SMTP-bearing host
(typically pollett.org) listening on port 5252 (literally
"2525 backwards"; well clear of the IANA well-known range
so common egress filters do not block it) and forwarding
to 127.0.0.1:25. A Yioop install running on a network
that blocks outbound port 25 then points its outbound
SmtpClient at pollett.org:5252 and reaches the
upstream Postfix transparently. CLI: php smtp_tunnel.php
[listen_port] [forward_host] [forward_port] [allow_csv]
where the optional fourth argument is a comma-separated
list of source IPs permitted to connect (empty means no
restriction). Operationally the tunnel is the
deployment-side complement to MailSite's outbound MTA
(item 55): MailSite resolves MX and tries 25→587 from
the client side; smtp_tunnel.php makes the listening
side reachable when the client side is firewalled.
Caveat: because the upstream Postfix sees the tunnel
host as the peer, allowing this through requires either
binding the upstream listener to loopback (the safe
shape) or restricting the tunnel's allow-list to the
operator's known cruise-ship egress IP -- otherwise the
tunnel becomes a small open relay onto port 25.
61. ✓ PHP 8.5 dynamic-property declaration on SmtpClient.
Declared public $last_response_body on SmtpClient;
it was being assigned in the constructor and in
readResponseGetCode without a declared property,
surfacing as "Creation of dynamic property
seekquarry\yioop\library\SmtpClient::$last_response_body
is deprecated" on every mail send under PHP 8.5. A
follow-up sweep should grep all classes for assigns to
undeclared $this->* so we catch the rest before they
hit users on 8.5 (the 8.2 deprecation is now noisy enough
that one bare mail-send test surfaces it).
62. ✓ Make the WebSite TCP connection lifecycle fully
non-blocking. Origin: observed bursty hangs on pollett.org,
where WebSite serves TLS directly on 443 (no Apache in
front). Root cause: the single-threaded accept/serve loop
blocked whenever one client stalled during a synchronous step
(TLS handshake, protocol detection, or a frame read), freezing
every other connection. Surfaced and fixed in the 2026-06-07
session. Verified in production with two standalone client
probes (diag_accept_blocking.php, diag_h2_frame_stall.php)
that stall N connections at a chosen lifecycle stage and time
a legitimate concurrent request. Sub-stages, each its own
verified patch on master:
a. Stage 1 — non-blocking accept + deferred TLS handshake.
Ported MailSite's deferred-handshake pattern: accept sets
the socket non-blocking immediately and returns secure
connections in a handshake protocol state with a
deadline; stepHandshake peeks for the 0x16 TLS record,
steps stream_socket_enable_crypto across select wakes
(done / failed / needs-more-IO), and promotes on
completion. HANDSHAKE_TIMEOUT on ConnectionAcceptor;
cullDeadStreams reaps pending handshakes by deadline (they
have no modified-time, so the generic idle timeout would
wrongly reap them instantly). Production: phase 3 went from
FAILED + an SSL: Broken pipe storm to ~0.9x baseline.
b. Stage 2 — deferred post-TLS protocol detection. The
post-handshake HTTP/2-magic read was itself blocking (a
client that finished TLS but sent no bytes blocked).
classifySecureFirstBytes accumulates the first decrypted
bytes in a per-connection detect buffer and decides h1/h2
non-blocking. Production ~1.0x.
c. Stage 3a — bound the select wait by handshake deadline.
With no timer alarms the main stream_select used a null
(infinite) timeout, so a silent stalled handshake was only
reaped when unrelated traffic woke the loop. Added
nextHandshakeDeadline() and clamp the wait to the soonest
pending deadline. Production ~0.9x.
d. Stage 3b — non-blocking WebSocket frame reads.
parseWsRequest had set the socket blocking and looped on
blocking reads, so a slow WS client dribbling a partial
frame blocked the loop. Replaced with a per-connection
input buffer plus a pure WebSocketFrame::parseBuffer
(returns frame + bytes consumed, or null for need-more);
per-frame handling moved to dispatchWsFrame. Public
WebSocket route API (onMessage / onClose / onError /
send / sendBinary / close, and WebSocketFrame::build)
unchanged, so the atto WebSocket example still works.
Verified with a real socketpair: a frame split across 3
reads buffers without blocking and dispatches only when
complete; two frames in one read both dispatch.
e. Stage 3c — non-blocking HTTP/2 frame reads. Evidence
first: diag_h2_frame_stall.php established real
H2-over-TLS connections (full preface + SETTINGS) then
stalled mid-frame; pre-fix this measured phase 3 at 146x
baseline (838 ms) with only 2 stall connections openable
before the loop choked. Fixed by a low-risk split: the
~190-line frame-processing body became processH2Frame
(byte-for-byte unchanged), and a new parseH2Request reads
available bytes non-blocking into a per-connection input
buffer, extracts each complete frame (9-byte header +
declared payload), and calls processH2Frame per frame,
draining multiple buffered frames in one pass. Verified a
real HPACK-encoded HEADERS request dispatches both whole
and when fragmented across reads. Production: phase 3
dropped from 146x to ~0.4x baseline, with all 20 stall
connections opening cleanly.
f. Stage 4 — H3Listener review (no code). Code-reviewed
the QUIC/HTTP-3 path: it is non-blocking by design — a
single UDP socket multiplexes all connections by connection
ID, accept drains datagrams with non-blocking
stream_socket_recvfrom, per-datagram processing is
bounded, and reapStaleConnections does RFC 9000
idle-timeout reaping, so one stalled H3 client cannot block
others. The only blocking read in the file
(Tls13Engine::fillExact / readRecord) is explicitly off
the QUIC path (a standalone TLS-over-TCP helper per its own
comment). No change needed.
*Open follow-up (optional, not blocking): `initH2Request`'s
read of the client's initial SETTINGS frame at H2
connection-establishment is still blocking. It is a distinct,
narrow window (only during H2 setup) and is not what the
diagnostic measured; converting it would mean deferring
establishment across select wakes like the handshake detect
phase. The two diagnostics are worth keeping as regression
checks. Note observed during testing: phase-3 latency spikes
can also come from HDD-under-load contention on response
serving (filesystem latency, independent of loop blocking) —
re-run a few times and compare typical figures, not a single
worst case.*
Production-driven work outside the originally planned phases, recorded as numbered items here and grouped under "Phase 2.5 — Ongoing production reliability" in the phased ordering below.
64. ✓ ACME certificate auto-renewal (2026-06-08). AcmeRenewJob
MediaJob, ARI-gated via AcmeManager::renewalWindow with a
ONE_MONTH expiry fallback, plus in-place
WebSite::reloadSecureCertificate driven by a once-a-minute
cert-watch timer in index.php, so the secure listeners pick up a
renewed certificate without a restart.
65. ✓ Per-domain Appearance (2026-06-09/10). Each custom-route
domain can override the full Appearance set (colors, logos,
favicon, background image, theme, landing page, searchbar), with a
global fallback when unset. New DOMAIN_APPEARANCE table
(migration 104), overlaid at render time, edited through a domain
picker in the Appearance activity; per-domain images live in
resources/<domain>/ served via a new pretty-URL route and applied
across all 13 logo/favicon surfaces through a new LogoHelper.
66. ✓ Restart Server made reliable (2026-06-13). Restart was
flaky — daemons sometimes killed-not-relaunched, the web server
sometimes killed-not-respawned. Root cause: the replacement secure
listener's bind-retry window (10 × 250 ms) was shorter than the
exiting TLS server's privileged-port teardown, so the replacement
exhausted its retries and exited, leaving no server;
BIND_ATTEMPTS widened 10→40 in WebSite. Supporting fixes:
interruptibleSleep($duration, $step, $wake_callback) in
Utility.php, with idle daemon waits in MediaUpdater / Mirror /
QueueServer / Fetcher converted to wake promptly on a stop
request instead of sleeping out a fixed interval.
CrawlDaemon lock files now store "timestamp pid";
isProcessAlive (POSIX posix_kill($pid, 0)) lets start()
clear a stale lock and proceed while never killing a live
daemon — removes the manual rm-the-lock footgun. statuses()
skips stale locks.
A non-blocking boot-drain state machine in index.php: a
1-second repeating timer advances one daemon per tick through
drain/relaunch, replacing a blocking sleep loop that had frozen
the freshly-restarted event loop for seconds per daemon.
A poll-based restart-status page (h2/h3-safe, replacing a
streaming page) with phase wording, including the &-in-URL
and CSRF-freshness fixes that had made the poll receive HTML
instead of JSON.
The configure-extensions check gained OpenSSL + Sodium (required) and POSIX (optional — enables graceful daemon shutdown and stale-lock clearing).
Side fix: a Media-Source Add/Edit form showed a phantom
unlabelled field because the show/hide JS referenced a
nonexistent path-terms id instead of the real path-regex.
Open follow-up: re-verify restart consistency across several
back-to-back restarts (a clean post-restart ps was observed —
web server present with the normal
secure-80-443 … PreviousSession.txt launch arg, daemons idle, no
spin); if the earlier intermittent lingering-old-process or
missing-replacement outcomes recur, the next suspect is the
boot-drain / relaunch state machine rather than the bind window.
67. ✓ HTTP/2 streaming delivery — window-sized spans
(2026-06-13). Large media and images stalled in Firefox under H2:
the transfer died at ~131 KB (NS_ERROR_NET_PARTIAL_TRANSFER),
then re-requested in a cascade. Root cause: the server committed a
whole large body to one stream and depended on the client extending
that stream's flow-control window; Firefox advertises the default
65535 per-stream window and withholds per-stream WINDOW_UPDATE
for a body it is not actively draining (the connection window
stayed at megabytes — only the per-stream window was pinned at 0).
H1 has no per-stream flow control (works); Safari advertises a 2 MB
window (works). Fix: a WebSite::peerStreamWindow() accessor
exposes the connection's per-stream initial window, and
ResourceController serves each span — byte-range and large
whole-file — at min(MAX_RANGE_STREAM_LEN, that window), so every
stream completes within the credit the client granted and closes,
the client fetching the next span the way it does against a
conventional server. On a 2 MB-window connection a response goes
out in one or two large spans (no slowdown); on a 65535-window
connection it serves ~64 KB spans. Interim hardening kept as
correct on its own: park-a-flow-blocked-stream-instead-of-resetting
(no stall-RST during paced playback) and holding an early
per-stream WINDOW_UPDATE that arrives before the response
registers the stream. Behaviour note: a large whole-file GET (no
Range header) now returns a capped 206 Partial Content +
Accept-Ranges so the client range-fetches the rest — browsers and
media elements handle this; spot-check any non-browser client that
specifically expects a 200 for a direct large-file download.
Follow-up resolved 2026-06-13 (commit 02f0561, "large download
full 200 on h1_1"): over HTTP/1.1 a large whole-file GET (no Range
header) now serves a full 200 OK streamed in chunks rather than
the capped 206, so non-browser clients that expect a 200 work;
the window-sized-span 206 path stays for H2/H3 where the
per-stream flow window requires it. Closes the spot-check above.
68. ✓ Event-loop CPU spins (2026-06-13). MailSite ran a forked
child at ~100% CPU: the select loop added every in-progress
TLS-handshake socket to the stream_select write set, and a
connected socket is almost always writable, so select returned
immediately every iteration while any handshake was pending — fixed
by leaving handshake sockets in the read set only (a "would block"
crypto step is waiting to read the peer's next record; a stalled
handshake is bounded by its deadline in cullDeadStreams). atto's
GopherSite spun at ~100% from a reversed timer subtraction
(microtime(true) - $next_alarm[0] → negative → clamped to a 0
select timeout) plus an always-zero fractional-seconds term; both
fixed (atto tag v2.2.1). The same fractional-seconds bug in
WebSite (correct operand order, so no continuous spin, but
sub-second timer waits collapsed to a 0 timeout) was fixed in
passing; MailSite's own timer math was already correct.
Fetcher / QueueServer / MediaUpdater / Mirror were checked
and are unaffected — sleep-based loops with no stream_select
timeout arithmetic. Reviewed and intentionally not changed:
H3Listener does not need the streaming or handshake fixes — it
already paces bodies incrementally with large (16 MB) default
per-stream windows, and QUIC does TLS in userspace over UDP so the
stream_select handshake-spin has no analogue. Backport status:
MailSite and WebSite work is backported to atto
(cpollett/atto); the streaming span-sizing logic is Yioop-side
(ResourceController) and does not backport.
69. ✓ First-run & single-secure-port startup robustness
(2026-06-13). A freshly-cloned, not-yet-configured instance can
now be started and brought up cleanly. Pieces: the crawl daemon
survives an unconfigured WORK_DIRECTORY instead of dying on launch
(CrawlDaemon lets the process live and warns rather than fatal,
commit e67c3c2); a single-secure-port mode (just 443, self-signed
cert auto-written) in index.php (e1fc4fd); maximum request
length derived from PHP's post_max_size so large uploads are not
silently truncated (c180065); the first-run Configure→running
handoff polls a real restart-status URL built through
controllerUrl with first-run constants in Config.php, replacing
a streaming page (70a7f97, c3eb12f, 0d25e11); and a guard that
warns when a second index server is started with a port already in
use (ada3e8f). This is the headless/browser bring-up path
referenced in the work-directory setup notes.
70. ✓ Command-line restart / status for the running server
(2026-06-13/14). index.php gained CLI restart and status
subcommands so the live server can be restarted and inspected from
a shell without the web UI: restart routes through the same
restart-server path the UI uses (66738f1), monitors restart
progress and reports phase status (283d957), names the services
it relaunched (241e740), a status command summarises the web
server + daemons (236735f), and bare/--help invocation prints
usage (0f63160). Supporting fix: CrawlDaemon::isProcessAlive
treats an EPERM from posix_kill($pid, 0) as alive-but-other-
user rather than dead (795957b), so process-liveness is correct
when the web user differs from the file owner — the same
cross-user (_www/www-data vs the owning account) hazard the
resource-UI Apache issue surfaced. Builds on item 66.
71. ✓ Headless ConfigureTool command-line mode (2026-06-13).
src/configs/ConfigureTool.php gained a non-interactive
one-command-per-invocation mode alongside the existing interactive
menu, so a fresh install is configurable with no browser:
work-dir [path], root-password <old> <new>, locale <tag>,
and the toggle commands debug / search-access /
page-elements (with +name/-name to turn settings on/off),
plus name-server and robot, each with an argument spec, count
checking, and a usage listing (71dc944, c9606ae, df25cdf,
12c71cd, aaf8f3b; readInput/readPassword consolidated in
Utility.php). This is the seam used to stand up a working
WORK_DIRECTORY + Profile.php + the two SQLite DBs headlessly
(e.g. so CodeTool unit tests can run). Document in the install
guide under item 34.
72. ✓ Page source/history/list visibility + per-domain robots.txt /
humans.txt (2026-06-16/17). Two related controls built on the
per-domain Appearance foundation (item 65) and the group OPTIONS
bitfield. (a) Visibility: SOCIAL_GROUPS.OPTIONS was refactored
from a packed (render_engine<<1)+encryption into a pure flag
bitfield, with render_engine split out to its own RENDER_ENGINE
column and new flags GROUP_OPTION_PAGE_SOURCE_ALLOWED /
GROUP_OPTION_PAGE_LIST_ALLOWED (migrations v106 / v107). A group
owner can now keep the public — and crawlers — out of wiki page
source and history (one master gating both: the page-level
public_source head var, the history endpoint, the
source/discuss/history nav, and the discussion-thread feed all honor
it) and out of the public page list. Owners stay exempt. (b) robots
/ humans: each web-server secure domain publishes its own files.
RobotsTxtGenerator (a pure library class) builds robots.txt from
per-domain choices stored in the DOMAIN_APPEARANCE blob (disallow
-all, a crawl delay with a None option, per-category crawlable
toggles for search / feeds / wiki with wiki sub-toggles for page
source / feed / list, plus free-form additional directives);
humans.txt is free-form text. robots.txt / humans.txt routes in
src/index.php serve the requesting host's stored text (falling back
to the built-in default), mirroring the .well-known / mta-sts route
pattern; the old static src/robots.txt was removed. Edited through a
new RobotssettingsElement page reached from each domain's
Server-Settings routing panel, with a live Preview. Commits
af06edee, d4a22b44, 7abf188a, 73b76b77, b2c8b7e8,
f70ded84, 87ec0bd0, c842350.
Phase 0 — Code hygiene + immediate hot-fixes (complete)
CodeTool needsdocs)✓ 41 (sweep and add the missing docblocks the tool surfaces;
0 findings remaining across src/. Tests tree also covered:
commit 048ce3a swept docblocks across tests/ to bring the
unit-test suite up to the same standard the production tree
now satisfies.)
✓ 31 partial: crawlAuthHash sibling added and CSRF token
migrated to it; checkCSRFToken now uses hash_equals for the
comparison and validates the new 54-char token layout; unit
test for the new primitive in UtilityTest::crawlAuthHashTestCase.
Indexing callsites of crawlHash untouched as planned.
✓ 31 partial: the security-relevant
crawlHash(AUTH_KEY . x) callsites (secret-ballot FORM_HASH,
csv_form_hash, page_hash, and the git app-code token) migrated to
crawlAuthHash; the CSV form fingerprint strengthened from the
8-byte crawlHash fold to hash("sha256", ...);
AUTH_KEY now generated from random_bytes; the ballot verify
and CSRF check compare with hash_equals. Two lingering
mt_rand git-archive temp names converted to
random_int. Arc-plan 4.1.a (crypto) and 4.1.b (constant-time)
done; indexing callsites left on crawlHash.
✓ 31 partial: arc-plan 4.1.c (drop dead
crypto fallbacks) — crawlCrypt was already on
random_bytes, and the last two stray mt_rand
git-archive temp names went to random_int. 4.1.d (cookie and
header hardening) — session cookie set HttpOnly,
SameSite=Lax, and Secure over HTTPS; CSP
frame-ancestors and nosniff were already
central. The cookie and header hardening lands in Phase 4.1, not Phase
5.
✓ 31 partial: arc-plan 4.1.d (response
and cookie hardening) done — session cookie HttpOnly, SameSite, and
Secure over HTTPS. 4.1.e (input and output paths) spot-checked: inputs go
through Controller::clean, no view echoes a raw request value, upload base
names use pathinfo basename, and resources are served read-only
(readfile/fread, never include/eval) so uploaded scripts do not run; the
one gap, a missing ".." strip in the resource-folder write
path, was closed.
✓ 33(a): new installs default to EMAIL_RECOVERY. Surgical
fix in ProfileModel::updateProfile — adds RECOVERY_MODE
to $not_null_fields, plus a shouldKeepZero policy helper
so an admin's deliberate NO_RECOVERY = 0 setting survives
saves. New tests/ProfileModelTest.php with 8 assertions.
✓ 14 (rename MailServer → BulkEmailClient and its file;
callers, test, autoload-by-namespace all updated)
✓ 12 partial: Message-ID, MIME-Version, Content-Type,
Content-Transfer-Encoding added to both branches of
MailServer::sendImmediate (PHP mail() and the SMTP path).
Helper MailServer::messageIdDomain factored out and unit-
tested. Full DKIM / List-Unsubscribe / Reply-To / Return-Path
still deferred to phase 2.
✓ 24 partial: fennec UA token dropped from the mobile
sniff in src/index.php (commit c590f5d; Fennec was
discontinued in 2020 when Mozilla replaced it with Fenix).
Full Sec-CH-UA-Mobile migration with Accept-CH header is
the remaining piece, deferred to phase 5.
✓ Ad-hoc: bumped session.gc_maxlifetime to match
AUTOLOGOUT in src/index.php (commit 9557889,
early-logout-fix tarball). Default PHP gc_maxlifetime is
1440 seconds (24 minutes), which was triggering early logouts
for users idle longer than that even though Yioop's intended
session timeout per AUTOLOGOUT is much longer. The fix sets
session.gc_maxlifetime and session.cookie_lifetime to
AUTOLOGOUT before session_start() so PHP's GC matches
Yioop's policy. Surfaced while testing the EMAIL_RECOVERY
default; not in the original phase 0 list. Note: this fixed
the obvious case but random logouts still occur on at least
one deployment, indicating an environment-level cause this
per-request ini_set cannot reach. See deferred item below
in Phase 3 for the deeper fix.
✓ Ad-hoc: PHP 8.5 forward-compat cleanups (commit
6e46da2). Dropped deprecated calls to imagedestroy() (GC is
automatic since PHP 8.0; the function emits an E_DEPRECATED in
8.5) and to xml_parser_free() (same story). Replaced one
list($a, $b) = ... destructure that no longer parsed cleanly
under 8.5's stricter list grammar with the bracketed
[$a, $b] = ... form. Surfaced when running yioop.com on
8.5 to validate the PHP 8.1 floor wasn't accidentally too low;
the same fixes are forward-compat work for whenever the floor
rises further.
✓ Ad-hoc: dropped empty HiTokenizer::partsOfSpeechTestCase
scaffolding (commit cc3776f). The Hindi tokenizer's test
class had a method named partsOfSpeechTestCase with no
assertions — leftover scaffolding from a never-implemented
POS-tagging feature. Removed rather than left as a 0-assertion
passing case that misleads about coverage.
Phase 1 — Codebase floor (complete)
✓ 38 (PHP minimum pinned at 8.1; composer.json / INSTALL.txt /
SystemComponent runtime check / 25 locale strings reconciled;
6 pre-existing locale translation bugs fixed in passing —
commit a1cd8f5)
✓ 31 partial (crawlCrypt fallbacks for mcrypt_create_iv
and mt_rand dropped now that PHP 8.1 guarantees random_bytes
— commit 1630deb. All 10 remaining mt_rand sites tree-wide
converted to random_int for defense in depth — commit
924ad9e. Two safety guards added: AdvertisementModel
now early-returns [] when TOTAL_AMOUNT < 1 instead of
silently calling mt_rand(1, 0); atto WebSite LRU eviction
now skips when $num_unmarked == 0 instead of silently
passing (0, -1). Both were pre-existing edge cases that
mt_rand masked but random_int would throw on. The
broader item 31 (full security audit) is still pending for
phase 4.)
✓ Ad-hoc: locale backslash-corruption fix. Surfaced while
doing the PHP-floor locale-string sweep: LocaleModel::update
Translation (line 783) was unconditionally applying addslashes
on every save, so any string with an apostrophe doubled its
backslash count on every round-trip through Manage Locales.
Some strings had accumulated hundreds of backslashes. Four
commits: 09165ed replaces the writer's addslashes with a
proper iniEscape helper (root-cause fix); 83819ca re-emits
every configure.ini through the new path (497 lines across 11
files; pure encoding normalization, semantically transparent);
f05b751 repairs 31 cross-contaminated keys across 7 locales
(de had Spanish strings, he had Japanese strings, pt was on
PHP 7.1 from a stale machine-translation pass, etc.) and resets
he to the en_US fallback for a native-speaker pass later
(item 42); 8d7b816 strips the runaway escape chains that the
normalization preserved (467 lines; \\{2,}' pattern collapses
to ', audit-verified safe against legit regex content like
vi_VN's \b word-boundary). Cleared 518 corruption sites
total. Pre-requisite for the broader item-42 translation audit.
✓ Ad-hoc: strpos/substr → str_contains/str_starts_with/
str_ends_with (145 sites across 49 files; commit 139f9e7).
Word-boundary \bstrpos\b anchor prevents accidental mb_strpos
matches; paren-aware non-greedy capture avoids the cross-line
matching trap; escape-aware length check for substr literals
("\r\n" with N=2 converts correctly). Skips stripos (semantic
change without mb_strtolower) and ($pos = strpos(...)) !== false
assignment-in-condition forms (need the position).
✓ Ad-hoc: switch → match for simple-expression arms
(10 sites across 11 files; commit c4f07c1). Audited all 96
switches; only 10 had value-returning arms suitable for match.
PhraseModel letter table, nl/it/en_US Tokenizer vowel/cons,
NWordGrams 4-arm via list-destructure, GroupModel EXIF
orientation (preserves no-op default), Utility CSS unit
conversion, PdfProcessor 2× char maps, LocaleModel direction,
PartialZipArchive decompression. Two follow-ups (f28cde9,
a5fb3f3, 7b40335) collapsed column-aligned => per the
fragile-layout rule in code I wrote, plus a sweep of the atto
files (back-ported to upstream atto).
✓ Ad-hoc: nullsafe ?-> for explicit null guards. 11 sites
total across 4 files in 2 commits (0bf2054, 9141f29).
if ($x) { $x->m(); } → $x?->m(); for fire-and-forget calls;
$x !== null ? $x->y : null → $x?->y; $x !== null && $x->y
→ $x?->y (preserving falsy short-circuit semantics). All
changes contained to atto WebSite / FeedArchiveBundle /
FeedDocumentBundle / MailSite. Wider scan found no further
clean candidates.
✓ Ad-hoc: constructor promotion across 45 classes. 4 commits
(f6dc46a, 628dc49, 4efe97b, e7963c3). Batch 1 (8 classes):
pure-assignment constructors where param names matched property
names — StaticTurnAuthenticator, Element, Transport, Flag,
Connection, ConnectionAcceptor, Listener, QuicStream. Batch 2 (14):
mixed-assignment constructors (some params promoted, body keeps
non-trivial setup) — Frame subclasses (HeaderFrame, DataFrame,
RstStreamFrame, PushPromiseFrame, PingFrame, GoAwayFrame,
WindowUpdateFrame, ContinuationFrame), WebSocket, FileAuthenticator,
StringArray, HashTable, PriorityQueue (also removed redundant
$num_values declaration that shadowed parent), SparseMatrix,
MediaJob, LRUCache, Trie, SuffixTree, PageIterator, View. Batch 3
(15): Frame, SettingsFrame, H3Connection, BulkEmailClient,
DocIterator, WordIterator, GroupIterator, MixArchiveBundleIterator,
IndexShard, LSMTree, Tier, IndexDictionary, MessagesBundle,
FileCache, VersionManager, Model. Batch 4 (8): WebSite (body
restructured so $this->base_path is updated in place by the
conditional default logic), WebArchiveBundle, IndexArchiveBundle,
CrawlQueueBundle, DoubleIndexBundle (also fixed the PHP-8.2
dynamic-property deprecation for $dir_name), BloomFilterBundle,
NetworkIterator, Bzip2BlockIterator. Skipped: name-mismatch
cases (Component, Layout, WebSite Listener at line 6725) that
would force visible API renames; constructors where the param
is transformed before assignment (ContextTagger's $lang,
StochasticTermSegmenter's $lang); cases where the param
shadows a parent property (TextArchiveBundleIterator,
DatabaseBundleIterator). Net effect: ~600 fewer lines across
src/library, constructors now self-documenting.
✓ Ad-hoc: removed unused Padding trait from atto WebSite.
Commit 36b6597 — 57 lines deleted. The trait had no use Padding;
consumers anywhere; PADDED-flag handling lives inline in
HeaderFrame/DataFrame parseBody methods. Pre-existing dead
code from the v9.5.3 baseline AI-generated atto scaffolding.
Back-ported to upstream atto.
✓ Ad-hoc: int-backed enums for QuicStream state machines.
Commit 4728703. Two file-scope enums in H3Listener.php:
RecvState (Open=0, SizeKnown=1, Done=2, Reset=3) and
SendState (Ready=0, Data=1, DataSent=2, Done=3, Reset=4).
Replaces 9 untyped int constants on QuicStream; typed properties
public RecvState $recv_state and public SendState $send_state;
12 internal references updated. Previously RECV_OPEN = 0 and
SEND_READY = 0 were interchangeable ints; now distinct types
catch cross-channel mix-ups at runtime. Surveyed wider codebase
for enum candidates — CrawlConstants is too tightly coupled
(inherited via implements by 56 classes) and other constant
clusters are config values, not closed sets. QuicStream is the
one clean win.
✓ 39 — Coding.pdf / Coding-guidelines wiki rewrite for the
PHP 8.1 floor. Surgical scope: dropped the PHP-5.4-compat rule
(PHP item 11) and replaced with an explicit PHP 8.1 floor plus
prefer-modern-form guidance covering match, str_contains/starts/
ends, nullsafe ?->, constructor promotion, enums, and random_int.
Updated PSR-12 reference to also cite PER Coding Style 3.0 (the
PHP-8.x extension of PSR-12). Bumped copyright year in the
example license block from 2023 to 2026 (two sites). Replaced
the .html-ltr / .html-rtl mandate with logical-property
guidance (preview of item 27) while keeping the legacy
selectors documented as still-supported. Added a parallel
"prefer media / container queries" note alongside the .mobile
selector (preview of items 23-25). Added a new Security
section between CSS and HTML codifying: no mt_rand for
security-relevant values; explicit range guards before
random_int when bounds derive from user-influenced data;
hash_equals for secret comparisons; HMAC over server-side
secret for auth tokens (not raw hash); password_hash /
password_verify only via crawlCrypt; no @ error suppression;
HTML/URL/SQL escaping rules; clean() the request supergloblals;
CSRF token verification for state-changing endpoints; do not
log credentials. Deliverable: full revised wiki source as
Coding-wiki-revised.txt — pasted whole-file into the wiki
editor; the wiki's built-in diff preserves the old version.
Re-render Coding.pdf from the revised wiki page as the final
step.
Phase 2 — Mail infrastructure (complete; the optional
mail-backend-search handler migration and the item-13 outbound
trim remain, plus item 12 below; item 11 is done and the
deliverability docs moved to Phase 11; the patch-12 visibility
cleanup and the MAIL_CLONE_JOB.LAST_ERROR column drop are both
done — verified in code 2026-06-07)
(Update 2026-06-14: the item-13 outbound trim is now done; only the
optional mail-backend-search handler migration and item 12's
RFC-5322/DKIM/List-Unsubscribe header work remain in this phase.)
(Update 2026-06-19: item 12 is now done too (verified in code) —
the email subscribe/unsubscribe arc landed the List-Unsubscribe
headers on top of the existing MimeMessage::build (RFC 5322) and
SmtpClient::dkimSign (DKIM); see item 12 above. Only the optional
mail-backend-search handler migration remains open in this phase.)
(Update 2026-06-19: Phase 2 is now COMPLETE. Its one remaining item,
the optional mail-backend-search webmail search UI, is deferred
Post-v10 — the backend primitive is already done
(MailBackend::search, implemented by both MailSiteMailBackend and
ImapMailBackend over the storage searchMessages); only the webmail
search box, the userMailSearch handler, and results rendering
(~150 lines) remain. It now lives with the other MailSite webmail
follow-ups (54/56/57/58) under Post-v10.)
Step 1 (config-only, no behaviour change):
Step 2 (UI affordance, no listener):
Step 3 (external IMAP first, validates the UI):
Step 4 (fill MailSite storage from disk, still no listener):
cpollett mailbox. Both Maildir
and mbox sources land; mbox export added. The tool was trimmed
to its durable command set import/export/verify — the
pre-production-only migrate/reuid/cleanup subcommands (and
their moveDirFiles/cleanupFolder helpers) were removed as
one-time states no current or future install reaches. Optional
live-IMAP source deferred (post-v10 nicety). See the status note
under item 48.Step 5 (MailSite-stored mail in the Mail activity):
mail-backend-search handler migration (patch j) is still
open. The patch-12 visibility cleanup (patch i) and the
MAIL_CLONE_JOB.LAST_ERROR column drop are done — the drop
ships as upgradeDatabaseVersion96 (table-rebuild with a
probe no-op guard), errors moved to the MAIL_CLONE_ERROR
table in v95, fresh schema in ProfileModel omits the column,
and DATABASE_VERSION is 102 so the migration runs; no code
reads/writes the dropped column.)Step 6 (the actual listener):
MachineController statuses and log paths
under the MailServer case).Step 7+ (everything else):
✓ 55 (MailSite outbound sending — landed during phase 2
ahead of the post-v10 schedule. Local recipients delivered
via MailSite::deliverMail; external recipients delivered
as peer MTA via SmtpClient with per-domain MX resolution
and port 25→587 fallback; opportunistic STARTTLS on the
outbound leg; sent copy archived to Sent via the MailSite
backend).
✓ 59 (Test Mail Mode + Test Mode Fallback Port: relaxes compose-form validation, skips MailSite FQDN-reject hook, enables an optional outbound port-25 fallback to a tunnel forwarder; replaces an earlier speculative 25→587 fallback).
✓ 60 (smtp_tunnel.php standalone byte-level TCP
forwarder for reaching port 25 from a network egress that
blocks it; lives outside the Yioop tree).
✓ 61 (PHP 8.5 dynamic-property declaration fix on
SmtpClient::$last_response_body).
✓ 13 (refactor outbound mail to share construction code
with MailSite when local-delivery applies; closed 2026-06-14 —
SmtpClient::sendImmediate builds via MimeMessage::build and
SmtpClient is trimmed to the wire exchange; see item 13).
✓ 12 full (RFC 5322 / DKIM / List-Unsubscribe headers) —
done; verified in code 2026-06-19. RFC 5322 message construction
is MimeMessage::build; DKIM signing is SmtpClient::dkimSign
wrapped around that built message in sendImmediate; the
List-Unsubscribe + List-Unsubscribe-Post headers come from
UnsubscribeToken::listUnsubscribeHeaders /
listUnsubscribeHeadersForEmail, attached on group/bulk send
(SocialComponent) and on registration mail (RegisterController).
Delivered by the 2026-06-17 email subscribe/unsubscribe arc (see
devlog/v10/2026-06-17 Email List Subscribe Unsubscribe/plan.html),
which also added the MAIL_SUPPRESSION store + send-path skip, the
web unsubscribe endpoint (ApiController::unsubscribe +
UnsubscribeView), the registration suppression gate + whole-site
signed token with its forgery guard, and the bot mailbox reader
(BulkEmailJob::processBotMailbox / applyUnsubscribeToken) that
acts on the mailto form of the header. Still open from that arc are
reader robustness (transient reconnect+retry, per-UID ratchet) and
account aliases; mail-account drag-drop reorder was already done
(verified 2026-06-07).
✓ 11 (webmail interface built — folders, list, read/compose/reply, flags, move/delete via MailBackend; further enhancements filed individually).
✓ 14 (rename MailServer → SmtpClient; done in
phase 0).
Phase 2.5 — Ongoing production reliability (complete to date; production-driven, runs alongside the planned phases as live issues surface). Full per-item detail in the "Production reliability & streaming" section above; the sub-blocks below carry the finer-grained hot-fix detail.
WebSite TCP connection lifecycle made fully non-blocking
(accept + TLS handshake, post-TLS protocol detection, WebSocket
frame reads, HTTP/2 frame reads; H3/QUIC reviewed and already
non-blocking). Arose from the observed pollett.org bursty-hang bug.
(2026-06-07)Production hot-fixes (2026-06-08, pollett.org) — each shipped as a
single-commit forward patch (see yioop-acme-prod-and-bugs-handoff.md):
✓ 64 — ACME certificate auto-renewal: AcmeRenewJob MediaJob
(ARI-gated via AcmeManager::renewalWindow, ONE_MONTH expiry
fallback) + in-place WebSite::reloadSecureCertificate on a
once-a-minute cert-watch timer in index.php.
✓ Manage-Machines under the secure server: WebSite
processInternalRequest inherits HTTPS/HOST/PORT so internal machine
self-calls no longer 301-redirect to https://0.0.0.0.
✓ Secure-domains apex suggestion strips leading mta-sts./www.;
MailSiteFactory::suggestedDnsRecords skips www./mta-sts. hosts
whose apex is already listed.
✓ 63 — Mail Banner / MAIL_HOST_NAME + suggested PTR.
serveRangeRequest loops drain each block through the atto streaming
flush (H2 DATA frame / H1 chunk) instead of the global flush the atto
buffer ignores — fixes OOM serving a ~696 MB .mp4;
WebSite::writeStreamingHead_H1 strips Content-Length when forcing
chunked; plus removal of the dead if ($start = 0) range-bound bug.Streaming engine + features (2026-06-09/10):
✓ HTTP/2 + HTTP/3 streaming engine hardening: RFC 7540 outbound send-window engine, slice-bounded interleave, busy-spin/OOM drain-check fix, HPACK curly-apostrophe crash fix; H3 connection-level flow control + handshake-over-TCP complete-write loop.
✓ 65 — Per-domain Appearance.
Restart, delivery, spins (2026-06-13):
✓ 66 — Restart Server reliability (bind-retry window + supporting daemon/boot-drain/status fixes).
✓ 67 — HTTP/2 window-sized span delivery.
Startup, sessions, CLI, resource UI (2026-06-13/15):
✓ 69 — First-run & single-secure-port startup robustness (daemon
survives an unconfigured work directory; single 443 port mode;
max-request-len from post_max_size; first-run restart status URL;
warn on a second index server with an in-use port).
✓ 70 — Command-line restart / status for the running server
(+ cross-user isProcessAlive EPERM fix; builds on 66).
✓ 71 — Headless ConfigureTool CLI (work-dir, root-password,
locale, debug/search-access/page-elements toggles,
name-server, robot) — configures a fresh install with no
browser.
✓ 47 — Database-backed session restore (also the Phase 3
random-logout item): SESSION_USER table + in-memory LRU heap with
DB rehydrate, DB version 104→105, so sessions survive a server
restart / memory eviction.
✓ 13 — Outbound mail construction dedup (SmtpClient::sendImmediate
→ MimeMessage::build; SmtpClient trimmed to the wire exchange).
✓ large whole-file download returns a full 200 over HTTP/1.1
(closes the item-67 spot-check).
▮ 17 — Page-resource folder UI: drag-drop, multifile/folder upload with progress, mail-style action toolbar, inline rename, parent-row move (incl. to root), single-resource delete, and the VersionManager stranded-lock root-cause fix all landed; multi-select + keyboard shortcuts and the CSS-Grid auto-fill rewrite remain (kept open under item 17 / Phase 7).
Group visibility + per-domain robots (2026-06-16/17):
RENDER_ENGINE column split, migrations v106 / v107) and
per-domain robots.txt / humans.txt (RobotsTxtGenerator +
DOMAIN_APPEARANCE storage + src/index.php host routes + a
Server-Settings edit page with Preview).Phase 3 — Auth modernisation (complete)
Phase 3.5 — Cooperative atto Audit and MailSite Daemon (complete; 2026-06-20 → 2026-06-21)
WebSite server was already cooperative from Phase 2.5 item
62). Full per-slice detail in the Phase 3.5 arc plan.Phase 3.75 — Authentication Controls and Password Policy (complete; 2026-06-25 → 2026-06-29)
LDAP_LOCAL_USER becomes a SigninModel method;
one email per account.Phase 4 — Security audit + wiki regex (closed 2026-07-10; started 2026-06-29)
Phase 4.5 — Settings refactor + live-site fixes (in progress; started 2026-07-10)
Phase 5 — CSS foundation + viewport
Phase 6 — Chat refresh
Phase 7 — Wiki UX
▮ 17 (page-resource folder UI — most of it landed 2026-06-14/15, logged under Phase 2.5; multi-select + keyboard shortcuts and the CSS-Grid auto-fill rewrite remain)
☐ 18, 19, 20, 21 (async secret-ballot witnesses), 37
Phase 8 — Calling
Phase 9 — Crawl / index health
Phase 10 — Final docs pass + release prep
☐ 34, 36, 42 (localization audit + escape-bug fix), 44 (trim VersionFunctions to v9+)
Item 36 (mail-deliverability docs) moved here 2026-06-19: it documents standards already delivered (DKIM/SPF/DMARC guidance, Banner / PTR), so it belongs in the final v10 documentation push rather than a phase of its own.
Phase 11 — Deliverability (the standards work is done — DKIM,
SPF, DMARC, MTA-STS, Banner / MAIL_HOST_NAME, PTR / FCrDNS all
shipped in Phase 2. Its one remaining piece was the documentation,
item 36, which moved 2026-06-19 into the final docs push (Phase 10).
This block is kept only as a record of what landed.)
36 (document mail deliverability requirements: DKIM, SPF, DMARC,
Banner / MAIL_HOST_NAME, PTR / FCrDNS — refresh the PDF
deliverability docs to match the wiki Mail Services page) — moved
to the final docs push, Phase 10.
Implementation prerequisite tracked under item 12 (RFC 5322 / DKIM signing / List-Unsubscribe headers, in Phase 2); document each piece here as it ships.
Already landed toward this (groundwork, for reference): item 63
(Banner / MAIL_HOST_NAME host identity + suggested PTR row in the
Suggested DNS Records panel; the wiki Mail Services page gained a
Banner section and PTR note). The HELO/PTR/FCrDNS half of the
deliverability story is in place; the RFC-5322/DKIM/List-Unsubscribe
half (item 12) and the PDF doc refresh (item 36) remain.
Log ongoing deliverability work as new bullets here as it lands.
Post-v10
☐ mail-backend-search (webmail search UI) — deferred from Phase 2
on 2026-06-19. The backend is done (MailBackend::search in both
backends, over the storage searchMessages); what remains is the
webmail search box, a userMailSearch handler that turns matched
uids into the same list/envelope structure MailElement renders,
and routing/locale/test (~150 lines). Belongs with the MailSite
webmail follow-ups (54/56/57/58) above.
☐ 43 follow-up (full IndexShard removal once the deprecation cycle has shipped)
☐ Mirror sync latent memory bug. ResourceController::syncList
builds the whole delta file-list for a mirror request and serves it
with base64_encode(gzcompress(serialize($info))) echoed
whole, holding the entire list (and its base64 copy) in memory at
once. On a large cache this is the same unbounded-memory exposure the
main resource serve was hardened against (range / offset-limit /
large-file streaming), so a mirror puller over a big cache could
exhaust the process the way the HTTP/2 scrub did. Latent for now
because pollett.org does not run a mirror. Either bound and stream the
sync list (page the delta in fixed spans, like the resource serve) or,
if the Mirror feature is being retired, remove it; surfaced
2026-06-23 during the HTTP/2 connection-memory work.
Write unit tests as code lands. Per Coding.pdf Code-base
Organization rule 10, every new library/ file should ship with
a sibling *Test.php. Run via
php executables/CodeTool.php unit (lists), unit ClassName
(run one class), or unit ClassName methodName (one method).
Note: UnitTest::setUp() and tearDown() are abstract — must
be implemented even when empty (skipping silently fatals out
with no diagnostic), which masked an earlier test scaffolding
attempt.
MailSite test seam. MailSiteFactory::setTestOverride($site)
replaces the factory's build() return value with a
pre-configured MailSite instance (typically wired to
RamMailStorage) so unit tests don't need a WORK_DIRECTORY.
Tests must clear the override in tearDown to avoid bleeding
into other tests. Used by MailSiteMailBackendTest.
Stale-item verification (2026-06-07). A pass over a few
"open" housekeeping items found several already done in the
code, so the list had drifted. Confirmed already complete:
(1) the dead clone locale key mail_element_clone_errors_toggle
does not exist in the locale files — the live keys
(mail_element_clone_errors_none,
mail_element_clone_error_resolved_via, etc.) are all
referenced in SocialComponent and mailclone.js;
(2) the MAIL_CLONE_JOB.LAST_ERROR column drop (see Step 5
note); (3) mail-account drag-drop reorder — reorderAccounts
handler in SocialComponent (with userMailReorderAccounts
persisting the order). Lesson: verify "open" items against the
code before scheduling work on them.
Stale-item verification (2026-06-19). Second pass, same
lesson. Item 12 (RFC 5322 / DKIM / List-Unsubscribe headers) was
still shown - [ ] open but is fully done in code — RFC 5322 via
MimeMessage::build, DKIM via SmtpClient::dkimSign,
List-Unsubscribe via the 2026-06-17 email subscribe/unsubscribe
arc; now checked off above. Re-confirmed already complete and not
to be relisted: the dead clone locale key
mail_element_clone_errors_toggle (absent from src/), the
MAIL_CLONE_JOB.LAST_ERROR drop (upgradeDatabaseVersion96;
MAIL_SCHEDULED.LAST_ERROR and MAIL_CLONE_ERROR.MESSAGE are
unrelated false positives), and mail-account drag-drop reorder.
Rule reinforced: this todo file is the canonical store, not chat
memory; an item only "counts" as tracked once it is checked off
here, so reconcile the todo against the code rather than re-raising
done work.
Phase-completion checkpoint. When we finish all items in a phase, Chris sends a fresh tarball of the project so Claude can verify its local working tree is in sync with origin/master before starting the next phase.
@return tag style. Keep tags concise: type plus a brief
prose phrase. For functions that return a value on some paths
and fall off the end on others, write
@return <type> <description of the failure value>; void on
the success path. For activity methods whose only return is
a redirectWithMessage(...) call, the established Yioop
convention is to omit @return entirely.
The Public, Help, Search built-in groups + per-user
Personal$<id> groups underpin chat, clipboard, and
context-sensitive help. Touching any chat / wiki-resources /
secret-ballot items must preserve this model.
All end-user-facing strings need to go through tl() /
LocaleModel.php. Code comments and log messages stay English.
Per Coding.pdf general rules: minimise external deps; UTF-8 no
BOM; 4-space indent; ≤80 col lines; phpDoc on public surfaces;
no @ error suppression.
Apache vs WebSite-mode hazards. Yioop runs under both
Apache (mod_php / php-fpm — output buffered, header() and
exit() work normally) and atto's WebSite (long-running
server, response stream starts early). Handler code that
emits non-HTML output must use $parent->web_site->header()
(delegates to plain header() under Apache) and
\seekquarry\atto\webExit() (falls back to real exit()
under Apache) rather than the bare PHP calls. Pre-MailSite
testing was under Apache; the same code paths under WebSite
warn-and-misbehave. Sweep done across SocialComponent JSON
handlers; other controllers may still have stale patterns.