Every Miss, Logged

Real-time alerts went silent from May 19 to May 26 — Postgres connection bug, now fixed

Dossier launched on schedule on May 19, 2026 (see #001). Alerts fired correctly for roughly 36 hours after launch. Then a Postgres connection bug in the wallet monitor caused it to crash and restart in a loop. Because the process kept restarting, the service appeared "online" on the Railway dashboard, but it was not actually capturing wallet activity. No new convergences entered the database, and therefore no Telegram alerts fired.

The system was effectively dead from approximately May 19, 11:00 PM CT through May 26, 5:43 PM CT — roughly seven days. During that window, the live ticker on this site continued to scroll older entries, and the 24-hour counter eventually fell to zero. Both were accurate. Neither was investigated quickly enough.

Root cause. The four backend services that talk to Postgres were using a single persistent connection per service, with no automatic reconnect when that connection was dropped. Railway's internal networking closes idle TCP connections after a period of inactivity, which surfaced as ECONNRESET errors that the services were not handling. Each service exited on the unhandled error and was restarted by Railway, only to fail the same way again. The wallet monitor was the visible symptom; the other services were silently vulnerable to the same failure.

Fix. All four services (wallet monitor, main detector, Handler detector, public API) were rewritten on May 28, 2026 to use a connection pool with built-in reconnect, TCP keepalive on the underlying socket, and a non-fatal error handler that logs and recovers rather than exiting the process. The change is a connection-handling fix only — no detection rules, wallet cohort, or alert formats were modified.

What is in place to prevent recurrence. The connection pool will transparently replace a dropped connection rather than crashing. Pool-level error events are logged but do not exit the process. The probe-on-boot pattern surfaces any database reachability issue immediately in the deploy logs rather than at the first user-visible failure. A separate follow-up will add automatic reconnect logic on the Helius WebSocket as well, addressing the other class of silent-death failure that this incident exposed.

What this correction does not cover. Dossier's monitoring was not strong enough to detect that the alert pipeline had stopped producing output for a week. That is a separate failure of operational diligence, and the lack of proactive uptime alerting is the underlying reason this took seven days to surface. Strengthening that is on the internal workplan, but no specific commitment is being made here — when one is, it will be logged as its own entry.

Second alert pipeline outage from May 30 to June 5 — WebSocket connection bug, now fixed

The Dossier alert pipeline went silent again from approximately the evening of May 30 (CT) through the morning of June 5. Both Telegram channels — Dossier Signals and Dossier Handler Signals — received no alerts during that window. This is the second multi-day outage in two weeks, following the May 19–26 outage logged in #007.

What happened. The wallet monitor's connection to our data provider (a long-lived WebSocket feed delivering Solana transaction events) stopped receiving data and never recovered. The Node process stayed alive, the Railway dashboard kept showing the service as "online," but no new trades were being recorded. Telegram alerts cannot fire on data that was never captured.

Root cause. Infrastructure log retention on the current tier had rolled over by the time the issue was discovered, so there is no definitive log from the moment the connection dropped. The failure pattern matches a class of issue commonly called "silent connection death" — the underlying network connection drops, the local process keeps believing it is still connected, but no data flows. The wallet monitor service had no logic to detect this state or to automatically reconnect when it occurred.

This is a different bug from #007. The May 19 outage was a database connection issue, which was fixed on May 28 with auto-reconnecting database pooling. The May 30 outage was a WebSocket connection issue, which that fix did not address. Two parallel failure modes in the same service. The first fix did what it was designed to do; it just was not enough on its own.

Fix. The wallet monitor was restarted on June 5 to recover the system. That brought it back online but did not fix the underlying cause. On June 6, an auto-reconnecting WebSocket layer was deployed with a heartbeat-based detector for silent connection death. The service now automatically reconnects on any disconnect (with exponential backoff), re-subscribes to all watched wallets on reconnect, and forces a fresh connection if it receives no data or heartbeat response within a configured window. The structural failure mode behind both #007 and #008 — silent connection failure with no auto-recovery — is now closed.

What this means in plain terms. The product was effectively offline for a substantial portion of the post-beta-launch period. Subscribers received no alerts during that time. The site dashboard continued to display historical data, which made "live" framing misleading. No promise is being made here that no other outages will occur — there are other ways a service can fail. The commitment is narrower and verifiable: every connection-class failure that produces silent downtime will get a corrections entry with root cause and fix, on the same week it is discovered.

A Helius API key was found hardcoded in a leftover file; rotated and the file deleted

During a security audit of the Dossier backend we found a Helius API key hardcoded in plaintext in a leftover file (an early learning script that was never part of the running service). All production code correctly reads credentials from environment variables; this one file predated that discipline and was missed. The repository’s secret-ignore rules were otherwise correct, so no environment file or other credential was exposed.

Impact. The exposed key was a Helius RPC key (read access to public Solana chain data). It could not move funds, access user data, or touch any wallet — Dossier holds no custody and stores no user secrets. The realistic worst case was consumption of our request quota by a third party. We have no evidence the key was used by anyone else.

What we did. Rotated the key immediately (the exposed key is now dead), re-applied access settings to the replacement, and deleted the leftover scripts from the repository. We log this even though it was low-impact and caught before any known misuse — which is exactly the kind of event a project is tempted not to disclose. That temptation is the reason to disclose it.

STATUS · Key rotated · leftover files deleted · no evidence of misuse

Rotating the exposed key (#014) took down the wallet monitor that was quietly using it; our own telemetry caught it

The key rotated in #014 turned out to also be the live credential the production wallet monitor was using. Rotating it killed the monitor’s authentication: its WebSocket to Helius began returning 401 (unauthorized) on every reconnect attempt, and new trades stopped being ingested. The interruption lasted roughly one day, until we identified the cause.

How it was caught. Our own status telemetry. The status page flipped to DEGRADED and stated plainly that the pipeline was online but the data feed was interrupted — new trades were not being ingested until it reconnected. This is the instrumentation we built after #007 and #008 specifically so a running process can never mask a stalled feed. It worked as designed: the failure was visible and honestly labeled, not hidden behind a green light.

Impact. Convergence detection paused during the interruption — the monitor could not see new trades, so no new signals could fire for that window. No user funds, custody, or data were involved. The gap is in ingestion coverage for the affected period, which we disclose here rather than pretend continuity we did not have.

What we did, and the process change. Updated the monitor to the current live key; ingestion resumed and status returned to LIVE. We then swept all backend services and confirmed every one is on the same live key. Going forward, a key rotation includes updating the credential everywhere it is used — code, deployment variables, and any local files — as a single checklist. #014 and this entry are the same event from two sides: rotating an exposed key without updating the service that depended on it.

STATUS · Resolved · all services on live key · rotation checklist adopted

Ingestion was dead for five days, and the detector kept alerting on frozen data the entire time

Dossier captured no wallet activity between 2026-07-22 18:32 CDT and 2026-07-27 13:41 CDT — four days and nineteen hours. The Helius WebSocket that feeds the wallet monitor was refused on every connection attempt, roughly once a minute, for the whole period. Every backend process stayed up and reported healthy throughout, because every process was healthy. The feed was not.

The detector did not stop. It scans on a rolling 24-hour lookback using the time a trade was recorded, not the time it happened on chain. When ingestion stopped, the trades table froze — and the detector kept scanning the same frozen rows every sixty seconds, kept finding convergences inside them, and kept alerting. 190 distinct stale convergences were alerted on, and alerting continued until 13:40:45 on 27 July — one minute before the feed recovered. Not for the first day of the outage. For all of it.

Those alerts described wallets that had “just bought” using data that was in some cases days old, with nothing in the message indicating it was stale. Against our own stated claim, they were not late. They were wrong. Anyone who acted on a Dossier alert between 22 and 27 July was acting on a snapshot of a market that had stopped being observed.

On the raw message count. The database records 1,242 alert messages sent in that window. We are publishing 190 as the number of stale convergences because the gap between those figures is a separate defect, disclosed in #017. Reporting 1,242 here would count that bug twice.

Root cause of the outage. Helius rejected the WebSocket upgrade with 401 Unauthorized (-32401) on every attempt, refused at their edge before reaching their origin. The same API key, from the same container, at the same moment, authenticated successfully over ordinary HTTPS and returned a healthy response — a test we ran from inside the running process specifically to separate the two cases. The key was not revoked, expired, restricted or out of credits, and the plan was active throughout. Issuing a new key and swapping it in restored the feed immediately, with no other change. We do not know why the original key was accepted for one protocol and refused for another, and we would rather say so than publish a tidy explanation we cannot support. The question is open with the provider.

Fix. Both detector tiers now check feed freshness before every scan. If the newest trade in the database is more than twenty minutes old, the scan is skipped entirely, the reason is logged, and the paused state is written into the heartbeat so it appears on /status rather than looking like an idle-but-healthy detector. A failure to read freshness also skips the scan: missing an alert is recoverable, sending a false one is not.

Data that is simply gone. The wallet monitor subscribes to a live stream with no historical replay. The trades never captured during those five days cannot be backfilled and no attempt will be made to reconstruct them. Any convergence that occurred in that window was missed and stays missed.

Two smaller things found while diagnosing, disclosed for completeness. A latent bug in the wallet monitor’s subscription bookkeeping could, under the right collision, have attributed one wallet’s trades to another or dropped a wallet’s stream silently; it has been closed, and unmapped events now log loudly instead of vanishing. Separately, a change made during this diagnosis to capture the rejection detail introduced a crash that took the wallet monitor down for several minutes and stopped its heartbeat with it — briefly making the outage less visible on this site rather than more. It was fixed the same hour, and the process now survives socket-layer errors instead of exiting on them.

What this entry does not fix. Our telemetry worked. The status page showed DEGRADED, refused to claim uptime it could not verify, and correctly separated process health from feed health — a running process never masked a dead feed. What failed is that nobody looked for five days. The instrumentation reports honestly to anyone who visits the page; it does not tell us. That gap is the real finding here, and no commitment is being made about it in this entry. When one is, it will be logged as its own entry with a date attached.

More than half of every alert we have ever sent was a duplicate of one sent minutes earlier

Auditing our own database after the outage, we measured what we had actually been sending. Over a normal, healthy eight-day period — 15 to 22 July — Dossier sent 7,825 alerts covering 2,113 distinct tokens. 4,238 of those messages repeated a token already alerted within the previous five minutes.

That is roughly 978 messages a day, over half of them re-sends of a call already delivered. This was not caused by the outage. It has been happening in normal operation for the entire beta, to every operator.

Cause. Every convergence is stored with a uniqueness key of token, wallet set and first-buy time, which is supposed to make a repeat impossible. The detector rescans every sixty seconds across a rolling window; as that window slides forward, the earliest qualifying buy changes, which changes the key, which reads as a brand-new convergence. The premium handler tier already carried a six-hour cooldown per token that suppressed this. The main tier had no cooldown at all. The protection existed and had simply never been applied where most of the volume was.

Against our own description. This site says “one signal when curated wallets independently move on the same token in a tight window. Convergence, not a feed.” A thousand messages a day, half of them repeats, is a feed. The claim was not aspirational marketing that ran ahead of the build — it was a straightforward description of what we believed the system did, and we had never checked.

Fix. The handler tier’s cooldown has been ported to the main detector: one alert per token per six-hour window, regardless of how many times a scan re-finds it. The convergence row is still recorded, so detection history stays complete — only the duplicate message is suppressed, and the scan log now reports how many were held back. Alert volume fell roughly 74% with no change whatsoever to what is detected. Deployed and verified live on 2026-07-28.

Every operator in the beta experienced this. If Dossier felt noisy to you, it was, and the reason was ours. Nobody reported it, which is worth noting: we found this by auditing our own database rather than by being told, and a system that only reveals its faults when someone complains will keep faults that nobody bothers to complain about.

Our published 6-hour convergence window is about thirty-six times wider than the events it catches. We picked it by intuition and never checked.

Dossier has always described itself as detecting wallets that bought the same token “within the same 6-hour window”. While auditing the alert stream we measured, for the first time, how long our convergences actually take. Across 9,723 convergences over fourteen days:

Median span 4.2 minutes. Seventy percent complete within fifteen minutes. Eighty-three percent within an hour. The ninety-ninth percentile is 315 minutes — which is to say our published window was calibrated, accidentally, to the outermost one percent of events.

Real convergences in our cohort are near-simultaneous. That fits what one would expect: wallets reacting to the same trigger within minutes, not independently arriving at the same token over an afternoon. The extra five hours and fifty minutes were not catching additional signal. They were admitting coincidence, and coincidence is precisely what this product claims to filter out.

A related measurement. Above a three-wallet threshold the window stops mattering almost entirely — three wallets agreeing inside fifteen minutes and three agreeing inside a full hour differ by about three events a day out of eighty-four. Three wallets agreeing is intrinsically near-simultaneous. The wallet threshold, not the window, is what actually governs how selective the product is, and we had the loosest possible setting on it.

No change has been made yet, deliberately. Altering the window or the wallet threshold changes the published definition of the product, and our practice is to publish a specification for comment before shipping a change of that kind rather than quietly adjusting a constant and updating the copy afterwards. That specification is being written. Until it is published and this entry is superseded, the site continues to describe the 6-hour window because that is what the system currently does.

The transferable lesson, since this log is meant to be useful. The 6-hour figure was chosen early, by judgement, and then repeated in every description of the product for months without anyone running a single query against it. It was not wrong through carelessness in the code. It was wrong because it was never measured. If you are building something that filters, measure the width of your filter against the thing you are filtering for, and do it before you write it on your homepage.

A second outage, five days after the first, and again nobody noticed — this time we are committing to a fix rather than describing the problem

Dossier captured no wallet activity at all on 2 August 2026. A full day, zero trades. We did not find out from an alert or from the status page. We found out on 3 August, running a routine query for something else, when the day was simply absent from the results.

Cause. Our data provider plan hit its credit ceiling and ingestion stopped. What drove the burn is the more useful part. Our wallet subscriptions fire on mentions — a transaction that merely references a watched wallet triggers a notification, whether or not that wallet traded. When one transaction referenced five wallets in our cohort, five separate notifications arrived, and each one independently spent a paid API request to fetch and parse the same transaction. Four of the five were then discarded by a uniqueness constraint and never stored. We were paying five times to record one row, and in the days before the outage that traffic rose about 60% while the share of it we could not classify at all climbed from roughly 15% to 45%.

In other words: the plan was exhausted largely by transactions our cohort did not trade in.

Fixed. The monitor now remembers signatures it has already handled and skips the duplicate API calls entirely. Nothing is lost, because the duplicates could never have been stored. The plan was also extended. Both are live.

The part that matters more. This is the second outage in two weeks that ran to completion without us knowing. The first lasted five days. Entry #016 named this exact gap — that our instrumentation reports honestly to anyone who visits the status page and tells us nothing — and deliberately made no commitment about it, on the grounds that we would log one when we were ready to date it. Two occurrences in two weeks is a pattern, not an incident, and an entry that names the same gap a second time without a date attached is where this log would start becoming decorative.

COMMITMENT. The detector’s freshness gate — which already suspends alerting when data stops arriving — now also notifies an operator directly. It fires when the newest trade passes thirty minutes old, repeats every four hours while the condition persists, and sends a recovery notice with the duration when the feed returns. It triggers on the symptom, so it catches any cause: a dropped connection, an exhausted plan, a crashed process. Deployed 2026-08-04. If a future outage again runs to completion without us knowing, that is a broken commitment and it will be logged here as one.

We were blind to a share of trades on one venue, which is why some watched wallets appeared to buy constantly and never sell

Auditing our trade capture, we found that roughly 12% of everything we recorded could not be classified as a buy or a sell at all, and that the rate was wildly uneven across the cohort. Several wallets showed hundreds of buys and almost no sells — one recorded 236 buys across 103 tokens and zero sells in a week. Wallets selected for realised profit cannot realise profit without selling. Something was missing.

What it turned out to be. Instrumenting the parser to report why it failed, rather than silently returning nothing, split that bucket into three. The largest part was legitimate: plain SOL movement, correctly not recorded as a token trade. A smaller and much more important part was not. Certain trades on one venue arrive labelled as swaps, with a token genuinely moving, but without the structured swap data our parser depended on. Those were real trades, and we were dropping them.

What that means for anything we have said. Any statement we might have made about how long the cohort holds positions would have been drawn from data missing a meaningful share of exits. We have not published such a figure, and this is why: an internal analysis produced one, we found this gap before it went out, and it was withheld. The measurement is being redone on clean data and will be published whatever it shows.

Fixed. The parser now falls back to reading the SOL side of the transaction when a token moves but cannot be tied to the wallet directly — a wallet that sent SOL bought, a wallet that received SOL sold. It also records why any remaining transaction was skipped, so what we are still missing is measurable in the data rather than only visible in logs. Live since 2026-08-03.

The transferable lesson. The parser had been failing silently for months. It returned nothing and moved on, and nothing anywhere counted how often that happened or asked why. The defect was not that the code could not handle every venue — no parser can. The defect was that it could not tell us where it was blind. If a component of your system can fail to understand its input, make it say so out loud, with a reason and a count. Otherwise your data looks complete and quietly is not.

More than a quarter of our alerts describe a position that is already closed — and we now know that is not something speed can fix

Entry #020 committed to redoing our hold-time measurement on clean data and publishing it whatever it showed. Here it is.

Measured across every alerted convergence since 4 August — the first data captured entirely by the corrected parser — 28.5% of the wallet positions that triggered an alert had already been closed by the time the alert was sent. Among positions we can observe closing, the median hold is 32 seconds.

The reason this is decisive rather than just bad. Two weeks ago the median gap between the last buy forming a convergence and the message going out was roughly twenty-three hours, an artefact of the duplicate-alerting defect in #017. That is now twenty seconds. We are approximately four thousand times faster than we were, essentially real-time — and more than a quarter of positions still close before the message lands.

This is not a latency problem. There is no further speed to find. It is also not something the detection thresholds can fix: raising the wallet count or narrowing the window selects for wallets that agree faster, and wallets that agree faster also exit faster. The gap between what this product claims and what it delivers is structural, not a tuning error.

What the measurement can and cannot support. The parser fix in #020 raised the share of positions with a recorded exit from 28.8% to 45.2%, which confirms the decoding gap was real and is substantially closed. But 54.8% of positions still show no recorded sell, and we cannot yet distinguish genuine holding from residual capture gaps. So the 32-second figure is the median among positions we can see close, not among all positions. Separately, about 18% of triggering positions could not be matched to a buy inside their own convergence window and are excluded — a gap we do not yet understand and are not going to paper over. Both caveats point the same way: the true picture is not better than what is published here, and may be worse.

Beta intake stays paused. It has been paused since 28 July. We are not going to charge for, or onboard anyone into, a product whose central claim we have just measured and found wanting.

What happens next, stated honestly and without a date. There are three routes and we have not chosen one. Change the cohort — the current wallets were selected on realised profit, which selects precisely for fast flippers; selecting for holding period instead would make convergence mean what this site says it means. Change the signal — alert when wallets have bought and are still holding after a set interval, which would be rare by construction and a genuine conviction signal rather than a speed contest. Change the claim — keep the product as built and describe it accurately as an attention signal rather than something actionable.

We are not committing to one today because we do not yet know which is right, and a commitment made to look decisive is worth less than an honest interval. When we choose, it will be logged here with a date attached, and the specification will be published for comment before anything is built — as with every other design decision on this project.

Why this entry exists at all. Nobody asked for this number. No operator complained. We could have quietly tightened the thresholds, published the improvement, and never mentioned the measurement. The reason we did not is that a corrections log which only contains failures somebody else would have found is not a corrections log — it is public relations with better formatting.

We kept a waitlist open, offering six months free at a launch we can no longer promise. It is closed as of today.

Until today the homepage carried this: “Real-time signal launches when 50 verified operators are on the line. Drop your email — early signups receive 6 months of Operative tier free at beta launch and a Telegram invite to the founder channel.”

What is wrong with that, stated plainly. It is not a queue. It is an offer, with a specific benefit attached, collected against a launch we have no date for and may never deliver in that form. Entry #021 published the measurement that put the product’s central claim in question, and no design decision has been made since. Every address collected in the meantime was given on the strength of a promise we are not currently in a position to keep.

It also stated the wrong condition. “Launches when 50 verified operators are on the line” says the launch is gated on how many people sign up. It is not, and it never was. It is gated on whether the signal means what we said it means. Framing intake as the bottleneck put the burden on the reader to unlock something that was never theirs to unlock.

How long it ran. Beta intake was described as paused on 2026-07-28 and the homepage has said so since. The waitlist form kept accepting submissions throughout, under the original offer text, for thirteen days after that. The pause notice and the sign-up panel were contradicting each other on the same page and we did not notice until we read the page rather than the log.

What we have done. The waitlist form is removed, not merely relabelled — no further addresses can be submitted. The homepage now states that intake is closed with no date, and points to this log and the public channel instead of to a form.

COMMITMENT. Everyone already on the waitlist will be contacted directly and told the position: the product is halted, the claim is under review, there is no launch date, and the six-month offer cannot be honoured against a launch that may not occur. Anyone who wants their address removed can have it removed. If the product does resume, people who signed up before today are not disadvantaged by this entry — the offer stands if there is ever something to attach it to. This is loggable: if that contact does not go out, it is a broken commitment and will be recorded here as one.

Why this is a disclosure rather than a fix. Nothing was taken from anyone and no money changed hands. It would have been easy to quietly delete the form and move on; the reason not to is that a waitlist is the one place a paused product still asks something of a reader, and continuing to ask while the thing being queued for is in doubt is the small version of exactly what this log exists to catch.

UPDATE · 2026-08-10, same day. The list held seven addresses. All seven were contacted directly by email, sent BCC, stating the halt, the measurement, the withdrawal of the six-month offer, and an unconditional offer to delete their address on reply. The commitment above is discharged.

A correction to this entry, in our own favour, which is why it is stated carefully. The entry above says the form kept accepting submissions for thirteen days after intake was publicly described as paused. That is true and the original text stands unchanged. But the submission dates show the most recent sign-up was 19 May 2026 — more than two months before the pause notice went up. Nobody joined during the window in which the page contradicted itself. The defect was real; the number of people affected by it was zero.

We are noting that because an update which makes us look better deserves at least the scrutiny of one that does not, and because the honest version is smaller than the original disclosure rather than larger. The contradiction on the page still should not have been there, and it was found by reading the site rather than by anyone reporting it.

The pricing page is being withdrawn. A halted product should not display a price list.

The homepage still carries a four-tier pricing surface — Recruit, Operative Beta with “FREE · 6 MO” attached, Operative, and Handler, with dollar prices — presented as the product’s access terms. That surface is being withdrawn from the active interface.

Why, stated plainly. Convergence detection is halted while the measurement published in #021 is worked through, and entry #023 already closed intake for the same reason. A price list is an offer. Continuing to display offer terms — including a free-period benefit — for a product whose central claim is under review asks the reader to price something we are not currently operating. The tier grid outlived the decisions that invalidated it, the same way the waitlist form did in #023; this is the same defect found one page section later.

What this is not. This is not a cancellation of Dossier, and it changes nothing about the measurement review. It also does not touch the standing position from #023: the people who signed up before that entry are not disadvantaged by this one, on the same terms stated there. The structural revenue commitment recorded in #006 is likewise unchanged — its terms were always conditional on paid tiers operating, and nothing here alters them.

What replaces it. A factual statement of the current access state — detection halted, intake closed, no terms on offer — pointing at this log. No new prices, no new tiers, and no reopening date are being implied, because none exist to imply.

The record. The withdrawn terms remain readable exactly as published: the corrections snapshot of 2026-08-10 is anchored in the public archive, and this log does not delete. Anyone who wants to check what was offered, and for how long, can.