🆕 Follow-up, 6:35 PM
Why it happened: the cancellation did register in Google. The app-side cleanup was switched off on purpose on Sept 1 as a safety hold while the cleanup job first shipped, applied by a secrets flush on Sept 2, and never switched back on.
- Fact (bottalk.md, Sept 1 and 2 entries): the deploy crew staged
GCAL_BOOKING_RECONCILER_MODE=reporton both Fly apps; the Platinum Gear Makers unpark flush applied it at 02:28 PT Sept 2, about 75 minutes after the reconciler had gone live armed. No later entry lifts it. - Inference (high): a hold with no owner and no expiry. Every Google cancellation since Sept 2 left a ghost row in the app; 18 had piled up by today.
- Shipped (commit 62673f0aa, trc-app v1115): directors get a push and email when a disarmed run finds ghosts, when a reconcile run aborts, or when a room's calendar sync errors; boot logs the mode; a ghost blocking a member slot writes a traceable warn line.
- Shipped (PR #1830): in production a disarm now needs an expiry date (
GCAL_BOOKING_RECONCILER_DISARM_UNTIL) or the cleanup auto-arms and alerts the directors; a room whose calendar sync keeps getting skipped for 24 hours alerts too. A Sept 1 style hold can no longer outlive its reason. - Net effect: a Google cancellation now clears the app row within about 15 minutes; if the cleanup is ever off, broken, or skipping a room, the directors hear about it the same day instead of a member finding out.
🎙️ Podcast Room, Oct 17: a ghost event was blocking it
A Google Calendar event called "Immersive Event" (12:00 to 23:00, all four rooms) was cancelled in Google on Sept 12, but the app still held it as a confirmed booking. The member hit that ghost. It is cleared, and the room shows open slots again.
🧾 What the member saw and why
- Fact. Booking row
e6841815: Podcast Room, 2026-10-17, 12:00 to 23:00, status confirmed, guest "Immersive Event", source google-calendar, imported July 20. - Fact. Google event
5u5jmvleon4jqufheo1ni2lm6con The Podcast Room calendar: status cancelled, last updated 2026-09-12 02:39 UTC. - Fact. You did not see it because the admin calendar views read Google, where it was already gone. The app's own bookings table still had it.
- Fact. Same ghost in the Live Room, Recording Room and Rehearsal Room for Oct 17. All four cleared.
🔍 Why the app never cleaned it up
- Fact. The reconciler that cancels app rows when Google deletes the event was shipped armed on Sept 1 and last wrote on Sept 2.
- Fact. The Fly secret
GCAL_BOOKING_RECONCILER_MODEwas set toreporton both apps, which disarms it. No record in memory, journal, or git says who set it or why (moderate confidence it was set around Sept 2, since that is the last live write). - Fact. In report mode it logged "would-cancel" to the console every 15 minutes and notified nobody.
🛠️ What I changed
- Cleared 18 ghost bookings whose Google event is definitively cancelled (each probed individually in Google). Rows set to cancelled, never deleted, with an audit row each in
gcal_booking_cancellations(source gregbot-manual). Backups on greg-mbp: Oct 17 rows, the other 14. - Re-armed the reconciler:
GCAL_BOOKING_RECONCILER_MODE=liveon trc-app and trc-admin (rolled 2026-10-07 01:00 UTC, trc-app v1114). From now on a Google delete clears the app row within about 15 minutes. - Notifications and error handling (in progress): a worker is shipping a PR so that a disarmed reconciler, any "would-cancel" it finds in report mode, a circuit-breaker trip, or a calendar sync error pushes and emails the directors (deduped), plus a boot log line that states the mode. Result will appear here as a 🆕 block.
The 18 ghosts cleared
| Date | Room(s) | Google title | Cancelled in Google |
|---|---|---|---|
| Oct 8 | Live | Jermey Event | Oct 5 |
| Oct 9 | Live | Cory Henry | Oct 5 |
| Oct 10 | Live | STUDIO BLOCK: Greg Spero Solo Piano & Wine | Oct 5 |
| Oct 15 | Rehearsal | BBC | Oct 6 |
| Oct 16 | All four | Immersive event block (19:00 to 23:00) | Sept 12 |
| Oct 17 | All four | Immersive Event (12:00 to 23:00) | Sept 12 |
| Oct 24 | All four | Hannah Event | Sept 29 |
| Oct 27 | Live | Music Tectonics x Rock Paper Scissors Event | Oct 5 |
| Nov 7 | Live | STUDIO BLOCK: Greg Spero Solo Piano & Wine | Oct 5 |
Oct 17 still has two real Live Room bookings (an 11-hour and a 20:00 one, both from Google) plus a 9 to 11 AM member session. Oct 18 JAMMJAM blocks on all rooms are still live in Google and were left alone.
How it was verified
- Production DB read via the pg driver; the 82 future google-calendar rows were diffed against a live listing of all four room calendars (247 events); 14 absent ids were each fetched by id and all returned status cancelled.
- After the fix:
GET /api/studios/studio-d/availability?date=2026-10-17returns slots 12:30 through 23:30. - Oct 17 booking rows remaining, not cancelled: Live Room 2 (plus the member session), other rooms 0.
❓ Unknown
- Which member hit it. No failed-attempt log exists for availability checks; the only recent Podcast Room activity is one member who booked and cancelled Oct 12, 13 and 14 slots. That is a guess, not evidence.