Arrival: 10.0 jobs / min
Measured from the creation timestamps of the jobs currently queued — sustained, with peaks of 45 in a single minute.
Production incident · 11 Aug 2026
We tripled the webhook load at 06:34 this morning
SugarCRM delivered portal webhooks in 1–4 minutes for eight straight days. At 06:34 today three new Web Logic Hooks were added, taking fan-out on surrogate leads from one hook to three. Within ninety minutes delivery was 110 minutes behind and the queue has grown ever since. Sugar did not change. Its capacity did not drop. We put three times the traffic through it.
Where it stands
Production, circlesurrogacy.sugarondemand.com, 18:36 UTC on 11 Aug 2026.
- 110min
- Delivery lag, and rising
- 1,113
- Dispatches queued
- +122/hr
- Net queue growth
- 0.03%
- Job failure rate
What actually happened
Four facts, each read directly from Sugar’s own tables. Together they leave one explanation.
- 1
Until this morning there was exactly ONE webhook
Sugar’s
WebLogicHooksmodule held a single record: Send New Lead to Stape.io, created 20 March by Frank Klemovitch. One dispatch job per surrogate-lead save. That is the configuration that ran for the previous eight days. - 2
At 06:33–06:34 three more were created
P2 Surrogate Lead Created and P2 Surrogate Lead Updated (both on
sl1_surrogate_leads), plus P2 Lead Debug on the stockLeadsmodule — which had never generated a dispatch before. Fan-out on surrogate leads went 1 → 3 in under a minute, and a fourth traffic source appeared from nothing. - 3
The queue crossed its capacity line at 14:00 UTC
The 13:00 hour carried 252 dispatches at a 4.5-minute wait. The 14:00 hour carried 955 — four times the busiest hour of the preceding week — and the median wait jumped to 42 minutes. It has not recovered since: 82 minutes, then 96, then 103.
- 4
The arithmetic was already decided at 06:34
Sugar clears about 480 jobs an hour. The old single-hook load peaked near 230/hr — 48% utilisation, comfortable, and exactly why nobody had ever looked at this queue. Triple it and you get ~690/hr, or 144% of capacity. A queue served slower than it is filled does not stabilise; it grows until the input stops.
Why the system had no room to absorb it
The headroom that looked comfortable at one hook was never as large as it appeared.
Capacity: 8.0 jobs / min
Sugar processes a fixed batch of 25 jobs per cron cycle, and cycles begin roughly every 3.05 minutes. 25 ÷ 3.05 = 8.0, regardless of how much is waiting.
The runner is idle 85% of the time
Each cycle finishes its 25 jobs in about 27 seconds, then does nothing for the remaining three minutes. This is a scheduling cadence, not a hardware, CPU or licence limit — which is why the ceiling is so low despite the machine being almost entirely unused.
Nothing is failing
One failure in 3,000 completed jobs (0.03%). No retry storm, no timeouts, no misconfiguration at either endpoint. This is purely a throughput deficit.
One save is not one webhook
The multiplier is the entire incident, and it is the thing we control.
- 1 → 3
- Hooks per surrogate-lead save, as of 06:34
- 4.4×
- Mean dispatches per record saved
- 36
- Most dispatches queued for one record
- 13.2%
- Saves that changed only “last modified”
What is in the queue
The 1,000 oldest waiting dispatches by destination, sampled before the debug hook was removed. Three of these four were created this morning.
29.2% · tracking tag
Server-side tracking at sst.circlesurrogacy.com. The only pre-existing hook — this was the entire load until today.
29.2% · lead status
sugar-surrogate-lead-status. New today. Fires on every surrogate-lead save.
29.1% · lead created
sugar-surrogate-lead-created. New today. Fires on the same save, for the same record, as the one to its left.
12.5% · debug capture
sugar-lead-debug on the Leads module. New today, capture-only, no business function. Removed from production on 11 Aug.
The fix
Undo the load we added. Everything else is secondary, and nothing here needs Sugar to change.
- 1
1 · Remove the debug hook — DONE, 11 Aug
P2 Lead Debughas been deleted from production Sugar. It was capture-only with no authentication — its URL carried no key — and it wrote full lead payloads and every request header into our traces. Effect: −12.5% of dispatches. It does not fix the lag on its own (arrival stays above capacity), but it was pure waste and a live PII exposure. - 2
2 · Collapse the two surrogate-lead hooks into one Sugar call
This is the one that restores health.
sugar-surrogate-lead-createdandsugar-surrogate-lead-statusare two Sugar hooks firing on the same save of the same record. Sugar charges its queue per hook; our side can fan out internally for free. Effect: −29% of dispatches, arrival 10.0 → ~5.9/min — below the 8.0 ceiling, so the backlog drains and stops recurring. - 3
3 · Do it with a `delegate` action, not a copy-paste merge
The two hooks cannot simply be pasted together. Sugar sends both byte-identical
after_savepayloads — there is no create-vs-update distinction at its end. The difference lives entirely in ouridempotency:block: created is keyed on the Sugar record id so an account is provisioned exactly once, while status is deliberately not idempotent so it catches every transition. Idempotency is hook-level, so one merged file structurally cannot be both, and a naive merge silently breaks one. Adelegateaction — one Sugar hook, fanning out to N named inbound webhooks that each keep their own semantics — solves it properly and makes every future Sugar-driven feature cost zero additional queue load. - 4
4 · Ask Northstar whether the tracking tag can batch or filter
sst.circlesurrogacy.com/sugarcrm_eventsis 29.2% of the queue and is not a portal hook. It was, however, the only load before today and the system was healthy with it — so it is an optimisation, not a cause. Owner: Northstar. - 5
5 · Then, separately, ask Brainsell about cadence
A runner that is idle 85% of the time and still caps at 8 jobs/min has no headroom for anything. Moving the cycle from ~3 minutes to 1 gives 25 jobs/min — 3.1× for free. Worth having. But raise it after our own load is corrected, never instead of it — asking a vendor to triple capacity because we tripled our traffic this morning is not a request that survives being checked.
What that adds up to
Step 1 is done and takes arrival from 10.0 to roughly 8.8 dispatches per minute — still above the 8.0 ceiling, so the queue keeps growing, more slowly. On its own it is not enough, which is precisely why it was never the headline.
Step 2 is the one that matters: it removes a further ~29%, putting arrival at ~5.9/min against 8.0 capacity. That is a surplus of roughly 126 jobs an hour, the existing backlog clears in about three hours, and the configuration is then permanently inside its own limits.
Step 5 is genuine headroom and worth pursuing on its own merits — a system running at 8.0 against 10.0 has none. It is listed last deliberately. The capacity was adequate this morning and it will be adequate again the moment our fan-out is corrected.
Method and pushback
How confident are you in the root cause?+
The hook creation timestamps come from Sugar’s WebLogicHooks records (06:33:44, 06:34:29, 06:34:58 PDT) and the wait-time collapse from 12,000 completed dispatch jobs going back to 3 August. Eight days at 1–4 minutes, then a 4× volume hour beginning within the hour of the change, then no recovery. The predicted load from tripling fan-out (~690/hr against 480/hr capacity) matches the observed sustained arrival. It is a correlation with a mechanism and matching arithmetic, not a coincidence.
Is this Sugar’s fault, or the geocoder’s?+
Neither. Of 3,000 jobs that ran in a 7.2-hour window, 76% were webhook dispatches; both geocoding schedulers together ran nine times, using 0.01% of runner time. Measured by duration rather than count, all non-webhook jobs are 23% of busy time and busy time is 15% of the clock — about 3.5% overall. They are victims of the backlog, each with a job stuck over 90 minutes.
Why didn’t the usual check catch it?+
“When did we last receive a webhook?” is structurally incapable of seeing a backlog — a saturated queue still drains continuously, so the newest arrival is always seconds old. That probe reported “lag effectively zero” during a real 53-minute backlog. The only honest measure is now minus the scheduled time of the oldest job still waiting.
Where do the numbers come from?+
Sugar’s SchedulersJobs and WebLogicHooks modules over REST, read directly from production — not from our logs or traces. Depth is Sugar’s server-side /count, not a row count, because a REST page stops at 1,000 and the backlog is past that. Capacity comes from grouping completed jobs by the cron process that executed them: 41 consecutive processes, no overlapping windows, 25 jobs each.
Can we unblock one specific person before the queue drains?+
Yes. Delivery is idempotent on the Sugar record id, so replaying a stuck record is safe. It does not help in aggregate — a replay adds a job to the same saturated queue.