How it works

The hard part is not connecting the systems. It is what happens when one of them breaks.

Any developer can post a form into a CRM. Integrations fall over on the second day, when the CRM times out, the provider re-sends the same webhook three times, and the same customer writes from two channels at once. This page is about those three cases.

• In plain words
• One event contract
• No duplicates
• Nothing lost
• Notifications
• Measured numbers
• What is not built

In plain words

The business result is short: a lead is never entered twice, and never disappears because a system was down. The sales rep sees one card per customer with the whole history, no matter which channel the person used.

One event contract for every channel

The website form, the Telegram bot, the phone system and partner sites all speak different formats. Each has a small adapter that converts its payload into one shape: who, how to reach them, in which language, from which campaign, plus channel-specific extras. Everything after the adapter is identical — which is why adding a fifth channel costs an hour, not a week.

Inbound URLs are stable and boring:

POST /hooks/website              # public form, no signature
POST /hooks/telephony            # PBX, signed
POST /hooks/telegram             # bot bridge, signed
POST /hooks/partner/<slug>       # other systems, signed, one secret each

Signed channels carry x-hub-signature: t=<unix>,v1=<hmac-sha256 of "t.body">. The timestamp is inside the signature, so a captured request cannot be replayed later — outside a 300-second window it is refused. The signature is checked against the raw bytes, before the JSON is parsed: re-serialising first would change key order and break the check.

A duplicate cannot become a second deal

Providers re-send. A phone system that does not get a fast answer will deliver the same call event again, sometimes three times. A person double-clicks Send. A partner retries after a timeout that only looked like a failure.

Every event gets an idempotency key, taken from the provider's own identifier where one exists:

• Phone call — call:<call_id> from the PBX
• Telegram — tg:<update_id>, unique by Telegram's own guarantee
• Partner — the event_id they send
• Web form — the Idempotency-Key header if the front end sends one, otherwise a hash of contact + message inside a 10-minute bucket, which is what catches double-clicks

The guarantee is a unique index in the database, not an if in the code:

CREATE UNIQUE INDEX uq_events_idem ON events(source, idem_key);

INSERT INTO events (...) VALUES (...)
ON CONFLICT(source, idem_key) DO NOTHING;

The difference matters under load. “Check whether it exists, then insert” has a gap between the two statements; two copies arriving in the same millisecond both pass the check and both insert. The database has no such gap. There is a test for exactly this: five identical webhooks fired in parallel, one accepted, four refused, one deal.

The refused copy is still written to the log — with status duplicate and a pointer to the original. “We drop duplicates” is a claim; a visible log line is proof.

The same person, not the same event

A second layer of de-duplication works on people. Phone numbers are normalised to E.164, so 067 000 17 42, +380670001742 and 00380670001742 are one contact. Emails are lower-cased and stripped of +tags; for Gmail, dots in the local part are removed too, because Gmail ignores them. Two more unique indexes make the database enforce it.

Matching stays deliberately narrow — only phone or email. Merging by name would glue two different customers together, and that error is far more expensive than a duplicate card.

Finally, a repeat request from a known contact within 72 hours joins the open deal instead of starting a new one, and bumps a touch counter. Once the deal is won or lost, the next request correctly starts a fresh one.

A lead survives the CRM being down

The webhook is acknowledged before the CRM is touched at all. The hub validates, stores the raw body and answers 202 in well under a millisecond. Only then does a worker deliver it onward.

This is not a shortcut, it is the requirement: providers treat a slow endpoint as a failed one and start their own retry storm. Fast acknowledgement plus a durable queue is the standard shape, and it is what makes the outage case boring.

When the CRM does not answer, the event moves to retrying with exponential backoff and jitter:

events       1s → 2s → 4s → 8s → 16s → 32s   (6 attempts, cap 45s)
deliveries   1.5s → 3s → 6s → 12s → 24s       (5 attempts, cap 30s)
jitter ±25%

The jitter is not decoration. Without it, everything that piled up during a one-minute outage hits the recovering CRM in the same millisecond and knocks it over again.

Retrying is also selective. A 5xx, a timeout, a 429 — those are worth repeating. A 422 is not: the payload is wrong and will still be wrong on the fifth attempt, so it goes straight to the dead-letter queue with its original body kept for a manual replay.

You can trigger this yourself. The “Take the CRM down” button on the panel injects real 503 responses at the CRM's REST boundary — the HTTP client genuinely fails, the retries are genuine. Measured on this stand: a 25-second outage, six attempts, the lead landed 33 seconds after it arrived. Zero events lost.

The notification cannot outlive the deal

A classic bug: the CRM write fails, but the “new lead” message was already sent, so the sales rep chases a deal that does not exist. Or the reverse — the deal is saved, the process restarts, and nobody is told.

Here the deal, the timeline entry and the queued notifications are written in one SQLite transaction (the transactional outbox pattern). Delivery happens afterwards from that queue. If the transaction rolls back, no message exists to send. If the process dies after it commits, the queue still holds the message and the worker picks it up on restart.

Each queued message has its own idempotency key, so a retried event cannot produce a second Telegram message about the same lead.

Measured, not estimated

Numbers below come from real requests to this stand (25 for the timings, 10 for the duplicate case), not from a local benchmark. You can repeat them; the ack time is returned in every response as ack_ms and as a Server-Timing header.

For the buyer the useful comparison is not milliseconds. It is this: a lead copied by hand from a mailbox into a CRM takes minutes to hours and is skipped entirely at night. The speed-to-lead literature is blunt about what that costs. This path takes single-digit milliseconds and does not sleep.

Tested offline, not by clicking

43 tests run without a network, in about 8 seconds. The ones that matter:

• the same webhook ten times in a row → one deal
• five identical webhooks in parallel → one accepted, one deal
• form, then a call from the same number in another format → one contact, one deal, two touches
• a contact with only a phone, later a form with an email → the email is added, no second contact
• CRM down → retries → the event lands after recovery
• CRM down past the attempt limit → status failed, original body kept
• a stale signature is refused; a tampered body is refused
• an unreachable notification target → retries → dead-letter queue → manual retry works
• an outbound webhook pointed at a private address is blocked

What is not built, and why

A demo that hides its gaps is not worth trusting, so here they are.

• No Telegram bot token on this stand yet. The bot client is written and polls the Bot API the moment a token is set. Until then the “Order in the bot” button sends the same signed webhook the bot would send, and the panel labels the channel bridge · no bot token rather than pretending.
• No SMTP relay on this server. Outgoing port 25 is blocked by the host and no relay account is configured yet. Emails are therefore queued with status held — visibly not sent. Marking them “delivered” would be a lie. When the relay is connected, one config line switches the sender and the queue flushes in order.
• The CRM here is mine, not KeyCRM. The stand needed an admin panel a guest could actually use, so the CRM is a small service of my own behind a REST API. The hub talks to it over HTTP exactly as it would to a vendor: same auth header, same timeouts, same retry logic. In production I have connected KeyCRM, SalesDrive, Bitrix24 and HubSpot through the same layer.
• SSRF protection is resolve-and-check. You can point the outgoing webhook at your own URL; private and link-local addresses are refused and redirects are not followed. A determined attacker could still win a DNS-rebinding race. For a demo with no internal network worth reaching, that trade-off is acceptable and it is written down here rather than hidden.
• The animation runs at a readable speed. Real hops take 2–40 ms, which is invisible. The packets move at a fixed pace; every number shown next to them is the real measurement.

Stack

Node 22, Fastify 5, TypeScript, SQLite (WAL). No frontend build step — the panel is vanilla ES modules and one 35 KB font. systemd units for the service, the 6-hour data reset and a 5-minute health check with a Telegram alert. About 4,300 lines of server code, 800 lines of tests, 43 test cases.

Want to see it break? Open the panel, press “Take the CRM down 25s”, then send a lead. Or send one from your own terminal — the commands are at the bottom of the panel.