> ## Documentation Index > Fetch the complete documentation index at: https://docs.autosend.com/llms.txt > Use this file to discover all available pages before exploring further. # Retries and Replays > Learn how AutoSend automatically handles failed webhook deliveries. export const APP_PATHS = { home: '/', quickstart: '/quickstart', domainConfiguration: '/domain', apiReference: '/api-reference', sendEmail: '/api-reference/mails/send', bulkSendEmail: '/api-reference/mails/bulk', upsertContactApiRef: '/api-reference/contacts/upsert-contact', transactional: '/transactional-emails', emailActivity: '/transactional-emails/email-activity', emailTemplates: '/transactional-emails/email-templates', sendingEmail: '/quickstart/email-using-api', transactionalTroubleshooting: '/transactional-emails/troubleshooting', marketing: '/marketing-emails', campaigns: '/marketing-emails/campaigns', contacts: '/marketing-emails/contacts', contactsIntroduction: '/marketing-emails/contacts/introduction', contactsImportCsv: '/marketing-emails/contacts/import-csv', contactsLists: '/marketing-emails/contacts/lists', contactsSegments: '/marketing-emails/contacts/segments', contactsCustomFields: '/marketing-emails/contacts/custom-fields', sender: '/marketing-emails/sender', unsubscribeGroups: '/others/unsubscribe-groups', webhookIntroduction: '/others/webhooks/introduction', webhookEventType: '/others/webhooks/event-type', webhookRetries: '/others/webhooks/retries', webhookVerifyRequests: '/others/webhooks/verify-requests', dynamicTemplates: '/dynamic-templates', guides: '/guides', sitemap: '/sitemap.xml', team: '/others/team', automations: '/automations', smtpIntroduction: '/quickstart/smtp', betterAuth: '/guides/better-auth', templateVariables: '/transactional-emails/variables', suppressions: '/others/suppressions', rateLimit: '/api-reference/rate-limit', nodejsSdk: '/sdk/nodejs', smtpIntegrationGuides: '/guides/smtp', apiKeys: '/api-keys', apiReferenceIntroduction: '/api-reference/introduction', lovableGuide: '/ai/integrations/lovable', aiIntroduction: '/ai/introduction', aiSkills: '/ai/skills', aiMcpServer: '/ai/mcp-server', aiLovable: '/ai/integrations/lovable', aiBolt: '/ai/integrations/bolt', aiV0: '/ai/integrations/v0', aiReplit: '/ai/integrations/replit', mcpClaude: '/ai/mcp-clients/claude', mcpCursor: '/ai/mcp-clients/cursor', mcpCopilot: '/ai/mcp-clients/copilot', mcpWindsurf: '/ai/mcp-clients/windsurf', mcpCodex: '/ai/mcp-clients/codex', mcpAntigravity: '/ai/mcp-clients/antigravity', mcpChatgpt: '/ai/mcp-clients/chatgpt', mcpRaycast: '/ai/mcp-clients/raycast', domainWarmup: '/marketing-emails/domain-warmup', projects: '/projects', createAutomationApi: '/api-reference/automations/create-automation' }; export const AUTOSEND_PATHS = { dashboard: 'https://autosend.com/dashboard', apiKey: 'https://autosend.com/account/api-key', faqs: 'https://autosend.com/faq', marketingEmails: 'https://autosend.com/marketing-emails', webhooks: 'https://autosend.com/webhooks', composeByAutoSend: 'https://autosend.com/compose', emailActivity: 'https://autosend.com/email-activities', team: 'https://autosend.com/settings/team', pricing: 'https://autosend.com/pricing', verifyEmail: 'https://autosend.com/compose/email-builder?template=verify-email', welcomeEmail: 'https://autosend.com/compose/email-builder?template=welcome-email', productUpdate: 'https://autosend.com/compose/email-builder?template=product-update', newsletter: 'https://autosend.com/compose/email-builder?template=newsletter', automations: 'https://autosend.com/automations', globalSuppressions: 'https://autosend.com/suppressions/global', signup: 'https://autosend.com/signup', domains: 'https://autosend.com/settings/domains', logoKit: 'https://asend.email/logo', contactsPage: 'https://autosend.com/contacts/list-and-segments' }; ## Automatic Retries AutoSend automatically retries webhook deliveries that fail due to network errors, timeouts, or non-2xx status codes from your endpoint. ### Retry Schedule If AutoSend does not receive a 2xx (200-299) response from your webhook endpoint, we will retry the webhook delivery using an **exponential backoff strategy**. | Attempt | Approximate Delay After Previous Failure | | --------- | ---------------------------------------- | | 1st Retry | \~5 seconds | | 2nd Retry | \~10 seconds | | 3rd Retry | \~20 seconds | **Total Retry Attempts**: 3 retries (4 total delivery attempts including the initial attempt) **Backoff Strategy**: Exponential backoff starting with a 5-second delay, doubling with each retry **Request Timeout**: Each delivery attempt will timeout after **10 seconds** if no response is received ### Example Timeline ``` Initial Attempt: 10:00:00 - Failed (500 Internal Server Error) 1st Retry: 10:00:05 - Failed (Timeout after 10s) 2nd Retry: 10:00:15 - Failed (503 Service Unavailable) 3rd Retry: 10:00:35 - Success (200 OK) ``` *** ## When Retries Occur ### Retry Triggers AutoSend will retry webhook deliveries when: * **Non-2xx status codes** are returned (400, 401, 403, 404, 500, 502, 503, 504, etc.) * **Network errors** occur (connection refused, DNS resolution failure, etc.) * **Timeouts** happen (no response within 10 seconds) * **SSL/TLS errors** are encountered * **Request aborted** due to timeout ### No Retry for Success If your endpoint returns any 2xx status code (200-299), the delivery is marked as successful and no retries will occur. ```javascript theme={null} theme={null} // ✅ These responses mark delivery as successful (no retry) res.status(200).json({ received: true }); res.status(201).json({ queued: true }); res.status(202).json({ accepted: true }); // ❌ These responses trigger retries res.status(400).json({ error: 'Bad request' }); res.status(401).json({ error: 'Unauthorized' }); res.status(500).json({ error: 'Internal error' }); res.status(503).json({ error: 'Service unavailable' }); ``` *** ## Webhook Headers Every webhook delivery includes these headers: | Header | Description | Example | | ----------------------- | ------------------------------------------------- | ------------------------ | | `Content-Type` | Always `application/json` | `application/json` | | `X-Webhook-Signature` | HMAC-SHA256 signature for verification | `a1b2c3d4e5f6...` | | `X-Webhook-Event` | The event type being delivered | `email.sent` | | `X-Webhook-Delivery-Id` | Unique ID for this delivery (same across retries) | `deliver-webhook-123456` | | `X-Webhook-Timestamp` | Unix timestamp (milliseconds) when sent | `1699876543210` | **Important**: The `X-Webhook-Delivery-Id` remains the same across all retry attempts for the same webhook event, making it perfect for implementing idempotency. *** ## After All Retries Fail After the conclusion of all retry attempts (initial + 3 retries = 4 total attempts), if the webhook still hasn't been delivered successfully, the delivery will be marked as **failed** in the system. ### What Happens Next * The delivery log will show the final failure status * The webhook's `failureCount` is incremented * The webhook's `lastFailedAt` timestamp is updated * Failed delivery records are kept for **48 hours** for debugging * AutoSend will **not** automatically retry it again ### Webhook Auto-Disable (Optional) By default, webhooks are NOT automatically disabled after consecutive failures. However, the system tracks: * `failureCount`: Number of consecutive delivery failures * `MAX_FAILURES` constant: Set to 5 (currently not enforced but available for future use) You can monitor these metrics to manually disable problematic webhooks. *** ## Delivery Logs ### Log Retention AutoSend keeps delivery attempt logs for debugging: * **Successful deliveries**: Retained for **24 hours** * **Failed deliveries**: Retained for **48 hours** * **Maximum completed jobs kept**: 1,000 most recent ### Log Information Each delivery log includes: * Webhook ID and organization/project IDs * Event type * Full payload sent * Destination URL * HTTP status code * Response body and headers * Success/failure status * Error message (if failed) * Number of attempts made * Duration of the request (in milliseconds) * Timestamp of the delivery attempt *** ## Monitoring Your Webhooks ### Check Webhook Status Monitor your webhooks from the AutoSend dashboard: * **Active** (green) - Webhook is enabled and functional * **Inactive** (gray) - Webhook is disabled * **Disabled** (red) - Webhook has been disabled due to issues ### Webhook Health Metrics Each webhook tracks important metrics: | Metric | Description | | ----------------- | ----------------------------------------- | | `failureCount` | Number of consecutive delivery failures | | `lastSuccessAt` | Timestamp of last successful delivery | | `lastFailedAt` | Timestamp of last failed delivery | | `lastDeliveredAt` | Timestamp of last delivery attempt (any) | | `isActive` | Whether the webhook is enabled | | `status` | Current status (active/inactive/disabled) | These metrics are updated automatically: * **On success**: `failureCount` resets to 0, `lastSuccessAt` and `lastDeliveredAt` are updated * **On failure**: `failureCount` increments by 1, `lastFailedAt` is updated ### Implement Your Own Monitoring Set up your own monitoring for webhook health: ```javascript Nodejs expandable theme={null} theme={null} // Example: Check webhook health periodically async function monitorWebhookHealth() { // Fetch webhook delivery logs from your database const recentDeliveries = await db.webhookDeliveryLogs.find({ createdAt: { $gte: new Date(Date.now() - 3600000) }, // Last hour }); const failureCount = recentDeliveries.filter((d) => !d.success).length; const successCount = recentDeliveries.filter((d) => d.success).length; const totalDeliveries = recentDeliveries.length; const successRate = totalDeliveries > 0 ? (successCount / totalDeliveries) * 100 : 100; // Calculate average response time const avgDuration = recentDeliveries.reduce((sum, d) => sum + d.duration, 0) / totalDeliveries; console.log({ totalDeliveries, successCount, failureCount, successRate: `${successRate.toFixed(2)}%`, avgResponseTime: `${avgDuration.toFixed(0)}ms`, }); // Alert if success rate drops if (successRate < 90) { await sendAlert({ title: 'Webhook Health Alert', message: `Webhook success rate dropped to ${successRate.toFixed(2)}%`, details: { successCount, failureCount, totalDeliveries, }, severity: 'warning', }); } // Alert if response time is slow if (avgDuration > 5000) { await sendAlert({ title: 'Webhook Performance Alert', message: `Average webhook response time is ${avgDuration.toFixed(0)}ms`, severity: 'warning', }); } } // Run every 5 minutes setInterval(monitorWebhookHealth, 5 * 60 * 1000); ``` *** ## Best Practices Always return a 2xx status code when your endpoint successfully receives and processes a webhook: ```javascript Nodejs expandable theme={null} theme={null} app.post("/webhooks/autosend", async (req, res) => { try { // Verify signature first const signature = req.headers["x-webhook-signature"]; const body = JSON.stringify(req.body); if (!verifySignature(signature, body, WEBHOOK_SECRET)) { // Return 401 for invalid signature (will retry) // Or return 200 to prevent retries for invalid signatures return res.status(401).json({ error: "Invalid signature" }); } // Queue for background processing await queue.add("process-webhook", { deliveryId: req.headers["x-webhook-delivery-id"], event: req.body.type, data: req.body.data, }); // Return 200 immediately - don't wait for processing res.status(200).json({ received: true }); } catch (error) { console.error("Webhook processing error:", error); // Return 500 for temporary errors (will retry) res.status(500).json({ error: "Internal error" }); } }); ``` Use the `X-Webhook-Delivery-Id` header to prevent duplicate processing during retries: ```javascript Nodejs expandable theme={null} theme={null} // Using Redis for idempotency tracking const redis = require("redis"); const client = redis.createClient(); app.post("/webhooks/autosend", async (req, res) => { const deliveryId = req.headers["x-webhook-delivery-id"]; const ttl = 86400; // 24 hours // Check if already processed const exists = await client.exists(`webhook:${deliveryId}`); if (exists) { console.log(`Duplicate delivery ${deliveryId}, skipping`); return res.status(200).json({ received: true, duplicate: true }); } try { // Process the webhook await processWebhook(req.body); // Mark as processed (with TTL to auto-cleanup) await client.setex(`webhook:${deliveryId}`, ttl, "processed"); res.status(200).json({ received: true }); } catch (error) { // Don't mark as processed on error so it can be retried console.error("Processing failed:", error); res.status(500).json({ error: "Processing failed" }); } }); ``` **Alternative: Database-based idempotency** ```javascript Nodejs expandable theme={null} theme={null} app.post("/webhooks/autosend", async (req, res) => { const deliveryId = req.headers["x-webhook-delivery-id"]; try { // Try to insert the delivery ID (unique constraint) await db.webhookDeliveries.insertOne({ deliveryId, receivedAt: new Date(), processed: false, }); } catch (error) { if (error.code === 11000) { // Duplicate key - already processed console.log(`Duplicate delivery ${deliveryId}`); return res.status(200).json({ received: true, duplicate: true }); } throw error; } try { await processWebhook(req.body); // Mark as processed await db.webhookDeliveries.updateOne( { deliveryId }, { $set: { processed: true, processedAt: new Date() } } ); res.status(200).json({ received: true }); } catch (error) { console.error("Processing failed:", error); res.status(500).json({ error: "Processing failed" }); } }); ``` Return appropriate status codes based on the type of error: ```javascript Nodejs expandable theme={null} theme={null} app.post("/webhooks/autosend", async (req, res) => { try { // Verify signature const signature = req.headers["x-webhook-signature"]; const isValid = verifySignature(signature, req.body, WEBHOOK_SECRET); if (!isValid) { // Invalid signature is a permanent error // Return 200 to prevent unnecessary retries return res.status(200).json({ error: "Invalid signature", retryable: false }); } // Process webhook await processWebhook(req.body); res.status(200).json({ received: true }); } catch (error) { console.error("Webhook error:", error); // Determine if error is temporary or permanent if (error.code === "ECONNREFUSED" || error.code === "ETIMEDOUT") { // Temporary database/service error - retry return res.status(500).json({ error: "Service temporarily unavailable", retryable: true }); } else if (error.name === "ValidationError") { // Permanent error - bad data, don't retry return res.status(200).json({ error: "Invalid data format", retryable: false }); } else { // Unknown error - retry to be safe return res.status(500).json({ error: "Internal error", retryable: true }); } } }); ``` Your endpoint MUST respond within 10 seconds or the request will timeout. Process webhooks asynchronously: ```javascript Nodejs expandable theme={null} theme={null} const Queue = require("bull"); const webhookQueue = new Queue("webhooks", { redis: { host: "localhost", port: 6379 }, }); // ❌ Bad - Synchronous processing (may timeout) app.post("/webhooks/autosend", async (req, res) => { await updateDatabase(req.body); // 2 seconds await sendToAnalytics(req.body); // 3 seconds await notifySlack(req.body); // 2 seconds await triggerWorkflow(req.body); // 4 seconds // Total: 11 seconds - WILL TIMEOUT! res.status(200).json({ received: true }); }); // ✅ Good - Asynchronous processing app.post("/webhooks/autosend", async (req, res) => { const deliveryId = req.headers["x-webhook-delivery-id"]; // Queue for background processing (fast!) await webhookQueue.add("process", { deliveryId, event: req.body.type, data: req.body.data, }); // Respond immediately (< 100ms) res.status(200).json({ received: true }); }); // Process in background worker webhookQueue.process("process", async (job) => { const { deliveryId, event, data } = job.data; console.log(`Processing webhook ${deliveryId} for event ${event}`); await updateDatabase(data); await sendToAnalytics(data); await notifySlack(data); await triggerWorkflow(data); console.log(`Completed processing webhook ${deliveryId}`); }); ``` Keep detailed logs of all webhook delivery attempts for debugging: ```javascript Nodejs expandable theme={null} theme={null} const winston = require("winston"); const logger = winston.createLogger({ level: "info", format: winston.format.json(), transports: [ new winston.transports.File({ filename: "webhooks.log" }), new winston.transports.Console(), ], }); app.post("/webhooks/autosend", async (req, res) => { const deliveryId = req.headers["x-webhook-delivery-id"]; const event = req.body.type; const timestamp = req.headers["x-webhook-timestamp"]; logger.info("Webhook received", { deliveryId, event, timestamp, receivedAt: new Date().toISOString(), }); try { await processWebhook(req.body); logger.info("Webhook processed successfully", { deliveryId, event, processingTime: Date.now() - parseInt(timestamp), }); res.status(200).json({ received: true }); } catch (error) { logger.error("Webhook processing failed", { deliveryId, event, error: error.message, stack: error.stack, }); res.status(500).json({ error: "Processing failed" }); } }); ``` Test how your endpoint handles retries in development: ```javascript Nodejs expandable theme={null} theme={null} // Simulate intermittent failures for testing const deliveryAttempts = new Map(); app.post("/webhooks/autosend", async (req, res) => { const deliveryId = req.headers["x-webhook-delivery-id"]; // Track attempts for this delivery const attempts = (deliveryAttempts.get(deliveryId) || 0) + 1; deliveryAttempts.set(deliveryId, attempts); console.log(`Delivery ${deliveryId} - Attempt ${attempts}`); // Simulate: Fail first 2 attempts, succeed on 3rd if (attempts <= 2) { console.log(`Simulating failure on attempt ${attempts}`); return res.status(500).json({ error: "Simulated error" }); } console.log(`Success on attempt ${attempts}`); // Process webhook await processWebhook(req.body); // Clean up tracking deliveryAttempts.delete(deliveryId); res.status(200).json({ received: true, attempts }); }); ``` AutoSend processes up to 5 webhooks concurrently. Ensure your endpoint can handle concurrent requests: ```javascript Nodejs expandable theme={null} theme={null} const express = require("express"); const cluster = require("cluster"); const os = require("os"); if (cluster.isMaster) { // Fork workers (one per CPU core) const numCPUs = os.cpus().length; console.log(`Master process starting ${numCPUs} workers`); for (let i = 0; i < numCPUs; i++) { cluster.fork(); } cluster.on("exit", (worker) => { console.log(`Worker ${worker.process.pid} died, starting new worker`); cluster.fork(); }); } else { // Worker process const app = express(); app.use(express.json()); app.post("/webhooks/autosend", async (req, res) => { try { await processWebhook(req.body); res.status(200).json({ received: true }); } catch (error) { res.status(500).json({ error: "Processing failed" }); } }); app.listen(3000, () => { console.log(`Worker ${process.pid} listening on port 3000`); }); } ``` *** ## Troubleshooting **Symptoms**: Webhooks failing frequently **Possible Causes**: * Endpoint is down or unreachable * Endpoint is timing out (>10 seconds) * Endpoint is returning non-2xx status codes * SSL/TLS certificate issues * Rate limiting on your server **Solutions**: 1. **Test your endpoint manually**: ```bash theme={null} theme={null} curl -X POST https://your-endpoint.com/webhooks/autosend \ -H "Content-Type: application/json" \ -H "X-Webhook-Signature: test_signature" \ -H "X-Webhook-Event: email.sent" \ -H "X-Webhook-Delivery-Id: test-123" \ -H "X-Webhook-Timestamp: $(date +%s)000" \ -d '{ "type": "email.sent", "createdAt": "2025-01-05T10:00:00.000Z", "data": { "emailId": "test123", "from": "test@example.com", "to": {"email": "user@example.com", "name": "Test User"}, "subject": "Test Email" } }' ``` 2. **Check your server logs** for error messages 3. **Verify SSL certificate is valid**: ```bash theme={null} theme={null} openssl s_client -connect your-endpoint.com:443 -servername your-endpoint.com ``` 4. **Check response time**: ```bash theme={null} theme={null} time curl -X POST https://your-endpoint.com/webhooks/autosend \ -H "Content-Type: application/json" \ -d '{"type":"test","data":{}}' ``` 5. **Monitor your server resources** (CPU, memory, disk) to ensure it's not overloaded 6. **Check for rate limiting** on your server or firewall **Symptoms**: All webhooks returning 401 or failing signature verification **Possible Causes**: * Using wrong webhook secret * Incorrect signature verification logic * Body parsing issues (modified body) * Character encoding issues **Solutions**: See Verify Webhook Requests for detailed troubleshooting. **Quick verification test**: ```javascript theme={null} theme={null} const crypto = require("crypto"); function verifyWebhookSignature(signature, body, secret) { const expectedSignature = crypto .createHmac("sha256", secret) .update(body) .digest("hex"); console.log("Received signature:", signature); console.log("Expected signature:", expectedSignature); console.log("Match:", signature === expectedSignature); return signature === expectedSignature; } app.post("/webhooks/autosend", express.raw({ type: "application/json" }), (req, res) => { const signature = req.headers["x-webhook-signature"]; const body = req.body.toString(); // Raw body as string if (!verifyWebhookSignature(signature, body, WEBHOOK_SECRET)) { return res.status(401).json({ error: "Invalid signature" }); } // Parse body after verification const data = JSON.parse(body); res.status(200).json({ received: true }); }); ``` **Symptoms**: Webhooks timing out (10 second timeout) **Possible Causes**: * Synchronous processing taking too long * Database queries are slow * External API calls are slow * No connection pooling * Inefficient code **Solutions**: 1. **Use background job queues**: ```javascript theme={null} theme={null} const Queue = require("bull"); const webhookQueue = new Queue("webhooks"); app.post("/webhooks/autosend", async (req, res) => { // Queue immediately (fast) await webhookQueue.add(req.body, { attempts: 3, backoff: { type: "exponential", delay: 2000 }, }); // Respond fast res.status(200).json({ received: true }); }); // Process in background webhookQueue.process(async (job) => { const { type, data } = job.data; // Time-consuming operations here await updateDatabase(data); await callExternalAPI(data); await generateReport(data); }); ``` 2. **Optimize database queries**: ```javascript theme={null} theme={null} // ❌ Slow - Sequential queries const user = await db.users.findOne({ email: data.to.email }); const campaign = await db.campaigns.findOne({ _id: data.campaignId }); const template = await db.templates.findOne({ _id: data.templateId }); // ✅ Fast - Parallel queries const [user, campaign, template] = await Promise.all([ db.users.findOne({ email: data.to.email }), db.campaigns.findOne({ _id: data.campaignId }), db.templates.findOne({ _id: data.templateId }), ]); ``` 3. **Add database indexes**: ```javascript theme={null} theme={null} // Create indexes on frequently queried fields db.webhookDeliveries.createIndex({ deliveryId: 1 }, { unique: true }); db.webhookDeliveries.createIndex({ createdAt: -1 }); db.webhookLogs.createIndex({ webhookId: 1, createdAt: -1 }); ``` 4. **Use connection pooling**: ```javascript theme={null} theme={null} const mongoose = require("mongoose"); // Configure connection pool mongoose.connect(MONGODB_URI, { poolSize: 10, socketTimeoutMS: 45000, family: 4, }); ``` 5. **Cache frequently accessed data**: ```javascript theme={null} theme={null} const NodeCache = require("node-cache"); const cache = new NodeCache({ stdTTL: 600 }); // 10 minute TTL async function getWebhookConfig(webhookId) { // Check cache first let config = cache.get(`webhook:${webhookId}`); if (!config) { // Fetch from database config = await db.webhooks.findOne({ _id: webhookId }); cache.set(`webhook:${webhookId}`, config); } return config; } ``` **Symptoms**: Same webhook processed multiple times **Possible Causes**: * Not implementing idempotency * Retry logic processing same delivery multiple times * Race conditions in distributed systems **Solutions**: Implement idempotency using `X-Webhook-Delivery-Id` (see Best Practices section above). **Symptoms**: Not receiving expected webhooks **Possible Causes**: * Webhook not configured for the event type * Webhook is inactive or disabled * Firewall blocking incoming requests * Incorrect URL configured **Solutions**: 1. **Check webhook configuration** in AutoSend dashboard 2. **Verify event types** are selected for the webhook 3. **Check webhook is active** (not disabled) 4. **Test webhook URL** is accessible from external networks 5. **Check firewall rules** allow incoming traffic on your endpoint port 6. **Review AutoSend delivery logs** for delivery attempts *** ## Related Resources Getting started with webhooks {' '} Complete list of webhook events {' '} Security and signature verification Manage your webhooks from the AutoSend sidebar