Retries and Replays

Automatic Retries

AutoSend automatically retries webhook deliveries that fail due to network errors, timeouts, or non-2xx status codes from your endpoint.

Retry Schedule

If AutoSend does not receive a 2xx (200-299) response from your webhook endpoint, we will retry the webhook delivery using an exponential backoff strategy.

Attempt	Approximate Delay After Previous Failure
1st Retry	~5 seconds
2nd Retry	~10 seconds
3rd Retry	~20 seconds

Total Retry Attempts: 3 retries (4 total delivery attempts including the initial attempt) Backoff Strategy: Exponential backoff starting with a 5-second delay, doubling with each retry Request Timeout: Each delivery attempt will timeout after 10 seconds if no response is received

Example Timeline

Initial Attempt:  10:00:00 - Failed (500 Internal Server Error)
1st Retry:        10:00:05 - Failed (Timeout after 10s)
2nd Retry:        10:00:15 - Failed (503 Service Unavailable)
3rd Retry:        10:00:35 - Success (200 OK)

When Retries Occur

Retry Triggers

AutoSend will retry webhook deliveries when:

Non-2xx status codes are returned (400, 401, 403, 404, 500, 502, 503, 504, etc.)
Network errors occur (connection refused, DNS resolution failure, etc.)
Timeouts happen (no response within 10 seconds)
SSL/TLS errors are encountered
Request aborted due to timeout

No Retry for Success

If your endpoint returns any 2xx status code (200-299), the delivery is marked as successful and no retries will occur.

// ✅ These responses mark delivery as successful (no retry)
res.status(200).json({ received: true });
res.status(201).json({ queued: true });
res.status(202).json({ accepted: true });

// ❌ These responses trigger retries
res.status(400).json({ error: 'Bad request' });
res.status(401).json({ error: 'Unauthorized' });
res.status(500).json({ error: 'Internal error' });
res.status(503).json({ error: 'Service unavailable' });

Webhook Headers

Every webhook delivery includes these headers:

Header	Description	Example
`Content-Type`	Always `application/json`	`application/json`
`X-Webhook-Signature`	HMAC-SHA256 signature for verification	`a1b2c3d4e5f6...`
`X-Webhook-Event`	The event type being delivered	`email.sent`
`X-Webhook-Delivery-Id`	Unique ID for this delivery (same across retries)	`deliver-webhook-123456`
`X-Webhook-Timestamp`	Unix timestamp (milliseconds) when sent	`1699876543210`

Important: The X-Webhook-Delivery-Id remains the same across all retry attempts for the same webhook event, making it perfect for implementing idempotency.

After All Retries Fail

After the conclusion of all retry attempts (initial + 3 retries = 4 total attempts), if the webhook still hasn’t been delivered successfully, the delivery will be marked as failed in the system.

What Happens Next

The delivery log will show the final failure status
The webhook’s failureCount is incremented
The webhook’s lastFailedAt timestamp is updated
Failed delivery records are kept for 48 hours for debugging
AutoSend will not automatically retry it again

Webhook Auto-Disable (Optional)

By default, webhooks are NOT automatically disabled after consecutive failures. However, the system tracks:

failureCount: Number of consecutive delivery failures
MAX_FAILURES constant: Set to 5 (currently not enforced but available for future use)

You can monitor these metrics to manually disable problematic webhooks.

Delivery Logs

Log Retention

AutoSend keeps delivery attempt logs for debugging:

Successful deliveries: Retained for 24 hours
Failed deliveries: Retained for 48 hours
Maximum completed jobs kept: 1,000 most recent

Log Information

Each delivery log includes:

Webhook ID and organization/project IDs
Event type
Full payload sent
Destination URL
HTTP status code
Response body and headers
Success/failure status
Error message (if failed)
Number of attempts made
Duration of the request (in milliseconds)
Timestamp of the delivery attempt

Monitoring Your Webhooks

Check Webhook Status

Monitor your webhooks from the AutoSend dashboard:

View your webhooks with their current status

Active (green) - Webhook is enabled and functional
Inactive (gray) - Webhook is disabled
Disabled (red) - Webhook has been disabled due to issues

Webhook Health Metrics

Each webhook tracks important metrics:

Metric	Description
`failureCount`	Number of consecutive delivery failures
`lastSuccessAt`	Timestamp of last successful delivery
`lastFailedAt`	Timestamp of last failed delivery
`lastDeliveredAt`	Timestamp of last delivery attempt (any)
`isActive`	Whether the webhook is enabled
`status`	Current status (active/inactive/disabled)

These metrics are updated automatically:

On success: failureCount resets to 0, lastSuccessAt and lastDeliveredAt are updated
On failure: failureCount increments by 1, lastFailedAt is updated

Implement Your Own Monitoring

Set up your own monitoring for webhook health:

Nodejs

// Example: Check webhook health periodically
async function monitorWebhookHealth() {
	// Fetch webhook delivery logs from your database
	const recentDeliveries = await db.webhookDeliveryLogs.find({
		createdAt: { $gte: new Date(Date.now() - 3600000) }, // Last hour
	});

	const failureCount = recentDeliveries.filter((d) => !d.success).length;
	const successCount = recentDeliveries.filter((d) => d.success).length;
	const totalDeliveries = recentDeliveries.length;
	const successRate = totalDeliveries > 0 ? (successCount / totalDeliveries) * 100 : 100;

	// Calculate average response time
	const avgDuration = recentDeliveries.reduce((sum, d) => sum + d.duration, 0) / totalDeliveries;

	console.log({
		totalDeliveries,
		successCount,
		failureCount,
		successRate: `${successRate.toFixed(2)}%`,
		avgResponseTime: `${avgDuration.toFixed(0)}ms`,
	});

	// Alert if success rate drops
	if (successRate < 90) {
		await sendAlert({
			title: 'Webhook Health Alert',
			message: `Webhook success rate dropped to ${successRate.toFixed(2)}%`,
			details: {
				successCount,
				failureCount,
				totalDeliveries,
			},
			severity: 'warning',
		});
	}

	// Alert if response time is slow
	if (avgDuration > 5000) {
		await sendAlert({
			title: 'Webhook Performance Alert',
			message: `Average webhook response time is ${avgDuration.toFixed(0)}ms`,
			severity: 'warning',
		});
	}
}

// Run every 5 minutes
setInterval(monitorWebhookHealth, 5 * 60 * 1000);

Best Practices

Return 2xx for Successful Processing

Always return a 2xx status code when your endpoint successfully receives and processes a webhook:

Nodejs

app.post("/webhooks/autosend", async (req, res) => {
  try {
    // Verify signature first
    const signature = req.headers["x-webhook-signature"];
    const body = JSON.stringify(req.body);

    if (!verifySignature(signature, body, WEBHOOK_SECRET)) {
      // Return 401 for invalid signature (will retry)
      // Or return 200 to prevent retries for invalid signatures
      return res.status(401).json({ error: "Invalid signature" });
    }

    // Queue for background processing
    await queue.add("process-webhook", {
      deliveryId: req.headers["x-webhook-delivery-id"],
      event: req.body.type,
      data: req.body.data,
    });

    // Return 200 immediately - don't wait for processing
    res.status(200).json({ received: true });
  } catch (error) {
    console.error("Webhook processing error:", error);
    // Return 500 for temporary errors (will retry)
    res.status(500).json({ error: "Internal error" });
  }
});

Implement Idempotency with Delivery ID

Use the X-Webhook-Delivery-Id header to prevent duplicate processing during retries:

Nodejs

// Using Redis for idempotency tracking
const redis = require("redis");
const client = redis.createClient();

app.post("/webhooks/autosend", async (req, res) => {
  const deliveryId = req.headers["x-webhook-delivery-id"];
  const ttl = 86400; // 24 hours

  // Check if already processed
  const exists = await client.exists(`webhook:${deliveryId}`);
  if (exists) {
    console.log(`Duplicate delivery ${deliveryId}, skipping`);
    return res.status(200).json({ received: true, duplicate: true });
  }

  try {
    // Process the webhook
    await processWebhook(req.body);

    // Mark as processed (with TTL to auto-cleanup)
    await client.setex(`webhook:${deliveryId}`, ttl, "processed");

    res.status(200).json({ received: true });
  } catch (error) {
    // Don't mark as processed on error so it can be retried
    console.error("Processing failed:", error);
    res.status(500).json({ error: "Processing failed" });
  }
});

Alternative: Database-based idempotency

Nodejs

app.post("/webhooks/autosend", async (req, res) => {
  const deliveryId = req.headers["x-webhook-delivery-id"];

  try {
    // Try to insert the delivery ID (unique constraint)
    await db.webhookDeliveries.insertOne({
      deliveryId,
      receivedAt: new Date(),
      processed: false,
    });
  } catch (error) {
    if (error.code === 11000) {
      // Duplicate key - already processed
      console.log(`Duplicate delivery ${deliveryId}`);
      return res.status(200).json({ received: true, duplicate: true });
    }
    throw error;
  }

  try {
    await processWebhook(req.body);

    // Mark as processed
    await db.webhookDeliveries.updateOne(
      { deliveryId },
      { $set: { processed: true, processedAt: new Date() } }
    );

    res.status(200).json({ received: true });
  } catch (error) {
    console.error("Processing failed:", error);
    res.status(500).json({ error: "Processing failed" });
  }
});

Differentiate Between Temporary and Permanent Errors

Return appropriate status codes based on the type of error:

Nodejs

app.post("/webhooks/autosend", async (req, res) => {
  try {
    // Verify signature
    const signature = req.headers["x-webhook-signature"];
    const isValid = verifySignature(signature, req.body, WEBHOOK_SECRET);

    if (!isValid) {
      // Invalid signature is a permanent error
      // Return 200 to prevent unnecessary retries
      return res.status(200).json({
        error: "Invalid signature",
        retryable: false
      });
    }

    // Process webhook
    await processWebhook(req.body);
    res.status(200).json({ received: true });
  } catch (error) {
    console.error("Webhook error:", error);

    // Determine if error is temporary or permanent
    if (error.code === "ECONNREFUSED" || error.code === "ETIMEDOUT") {
      // Temporary database/service error - retry
      return res.status(500).json({
        error: "Service temporarily unavailable",
        retryable: true
      });
    } else if (error.name === "ValidationError") {
      // Permanent error - bad data, don't retry
      return res.status(200).json({
        error: "Invalid data format",
        retryable: false
      });
    } else {
      // Unknown error - retry to be safe
      return res.status(500).json({
        error: "Internal error",
        retryable: true
      });
    }
  }
});

Respond Quickly (Under 10 Seconds)

Your endpoint MUST respond within 10 seconds or the request will timeout. Process webhooks asynchronously:

Nodejs

const Queue = require("bull");
const webhookQueue = new Queue("webhooks", {
  redis: { host: "localhost", port: 6379 },
});

// ❌ Bad - Synchronous processing (may timeout)
app.post("/webhooks/autosend", async (req, res) => {
  await updateDatabase(req.body);      // 2 seconds
  await sendToAnalytics(req.body);     // 3 seconds
  await notifySlack(req.body);         // 2 seconds
  await triggerWorkflow(req.body);     // 4 seconds
  // Total: 11 seconds - WILL TIMEOUT!

  res.status(200).json({ received: true });
});

// ✅ Good - Asynchronous processing
app.post("/webhooks/autosend", async (req, res) => {
  const deliveryId = req.headers["x-webhook-delivery-id"];

  // Queue for background processing (fast!)
  await webhookQueue.add("process", {
    deliveryId,
    event: req.body.type,
    data: req.body.data,
  });

  // Respond immediately (< 100ms)
  res.status(200).json({ received: true });
});

// Process in background worker
webhookQueue.process("process", async (job) => {
  const { deliveryId, event, data } = job.data;

  console.log(`Processing webhook ${deliveryId} for event ${event}`);

  await updateDatabase(data);
  await sendToAnalytics(data);
  await notifySlack(data);
  await triggerWorkflow(data);

  console.log(`Completed processing webhook ${deliveryId}`);
});

Log All Delivery Attempts

Keep detailed logs of all webhook delivery attempts for debugging:

Nodejs

const winston = require("winston");

const logger = winston.createLogger({
  level: "info",
  format: winston.format.json(),
  transports: [
    new winston.transports.File({ filename: "webhooks.log" }),
    new winston.transports.Console(),
  ],
});

app.post("/webhooks/autosend", async (req, res) => {
  const deliveryId = req.headers["x-webhook-delivery-id"];
  const event = req.body.type;
  const timestamp = req.headers["x-webhook-timestamp"];

  logger.info("Webhook received", {
    deliveryId,
    event,
    timestamp,
    receivedAt: new Date().toISOString(),
  });

  try {
    await processWebhook(req.body);

    logger.info("Webhook processed successfully", {
      deliveryId,
      event,
      processingTime: Date.now() - parseInt(timestamp),
    });

    res.status(200).json({ received: true });
  } catch (error) {
    logger.error("Webhook processing failed", {
      deliveryId,
      event,
      error: error.message,
      stack: error.stack,
    });

    res.status(500).json({ error: "Processing failed" });
  }
});

Test Retry Behavior

Test how your endpoint handles retries in development:

Nodejs

// Simulate intermittent failures for testing
const deliveryAttempts = new Map();

app.post("/webhooks/autosend", async (req, res) => {
  const deliveryId = req.headers["x-webhook-delivery-id"];

  // Track attempts for this delivery
  const attempts = (deliveryAttempts.get(deliveryId) || 0) + 1;
  deliveryAttempts.set(deliveryId, attempts);

  console.log(`Delivery ${deliveryId} - Attempt ${attempts}`);

  // Simulate: Fail first 2 attempts, succeed on 3rd
  if (attempts <= 2) {
    console.log(`Simulating failure on attempt ${attempts}`);
    return res.status(500).json({ error: "Simulated error" });
  }

  console.log(`Success on attempt ${attempts}`);

  // Process webhook
  await processWebhook(req.body);

  // Clean up tracking
  deliveryAttempts.delete(deliveryId);

  res.status(200).json({ received: true, attempts });
});

Handle Concurrent Deliveries

AutoSend processes up to 5 webhooks concurrently. Ensure your endpoint can handle concurrent requests:

Nodejs

const express = require("express");
const cluster = require("cluster");
const os = require("os");

if (cluster.isMaster) {
  // Fork workers (one per CPU core)
  const numCPUs = os.cpus().length;
  console.log(`Master process starting ${numCPUs} workers`);

  for (let i = 0; i < numCPUs; i++) {
    cluster.fork();
  }

  cluster.on("exit", (worker) => {
    console.log(`Worker ${worker.process.pid} died, starting new worker`);
    cluster.fork();
  });
} else {
  // Worker process
  const app = express();
  app.use(express.json());

  app.post("/webhooks/autosend", async (req, res) => {
    try {
      await processWebhook(req.body);
      res.status(200).json({ received: true });
    } catch (error) {
      res.status(500).json({ error: "Processing failed" });
    }
  });

  app.listen(3000, () => {
    console.log(`Worker ${process.pid} listening on port 3000`);
  });
}

Troubleshooting

High Failure Rate

Symptoms: Webhooks failing frequentlyPossible Causes:

Endpoint is down or unreachable
Endpoint is timing out (>10 seconds)
Endpoint is returning non-2xx status codes
SSL/TLS certificate issues
Rate limiting on your server

Solutions:

Test your endpoint manually:

curl -X POST https://your-endpoint.com/webhooks/autosend \
  -H "Content-Type: application/json" \
  -H "X-Webhook-Signature: test_signature" \
  -H "X-Webhook-Event: email.sent" \
  -H "X-Webhook-Delivery-Id: test-123" \
  -H "X-Webhook-Timestamp: $(date +%s)000" \
  -d '{
    "type": "email.sent",
    "createdAt": "2025-01-05T10:00:00.000Z",
    "data": {
      "emailId": "test123",
      "from": "[email protected]",
      "to": {"email": "[email protected]", "name": "Test User"},
      "subject": "Test Email"
    }
  }'

Check your server logs for error messages
Verify SSL certificate is valid:

openssl s_client -connect your-endpoint.com:443 -servername your-endpoint.com

Check response time:

time curl -X POST https://your-endpoint.com/webhooks/autosend \
  -H "Content-Type: application/json" \
  -d '{"type":"test","data":{}}'

Monitor your server resources (CPU, memory, disk) to ensure it’s not overloaded
Check for rate limiting on your server or firewall

Signature Verification Failures

Symptoms: All webhooks returning 401 or failing signature verificationPossible Causes:

Using wrong webhook secret
Incorrect signature verification logic
Body parsing issues (modified body)
Character encoding issues

Solutions:See Verify Webhook Requests for detailed troubleshooting.Quick verification test:

const crypto = require("crypto");

function verifyWebhookSignature(signature, body, secret) {
  const expectedSignature = crypto
    .createHmac("sha256", secret)
    .update(body)
    .digest("hex");

  console.log("Received signature:", signature);
  console.log("Expected signature:", expectedSignature);
  console.log("Match:", signature === expectedSignature);

  return signature === expectedSignature;
}

app.post("/webhooks/autosend", express.raw({ type: "application/json" }), (req, res) => {
  const signature = req.headers["x-webhook-signature"];
  const body = req.body.toString(); // Raw body as string

  if (!verifyWebhookSignature(signature, body, WEBHOOK_SECRET)) {
    return res.status(401).json({ error: "Invalid signature" });
  }

  // Parse body after verification
  const data = JSON.parse(body);
  res.status(200).json({ received: true });
});

Timeout Issues

Symptoms: Webhooks timing out (10 second timeout)Possible Causes:

Synchronous processing taking too long
Database queries are slow
External API calls are slow
No connection pooling
Inefficient code

Solutions:

Use background job queues:

const Queue = require("bull");
const webhookQueue = new Queue("webhooks");

app.post("/webhooks/autosend", async (req, res) => {
  // Queue immediately (fast)
  await webhookQueue.add(req.body, {
    attempts: 3,
    backoff: { type: "exponential", delay: 2000 },
  });

  // Respond fast
  res.status(200).json({ received: true });
});

// Process in background
webhookQueue.process(async (job) => {
  const { type, data } = job.data;

  // Time-consuming operations here
  await updateDatabase(data);
  await callExternalAPI(data);
  await generateReport(data);
});

Optimize database queries:

// ❌ Slow - Sequential queries
const user = await db.users.findOne({ email: data.to.email });
const campaign = await db.campaigns.findOne({ _id: data.campaignId });
const template = await db.templates.findOne({ _id: data.templateId });

// ✅ Fast - Parallel queries
const [user, campaign, template] = await Promise.all([
  db.users.findOne({ email: data.to.email }),
  db.campaigns.findOne({ _id: data.campaignId }),
  db.templates.findOne({ _id: data.templateId }),
]);

Add database indexes:

// Create indexes on frequently queried fields
db.webhookDeliveries.createIndex({ deliveryId: 1 }, { unique: true });
db.webhookDeliveries.createIndex({ createdAt: -1 });
db.webhookLogs.createIndex({ webhookId: 1, createdAt: -1 });

Use connection pooling:

const mongoose = require("mongoose");

// Configure connection pool
mongoose.connect(MONGODB_URI, {
  poolSize: 10,
  socketTimeoutMS: 45000,
  family: 4,
});

Cache frequently accessed data:

const NodeCache = require("node-cache");
const cache = new NodeCache({ stdTTL: 600 }); // 10 minute TTL

async function getWebhookConfig(webhookId) {
  // Check cache first
  let config = cache.get(`webhook:${webhookId}`);

  if (!config) {
    // Fetch from database
    config = await db.webhooks.findOne({ _id: webhookId });
    cache.set(`webhook:${webhookId}`, config);
  }

  return config;
}

Duplicate Processing

Symptoms: Same webhook processed multiple timesPossible Causes:

Not implementing idempotency
Retry logic processing same delivery multiple times
Race conditions in distributed systems

Solutions:Implement idempotency using X-Webhook-Delivery-Id (see Best Practices section above).

Missing Webhooks

Symptoms: Not receiving expected webhooksPossible Causes:

Webhook not configured for the event type
Webhook is inactive or disabled
Firewall blocking incoming requests
Incorrect URL configured

Solutions:

Check webhook configuration in AutoSend dashboard
Verify event types are selected for the webhook
Check webhook is active (not disabled)
Test webhook URL is accessible from external networks
Check firewall rules allow incoming traffic on your endpoint port
Review AutoSend delivery logs for delivery attempts

Introduction

Getting started with webhooks

Event Types

Complete list of webhook events

Verify Webhook Requests

Security and signature verification

Webhooks

Manage your webhooks from the AutoSend sidebar

Getting Started

Quickstart

Transactional Emails

Marketing Emails

Email Automations

Inbound

Others

Integrations

Migration

Automatic Retries

Retry Schedule

Example Timeline

When Retries Occur

Retry Triggers

No Retry for Success

Webhook Headers

After All Retries Fail

What Happens Next

Webhook Auto-Disable (Optional)

Delivery Logs

Log Retention

Log Information

Monitoring Your Webhooks

Check Webhook Status

Navigate to Webhooks from the AutoSend sidebar

View your webhooks with their current status

Webhook Health Metrics

Implement Your Own Monitoring

Best Practices

Troubleshooting

Introduction

Event Types

Verify Webhook Requests

Webhooks

​Automatic Retries

​Retry Schedule

​Example Timeline

​When Retries Occur

​Retry Triggers

​No Retry for Success

​Webhook Headers

​After All Retries Fail

​What Happens Next

​Webhook Auto-Disable (Optional)

​Delivery Logs

​Log Retention

​Log Information

​Monitoring Your Webhooks

​Check Webhook Status

Navigate to Webhooks from the AutoSend sidebar

View your webhooks with their current status

​Webhook Health Metrics

​Implement Your Own Monitoring

​Best Practices

​Troubleshooting

​Related Resources

Introduction

Event Types

Verify Webhook Requests

Webhooks

Automatic Retries

Retry Schedule

Example Timeline

When Retries Occur

Retry Triggers

No Retry for Success

Webhook Headers

After All Retries Fail

What Happens Next

Webhook Auto-Disable (Optional)

Delivery Logs

Log Retention

Log Information

Monitoring Your Webhooks

Check Webhook Status

Webhook Health Metrics

Implement Your Own Monitoring

Best Practices

Troubleshooting

Related Resources