Produce jobs from AWS Lambda

This guide is the producer-side recipe: short-lived Lambda functions enqueueing jobs into ChasquiMQ on ElastiCache (or any TLS-fronted Redis 8.6+) for a long-running worker fleet to drain. It does not cover running ChasquiMQ workers in Lambda — see When not to use Lambda at the end.

Connection shape

ChasquiMQ’s defaults (from PRs #114–#117) are tuned for hosts that may be frozen or terminated between invocations:

TCP keepalive on (60s probe interval, well under AWS NAT’s 350s idle cutoff).
Exponential reconnect with unbounded attempts (100 ms → 30 s, base 2, ±50 ms jitter).
10 s connection timeout per handshake.
Reconnect on auth error.
Every Producer::add awaits the Redis XADD response before resolving — bytes are committed to Redis before your handler returns.

You don’t need to tune these. What you do need to wire is TLS, IAM auth, and a module-scope Queue so warm invocations reuse the producer pool.

Module-scope client

Build the Queue once at module scope, never inside the handler. Lambda freezes the execution environment between invocations and thaws it on the next; sockets in module scope survive. Building per-invoke pays the TCP+TLS+AUTH handshake every time and bottoms out your throughput.

Node
Python

import { Queue } from "chasquimq";

const queue = new Queue("emails", {
  connection: {
    url: process.env.REDIS_URL,
    tls: true,
  },
});

export const handler = async (event: { to: string }) => {
  await queue.add("welcome", { to: event.to });
  return { ok: true };
};

tls: true upgrades a redis:// or schemeless host:port URL to rediss:// in place. If REDIS_URL is already rediss://... it’s left untouched.

Do not wrap the Queue in await using here, even though that’s the recommended pattern in the Node README. await using calls close() at scope exit, which closes the underlying connection pool — fine for a CLI script, wrong for a Lambda handler where you want the warm container to reuse the pool on the next invocation. Build the Queue once at module scope, never close it from the handler.

import os
from chasquimq import Queue

_queue = Queue(
    "emails",
    redis_url=os.environ["REDIS_URL"],
    tls=True,
)


async def handler(event):
    await _queue.add("welcome", {"to": event["to"]})
    return {"ok": True}

tls=True upgrades a redis:// or schemeless input to rediss:// in place.

Do not wrap the Queue in async with here, even though that’s the pattern in the Python README. async with calls close() at scope exit, which closes the underlying connection pool — fine for a script, wrong for a Lambda handler where you want the warm container to reuse the pool on the next invocation. Build the Queue once at module scope, never close it from the handler.

ElastiCache + IAM auth

For ElastiCache with encryption-in-transit (the secure default; required for IAM auth):

Put the Lambda in a VPC with subnets that can reach the ElastiCache cluster’s security group on port 6379. The Hyperplane shared-ENI architecture makes the VPC cold-start tax small (tens of milliseconds, not the 10 s of pre-2019 Lambda).
The cluster URL the AWS console shows is schemeless (e.g. my-cluster.cache.amazonaws.com:6379). With tls: true, the shim prepends rediss://.
ChasquiMQ’s enable-rustls-ring TLS backend reads platform trust roots via rustls-native-certs. AWS Trust CA-signed endpoints work out of the box. For private CAs, set SSL_CERT_FILE to a PEM bundle in the Lambda environment variables — that env var takes precedence over the platform store.
Use multiple subnets in your Lambda VPC config to spread connections across Hyperplane ENIs (the per-subnet-SG combo has a 65k-connection cap; under burst scale-out you can hit it).

Rotating IAM tokens

ElastiCache IAM auth issues short-lived tokens (~15 min). The engine has a CredentialProvider hook that fred calls before every AUTH / HELLO — initial connect and every reconnect — and both shims now expose it. Combined with the engine’s reconnect_on_auth_error default, a module-scope Queue survives token rotation without rebuilding: fred re-invokes your callback to mint a fresh token on each reconnect.

The callback is handed the resolving host (or null/None) and returns the credentials to authenticate with. Generate the IAM auth token with the AWS SDK inside it:

Node
Python

import { Queue } from "chasquimq";

const queue = new Queue("emails", {
  connection: {
    url: process.env.REDIS_URL,
    tls: true,
    // Called before every AUTH/HELLO — initial connect and every reconnect.
    credentialProvider: async (host) => ({
      username: process.env.ELASTICACHE_USER,
      password: await mintElastiCacheIamToken(host), // AWS SDK signer
    }),
  },
});

import os
from chasquimq import Queue

async def fetch_credentials(host: str | None) -> tuple[str | None, str | None]:
    # Called before every AUTH/HELLO — initial connect and every reconnect.
    token = await mint_elasticache_iam_token(host)  # AWS SDK signer
    return (os.environ["ELASTICACHE_USER"], token)

_queue = Queue(
    "emails",
    redis_url=os.environ["REDIS_URL"],
    tls=True,
    credential_provider=fetch_credentials,
)

On Python the Producer connects lazily on the first awaited method (a deferred-construction path that avoids an asyncio-loop deadlock when the callback is itself a coroutine); this is internal, but it means the first await _queue.add(...) is where the initial token fetch happens.

Caveat: reconnect_max_attempts is not yet exposed to either shim, so a credential callback that always rejects (expired IAM role, revoked user) loops forever on reconnect rather than failing fast. Make the token mint inside the callback resilient, and alert on sustained reconnect churn.

For Rust producers on Lambda (the rare case): plug an Arc<dyn fred::types::config::CredentialProvider> into ProducerConfig::connection.credential_provider directly. See the Rust API reference for the trait shape, the connection options reference for the per-language surface, and engine internals for the rotation semantics.

If you cannot run a token signer in-process, the pre-callback fallback still works: rotate REDIS_URL externally (sidecar writes a fresh URL to Secrets Manager / Parameter Store; rebuild the Queue when the cached instance ages past the token TTL). The callback is the better path — fewer reconnects, no rebuild — but the external-rotation pattern remains valid where an in-process signer isn’t an option.

Result-back patterns

Three shapes for “Lambda enqueues, gets a result”:

Fire-and-forget (default). await queue.add(...); return from the handler. The bytes are committed before add resolves — safe to return immediately.
Block on the result for sync-style API Gateway integrations. await queue.add(...) returns a Job; call await job.waitForResult({ timeoutMs }). Cap timeoutMs well under your Lambda timeout (default 3s, max 15 min).
Async pickup elsewhere. Producer Lambda enqueues with a stable jobId (Queue.addUnique) and returns immediately; a different process — most realistically your existing worker fleet, since it already drains the queue — handles the result and pushes it onto whatever channel the original caller was waiting on (SNS, S3-webhook, a websocket gateway, a row in a results table). The events stream ({chasqui:<queue>}:events) carries completed / failed transitions for cross-process subscribers, but it’s a Redis Stream — not an AWS event source — so a Lambda can’t be triggered by it directly. Either subscribe with QueueEvents from a long-running process, or have the worker publish the result-channel push itself.

For (2) and (3), the consumer side must run with storeResults: true / store_results=True.

Operational gotchas

NAT 350 s idle timeout. Lambdas in a private subnet egressing through a NAT gateway to non-VPC Redis silently lose TCP after 350 s idle — the next invocation hangs until reconnect. ChasquiMQ’s 60 s TCP keepalive default handles this; if you’ve overridden tcp_keepalive_secs, keep it under 350.
Connection storm on cold-start spikes. Each new execution environment opens its own producer pool (engine default: 8 sockets). There is no Redis equivalent of RDS Proxy; ElastiCache’s maxclients becomes your ceiling. Plan reserved concurrency around maxclients ÷ pool_size headroom. Pool size is exposed only on the Rust ProducerConfig today; for Node and Python producers, treat the 8-socket-per-execution-environment count as fixed when sizing reserved concurrency.
Redis Cluster is not supported today. The engine pulls fred without the cluster feature, so it speaks plain Redis. A single-shard ElastiCache or MemoryDB cluster (or any non-clustered Redis) works; multi-shard cluster topologies will not route correctly. If you need to scale beyond one Redis instance, that’s a roadmap gap rather than a knob.
Don’t close() between invocations. Closing the cached Queue forfeits the warm-socket benefit and pays the handshake on every invoke. The TCP socket survives the freeze; let it.

When not to use Lambda

ChasquiMQ workers are a long-running tokio::XREADGROUP BLOCK loop with a connection-multiplexed pool, batched pipelined XACK, and graceful shutdown. Everything that makes them fast is amortized across many jobs. Lambda gives you per-event handlers with a 15-minute ceiling and no graceful-shutdown signal. The mismatch is structural, not tunable:

For Lambda-driven processing, use SQS + Lambda event source mapping. AWS owns the pollers, scaling, dead-lettering, concurrency caps. ChasquiMQ is the wrong tool when your only requirement is “run this function in response to a queued message on AWS.”
For ChasquiMQ workers in AWS, use Fargate, ECS, or EC2. The engine is built to live for hours, not seconds.

This guide is the right shape only when you already have ChasquiMQ workers running somewhere durable and want Lambda functions to participate as producers (cross-cloud, on-prem hybrid, or a Rust hot path). For “everything in AWS, Redis-only,” the answer is usually still ChasquiMQ on Fargate with Lambda producers — but think twice if the producer side is also greenfield.