Lambda Timeout, Memory & Response Settings for Bedrock Streaming
This guide explains how to choose the right Lambda timeout, memory, and response settings when building a streaming application with Amazon Bedrock. You’ll learn what each setting does, how they interact, and how to pick values that keep your streaming responses fast and reliable without wasting money.
What You’ll Need
- An AWS account with access to Amazon Bedrock and AWS Lambda.
- A basic understanding of Lambda functions and IAM roles.
- Familiarity with the Bedrock Runtime API and its streaming response format (e.g.,
InvokeModelWithResponseStream). - The AWS CLI or SDK configured in your development environment.
Step-by-Step
- Understand the streaming flow. When you call Bedrock with streaming enabled, the model sends tokens incrementally over an HTTP/2 stream. Lambda must receive these chunks and forward them to your client (e.g., via API Gateway or a WebSocket). The Lambda function stays active for the entire duration of the stream.
- Estimate your model’s response time. The time it takes to generate a full response depends on the model, prompt complexity, and output length. A typical streaming response for a short paragraph might take 5–10 seconds; longer outputs (e.g., a 500-word article) can take 30+ seconds. Use a small test prompt to measure actual latency.
- Set the Lambda timeout. The timeout must be longer than your maximum expected streaming duration. For most use cases, start with 1 minute (60 seconds). If your model can generate longer responses, increase it to 5 minutes (300 seconds) – the current maximum. Remember, the timeout covers the entire stream, not just the first token.
- Choose memory based on concurrency and buffering. Lambda memory also determines CPU allocation. Streaming itself is I/O-bound, not CPU-heavy, but you need enough memory to handle the response buffer and any SDK overhead. Start with 512 MB. If you’re using a language like Node.js or Python, 512 MB is usually sufficient. For heavy JSON processing or if you see memory errors, bump to 1024 MB or 2048 MB.
- Configure the response stream in your Lambda handler. When using the Bedrock Runtime, set
streamtotruein the request body. Your Lambda must iterate over the event stream and forward each chunk to the client. Do not accumulate the entire response in memory before sending – that defeats streaming. - Set up the API Gateway (or proxy) correctly. If you’re using API Gateway REST API or HTTP API, enable
proxymode and setTransfer-Encoding: chunkedor use WebSocket for real-time delivery. For HTTP API, you may need to setstreamingresponse support (available for HTTP API with thepayloadFormatVersionset to1.0). - Handle backpressure and errors. Your Lambda should catch Bedrock exceptions (like throttling) and send an error event to the client. Also, if the client disconnects, stop the stream to avoid wasted compute.
- Test with realistic payloads. Run load tests with your expected prompt sizes and output lengths. Monitor Lambda metrics: duration, memory usage, and invocation count. Adjust timeout and memory based on the 99th percentile duration.
- Optimize costs. Lambda billing is based on memory and duration. Streaming longer responses costs more. If you see high costs, consider using a smaller model, limiting output tokens, or using a dedicated streaming service like AWS AppSync for real-time data.
Common Mistakes
- Setting timeout too low. If the timeout is shorter than the stream duration, the Lambda function is killed mid-stream, and the client gets an incomplete response.
- Buffering the entire response. Some developers collect all chunks in a list and return the full body. This defeats streaming and can cause memory exhaustion for long outputs.
- Ignoring API Gateway limits. API Gateway has a 30-second integration timeout for REST APIs. If your Lambda runs longer, the client will see a 504 error. Use HTTP API with streaming or WebSocket to avoid this.
- Using too much memory unnecessarily. Higher memory increases cost. Streaming doesn’t need 3 GB unless you’re doing heavy processing.
- Not handling client disconnects. If the client closes the connection, your Lambda may continue sending to a dead socket, wasting resources.
Tips
- Start with a timeout of 60 seconds and memory of 512 MB, then adjust based on real measurements.
- Use the
InvokeModelWithResponseStreamAPI (or the SDK’sinvoke_model_with_response_stream) to get a proper event stream. - For API Gateway, use HTTP API with
payloadFormatVersion: 1.0to enable streaming responses. REST API does not support streaming responses. - Consider using Lambda Response Streaming (for Node.js and Python) to stream the response directly from Lambda, which works well with API Gateway HTTP API.
- Monitor CloudWatch logs for
Task timed outerrors – they indicate your timeout is too short. - Set a maximum output token count in your Bedrock request to control response length and cost.
Troubleshooting
- Client receives a 504 from API Gateway. This is likely the API Gateway integration timeout (30s for REST). Switch to HTTP API with streaming or increase the Lambda timeout and ensure the API Gateway is configured for streaming.
- Lambda times out mid-stream. Increase the timeout. Check the CloudWatch logs for the exact time the function was killed. If it’s consistently near the limit, add headroom.
- Memory errors (e.g.,
OutOfMemoryError). Increase memory. This can happen if you’re accumulating chunks or using a heavy runtime. Also, review your code for memory leaks. - Stream starts but then stops. This could be a Bedrock throttling issue. Implement exponential backoff in your SDK calls, and consider enabling provisioned throughput for consistent performance.
- Client sees no data until the end. Your code might be buffering. Ensure you’re flushing each chunk to the response as soon as it arrives.
Frequently Asked Questions
What is the maximum Lambda timeout for streaming?
The maximum timeout for a Lambda function is 15 minutes (900 seconds). For Bedrock streaming, you’ll rarely need more than a few minutes, but you can set up to 900 seconds.
Does memory affect streaming performance?
Memory allocation also determines CPU. Streaming is mostly I/O-bound, so extra memory doesn’t speed up token generation. However, insufficient memory can cause crashes if your runtime uses more memory than allocated.
Can I use API Gateway REST API for streaming?
No, REST API does not support streaming responses. Use HTTP API with payloadFormatVersion: 1.0 or a WebSocket API.
How do I send an error to the client mid-stream?
In your Lambda, catch the exception and write an error event to the response stream (e.g., a JSON object with an error field). The client should be coded to handle that event.
Is it better to use Lambda Response Streaming or API Gateway?
Lambda Response Streaming (for Node.js and Python) is simpler and works well with HTTP API. It lets you stream directly from Lambda without a separate proxy. If you need advanced features like auth or rate limiting, API Gateway is still useful.
How do I estimate the right timeout?
Run a test with your largest expected prompt and measure the time to receive the last token. Add a 20–30% buffer. For example, if the test takes 40 seconds, set timeout to 60 seconds.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.