AWS Lambda Runtime Emulator for Local and Remote Debugging

This emulator allows running Lambda functions locally with either a local payload from a file or a remote payload from AWS as if the local lambda was running there.

Installation and usage

cargo install lambda-debugger

Debugging with local payload

Use this method for simple use cases where a single static payload is sufficient.

1. Save your payload into a file, e.g. save {"command": "echo"} into test-payload.json file
2. Start the emulator with the payload file name as its only param, e.g. cargo lambda-debugger test-payload.json
3. Add env vars printed by the emulator and start your lambda with cargo run in a separate terminal

The lambda will connect to the emulator and receive the payload. You can re-run your lambda with the same payload as many times as needed.

Debugging with remote payload

Use this method to get dynamic payload from other AWS services or when you need to send back a dynamic response, e.g. to process a request triggered by a user action on a website involving API Gateway as in the following diagram:

Remote debugging configuration

This project provides the tools necessary to bring the AWS payload to your local machine in real-time, run the lambda and send back the response as if the local lambda was running on AWS.

• proxy-lambda forwards Lambda requests and responses between AWS and your development machine in real time
• lambda-debugger provides Lambda APIs to run a lambda function locally and exchange payloads with AWS

Limitations

This Lambda emulator does not provide the full runtime capabilities of AWS:

• no environment constraints, e.g. memory or execution time
• panics are not reported back to AWS
• no concurrent request handling
• no support for X-Trace or Extensions APIs
• no stream responses
• smaller maximum payload size

Getting started with remote debugging

• create request and response queues in SQS with IAM permissions
• deploy proxy-lambda in place of the function you need to debug
• run the emulator locally as a binary or with cargo run
• run your lambda locally with cargo run

SQS configuration

Create two SQS queues with an identical configuration:

• proxy_lambda_req for requests to be sent from AWS to your local lambda under debugging. Required.
• proxy_lambda_resp if you want responses from your local lambda to be returned to the caller. Optional.

See Advanced setup section for more info on how to customize queue names and other settings.

Recommended queue settings:

• Queue type: Standard
• Maximum message size: 256 KB
• Default visibility timeout: 10 Seconds
• Message retention period: 1 Hour
• Receive message wait time: 20 Seconds

This IAM policy grants proxy-lambda access to the queues. It assumes that you already have sufficient privileges to access Lambda and SQS from your local machine.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::512295225992:role/lambda_basic"
      },
      "Action": [
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes",
        "sqs:ReceiveMessage",
        "sqs:SendMessage"
      ],
      "Resource": "arn:aws:sqs:us-east-1:512295225992:proxy_lambda_req"
    }
  ]
}

Modifying the policy with your IDs

You need to replace Principal and Resource IDs with your values before adding the above policy to your queues:

• Principal - is the IAM role your lambda assumes (check Lambda's Permissions Config tab in the AWS console to find the value)
• Resource - the ARN of the queue the policy is attached to (see the queue details page in the AWS console to find the value)

Use different Resource values for request and response queues:

• arn:aws:sqs:[your_region]:[your_aws_account]:proxy_lambda_req for proxy_lambda_req queue
• arn:aws:sqs:[your_region]:[your_aws_account]:proxy_lambda_resp for proxy_lambda_resp queue

Building and deploying proxy-lambda

The proxy-lambda function should be deployed to AWS in place of the function you want to debug.

Replace the following parts of the bash script below with your values before running it from the project root:

• target - the architecture of the lambda function on AWS, e.g. x86_64-unknown-linux-gnu
• region - the region of the lambda function, e.g. us-east-1
• name - the name of the lambda function you want to replace with the proxy, e.g. my-lambda

target=x86_64-unknown-linux-gnu 
region=us-east-1
name=my-lambda

cargo build --release --target $target
cp ./target/$target/release/proxy-lambda ./bootstrap && zip proxy.zip bootstrap && rm bootstrap
aws lambda update-function-code --region $region --function-name $name --zip-file fileb://proxy.zip

A deployed proxy-lambda should return OK or time out waiting for a response if you run it with a test event from the AWS console. Check CloudWatch logs for a detailed execution report.

Debugging

Pre-requisites:

• proxy-lambda was deployed to AWS
• SQS queues were created with the recommended access policies

Launching the local lambda:

• run cargo lambda-debugger in a separate terminal
• add environmental variables from the prompt printed by the emulator
• start your lambda with cargo run in the same terminal where you added the env vars

Debugging:

• trigger the event on AWS as part of your normal data flow, e.g. by a user action on a webpage
• the emulator should display the lambda payload and forward it to your local lambda for processing
• debug as needed

Success, failure and replay:

• successful responses are sent back to the caller if the response queue is configured (proxy_lambda_resp)
• panics or handler errors are not sent back to AWS
• the same incoming SQS message is reused until the lambda completes successfully
• lambda-debugger deletes the request message from proxy_lambda_req queue when the local lambda completes successfully
• proxy-lambda deletes the response message from proxy_lambda_resp queue after forwarding it to the caller, e.g. to API Gateway
• proxy-lambda purges proxy_lambda_resp queue before sending a new request to proxy_lambda_resp
• you have to purge proxy_lambda_req queue manually to delete stale requests

If the local lambda fails, terminates or panics, you can make changes to its code and run it again to reuse the same incoming payload from the request queue.

Advanced remote debugging setup

Custom SQS queue names

By default, proxy-lambda and the local lambda-debugger attempt to connect to proxy_lambda_req and proxy_lambda_resp queues in the same region.

Provide these env vars to proxy-lambda and lambda-debugger if your queue names differ from the defaults:

• PROXY_LAMBDA_REQ_QUEUE_URL - request queue, e.g. https://sqs.us-east-1.amazonaws.com/512295225992/debug_request
• PROXY_LAMBDA_RESP_QUEUE_URL - response queue, e.g. https://sqs.us-east-1.amazonaws.com/512295225992/debug_response

Late responses

Debugging the local lambda may take longer than the AWS service is willing to wait. For example, proxy-lambda function can be configured to wait for up to 15 minutes, but the AWS API Gateway wait time is limited to 30 seconds.

Assume that it took you 5 minutes to fix the lambda code and return the correct response. If proxy-lambda was configured to wait for that long it would still forward the response to the API Gateway which timed out 4.5 min earlier. In that case, you may need to trigger another request for it to complete successfully end-to-end.

Not waiting for responses from local lambda

It may be inefficient to have proxy-lambda waiting for a response from the local lambda because it takes too long or no response is necessary. Both proxy-lambda and lambda-debugger would not expect a response if the response queue is inaccessible.

Option 1: delete proxy_lambda_resp queue

Option 2: add PROXY_LAMBDA_RESP_QUEUE_URL env var with no value to proxy-lambda and lambda-debugger

Option 3: make proxy_lambda_resp queue inaccessible by changing its IAM policy. E.g. change the resource name from the correct queue name "Resource": "arn:aws:sqs:us-east-1:512295225992:proxy_lambda_resp" to a non-existent name like this "Resource": "arn:aws:sqs:us-east-1:512295225992:proxy_lambda_resp_BLOCKED". Both proxy-lambda and lambda-debugger treat the access error as a hint to not expect a response.

Canceling long proxy-lambda wait

If your proxy-lambda is configured to expect a long debugging time, e.g. 30 minutes, you may want to cancel the wait for a rerun. Since it is impossible to kill a running lambda instance on AWS, the easiest way to cancel the wait is to send a random message to proxy_lambda_resp queue via the AWS console. The waiting proxy-lambda will forward it to the caller and become available for a new request.

Large payloads and data compression

The size of the SQS payload is limited to 262,144 bytes by SQS while Lambda allows up to 6MB. proxy-lambda and lambda-debugger compress oversized payloads using flate2 crate and send them as an encoded Base58 string to get around that limitation.

The data compression can take up to a minute in debug mode. It is significantly faster with release builds.

Logging

Both proxy-lambda and lambda-debugger use RUST_LOG env var to set the logging level and filters. If RUST_LOG is not present or is empty, both crates log at the INFO level and suppress logging from their dependencies. See [https://docs.rs/tracing-subscriber/latest/tracing_subscriber/filter/struct.EnvFilter.html#example-syntax] for more info.

Examples of RUST_LOG values:

• error - log errors only from all crates and dependencies
• warn,lambda_debugger=info - INFO level for the lambda-debugger, WARN level for everything else
• proxy=debug - detailed logging in proxy-lambda