Docker 4.33 AWS Lambda incompatibility with invocation

[ChronosMasterOfAllTime]

Description

I opened an issue on

But Since upgrading to docker 4.33, it breaks the invocation of lambdas when using sam local invoke or sam local start api

Getting the following error:

2024-08-07 09:21:09,933 | Using local image: public.ecr.aws/lambda/provided:al2-rapid-x86_64.                                                                              
                                                                                                                                                                           
2024-08-07 09:21:09,935 | Mounting /Users/chronos/REDACTED/bin as /var/task:ro,delegated, inside runtime container               
2024-08-07 09:21:10,297 | Starting a timer for 900 seconds for function 'REDACTED'                                                                                        
2024-08-07 09:21:10,299 | Getting lock for the key localhost-7971                                                                                                          
2024-08-07 09:21:10,300 | Waiting to retrieve the lock (localhost-7971) to start invocation                                                                                
START RequestId: 69028466-59f4-401e-b785-5188b26c4e07 Version: $LATEST
07 Aug 2024 14:21:10,307 [ERROR] (rapid) Init failed error=fork/exec /var/task/bootstrap: invalid argument InvokeID=
2024-08-07 09:21:10,312 | Failed to deserialize response from RIE, returning the raw response as is                                                                        
07 Aug 2024 14:21:10,310 [ERROR] (rapid) Invoke failed error=fork/exec /var/task/bootstrap: invalid argument InvokeID=f21b3f5c-2daf-4002-ba71-30b0c91ea19e
07 Aug 2024 14:21:10,310 [ERROR] (rapid) Invoke DONE failed: Runtime.InvalidEntrypoint
This works fine with docker 4.32. Can you please investigate what behaviorally changed between the versions?

Reproduce

Install AWS SAM CLI
Install Golang (latest is fine)
Install Docker 4.33

Clone cd into aws-sam-examples/tree/master/samples_1/go-lambda

go mod init test
go mod tidy
GOOS=linux go build -o bootstrap main.go
sam local invoke

modify the template.yml file to the following:

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: Hello World Go Lambda function

Resources:
  Function:
    Type: AWS::Serverless::Function
    Properties:
      Handler: bootstrap
      CodeUri: ./
      Runtime: go1.x

Expected behavior

Output as described in the README

docker version

Client:
 Version:           27.1.1
 API version:       1.46
 Go version:        go1.21.12
 Git commit:        6312585
 Built:             Tue Jul 23 19:54:12 2024
 OS/Arch:           darwin/amd64
 Context:           desktop-linux

Server: Docker Desktop 4.33.0 (160616)
 Engine:
  Version:          27.1.1
  API version:      1.46 (minimum version 1.24)
  Go version:       go1.21.12
  Git commit:       cc13f95
  Built:            Tue Jul 23 19:57:19 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.19
  GitCommit:        2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
 runc:
  Version:          1.7.19
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    27.1.1
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.1-desktop.1
    Path:     /Users/a6040963/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.1-desktop.1
    Path:     /Users/a6040963/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.34
    Path:     /Users/a6040963/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Alpha) (Docker Inc.)
    Version:  v0.0.14
    Path:     /Users/a6040963/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/a6040963/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.25
    Path:     /Users/a6040963/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/a6040963/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.3.0
    Path:     /Users/a6040963/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/a6040963/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.11.0
    Path:     /Users/a6040963/.docker/cli-plugins/docker-scout

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 10
 Server Version: 27.1.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.10.0-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.62GiB
 Name: docker-desktop
 ID: f4d0632b-21c0-406a-a049-3b60d4000856
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/a6040963/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

Diagnostics ID

957483F6-D02A-46D9-A7D0-6228E0795736/20240808163555

Additional Info

Mac OS Ventura 13.6.7 X86_64

[andrea-reale]

Hello there,

Thanks for reporting the issue!

It looks like this is a regression on Docker Desktop's side related to exposing file system extended attributes on bind mounts. It is only triggered in rare circumstances but the combination of sam local and the Lambda RIE containers seems one of them.

Please, feel free to try out this development build that should fix the issue for 4.33 and let us know how that works out. The fix will be shipped with the next Docker Desktop release.

• MacOS arm64 (DMG)
• Windows amd64 (installer)

[ChronosMasterOfAllTime]

Hello there,

Thanks for reporting the issue!

It looks like this is a regression on Docker Desktop's side related to exposing file system extended attributes on bind mounts. It is only triggered in rare circumstances but the combination of sam local and the Lambda RIE containers seems one of them.

Please, feel free to try out this development build that should fix the issue for 4.33 and let us know how that works out. The fix will be shipped with the next Docker Desktop release.

• MacOS arm64 (DMG)
• Windows amd64 (installer)

Great! Thank you for looking into this.

[MihaelaStoica]

A fix for this issue has been included in the 4.34.0 release, more information in the release notes.

Closing this issue, but if the problem persists please open a new one.

[afbarbaro]

I think this may have broken something else with setting extended attributes.

This python call works in the previous version 4.33.0 and breaks in 4.34.0 which includes this change. I verified by rolling back and forward 😞. I'll report as a new issue soon.

os.setxattr(file_path, attribute_name, attribute_value_bytes)

[0Qaismoradi]

"imageList": {
"images": [
"docker.io/localstack/localstack:",
"docker.io/testcontainers/ryuk:",
"docker:cli"
]
}

[0Qaismoradi]

os.setxattr(file_path, attribute_name, attribute_value_bytes)