Error while deploying Rails API: CREATE_FAILED AWS::CloudFormation::Stack JetsRackController Embedded


#1

Rails API 5
Jets 1.5.3

Deploying CloudFormation stack with jets app!
09:40:51PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack mikweb-api-dev User Initiated
09:40:57PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack ApiGateway 
09:40:57PM CREATE_IN_PROGRESS AWS::IAM::Role IamRole 
09:40:57PM CREATE_IN_PROGRESS AWS::Lambda::LayerVersion GemLayer 
09:40:58PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack ApiGateway Resource creation Initiated
09:40:59PM CREATE_IN_PROGRESS AWS::IAM::Role IamRole Resource creation Initiated
09:41:17PM CREATE_COMPLETE AWS::IAM::Role IamRole 
09:41:20PM CREATE_COMPLETE AWS::CloudFormation::Stack ApiGateway 
09:41:22PM CREATE_IN_PROGRESS AWS::Lambda::LayerVersion GemLayer Resource creation Initiated
09:41:22PM CREATE_COMPLETE AWS::Lambda::LayerVersion GemLayer 
09:41:25PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack JetsPreheatJob 
09:41:26PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack JetsRackController 
09:41:26PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack JetsPreheatJob Resource creation Initiated
09:41:27PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack JetsRackController Resource creation Initiated
09:41:49PM CREATE_FAILED AWS::CloudFormation::Stack JetsRackController Embedded stack arn:aws:cloudformation:sa-east-1:824805048872:stack/mikweb-api-dev-JetsRackController-1HVLIGOQ89ZMH/4abfb160-185e-11e9-8176-50a68a7fb8c5 was not successfully created: The following resource(s) failed to create: [ProcessLambdaFunction]. 
09:41:50PM CREATE_FAILED AWS::CloudFormation::Stack JetsPreheatJob Resource creation cancelled
09:41:51PM UPDATE_ROLLBACK_IN_PROGRESS AWS::CloudFormation::Stack mikweb-api-dev The following resource(s) failed to create: [JetsPreheatJob, JetsRackController]. 
09:42:05PM UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS AWS::CloudFormation::Stack mikweb-api-dev 
09:42:07PM DELETE_IN_PROGRESS AWS::CloudFormation::Stack JetsPreheatJob 
09:42:07PM DELETE_IN_PROGRESS AWS::CloudFormation::Stack JetsRackController 
09:42:18PM DELETE_COMPLETE AWS::CloudFormation::Stack JetsRackController 
09:42:19PM DELETE_IN_PROGRESS AWS::CloudFormation::Stack ApiGateway 
09:42:30PM DELETE_COMPLETE AWS::CloudFormation::Stack ApiGateway 
09:42:41PM DELETE_COMPLETE AWS::CloudFormation::Stack JetsPreheatJob 
09:42:43PM DELETE_IN_PROGRESS AWS::Lambda::LayerVersion GemLayer 
09:42:43PM DELETE_IN_PROGRESS AWS::IAM::Role IamRole 
09:42:45PM DELETE_COMPLETE AWS::Lambda::LayerVersion GemLayer 
09:42:45PM DELETE_COMPLETE AWS::IAM::Role IamRole 
09:42:46PM UPDATE_ROLLBACK_COMPLETE AWS::CloudFormation::Stack mikweb-api-dev 
Stack rolled back: UPDATE_ROLLBACK_COMPLETE
Time took for stack deployment: 1m 55s.
Prewarming application.
The function mikweb-api-dev-jets-preheat_job-warm was not found.  Maybe check the spelling or the AWS_PROFILE?
The function mikweb-api-dev-jets-preheat_job-warm was not found.  Maybe check the spelling or the AWS_PROFILE?

#2

@josuetex When you get a chance, can you click on the JetsRackController stack while it’s deploying and screenshot that. It might show some more detailed information on the specific child stack. :ok_hand:


#3

I have identified that when I add some environment variable in the .env file the error happens.


#4

Problem found:

Lambda was unable to configure your environment variables because the environment variables you have provided contains reserved keys that are currently not supported for modification. Reserved keys used in this request: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: 21300da3-186d-11e9-b33f-d3ab82174bca)


#5

@josuetex Thanks so much for following up and explaining the issue. This is helpful! You’re awesome :grin: Someone else ran into this also. This feels common enough to add some checking into Jets. It’ll save everyone time!


#6

You are welcome.

Now, i am with other problem:

Can’t connect to local MySQL server through socket ‘/var/lib/mysql/mysql.sock’ (2)

RDS Mysql Database

database.yml

default: &default
encoding: utf8
adapter: mysql2

development:
<<: *default
host: <%= ENV['DATABASE_HOST'] %>
database: <%= ENV['DATABASE_NAME'] %>
username: <%= ENV['DATABASE_USER'] %>
password: <%= ENV['DATABASE_PASSWORD'] %>
pool: 10

test:
<<: *default
host: <%= ENV['DATABASE_HOST'] %>
database: <%= ENV['DATABASE_NAME'] %>
username: <%= ENV['DATABASE_USER'] %>
password: <%= ENV['DATABASE_PASSWORD'] %>
pool: 10

staging:
<<: *default
host: <%= ENV['DATABASE_HOST'] %>
database: <%= ENV['DATABASE_NAME'] %>
username: <%= ENV['DATABASE_USER'] %>
password: <%= ENV['DATABASE_PASSWORD'] %>
pool: 10

production:
<<: *default
host: <%= ENV['DATABASE_HOST'] %>
database: <%= ENV['DATABASE_NAME'] %>
username: <%= ENV['DATABASE_USER'] %>
password: <%= ENV['DATABASE_PASSWORD'] %>
pool: 10

.env

DATABASE_NAME=xxxxxxxxxxxxxxx
DATABASE_USER=xxxxxxxxx
DATABASE_PASSWORD=xxxxxxxxxxx
DATABASE_HOST=xxxxxxxxxxxxxxxxxxxxxxxxxxx

#7

:thinking: It seems like everything looks good. Wondering if you can check the Lambda Console function properties and confirmed that the DATABASE_* environment variables are set, just in case.

The error message:

Can’t connect to local MySQL server through socket ‘/var/lib/mysql/mysql.sock’ (2)

Would show usually if the db host is not set and then the default behavior of rails and the mysql2 client I believe it to connect via a local socket instead of a host, which is what you’re seeing.


#8

Problem resolved and deployed.

Now, other problem:

START RequestId: 1a50a20a-ef29-45db-8326-68d86afaa4ee Version: $LATEST

Init error when loading handler handlers/controllers/jets/rack_controller.process

{

"errorMessage": "undefined method `layers=' for nil:NilClass",

"errorType": "Init&lt;NoMethodError&gt;",

"stackTrace": [

"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/application.rb:127:in `instance_eval'",

"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/application.rb:11:in `instance_eval'",

"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/application.rb:11:in `configure'",

"(eval):1:in `load_environments_config'",

"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/application.rb:127:in `instance_eval'",

"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/application.rb:127:in `load_environments_config'",

"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/application.rb:134:in `load_configs'",

"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/application.rb:15:in `setup!'",

"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/booter.rb:11:in `boot!'",

"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/core.rb:18:in `boot'",

"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/core.rb:197:in `once'",

"/var/task/handlers/controllers/jets/rack_controller.rb:3:in `&lt;top (required)&gt;'",

"/var/lang/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'",

"/var/lang/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'"

]

}

END RequestId: 1a50a20a-ef29-45db-8326-68d86afaa4ee

REPORT RequestId: 1a50a20a-ef29-45db-8326-68d86afaa4ee Duration: 2022.09 ms Billed Duration: 2100 ms Memory Size: 1536 MB Max Memory Used: 73 MB

Unknown application error occurred
Init&lt;NoMethodError&gt;

Init error when loading handler handlers/controllers/jets/rack_controller.process

{

"errorMessage": "undefined method `layers=' for nil:NilClass",

"errorType": "Init&lt;NoMethodError&gt;",

"stackTrace": [

"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/application.rb:127:in `instance_eval'",

"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/application.rb:11:in `instance_eval'",

"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/application.rb:11:in `configure'",

"(eval):1:in `load_environments_config'",

"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/application.rb:127:in `instance_eval'",

"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/application.rb:127:in `load_environments_config'",

"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/application.rb:134:in `load_configs'",

"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/application.rb:15:in `setup!'",

"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/booter.rb:11:in `boot!'",

"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/core.rb:18:in `boot'",

"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/core.rb:197:in `once'",

"/var/task/handlers/controllers/jets/rack_controller.rb:3:in `&lt;top (required)&gt;'",

"/var/lang/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'",

"/var/lang/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'"

#9

Noticed jets is on version 1.3.9.

Lets try upgrading and we’ll go from there. :ok_hand:


#10

Also, if you can, what was the problem?


#11

When trying to access the api, the error reported in CloudWatch occurs.


#12

I see. Just in case here are the Minimal Deploy IAM docs You may want to double check that the IAM user you are using to run jets deploy has all the necessary permissions. Though you might already be good. :ok_hand:

Let’s see how it goes after the upgrade.


#13

User with full access.

I’m using the version 1.5.3 from jets CLI.


#14

On deploy, the CLI jets install the version 1.3.9.

Fetching jets 1.3.9
Installing jets 1.3.9


#15

I see. Think this is a relatively new bug. Will dig into it to confirm.


#16

Fixed in https://github.com/tongueroo/jets/pull/147

Released in v1.5.4 CHANGELOG

Try updating the jets cli again. This should upgrade the jets version that gets installed part the Jets deploy. It’ll still be a version back but the 1.5.3 version should work.

Essentially, The version of Jets afterburner was using did not yet support Custom Lambda Layers Jets afterburner mode uses a locked version of jets to ensure some level of control. Might change this in the future.


#17

Now the problem is:

5.0/gems/jets-1.5.3/lib/jets/processors/main_processor.rb:32:in `run'
  /opt/ruby/gems/2.5.0/gems/jets-1.5.3/lib/jets/core.rb:185:in `process'
  /opt/ruby/gems/2.5.0/gems/jets-1.5.3/lib/jets/core.rb:193:in `block in handler'
  /var/runtime/lib/lambda_handler.rb:24:in `call_handler'
  /var/runtime/lib/runtime.rb:42:in `<main>'
Error raised from handler method
{
  "errorMessage": "Failed to open TCP connection to localhost:9292 (Connection refused - connect(2) for \"localhost\" port 9292)",
  "errorType": "Function<Errno::ECONNREFUSED>",
  "stackTrace": [
    "/var/lang/lib/ruby/2.5.0/net/http.rb:939:in `rescue in block in connect'",
    "/var/lang/lib/ruby/2.5.0/net/http.rb:936:in `block in connect'",
    "/var/lang/lib/ruby/2.5.0/timeout.rb:93:in `block in timeout'",
    "/var/lang/lib/ruby/2.5.0/timeout.rb:103:in `timeout'",
    "/var/lang/lib/ruby/2.5.0/net/http.rb:935:in `connect'",
    "/var/lang/lib/ruby/2.5.0/net/http.rb:920:in `do_start'",
    "/var/lang/lib/ruby/2.5.0/net/http.rb:909:in `start'",
    "/var/lang/lib/ruby/2.5.0/net/http.rb:1455:in `request'",
    "/opt/ruby/gems/2.5.0/gems/jets-1.5.3/lib/jets/mega/request.rb:47:in `proxy'",
    "/opt/ruby/gems/2.5.0/gems/jets-1.5.3/lib/jets/internal/app/controllers/jets/rack_controller.rb:23:in `mega_request'",
    "/opt/ruby/gems/2.5.0/gems/jets-1.5.3/lib/jets/internal/app/controllers/jets/rack_controller.rb:7:in `process'",
    "/opt/ruby/gems/2.5.0/gems/jets-1.5.3/lib/jets/controller/base.rb:41:in `dispatch!'",
    "/opt/ruby/gems/2.5.0/gems/jets-1.5.3/lib/jets/internal/app/controllers/jets/rack_controller.rb:16:in `process!'",
    "/opt/ruby/gems/2.5.0/gems/jets-1.5.3/lib/jets/controller/base.rb:82:in `process'",
    "./app/controllers/jets/rack_controller.rb:1:in `run'",
    "/opt/ruby/gems/2.5.0/gems/jets-1.5.3/lib/jets/processors/main_processor.rb:32:in `instance_eval'",
    "/opt/ruby/gems/2.5.0/gems/jets-1.5.3/lib/jets/processors/main_processor.rb:32:in `run'",
    "/opt/ruby/gems/2.5.0/gems/jets-1.5.3/lib/jets/core.rb:185:in `process'",
    "/opt/ruby/gems/2.5.0/gems/jets-1.5.3/lib/jets/core.rb:193:in `block in handler'"
  ]
}
{

#18

This indicates that the rails app is not starting up. With Jets Afterburner mode, it generates a wrapper Jets app, places the rails app in a rack subfolder, and proxies to the rails app. This is known as Mega Mode and what makes it all possible.

So what’s going on is the rack (Rails) application is failing to start. Once you can get it started successfully then it should be good.

Try going up farther in the CloudWatch logs and see if you can find an error about rackup failing to start. That is the key. Was able to just deploy the tongueroo/demo-rails-api app with Jets Afterburner mode successfully. So, at this point think it’s project specific issue :face_with_monocle: Try to go farther up the CloudWatch logs to find out where rackup is failing to start.

The startup process only happens on the first request for each concurrent lambda function. So you might have better luck finding the error on a fresh deploy. So deploy again, hit app with a request, and then look at the CloudWatch logs around the time at the deploy finishes. The prewarming requests as part of the deploy will trigger all of this.


#19

The problem is in the custom layer.

[0;32;49m=> cd ./rack && bin/rackupe[0m

/var/runtime/gems/bundler-1.17.1/lib/bundler/spec_set.rb:91:in `block in materialize’: Could not find curb-0.9.6 in any of the sources (Bundler::GemNotFound)

How to packaging a custom layer?


#20

Custom layers are documented here: https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html :ok_hand: