Rails API 5
Jets 1.5.3
Deploying CloudFormation stack with jets app!
09:40:51PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack mikweb-api-dev User Initiated
09:40:57PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack ApiGateway
09:40:57PM CREATE_IN_PROGRESS AWS::IAM::Role IamRole
09:40:57PM CREATE_IN_PROGRESS AWS::Lambda::LayerVersion GemLayer
09:40:58PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack ApiGateway Resource creation Initiated
09:40:59PM CREATE_IN_PROGRESS AWS::IAM::Role IamRole Resource creation Initiated
09:41:17PM CREATE_COMPLETE AWS::IAM::Role IamRole
09:41:20PM CREATE_COMPLETE AWS::CloudFormation::Stack ApiGateway
09:41:22PM CREATE_IN_PROGRESS AWS::Lambda::LayerVersion GemLayer Resource creation Initiated
09:41:22PM CREATE_COMPLETE AWS::Lambda::LayerVersion GemLayer
09:41:25PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack JetsPreheatJob
09:41:26PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack JetsRackController
09:41:26PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack JetsPreheatJob Resource creation Initiated
09:41:27PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack JetsRackController Resource creation Initiated
09:41:49PM CREATE_FAILED AWS::CloudFormation::Stack JetsRackController Embedded stack arn:aws:cloudformation:sa-east-1:824805048872:stack/mikweb-api-dev-JetsRackController-1HVLIGOQ89ZMH/4abfb160-185e-11e9-8176-50a68a7fb8c5 was not successfully created: The following resource(s) failed to create: [ProcessLambdaFunction].
09:41:50PM CREATE_FAILED AWS::CloudFormation::Stack JetsPreheatJob Resource creation cancelled
09:41:51PM UPDATE_ROLLBACK_IN_PROGRESS AWS::CloudFormation::Stack mikweb-api-dev The following resource(s) failed to create: [JetsPreheatJob, JetsRackController].
09:42:05PM UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS AWS::CloudFormation::Stack mikweb-api-dev
09:42:07PM DELETE_IN_PROGRESS AWS::CloudFormation::Stack JetsPreheatJob
09:42:07PM DELETE_IN_PROGRESS AWS::CloudFormation::Stack JetsRackController
09:42:18PM DELETE_COMPLETE AWS::CloudFormation::Stack JetsRackController
09:42:19PM DELETE_IN_PROGRESS AWS::CloudFormation::Stack ApiGateway
09:42:30PM DELETE_COMPLETE AWS::CloudFormation::Stack ApiGateway
09:42:41PM DELETE_COMPLETE AWS::CloudFormation::Stack JetsPreheatJob
09:42:43PM DELETE_IN_PROGRESS AWS::Lambda::LayerVersion GemLayer
09:42:43PM DELETE_IN_PROGRESS AWS::IAM::Role IamRole
09:42:45PM DELETE_COMPLETE AWS::Lambda::LayerVersion GemLayer
09:42:45PM DELETE_COMPLETE AWS::IAM::Role IamRole
09:42:46PM UPDATE_ROLLBACK_COMPLETE AWS::CloudFormation::Stack mikweb-api-dev
Stack rolled back: UPDATE_ROLLBACK_COMPLETE
Time took for stack deployment: 1m 55s.
Prewarming application.
The function mikweb-api-dev-jets-preheat_job-warm was not found. Maybe check the spelling or the AWS_PROFILE?
The function mikweb-api-dev-jets-preheat_job-warm was not found. Maybe check the spelling or the AWS_PROFILE?
tung
January 15, 2019, 2:23am
2
@josuetex When you get a chance, can you click on the JetsRackController stack while it’s deploying and screenshot that. It might show some more detailed information on the specific child stack.
I have identified that when I add some environment variable in the .env file the error happens.
Problem found:
Lambda was unable to configure your environment variables because the environment variables you have provided contains reserved keys that are currently not supported for modification. Reserved keys used in this request: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: 21300da3-186d-11e9-b33f-d3ab82174bca)
tung
January 15, 2019, 2:42am
5
@josuetex Thanks so much for following up and explaining the issue. This is helpful! You’re awesome Someone else ran into this also. This feels common enough to add some checking into Jets. It’ll save everyone time!
You are welcome.
Now, i am with other problem:
Can’t connect to local MySQL server through socket ‘/var/lib/mysql/mysql.sock’ (2)
RDS Mysql Database
database.yml
default: &default
encoding: utf8
adapter: mysql2
development:
<<: *default
host: <%= ENV['DATABASE_HOST'] %>
database: <%= ENV['DATABASE_NAME'] %>
username: <%= ENV['DATABASE_USER'] %>
password: <%= ENV['DATABASE_PASSWORD'] %>
pool: 10
test:
<<: *default
host: <%= ENV['DATABASE_HOST'] %>
database: <%= ENV['DATABASE_NAME'] %>
username: <%= ENV['DATABASE_USER'] %>
password: <%= ENV['DATABASE_PASSWORD'] %>
pool: 10
staging:
<<: *default
host: <%= ENV['DATABASE_HOST'] %>
database: <%= ENV['DATABASE_NAME'] %>
username: <%= ENV['DATABASE_USER'] %>
password: <%= ENV['DATABASE_PASSWORD'] %>
pool: 10
production:
<<: *default
host: <%= ENV['DATABASE_HOST'] %>
database: <%= ENV['DATABASE_NAME'] %>
username: <%= ENV['DATABASE_USER'] %>
password: <%= ENV['DATABASE_PASSWORD'] %>
pool: 10
.env
DATABASE_NAME=xxxxxxxxxxxxxxx
DATABASE_USER=xxxxxxxxx
DATABASE_PASSWORD=xxxxxxxxxxx
DATABASE_HOST=xxxxxxxxxxxxxxxxxxxxxxxxxxx
tung
January 15, 2019, 3:08am
7
It seems like everything looks good. Wondering if you can check the Lambda Console function properties and confirmed that the DATABASE_*
environment variables are set, just in case.
The error message:
Can’t connect to local MySQL server through socket ‘/var/lib/mysql/mysql.sock’ (2)
Would show usually if the db host is not set and then the default behavior of rails and the mysql2 client I believe it to connect via a local socket instead of a host, which is what you’re seeing.
Problem resolved and deployed.
Now, other problem:
START RequestId: 1a50a20a-ef29-45db-8326-68d86afaa4ee Version: $LATEST
Init error when loading handler handlers/controllers/jets/rack_controller.process
{
"errorMessage": "undefined method `layers=' for nil:NilClass",
"errorType": "Init<NoMethodError>",
"stackTrace": [
"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/application.rb:127:in `instance_eval'",
"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/application.rb:11:in `instance_eval'",
"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/application.rb:11:in `configure'",
"(eval):1:in `load_environments_config'",
"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/application.rb:127:in `instance_eval'",
"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/application.rb:127:in `load_environments_config'",
"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/application.rb:134:in `load_configs'",
"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/application.rb:15:in `setup!'",
"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/booter.rb:11:in `boot!'",
"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/core.rb:18:in `boot'",
"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/core.rb:197:in `once'",
"/var/task/handlers/controllers/jets/rack_controller.rb:3:in `<top (required)>'",
"/var/lang/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'",
"/var/lang/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'"
]
}
END RequestId: 1a50a20a-ef29-45db-8326-68d86afaa4ee
REPORT RequestId: 1a50a20a-ef29-45db-8326-68d86afaa4ee Duration: 2022.09 ms Billed Duration: 2100 ms Memory Size: 1536 MB Max Memory Used: 73 MB
Unknown application error occurred
Init<NoMethodError>
Init error when loading handler handlers/controllers/jets/rack_controller.process
{
"errorMessage": "undefined method `layers=' for nil:NilClass",
"errorType": "Init<NoMethodError>",
"stackTrace": [
"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/application.rb:127:in `instance_eval'",
"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/application.rb:11:in `instance_eval'",
"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/application.rb:11:in `configure'",
"(eval):1:in `load_environments_config'",
"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/application.rb:127:in `instance_eval'",
"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/application.rb:127:in `load_environments_config'",
"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/application.rb:134:in `load_configs'",
"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/application.rb:15:in `setup!'",
"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/booter.rb:11:in `boot!'",
"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/core.rb:18:in `boot'",
"/opt/ruby/gems/2.5.0/gems/jets-1.3.9/lib/jets/core.rb:197:in `once'",
"/var/task/handlers/controllers/jets/rack_controller.rb:3:in `<top (required)>'",
"/var/lang/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'",
"/var/lang/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'"
tung
January 15, 2019, 3:38am
9
Noticed jets is on version 1.3.9.
josuetex:
jets-1.3.9
Lets try upgrading and we’ll go from there.
tung
January 15, 2019, 3:39am
10
Also, if you can, what was the problem?
When trying to access the api, the error reported in CloudWatch occurs.
tung
January 15, 2019, 3:50am
12
I see. Just in case here are the Minimal Deploy IAM docs You may want to double check that the IAM user you are using to run jets deploy has all the necessary permissions. Though you might already be good.
Let’s see how it goes after the upgrade.
User with full access.
I’m using the version 1.5.3 from jets CLI.
On deploy, the CLI jets install the version 1.3.9.
Fetching jets 1.3.9
Installing jets 1.3.9
tung
January 15, 2019, 4:32am
15
I see. Think this is a relatively new bug. Will dig into it to confirm.
1 Like
tung
January 15, 2019, 5:42am
16
Fixed in https://github.com/tongueroo/jets/pull/147
Released in v1.5.4 CHANGELOG
Try updating the jets cli again. This should upgrade the jets version that gets installed part the Jets deploy. It’ll still be a version back but the 1.5.3 version should work.
Essentially, The version of Jets afterburner was using did not yet support Custom Lambda Layers Jets afterburner mode uses a locked version of jets to ensure some level of control. Might change this in the future.
1 Like
Now the problem is:
5.0/gems/jets-1.5.3/lib/jets/processors/main_processor.rb:32:in `run'
/opt/ruby/gems/2.5.0/gems/jets-1.5.3/lib/jets/core.rb:185:in `process'
/opt/ruby/gems/2.5.0/gems/jets-1.5.3/lib/jets/core.rb:193:in `block in handler'
/var/runtime/lib/lambda_handler.rb:24:in `call_handler'
/var/runtime/lib/runtime.rb:42:in `<main>'
Error raised from handler method
{
"errorMessage": "Failed to open TCP connection to localhost:9292 (Connection refused - connect(2) for \"localhost\" port 9292)",
"errorType": "Function<Errno::ECONNREFUSED>",
"stackTrace": [
"/var/lang/lib/ruby/2.5.0/net/http.rb:939:in `rescue in block in connect'",
"/var/lang/lib/ruby/2.5.0/net/http.rb:936:in `block in connect'",
"/var/lang/lib/ruby/2.5.0/timeout.rb:93:in `block in timeout'",
"/var/lang/lib/ruby/2.5.0/timeout.rb:103:in `timeout'",
"/var/lang/lib/ruby/2.5.0/net/http.rb:935:in `connect'",
"/var/lang/lib/ruby/2.5.0/net/http.rb:920:in `do_start'",
"/var/lang/lib/ruby/2.5.0/net/http.rb:909:in `start'",
"/var/lang/lib/ruby/2.5.0/net/http.rb:1455:in `request'",
"/opt/ruby/gems/2.5.0/gems/jets-1.5.3/lib/jets/mega/request.rb:47:in `proxy'",
"/opt/ruby/gems/2.5.0/gems/jets-1.5.3/lib/jets/internal/app/controllers/jets/rack_controller.rb:23:in `mega_request'",
"/opt/ruby/gems/2.5.0/gems/jets-1.5.3/lib/jets/internal/app/controllers/jets/rack_controller.rb:7:in `process'",
"/opt/ruby/gems/2.5.0/gems/jets-1.5.3/lib/jets/controller/base.rb:41:in `dispatch!'",
"/opt/ruby/gems/2.5.0/gems/jets-1.5.3/lib/jets/internal/app/controllers/jets/rack_controller.rb:16:in `process!'",
"/opt/ruby/gems/2.5.0/gems/jets-1.5.3/lib/jets/controller/base.rb:82:in `process'",
"./app/controllers/jets/rack_controller.rb:1:in `run'",
"/opt/ruby/gems/2.5.0/gems/jets-1.5.3/lib/jets/processors/main_processor.rb:32:in `instance_eval'",
"/opt/ruby/gems/2.5.0/gems/jets-1.5.3/lib/jets/processors/main_processor.rb:32:in `run'",
"/opt/ruby/gems/2.5.0/gems/jets-1.5.3/lib/jets/core.rb:185:in `process'",
"/opt/ruby/gems/2.5.0/gems/jets-1.5.3/lib/jets/core.rb:193:in `block in handler'"
]
}
{
tung
January 15, 2019, 6:52am
18
This indicates that the rails app is not starting up. With Jets Afterburner mode, it generates a wrapper Jets app, places the rails app in a rack subfolder, and proxies to the rails app. This is known as Mega Mode and what makes it all possible.
So what’s going on is the rack (Rails) application is failing to start. Once you can get it started successfully then it should be good.
Try going up farther in the CloudWatch logs and see if you can find an error about rackup failing to start. That is the key. Was able to just deploy the tongueroo/demo-rails-api app with Jets Afterburner mode successfully. So, at this point think it’s project specific issue Try to go farther up the CloudWatch logs to find out where rackup is failing to start.
The startup process only happens on the first request for each concurrent lambda function. So you might have better luck finding the error on a fresh deploy. So deploy again, hit app with a request, and then look at the CloudWatch logs around the time at the deploy finishes. The prewarming requests as part of the deploy will trigger all of this.
The problem is in the custom layer.
[0;32;49m=> cd ./rack && bin/rackupe[0m
/var/runtime/gems/bundler-1.17.1/lib/bundler/spec_set.rb:91:in `block in materialize’: Could not find curb-0.9.6 in any of the sources (Bundler::GemNotFound)
How to packaging a custom layer?
tung
January 15, 2019, 7:00am
20
1 Like