I made an endpoint and it was working on my local however after deployment to AWS I got this issue. This is happening at actionmailer’s initialization in Jets.once method in the lambda function. I do not use any mailer and I have no clue so far. Any help would be appreciated. Thank you.
Thanks for the report. Interestingly, when digging into Ruby 2.7 a few days ago, also ran into this. But deployed again and then was unable to reproduce. It seems like one of those tough bugs that happens intermittently.
Seems like Lambda Environment can get slightly different environments, maybe based on different AMIs? And depending on which AMI the lambda function happens to get the bug surfaces.
Thought it was specific to the Lambda Ruby 2.7 runtime at the time though. But this makes it obvious that its also happening in the Ruby 2.5 runtime. Bummer!
Released temp fix https://github.com/tongueroo/jets/pull/446 Am not a fan of the fix because it actually hides the issue. But better than a crash for actionmailer, which is not as used much. Will dig into a better fix later.
This is my first post, so, first, thanks a lot for this great Jets tool/framework!
It seems that we are having the exact same problem with our application, but… it doesn’t seem to be random, it seems to happen all the time now.
{“errorMessage”:“undefined method default_url_options' for nil:NilClass","errorType":"Init\u003cNoMethodError\u003e","stackTrace":["/opt/ruby/gems/2.5.0/gems/jets-1.9.16/lib/jets/internal/turbines/jets/mailer.rb:12:inblock in \u003cclass:Mailer\u003e’”,"/opt/ruby/gems/2.5.0/gems/jets-1.9.16/lib/jets/booter.rb:66:in block (2 levels) in run_turbines'","/opt/ruby/gems/2.5.0/gems/jets-1.9.16/lib/jets/booter.rb:65:ineach’","/opt/ruby/gems/2.5.0/gems/jets-1.9.16/lib/jets/booter.rb:65:in block in run_turbines'","/opt/ruby/gems/2.5.0/gems/jets-1.9.16/lib/jets/booter.rb:63:ineach’","/opt/ruby/gems/2.5.0/gems/jets-1.9.16/lib/jets/booter.rb:63:in run_turbines'","/opt/ruby/gems/2.5.0/gems/jets-1.9.16/lib/jets/booter.rb:19:inboot!’","/opt/ruby/gems/2.5.0/gems/jets-1.9.16/lib/jets/core.rb:18:in boot'","/opt/ruby/gems/2.5.0/gems/jets-1.9.16/lib/jets/core.rb:125:inonce’","/var/task/handlers/controllers/users_controller.rb:3:in \u003ctop (required)\u003e'","/var/lang/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:92:inrequire’","/var/lang/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:92:in `require’"]}
We are using Jets v1.9.16 (I know, a bit old :-/ )
And ruby 2.5.7
But… we are also using action_mailer (to send emails through SMTP) and properly configured it (it was working untill we deployed a new version of our app recently).
So, what should we do?
Should we upgrade to last jets version? If so, will we still be able to use action_mailer properly?
Dug into this. This one has been annoyingly difficult to reproduce. Deployed a test app to 6 different AWS regions in case it was region specific. Got emails from all 6 regions.
Hoping someone who is getting this issue can deploy it. Wondering if you’re seeing if the config.object_id change within the same “container” consistently. Here’s my a debugging session, where the lambda function is called repeatedly with:
jets call hard_job-dig --show-log
And it shows the same config.object_id.
$ aws-logs tail /aws/lambda/mailer-dev-hard_job-dig | grep 'config.object_id'
2020-04-15 18:50:07 UTC 2020/04/15/[$LATEST]c68e16ddfa204a30b34abdf36c52aefa application.rb load_default_config2 @config.object_id 47006458941220
2020-04-15 18:50:07 UTC 2020/04/15/[$LATEST]c68e16ddfa204a30b34abdf36c52aefa application.rb setup! @config.object_id 47006458941220
2020-04-15 18:50:07 UTC 2020/04/15/[$LATEST]c68e16ddfa204a30b34abdf36c52aefa mailer.rb initializer config.object_id 47006458941220
2020-04-15 18:50:08 UTC 2020/04/15/[$LATEST]c68e16ddfa204a30b34abdf36c52aefa config/application/development.rb config.object_id: 47006458941220
2020-04-15 18:50:08 UTC 2020/04/15/[$LATEST]c68e16ddfa204a30b34abdf36c52aefa application.rb configs! @config.object_id 47006458941220
2020-04-15 18:50:08 UTC 2020/04/15/[$LATEST]c68e16ddfa204a30b34abdf36c52aefa Jets.config.object_id 47006458941220
2020-04-15 18:50:08 UTC 2020/04/15/[$LATEST]c68e16ddfa204a30b34abdf36c52aefa HardJob Jets.config.object_id 47006458941220
2020-04-15 18:50:38 UTC 2020/04/15/[$LATEST]c68e16ddfa204a30b34abdf36c52aefa HardJob Jets.config.object_id 47006458941220
2020-04-15 18:51:00 UTC 2020/04/15/[$LATEST]c68e16ddfa204a30b34abdf36c52aefa HardJob Jets.config.object_id 47006458941220
2020-04-15 18:51:21 UTC 2020/04/15/[$LATEST]c68e16ddfa204a30b34abdf36c52aefa HardJob Jets.config.object_id 47006458941220
2020-04-15 18:51:28 UTC 2020/04/15/[$LATEST]c68e16ddfa204a30b34abdf36c52aefa HardJob Jets.config.object_id 47006458941220
2020-04-15 18:51:36 UTC 2020/04/15/[$LATEST]c68e16ddfa204a30b34abdf36c52aefa HardJob Jets.config.object_id 47006458941220
2020-04-15 18:51:47 UTC 2020/04/15/[$LATEST]c68e16ddfa204a30b34abdf36c52aefa HardJob Jets.config.object_id 47006458941220
Remember to check that you’re looking at the logs from the same Lambda container. IE: 02b1d85a58024220b85df3aa7036c8f7 in this case. Because the object_id will be different from different containers, but it should stay the same within the same Lambda container.
If you do not see the object_id change with the test app. Try switching your own project to use the debug-actionmailer branch of jets and deploying that. You can see how to do that here: https://github.com/tongueroo/jets-debug-actionmailer/blob/master/Gemfile#L4 That will indicate if it’s something specific to how the project is set up. Then we just need to track down exactly what it is.
Note, you can use JETS_ENV_EXTRA to create a new environment with the same code. Example:
RE: So maybe this issue is linked to the use of Devise, isn’t it?
Dont believe so, but unsure.
Took another stab at this. Even removed the hack with app.config.action_mailer ||= ActiveSupport::OrderedOptions.new in the latest debug-actionmailer branch. Here’s the AWS Lambda logs with 3 successive calls to jets call hard_job-dig
The first call is the cold-start, the 2nd and 3rd call is without the cold-start.
The Jets.config.object_id and Jets.config.action_mailer.object_id are 47338339889800 and 47338333821080 respectively.
$ aws-logs tail /aws/lambda/mailer-dev-hard_job-dig
2020-04-15 19:10:58 UTC 2020/04/15/[$LATEST]510107f8f4b3439c8950cd5df46afe4d START RequestId: dbe1b9ec-cab0-461c-b7db-db9b3804a62a Version: $LATEST
2020-04-15 19:10:58 UTC 2020/04/15/[$LATEST]510107f8f4b3439c8950cd5df46afe4d application.rb load_default_config2 @config.object_id 47338339889800
2020-04-15 19:10:58 UTC 2020/04/15/[$LATEST]510107f8f4b3439c8950cd5df46afe4d application.rb setup! @config.object_id 47338339889800
2020-04-15 19:10:58 UTC 2020/04/15/[$LATEST]510107f8f4b3439c8950cd5df46afe4d mailer.rb turbine loaded
2020-04-15 19:10:58 UTC 2020/04/15/[$LATEST]510107f8f4b3439c8950cd5df46afe4d mailer.rb initializer config.object_id 47338339889800
2020-04-15 19:10:59 UTC 2020/04/15/[$LATEST]510107f8f4b3439c8950cd5df46afe4d application.rb configs! @config.object_id 47338339889800
2020-04-15 19:10:59 UTC 2020/04/15/[$LATEST]510107f8f4b3439c8950cd5df46afe4d mailer.rb Jets.config.object_id 47338339889800
2020-04-15 19:10:59 UTC 2020/04/15/[$LATEST]510107f8f4b3439c8950cd5df46afe4d mailer.rb Jets.config.action_mailer.object_id 47338333821080 (config.action_mailer)
2020-04-15 19:10:59 UTC 2020/04/15/[$LATEST]510107f8f4b3439c8950cd5df46afe4d mailer.rb options 47338333821080 (config.action_mailer)
2020-04-15 19:10:59 UTC 2020/04/15/[$LATEST]510107f8f4b3439c8950cd5df46afe4d HardJob Jets.config.object_id 47338339889800
2020-04-15 19:10:59 UTC 2020/04/15/[$LATEST]510107f8f4b3439c8950cd5df46afe4d HardJob Jets.config.action_mailer.object_id 47338333821080 (config.action_mailer)
2020-04-15 19:10:59 UTC 2020/04/15/[$LATEST]510107f8f4b3439c8950cd5df46afe4d END RequestId: dbe1b9ec-cab0-461c-b7db-db9b3804a62a
2020-04-15 19:10:59 UTC 2020/04/15/[$LATEST]510107f8f4b3439c8950cd5df46afe4d REPORT RequestId: dbe1b9ec-cab0-461c-b7db-db9b3804a62a Duration: 4.99 ms Billed Duration: 100 ms Memory Size: 1536 MB Max Memory Used: 157 MB Init Duration: 2439.40 ms
2020-04-15 19:11:19 UTC 2020/04/15/[$LATEST]510107f8f4b3439c8950cd5df46afe4d START RequestId: 932d8b55-5fce-46ca-851c-e29b276d5f73 Version: $LATEST
2020-04-15 19:11:19 UTC 2020/04/15/[$LATEST]510107f8f4b3439c8950cd5df46afe4d HardJob Jets.config.object_id 47338339889800
2020-04-15 19:11:19 UTC 2020/04/15/[$LATEST]510107f8f4b3439c8950cd5df46afe4d HardJob Jets.config.action_mailer.object_id 47338333821080 (config.action_mailer)
2020-04-15 19:11:19 UTC 2020/04/15/[$LATEST]510107f8f4b3439c8950cd5df46afe4d END RequestId: 932d8b55-5fce-46ca-851c-e29b276d5f73
2020-04-15 19:11:19 UTC 2020/04/15/[$LATEST]510107f8f4b3439c8950cd5df46afe4d REPORT RequestId: 932d8b55-5fce-46ca-851c-e29b276d5f73 Duration: 2.17 ms Billed Duration: 100 ms Memory Size: 1536 MB Max Memory Used: 157 MB
2020-04-15 19:11:27 UTC 2020/04/15/[$LATEST]510107f8f4b3439c8950cd5df46afe4d START RequestId: 24f44f7b-09a9-453f-8e89-d19cedb6ac0b Version: $LATEST
2020-04-15 19:11:27 UTC 2020/04/15/[$LATEST]510107f8f4b3439c8950cd5df46afe4d HardJob Jets.config.object_id 47338339889800
2020-04-15 19:11:27 UTC 2020/04/15/[$LATEST]510107f8f4b3439c8950cd5df46afe4d HardJob Jets.config.action_mailer.object_id 47338333821080 (config.action_mailer)
2020-04-15 19:11:27 UTC 2020/04/15/[$LATEST]510107f8f4b3439c8950cd5df46afe4d END RequestId: 24f44f7b-09a9-453f-8e89-d19cedb6ac0b
2020-04-15 19:11:27 UTC 2020/04/15/[$LATEST]510107f8f4b3439c8950cd5df46afe4d REPORT RequestId: 24f44f7b-09a9-453f-8e89-d19cedb6ac0b Duration: 2.14 ms Billed Duration: 100 ms Memory Size: 1536 MB Max Memory Used: 157 MB
Wondering what am missing here This has been tough to reproduce. Hoping that someone with the issue can run this jets debug-actionmailer branch on their project and see what the logging shows. That may lead to better clues.