Init error when loading handler - errorType": "Init<Gem::LoadError>


#1

Hi Gurus,

This morning I had the bad surprise that my Web App in Production environment is not working anymore.

I did not push any new version of my app since one month.

Please find the log below:

Blockquote---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| @timestamp | @message |
|-------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|
| 2020-04-11 08:30:04.417 | Init error when loading handler handlers/controllers/sessions_controller.create |
| 2020-04-11 08:30:04.417 | { |
| 2020-04-11 08:30:04.417 | “errorMessage”: “You have already activated json 2.3.0, but your Gemfile requires json 2.2.0. Prepending bundle exec to your command may solve this.”, |
| 2020-04-11 08:30:04.417 | “errorType”: “InitGem::LoadError”, |
| 2020-04-11 08:30:04.417 | “stackTrace”: [ |
| 2020-04-11 08:30:04.417 | “/var/runtime/gems/bundler-2.1.4/lib/bundler/runtime.rb:312:in check_for_activated_spec!'", | | 2020-04-11 08:30:04.417 | "/var/runtime/gems/bundler-2.1.4/lib/bundler/runtime.rb:31:inblock in setup’”, |
| 2020-04-11 08:30:04.417 | “/var/runtime/gems/bundler-2.1.4/lib/bundler/spec_set.rb:147:in each'", | | 2020-04-11 08:30:04.417 | "/var/runtime/gems/bundler-2.1.4/lib/bundler/spec_set.rb:147:ineach’”, |
| 2020-04-11 08:30:04.417 | “/var/runtime/gems/bundler-2.1.4/lib/bundler/runtime.rb:26:in map'", | | 2020-04-11 08:30:04.417 | "/var/runtime/gems/bundler-2.1.4/lib/bundler/runtime.rb:26:insetup’”, |
| 2020-04-11 08:30:04.417 | “/var/runtime/gems/bundler-2.1.4/lib/bundler.rb:149:in setup'", | | 2020-04-11 08:30:04.417 | "/var/runtime/gems/bundler-2.1.4/lib/bundler/setup.rb:20:inblock in <top (required)>’”, |
| 2020-04-11 08:30:04.417 | “/var/runtime/gems/bundler-2.1.4/lib/bundler/ui/shell.rb:136:in with_level'", | | 2020-04-11 08:30:04.417 | "/var/runtime/gems/bundler-2.1.4/lib/bundler/ui/shell.rb:88:insilence’”, |
| 2020-04-11 08:30:04.417 | “/var/runtime/gems/bundler-2.1.4/lib/bundler/setup.rb:20:in <top (required)>'", | | 2020-04-11 08:30:04.417 | "/var/task/handlers/controllers/sessions_controller.rb:1:inrequire’”, |
| 2020-04-11 08:30:04.417 | “/var/task/handlers/controllers/sessions_controller.rb:1:in <top (required)>'", | | 2020-04-11 08:30:04.417 | "/var/lang/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:92:inrequire’”, |
| 2020-04-11 08:30:04.417 | “/var/lang/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:92:in `require’” |
| 2020-04-11 08:30:04.417 | ] |
| 2020-04-11 08:30:04.417 | } |
| 2020-04-11 08:30:04.108 | END RequestId: 607273bb-9f0d-4752-b415-33f2de695147 |
| 2020-04-11 08:30:04.108 | REPORT RequestId: 607273bb-9f0d-4752-b415-33f2de695147 Duration: 389.25 ms Billed Duration: 400 ms Memory Size: 1536 MB Max Memory Used: 21 MB |
| 2020-04-11 08:30:04.108 | Unknown application error occurred Function |
| 2020-04-11 08:30:04.061 | }

Any idea to solve this issue? and why it happens so suddenly whereas I do not push any new code or version?

Cheers,

Fabien.


#2

Hi,

I just get a feedback from aws support :slight_smile:
Hi Fabien,

Thank you for contacting AWS Premium Support. I hope you are keeping well! My name is Wassim and I will be assisting you today.

I understand you noticed that your Lambda function started to fail execution without performing any code change.

Thanks for sharing the error message, that was of great help to initiate the investigation. The error suggest that the ‘json’ library was upgraded to version “ 2.3.0”, but your code was expecting version “2.2.0” ( as specified by the error message, “You have already activated json 2.3.0, but your Gemfile requires json 2.2.0” ). The default Lambda environment will tend to use always the latest libraries to enable the latest set of features and security updates as specify in this documentation[1], hence , I do believe that the Lambda team have upgraded the library on the default Lambda environment which caused this issue.

In order to avoid such behavior, you need to package all of your dependencies in deployment package, this will allow you to have a full control of the environment and avoid being affected by libraries changes. I have this link[2] for your convenience for additional information regarding creating deployment package for Ruby.

As a quick workaround here, I would recommend creating a layer[3] with the json 2.2.0 Gem, Ruby 2.5.0 uses “Amazon Linux" as an Operating System[4], I would recommend you creating the layer on an “Amazon Linux" EC2 instance then upload it to Lambda to avoid any compatibility issues.

Further, I checked on Lambda function using command “puts Gem.loaded_specs[‘json’].version” and I see the environment is indeed using json library 2.3.0.

I hope this was helpful, please don’t hesitate to reach out back to me if you need any additional information, I will be more than happy to assist you further Fabien!

Have a great day !

[1] https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html#function-code
[2] https://docs.aws.amazon.com/lambda/latest/dg/ruby-package.html#ruby-package-dependencies
[3] https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-path
[4] https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html

Best regards,

Wassim J.
Amazon Web Services

Maybe it can help other people.

Fabien.


#3

Reading the Jets documentation I found that Jets already use Gem Layer https://rubyonjets.com/docs/extras/gem-layer/

So I should not get this error and I should not have to package a specific layer, shouldn’t I ?


#4

Same thing is happening to me. Unsure how to fix it exactly.


#5

@Fabien I did a fresh deploy and it fixed itself. I too hadn’t deployed in weeks. No idea what caused it or why a fresh bundle/deploy suddenly fixed it, but give that a try.


#6

Hi all,

I get two other answers from AWS support:

1/
Hi Fabien,

Thanks for reaching back out to me and confirming you were indeed including this in the layer, As per the privacy policy agreed with AWS[1] we don’t have any visibility into your internal data, therefore we can’t see the content of the layer/ Lambda code.

I went ahead and created a layer with the json 2.2.0 gem and I confirm the function kept referencing the json 2.3.0 instead. I believe this is an unexpected behavior. I have escalated this to the Lambda developers team to further investigate into this issue as they do have permission on the service more than I do. I assigned your ticket the highest priority.

I will reach out back to you as soon as I have additional information, while I am working on this issue internally with the Lambda team, please don’t hesitate to contact me again if you need any additional information, I will be more than happy to assist you further !

Talk to you soon !

Best regards,

Wassim J.
Amazon Web Services

2/
Hi Fabien,

Much appreciate your patience!

The Lambda developer team confirmed they are enforcing now the use of JSON 2.3.0, this was as part of a security update that was recently introduced to Lambda. As this is used by the runtime, there is no way ,unfortunately, to use JSON 2.2.0, the only workaround for this at the present moment is to update your environment to use JSON 2.3.0. Also, as the environment will be forced to use whatever JSON version used by the runtime, you can rely on that version instead of importing it in your environment. I do appreciate this may be a bit of work to do on your end to get back the function up and running , however this is really recommended for security reasons.

Further, I have sent a feedback to the team to send communication for similar security updates in advance to avoid. Apologies for inconvenience may this caused to you.

Please, let me know if you have any queries or concerns and I will pass them on to the Lambda developers team.

Have a nice day !

Best regards,

Wassim J.
Amazon Web Services

So to sum-up AWS lambda imposes to use specific versions of component and you can’t do anything…

It is really annoying in a production environment!!!


#7

Thank very much for the details.

Wow. Feel like ran into something similar a week ago or so with ruby, lambda, and json, and lono actually. Using lono for lambda functions also. At the time, had to removed the json gem from the Gemfile and that allows ruby to use whatever system-level json gem was installed on the AWS Lambda runtime. So it was more “free” at that point. Guessing AWS is now straight up vendorizing json 2.3.0 and enforcing it.

Bummer about this whole thing. Especially, since it’s was on production. Am also a little sympathetic to the AWS Lambda team. I vaguely remember having ruby and json issues a long time ago. At some point, it seems like ruby team itself decided to include json into stdlib core. That seem to cause issues. Don’t even remember the exact details because it was pretty confusing at the time. Am also sympathetic to the Ruby core team. It seems like a difficult problem.

Thanks for posting again. It’s useful.


#8

This happened to the Jets app that BoltOps is running. Actually run quite a few of them now. Looks like AWS is rolled out the update more generally on 4/16/2020. Thanks to this report knew exactly what to do! Upgrade json 2.3. Bummer again about it!

Also, this PR should migrate the issue somewhat https://github.com/tongueroo/jets/pull/470 since it removes json from Gemfile.lock. Though projects or other gems can still add json, it at least migrates it.