Something special about environment named 'dev'?

I have a pretty simple pipeline running with 2 environments both building and deploying from master: dev and ci.

They are essentially the same other than a few environment variables: db name, db url, etc.

For some reason…dev will not build but ci will.

The following command succeeds:

JETS_ENV=ci JETS_AGREE=no AWS_REGION=us-west-2 bundle exec jets build --templates

However, the next command fails( see below for error )
JETS_ENV=dev JETS_AGREE=no AWS_REGION=us-west-2 bundle exec jets build --templates

I’ve even tried ensuring that all environment variables exactly match ci build building dev just to see if something in there might be causing problems. That doesn’t seem to help.

Note: I’ve tried this with Jets 1.9.10 and 1.9.16
I’m at a loss. Any insights?

rich$ JETS_ENV=dev JETS_AGREE=no AWS_REGION=us-west-2 bundle exec jets build --templates
Building project for Lambda...
Building CloudFormation templates.
Traceback (most recent call last):
        47: from /Users/rich/.rvm/gems/ruby-2.5.3/bin/ruby_executable_hooks:24:in `<main>'
        46: from /Users/rich/.rvm/gems/ruby-2.5.3/bin/ruby_executable_hooks:24:in `eval'
        45: from /Users/rich/.rvm/gems/ruby-2.5.3/bin/jets:23:in `<main>'
        44: from /Users/rich/.rvm/gems/ruby-2.5.3/bin/jets:23:in `load'
        43: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/exe/jets:14:in `<top (required)>'
        42: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/cli.rb:5:in `start'
        41: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/cli.rb:28:in `start'
        40: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/commands/base.rb:27:in `perform'
        39: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/commands/base.rb:38:in `dispatch'
        38: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
        37: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
        36: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
        35: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/commands/main.rb:12:in `build'
        34: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/commands/build.rb:18:in `run'
        33: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/commands/build.rb:23:in `build'
        32: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/commands/build.rb:34:in `build_templates'
        31: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/commands/build.rb:44:in `build_all_templates'
        30: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/commands/build.rb:58:in `build_api_gateway_templates'
        29: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/cfn/builders/interface.rb:13:in `build'
        28: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/cfn/builders/api_gateway_builder.rb:15:in `compose'
        27: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/cfn/builders/api_gateway_builder.rb:33:in `add_gateway_rest_api'
        26: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/cfn/builders/interface.rb:86:in `add_resource'
        25: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/resource/base.rb:16:in `logical_id'
        24: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/memoist-0.16.0/lib/memoist.rb:170:in `resource'
        23: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/resource/base.rb:20:in `resource'
        22: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/resource/api_gateway/rest_api.rb:11:in `definition'
        21: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/resource/api_gateway/rest_api.rb:19:in `internal_logical_id'
        20: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/resource/api_gateway/rest_api.rb:23:in `logical_id'
        19: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/resource/api_gateway/rest_api.rb:28:in `internal_logical_id'
        18: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/resource/api_gateway/rest_api/logical_id.rb:9:in `get'
        17: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/resource/api_gateway/rest_api/logical_id.rb:18:in `changed?'
        16: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/resource/api_gateway/rest_api/change_detection.rb:9:in `changed?'
        15: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/resource/api_gateway/rest_api/routes.rb:4:in `changed?'
        14: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/resource/api_gateway/rest_api/routes/change.rb:5:in `changed?'
        13: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/resource/api_gateway/rest_api/routes/change/to.rb:5:in `changed?'
        12: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/resource/api_gateway/rest_api/routes/change/to.rb:9:in `changed?'
        11: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/memoist-0.16.0/lib/memoist.rb:170:in `deployed_routes'
        10: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/resource/api_gateway/rest_api/routes/change/base.rb:21:in `deployed_routes'
         9: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/resource/api_gateway/rest_api/routes/change/base.rb:21:in `each'
         8: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/resource/api_gateway/rest_api/routes/change/base.rb:25:in `block in deployed_routes'
         7: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/resource/api_gateway/rest_api/routes/change/base.rb:25:in `each'
         6: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/resource/api_gateway/rest_api/routes/change/base.rb:35:in `block (2 levels) in deployed_routes'
         5: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/resource/api_gateway/rest_api/routes/change/base.rb:52:in `to'
         4: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/resource/api_gateway/rest_api/routes/change/base.rb:72:in `recreate_to'
         3: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/resource/api_gateway/rest_api/routes/change/base.rb:80:in `get_controller_action'
         2: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/resource/api_gateway/rest_api/routes/change/base.rb:90:in `controller_action_from_api'
         1: from /Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/resource/api_gateway/rest_api/routes/change/base.rb:106:in `lambda_function_description'
/Users/rich/.rvm/gems/ruby-2.5.3/gems/jets-1.9.16/lib/jets/resource/api_gateway/rest_api/routes/change/base.rb:106:in `lambda': tried to create Proc object without a block (ArgumentError)

I see, the error doesn’t have to do with JETS_ENV=dev. It has to do with a bug with routes change detection on that environment. Fixed in: https://github.com/tongueroo/jets/pull/292 Released in v1.9.17

Also, for those wondering what route change detection is, it’s discuss here Jets deploy: CREATE_FAILED AWS::CloudFormation::Stack ApiGateway Embedded

Thanks for the fix…I’ll definitely go through the docs shortly but is there some changes that I should be looking out for that would affect deployments? After updating Jets, I cannot get any of my deployments to work. I’ve even tried deleting and starting from scratch and right now I’m failing on a base path role.

Wondering what the CloudWatch log associated with the base path shows. The base mapping log is created as part of the Custom Domains logic The log group looks something like this:

When you get a chance, can you check that and show that? Think there’s some edge case bug with custom domains. Someone reported something a while back, but haven’t been able to reproduce this yet. So am very interested in the base path error log.

FWIW, created, deleted, and recreated a demo app configured with a custom domain 3 times in a row with no success in reproducing the error. To be clear, believe it’s happening, just having trouble reproducing it for some reason. Hoping the logs will be helpful.

Also, would then try to:

  1. Comment out the custom domain in config/application.rb and deploy again without the custom domain. That should work.
  2. Add back in the custom domain and deploy again. That may error again but at least we’re isolating it to the error. Then we’ll go from there.

Thanks for looking more into this @tung. You are correct, the problem I’m having now is not related to the custom domain name. Rather it appears to be related to deploying the lambdas within a VPC.
I know, stay out of the VPC :slight_smile:, unfortunately we are using RDS and have a requirement of no public route to the database.

Anyways, looks like a malformed policy in the xxx-dev-api-deployment.yml template around ec2 permissions added when putting the lambdas in a VPC.

I’ve included what that policy looks like below. As you can see it appears the ec2:* were not added appropriately.

        PolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Action:
            - logs:*
            Effect: Allow
            Resource: arn:aws:logs:us-west-2:123456789000:log-group:/aws/lambda/myapp-dev-*
          - Action:
            - s3:Get*
            - s3:List*
            Effect: Allow
            Resource: arn:aws:s3:::myapp-dev-s3bucket-5ush90hw7s9z*
          - Action:
            - s3:ListAllMyBuckets
            - s3:HeadBucket
            Effect: Allow
            Resource: arn:aws:s3:::*
          - ec2:CreateNetworkInterface
          - ec2:DeleteNetworkInterface
          - ec2:DescribeNetworkInterfaces
          - ec2:DescribeVpcs
          - ec2:DescribeSubnets
          - ec2:DescribeSecurityGroups
          - Action:
            - apigateway:*
            Effect: Allow
            Resource: arn:aws:apigateway:us-west-2::/restapis/*
          - Action:
            - apigateway:*
            Effect: Allow
            Resource: arn:aws:apigateway:us-west-2::/domainnames/*
          - Action:
            - cloudformation:DescribeStacks
            Effect: Allow
            Resource: arn:aws:cloudformation:us-west-2:12345678900:stack/myapp-dev*

How weird. When I saw this, first thought it was the auto-generated vpc iam policy that was added in v1.9.19 from PR 296. Did this as a part of WarmLambdaFunction, TorchLambdaFunction dose not use my configured role The permissions that are incorrectly injected in your example are:

  - ec2:CreateNetworkInterface
  - ec2:DeleteNetworkInterface
  - ec2:DescribeNetworkInterfaces
  - ec2:DescribeVpcs
  - ec2:DescribeSubnets
  - ec2:DescribeSecurityGroups

These permissions exactly match the auto-generated vpc iam permissions. So feel that it is very likely related.

Wasn’t able to reproduce the invalid structure though. :thinking:

The other thing that is odd that the xxx-dev-api-deployment.yml template usually does not produce any IAM policies. Unsure why it produced one with your app :face_with_monocle: Added custom IAM policies at all 3 levels: application-wide, class, and function level. Also, configured the app to use a vpc in config/application.rb. Am unable to reproduce at the moment. Wondering if you could put together a small example app that reproduces it.

Here’s a trick to help generate the CloudFormation templates without an actual deploy.

jets build --templates

It’ll help speed up debugging this issue.

P.S. Also, while looking at this, shorten the auto-generated vpc IAM policy and released it in v1.9.21.

Longer Auto-Generated VPC IAM Permissions: Before Improvement
  - Action:
    - ec2:CreateNetworkInterface
    Effect: Allow
    Resource: "*"
  - Action:
    - ec2:DeleteNetworkInterface
    Effect: Allow
    Resource: "*"
  - Action:
    - ec2:DescribeNetworkInterfaces
    Effect: Allow
    Resource: "*"
  - Action:
    - ec2:DescribeVpcs
    Effect: Allow
    Resource: "*"
  - Action:
    - ec2:DescribeSubnets
    Effect: Allow
    Resource: "*"
  - Action:
    - ec2:DescribeSecurityGroups
    Effect: Allow
    Resource: "*"
Shorter Auto-Generated VPC IAM Permissions in v1.9.21
  - Action:
    - ec2:CreateNetworkInterface
    - ec2:DeleteNetworkInterface
    - ec2:DescribeNetworkInterfaces
    - ec2:DescribeVpcs
    - ec2:DescribeSubnets
    - ec2:DescribeSecurityGroups
    Effect: Allow
    Resource: "*"

Ok, interestingly, 1.9.21 cleared up my issue with the odd PolicyDocument. That said…1.9.19 still exhibits the issue, I can put together a repo which exhibits the error if you want to dig deeper…or we can ignore 1.9.19 and everyone should just move to 1.9.21 :slight_smile:

Your call!!!

I’m back up and CDing :+1:

Lets ignore and move on :tada:

1 Like