Jets deploy: CREATE_FAILED AWS::CloudFormation::Stack ApiGateway Embedded

Hey,

Everything was working fine, then suddenly my jets deploy started failing:

Deploying CloudFormation stack with jets app!
09:34:13PM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack patchkit-app-catalog-dev User Initiated
09:34:18PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack ApiGateway 
09:34:18PM CREATE_IN_PROGRESS AWS::IAM::Role IamRole 
09:34:18PM CREATE_IN_PROGRESS AWS::Lambda::LayerVersion GemLayer 
09:34:19PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack ApiGateway Resource creation Initiated
09:34:19PM CREATE_IN_PROGRESS AWS::IAM::Role IamRole Resource creation Initiated
09:34:30PM CREATE_COMPLETE AWS::IAM::Role IamRole 
09:34:31PM CREATE_IN_PROGRESS AWS::Lambda::LayerVersion GemLayer Resource creation Initiated
09:34:31PM CREATE_COMPLETE AWS::Lambda::LayerVersion GemLayer 
09:34:34PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack JetsPreheatJob 
09:34:35PM CREATE_IN_PROGRESS AWS::CloudFormation::Stack JetsPreheatJob Resource creation Initiated
09:34:41PM CREATE_FAILED AWS::CloudFormation::Stack ApiGateway Embedded stack arn:aws:cloudformation:us-west-2:470835755211:stack/patchkit-app-catalog-dev-ApiGateway-1N4RAIGOAO4B1/9a12ba60-1772-11e9-bdaf-0af0816e310e was not successfully created: The following resource(s) failed to create: [CatalogsCatalogIdApiResource]. 
09:34:41PM CREATE_FAILED AWS::CloudFormation::Stack JetsPreheatJob Resource creation cancelled

This is how it looks like on CloudFormation event log:

I tried removing the stack, clearing the /tmp/jets cache, reverting to previous working releases, but nothing seems to work. I’m using AWS root account and I can’t remember doing any changes to my environment.

How can I debug it?

I believe I might just found a working commit of my project. It was a lot further that I thought. I’m looking for the culprit now…

OK, I found it! It’s pretty interesting.

If I define a route like that:

get 'catalogs/:id', to: 'catalogs#show'

Everything seems to be working well, but as soon as try to rename :id to :catalog_id:

get 'catalogs/:catalog_id', to: 'catalogs#show'

jets deploy fails with the following error:

10:15:15PM UPDATE_FAILED AWS::CloudFormation::Stack ApiGateway Embedded stack arn:aws:cloudformation:us-west-2:470835755211:stack/patchkit-app-catalog-dev-ApiGateway-1ESXLW1Z66X0L/9d8f4c10-1774-11e9-a229-02a5b7e2d79e was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [CatalogsCatalogIdApiResource]. 
10:15:26PM UPDATE_FAILED AWS::CloudFormation::Stack JetsPreheatJob Resource update cancelled

Is this a Jets bug?

Edit: It seems to have an issue with everything else than :id. Tried using :cid with the same result. Compared result CloudFormation templates from :id and :cid attempt, and I can’t find anything wrong with it.

@genail Odd. Unsure why its rolling back for you. Created a similar demo app to debug this: https://github.com/tongueroo/jets-issue-path-variables

So using posts/:post_id and it deployed successfully.

Here’s the actually API Gateway endpoint: https://garmaxmbm1.execute-api.us-west-2.amazonaws.com/dev/posts

So unsure right now :thinking:

When you get a chance, can you click on the ApiGateway stack while it’s deploying and screenshot that. It might show some more detailed information on the specific child stack. :ok_hand:

Damn! I totally forgot that Jets is creating nested stacks!

This is a screenshot where I’m trying to use :cid (with ‘c’):

I don’t fully understand it yet, but it’s a good lead.

OK I think I know exactly what is going on. I’ve encountered two issues at once, but both were caused by a rule, that you cannot define in API Gateway two resources with variable on the same level, like:

/resource/{id}
/resource/{another_id}

It makes sense since ApiGateway is unable to tell which one to use.

Moving forward:

  1. First issue: I tried to delete the stack and install it anew. It should have helped, right? Wrong. At the time I tried to deploy a stack with this in my routes.rb:
  get 'catalogs/:id', to: 'catalogs#show'
  post 'catalogs/:catalog_id/apps', to: 'catalogs/apps#create'

And the stack creation error is “A sibling ({catalog_id}) of this resource already has a variable path part – only one is allowed”. It makes sense.

  1. Second issue: I tried to rename path parameter on the already existing stack. I suspect that ApiGateway first tries to create the new resource (and connect Lambda to it) and remove the old resource at the end. Of course, it does not allow two resource variables on the same level so it failed.

Now I’m thinking how it can be fixed. I see two options:

  1. Instruct the user that in routes.rb path variables on the same level should be called the same. Unfortunately, this won’t fix the rename issue and deploy will still fail if someone tries to rename the path variable. Is there a way around that? It seems like serverless framework has the same exact issue with no clean solution whatsoever.
  2. In the deployment template, map routes.rb variable names to something constant like var_X (/apps/:app_id/comments/:id to /apps/:var_1/comments/:var_3) where X is a resource depth index (/0/1/2/3). This will fix the renaming issue.
    Naturally, Jets would need a translator encoded in the base controller to translate it back. I’m not sure if it will not introduce any new issues on the road.

What do you think @tung ? If I am right, I could try making a PR for the second approach.

Thanks so much for debugging this and providing this helpful info :+1: Summarizing for myself:

The specific route combination that reproduces the issue:

get 'catalogs/:id', to: 'catalogs#show'
post 'catalogs/:catalog_id/apps', to: 'catalogs/apps#create'

Think see what’s going on here. This is because API Gateway only allows one path variable under the same in the parent route node. Here’s a reproduction of the issue manually.

RE: 1. Instruct the user that in routes.rb path variables on the same level should be called the same.

  1. Yup, think we should add docs noting this API Gateway constraint and how to currently avoid it. That’s a big win and will help save folks time right now.
  2. Upon route building, check for “multiple variable part path definitions” that collide and error with an informative message to the user. So the user finds out about this early as we can help with the process, locally. Even before they try to deploy. When they deploy check the same logic and also provide same information message in case they deploy without checking locally. Something like

API Gateway only allows one unique variable path. [screenshot link]. You must use the same variable name within the same parent route path. Example: /posts/:id and /posts/:post_id/reveal should both be /posts/:id and /posts/:id/reveal.

Something like that, the message can be improved and can tell the users how to fix it.

Created an issue: https://github.com/tongueroo/jets/issues/143

RE: Unfortunately, this won’t fix the rename issue and deploy will still fail if someone tries to rename the path variable. Is there a way around that?

So Jets actually addresses the rename issue for most cases already :grin: rest_api/change_detection.rb Was a pretty annoying issue so I introduced route change detection in version 1.2 CHANGELOG At the same time, also introduced Custom Domain support and Automated Blue-Green Deployments. Since when routes changed Jets builds an entirely new API Gateway with a new endpoint. Custom domains help to make make it transparent.

But you’re right, the rename detection does not properly detect the case of a path variable rename. So that needs to be improved.

Created as issue: https://github.com/tongueroo/jets/issues/144

RE: 2. In the deployment template, map routes.rb variable names to something constant like var_X(/apps/:app_id/comments/:id to /apps/:var_1/comments/:var_3) where X is a resource depth index (/0/1/2/3). This will fix the renaming issue.
Naturally, Jets would need a translator encoded in the base controller to translate it back. I’m not sure if it will not introduce any new issues on the road.

RE: handling multiple variable part path definitions with a jets routing translation layer.

Unsure. Maybe one day API Gateway will add support for multiple path variables. :thinking: These things are so hard to tell without actually implementing enough of it before we can ascertain if it’s worth it. Sometimes I’ve even built prototypes successfully, and then decided to throw it all away in the end. The complexity and maintenance is sometimes not worth the feature :face_with_monocle:

If we did do add variable path support internally to Jets. Think this would be a simpler approach. Take:

get 'catalogs/:id', to: 'catalogs#show'
post 'catalogs/:catalog_id/apps', to: 'catalogs/apps#create'

Translate it to:

get 'catalogs/:multiple', to: 'catalogs#show'
post 'catalogs/:multiple/apps', to: 'catalogs/apps#create'

So API Gateway would create catalogs/:multiple as the resource path successfully. multiple would be a reserved path variable name for this special case.

Then internally within Jets, route the :multiple variable back to original routes and parameters using the routes definition.

Could also be all in vain as API Gateway could add multiple variable route support. :thinking: Gut tells me it might not be worth it? Will have to play with it more to really understand this better.

Hi @tung,
Thanks for a detailed answer!

Created as issue: https://github.com/tongueroo/jets/issues/144

Great, thanks! I’ve heard about the change detector and I was wondering how it is actually working.

So API Gateway would create catalogs/:multiple as the resource path successfully. multiple would be a reserved path variable name for this special case.

Unfortunately, this won’t work. This is another Api Gateway constraint. You can’t have two path variables called the same in a single path. Here I’ve tried to create /test/{var}/another/{var}:

That’s why I proposed variable name + level since there won’t be conflicts.

Sorry if I missed anything.

Oh duh. We need different variable names in the same path. Good catch. :+1: You didn’t miss anything. Thanks for testing and providing the screenshot. Cheers!