Jets deploy doesn't work after deleting AWS functions: UPDATE_FAILED AWS::CloudFormation::Stack

Jets
1

Hello, I’m currently trying out Jets Framework, been awesome so far.

However, after deploying there was some kind of error with the cache in the tmp folder that prevented subsequent deploys from working. I deleted the aws lambda functions (I shouldn’t have haha) and now i’m getting this error:
UPDATE_FAILED AWS::CloudFormation::Stack JetsPreheatJob
Because it’s not finding the functions.

Is there a way to do something like jets deploy --rebuild ? like redo the whole deployment and functions.

I deployed with another environment to keep working, but maybe someone can give me some insight with this.

I actually fixed the cache error with this command:
rm -rf /tmp/jets/project-name/cache

However, the staging env is pretty much not working, non deployable.

2

Thanks for the kind words.

Yup, the CloudFormation stack is out of sync with the actual resources. In these cases, it’s a bit of a pain. But the silver lining is that you get to learn how out to get out of these situations. I’ve learned a lot about CloudFormation and all sorts of situations with experience, not just Jets related. I usually end up deleting everything to get back to a clean slate.

General Steps

Go to the CloudFormation console and delete stacks manually to get unstuck.

  • Start by manually to deleting the parent stack (it will fail)
  • Particularly, the nested child stack with the out-of-sync functions will fail.
  • Then manually delete each child stack by clicking it and deleting each one that is failing.
  • If you delete the same stack multiple times, then CloudFormation will eventually (usually the 2nd time) give you a checkbox to “keep” the resources CloudFormation cannot successfully delete. In your case, it will be the functions that have already been deleted that CloudFormation cannot find. So you’re not really “keeping” the functions, you’re just telling CloudFormation to “ignore” it and move on to delete the other resources it knows about.
  • You repeat this for until the misbehaving child stacks are gone.
  • Eventually you’ll only have the parent stack left and in-sync child stacks left.
  • You can try a jets delete again and might be done.
  • If you keep going until you get all the way to just the parent stack by itself. When you try to delete that the parent stack, it will eventually say that it cannot delete the stack because the s3 bucket that is created by the parent stack and managed by Jets is not empty.

At this point you have to 2 options:

  • Run jets delete. It should 1. empty out the s3 bucket 2. delete the parent stack 3. and clean out the CloudWatch logs.

Or:

  • Manually empty the S3 bucket
  • Manually delete the parent stack.
  • Manually delete the CloudWatch logs associated with that Jets environment.

Here’s a loop might be helpful:

for i in $(aws logs describe-log-groups | jq -r '.logGroups[].logGroupName' | grep demo-dev) ; do echo $i ; aws logs delete-log-group --log-group-name $i ; done

Note the grep demo-dev. Replace it with the project namespace that you are using.

You can build up to the command by running it separately first:

aws logs describe-log-groups | jq -r '.logGroups[].logGroupName' | grep demo-dev # separately first

Then run the loop:

for i in $(aws logs describe-log-groups | jq -r '.logGroups[].logGroupName' | grep demo-dev) ; do echo $i ; aws logs delete-log-group --log-group-name $i ; done

Hope that helps.

2 Likes
3

There are other ways aside from deleting everything. But it’s tricky. One thing you can try is:

  1. Delete the controller and routes containing the lambda functions from your code that you have accidentally deleted on the console.
  2. Deploy with the resources deleted, this tricks CloudFormation to deleting that child stack.
  3. If that is successful, then your CloudFormation is back in sync with your code!
  4. Then add back the controller and routes that were deleted.
  5. Deploy again and everything is restored.

So you have to trick CloudFormation into doing the right thing. Usually, I just spinning up another environment and start fresh instead of dealing with the one that is out-of-sync.

1 Like
4

Thanks a lot for the help Tung!

Deleting controllers to re-sync cloudformation was my second idea,
but I ended deleting manually the S3 bucket and then ran jets delete development, and it successfully removed everything related to this environment.
I just had to re deploy everything in that environment afterwards, and it worked.

I’ll keep working with Jets on internal projects at my job :slight_smile: and post in the forums if I run into any trouble.

Thanks.

1 Like