Route is destroyed and recreated when upgrading from 1.2 to 1.3

[jordanbd]

I'd like some assistance upgrading from 1.2 to 1.3 when using the fck-nat TF module in combination with the TF vpc module.

When I upgrade from 1.2 to 1.3 I get the following error because TF is attempting to recreate my routes: Error: api error RouteAlreadyExists: Route in Route Table (rtb-xxx) with destination (0.0.0.0/0) already exists. The TF logs show that the route is being destroyed and recreated:

  # module.fck-nat[0].aws_route.main[0] will be destroyed
  # (because resource does not use count)
  - resource "aws_route" "main" {
      - destination_cidr_block      = "0.0.0.0/0" -> null
      - id                          = "r-rtb-0ed892a98ed869c161080289494" -> null
      - instance_id                 = "i-0233e87ff9c6e50c3" -> null
      - instance_owner_id           = "443495191585" -> null
      - network_interface_id        = "eni-0066c806c4ac12383" -> null
      - origin                      = "CreateRoute" -> null
      - route_table_id              = "rtb-0ed892a98ed869c16" -> null
      - state                       = "active" -> null
        # (11 unchanged attributes hidden)
    }
  # module.fck-nat[0].aws_route.main["RESERVED_FKC_NAT"] will be created
  + resource "aws_route" "main" {
      + destination_cidr_block = "0.0.0.0/0"
      + id                     = (known after apply)
      + instance_id            = (known after apply)
      + instance_owner_id      = (known after apply)
      + network_interface_id   = "eni-0066c806c4ac12383"
      + origin                 = (known after apply)
      + route_table_id         = "rtb-0ed892a98ed869c16"
      + state                  = (known after apply)
    }

Ideally I would like to upgrade from 1.2 to 1.3 without TF attempting to destroy my routes and recreate them. To achieve this I believe I need to migrate from the deprecated update_route_table and route_table_id properties which were changed in this commit.

I am having trouble understanding what I needs to change, hence this ticket.

I have currently defined fck-nat as follows:

module "fck-nat" {
  count = length(module.vpc.public_subnets)
  source = "RaJiska/fck-nat/aws"

  name                 = "${local.prefix}-fck-nat-${count.index}"
  vpc_id               = module.vpc.vpc_id
  subnet_id            = module.vpc.public_subnets[count.index]

  instance_type        = var.fck_nat_instance_type
  ha_mode              = true
  use_cloudwatch_agent = true

  update_route_table   = true
  route_table_id       = module.vpc.private_route_table_ids[count.index]
}

My VPC looks something like this:

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name                    = "${local.prefix}-vpc"
  cidr                    = "10.5.0.0/16"
  azs                     = ["ap-southeast-2a", "ap-southeast-2b"]
  private_subnets         = ["10.5.2.0/24", "10.5.3.0/24"]
  public_subnets          = ["10.5.0.0/24", "10.5.1.0/24"]
  enable_dns_hostnames    = true
  enable_dns_support      = true
  enable_nat_gateway      = false
  map_public_ip_on_launch = true
}

My goal is to change my fck-nat config in such a way that prevents TF from deleting and recreating my routes (i.e. keeping the name module.fck-nat[0].aws_route.main[0] or module.fck-nat[0].aws_route.main[1] instead of module.fck-nat[0].aws_route.main["RESERVED_FKC_NAT"]. Assistance is appreciated!

[jordanbd]

I did some more playing around with this on another of our less critical services. After I got the error I reran Terraform and it completely, successfully creating the routes. However during the period between the failure and the rerun there was no 0.0.0.0/0 route so all outgoing traffic over NAT was not working. Depending on your service this may or may not be acceptable.

So provided you immediately re-run terraform apply you can minimize the period that you don't have the required fck-nat routes and everything appears to be fine.

I would still like to understand if there's anything I can do to prevent the failure entirely, as a minute or two of missing nat routes on some of our services will cause a minor ruckus.

[efitzkiwi]

I have the same problem. I think I got around it just by pointing the route to the module eni_id and not using the update route config input:

  dynamic "route" {
      for_each = var.use_fck_nat ? toset([1]) : toset([])
      content {
        cidr_block     = "0.0.0.0/0"
        network_interface_id = module.fck-nat[0].eni_id
      }      
  }

It's probably a weird terraform bug.

[RaJiska]

Hi @jordanbd ,

To achieve this without recreating the route you could either edit the state (via terraform state rm and terraform import), or do the change in-code via the moved block. For your specific case with the former, you could do the following:

moved {
  from = module.fck-nat.aws_route.main[0]
  to   = module.fck-nat.aws_route.main["RESERVED_FKC_NAT"]
}

If you are to do this, you may as well update to remove the deprecated code, combining with the moved block as follow:

module "fck-nat" {
  [...]
  update_route_tables = true
  route_tables_ids = {
    "private" = aws_route_table.private.id
  }
}

moved {
  from = module.fck-nat.aws_route.main[0]
  to     = module.fck-nat.aws_route.main["private"]
}

You will have to update the above with your use case.

Unfortunately I couldn't add the moved block in-module as I used the same resource name when deprecating the previous route arguments which with hindsight, shouldn't have done.

[jordanbd]

@RaJiska Apologies for the lateness of my reply. Your suggestions have worked - I went with your second block where I also removed the deprecated code. Thanks again.