Link Search Menu Expand Document

AWS CDK - Computing Diff Between Infrastructure Templates [TypeScript]

Status
PUBLISHED
Project
AWS CDK
Project home page
https://github.com/aws/aws-cdk
Language
TypeScript
Tags
#aws #cloud #diff #infrastructure-as-code

Help Code Catalog grow: suggest your favorite code or weight in on open article proposals.

Table of contents
  1. Context
  2. Problem
  3. Overview
  4. Implementation details
  5. Testing
  6. Observations
  7. Related
  8. References
  9. Copyright notice

Context

CDK (Cloud Development Kit) is an infrastructure as code software tool created by AWS. CDK is used to synthesize and deploy CloudFormation infrastructure templates.

Infrastructures are made up of resources (virtual machines, database tables, load balancers, etc.). Resources can depend on other resources.

Problem

Synthesized infrastructure templates need to be compared to the existing state of the infrastructure to see what resources will be created, updated or deleted if the template is deployed. The diff algorithm needs to be aware of the template semantics.

Overview

There are different diff handlers for the 9 top-level keys (AWSTemplateFormatVersion, Description, Metadata, Parameters, Mappings, Conditions, Transform, Resources, Outputs).

It calculates what was added, removed or updated. For each changed resource it decides the impact: if it will be updated, destroyed, orphaned (excluded from the template but not actually deleted).

Changes to one resource can trigger changes to resources dependent on it. These changes are propagated until convergence.

There’s a method to print the diff in a human-readable format.

Implementation details

The implementation is rather long; we are just scratching the surface in this review.

The main method. First it actually calculates the diff and then propagates replacements for replaced resources until it converges.

/**
 * Compare two CloudFormation templates and return semantic differences between them.
 *
 * @param currentTemplate the current state of the stack.
 * @param newTemplate     the target state of the stack.
 *
 * @returns a +types.TemplateDiff+ object that represents the changes that will happen if
 *      a stack which current state is described by +currentTemplate+ is updated with
 *      the template +newTemplate+.
 */
export function diffTemplate(currentTemplate: { [key: string]: any }, newTemplate: { [key: string]: any }): types.TemplateDiff {
  // Base diff
  const theDiff = calculateTemplateDiff(currentTemplate, newTemplate);

  // We're going to modify this in-place
  const newTemplateCopy = deepCopy(newTemplate);

  let didPropagateReferenceChanges;
  let diffWithReplacements;
  do {
    diffWithReplacements = calculateTemplateDiff(currentTemplate, newTemplateCopy);

    // Propagate replacements for replaced resources
    didPropagateReferenceChanges = false;
    if (diffWithReplacements.resources) {
      diffWithReplacements.resources.forEachDifference((logicalId, change) => {
        if (change.changeImpact === types.ResourceImpact.WILL_REPLACE) {
          if (propagateReplacedReferences(newTemplateCopy, logicalId)) {
            didPropagateReferenceChanges = true;
          }
        }
      });
    }
  } while (didPropagateReferenceChanges);

  // Copy "replaced" states from `diffWithReplacements` to `theDiff`.
  diffWithReplacements.resources
    .filter(r => isReplacement(r!.changeImpact))
    .forEachDifference((logicalId, downstreamReplacement) => {
      const resource = theDiff.resources.get(logicalId);

      if (resource.changeImpact !== downstreamReplacement.changeImpact) {
        propagatePropertyReplacement(downstreamReplacement, resource);
      }
    });

  return theDiff;
}

Diffing templates (without propagation). Most of the work is delegated to DIFF_HANDLERS.

function calculateTemplateDiff(currentTemplate: { [key: string]: any }, newTemplate: { [key: string]: any }): types.TemplateDiff {
  const differences: types.ITemplateDiff = {};
  const unknown: { [key: string]: types.Difference<any> } = {};
  for (const key of unionOf(Object.keys(currentTemplate), Object.keys(newTemplate)).sort()) {
    const oldValue = currentTemplate[key];
    const newValue = newTemplate[key];
    if (deepEqual(oldValue, newValue)) { continue; }
    const handler: DiffHandler = DIFF_HANDLERS[key]
                  || ((_diff, oldV, newV) => unknown[key] = impl.diffUnknown(oldV, newV));
    handler(differences, oldValue, newValue);

  }
  if (Object.keys(unknown).length > 0) { differences.unknown = new types.DifferenceCollection(unknown); }

  return new types.TemplateDiff(differences);
}

Diffing two resources:

export function diffResource(oldValue?: types.Resource, newValue?: types.Resource): types.ResourceDifference {
  const resourceType = {
    oldType: oldValue && oldValue.Type,
    newType: newValue && newValue.Type,
  };
  let propertyDiffs: { [key: string]: types.PropertyDifference<any> } = {};
  let otherDiffs: { [key: string]: types.Difference<any> } = {};

  if (resourceType.oldType !== undefined && resourceType.oldType === resourceType.newType) {
    // Only makes sense to inspect deeper if the types stayed the same
    const typeSpec = cfnspec.filteredSpecification(resourceType.oldType);
    const impl = typeSpec.ResourceTypes[resourceType.oldType];
    propertyDiffs = diffKeyedEntities(oldValue!.Properties,
      newValue!.Properties,
      (oldVal, newVal, key) => _diffProperty(oldVal, newVal, key, impl));

    otherDiffs = diffKeyedEntities(oldValue, newValue, _diffOther);
    delete otherDiffs.Properties;
  }

  return new types.ResourceDifference(oldValue, newValue, {
    resourceType, propertyDiffs, otherDiffs,
  });

  function _diffProperty(oldV: any, newV: any, key: string, resourceSpec?: cfnspec.schema.ResourceType) {
    let changeImpact = types.ResourceImpact.NO_CHANGE;

    const spec = resourceSpec && resourceSpec.Properties && resourceSpec.Properties[key];
    if (spec && !deepEqual(oldV, newV)) {
      switch (spec.UpdateType) {
        case cfnspec.schema.UpdateType.Immutable:
          changeImpact = types.ResourceImpact.WILL_REPLACE;
          break;
        case cfnspec.schema.UpdateType.Conditional:
          changeImpact = types.ResourceImpact.MAY_REPLACE;
          break;
        default:
          // In those cases, whatever is the current value is what we should keep
          changeImpact = types.ResourceImpact.WILL_UPDATE;
      }
    }

    return new types.PropertyDifference(oldV, newV, { changeImpact });
  }

  function _diffOther(oldV: any, newV: any) {
    return new types.Difference(oldV, newV);
  }
}

Rendering diffs in a human-readable form (not listed).

Testing

The test suite is quite comprehensive.

Basic test for adding a resource:

test('when a resource is created', () => {
  const currentTemplate = { Resources: {} };

  const newTemplate = { Resources: { BucketResource: { Type: 'AWS::S3::Bucket' } } };

  const differences = diffTemplate(currentTemplate, newTemplate);
  expect(differences.differenceCount).toBe(1);
  expect(differences.resources.differenceCount).toBe(1);
  const difference = differences.resources.changes.BucketResource;
  expect(difference).not.toBeUndefined();
  expect(difference?.isAddition).toBeTruthy();
  expect(difference?.newResourceType).toEqual('AWS::S3::Bucket');
  expect(difference?.changeImpact).toBe(ResourceImpact.WILL_CREATE);
});

Test cascading changes:

test('resource replacement is tracked through references', () => {
  // If a resource is replaced, then that change shows that references are
  // going to change. This may lead to replacement of downstream resources
  // if the reference is used in an immutable property, and so on.

  // GIVEN
  const currentTemplate = {
    Resources: {
      Bucket: {
        Type: 'AWS::S3::Bucket',
        Properties: { BucketName: 'Name1' }, // Immutable prop
      },
      Queue: {
        Type: 'AWS::SQS::Queue',
        Properties: { QueueName: { Ref: 'Bucket' } }, // Immutable prop
      },
      Topic: {
        Type: 'AWS::SNS::Topic',
        Properties: { TopicName: { Ref: 'Queue' } }, // Immutable prop
      },
    },
  };

  // WHEN
  const newTemplate = {
    Resources: {
      Bucket: {
        Type: 'AWS::S3::Bucket',
        Properties: { BucketName: 'Name2' },
      },
      Queue: {
        Type: 'AWS::SQS::Queue',
        Properties: { QueueName: { Ref: 'Bucket' } },
      },
      Topic: {
        Type: 'AWS::SNS::Topic',
        Properties: { TopicName: { Ref: 'Queue' } },
      },
    },
  };
  const differences = diffTemplate(currentTemplate, newTemplate);

  // THEN
  expect(differences.resources.differenceCount).toBe(3);
});

Testing that it understands that the order of elements in an array matters in some places and doesn’t matter in others:

test('array equivalence is independent of element order in DependsOn expressions', () => {
  // GIVEN
  const currentTemplate = {
    Resources: {
      BucketResource: {
        Type: 'AWS::S3::Bucket',
        DependsOn: ['SomeResource', 'AnotherResource'],
      },
    },
  };

  // WHEN
  const newTemplate = {
    Resources: {
      BucketResource: {
        Type: 'AWS::S3::Bucket',
        DependsOn: ['AnotherResource', 'SomeResource'],
      },
    },
  };

  let differences = diffTemplate(currentTemplate, newTemplate);
  expect(differences.resources.differenceCount).toBe(0);

  differences = diffTemplate(newTemplate, currentTemplate);
  expect(differences.resources.differenceCount).toBe(0);
});

test('arrays of different length are considered unequal in DependsOn expressions', () => {
  // GIVEN
  const currentTemplate = {
    Resources: {
      BucketResource: {
        Type: 'AWS::S3::Bucket',
        DependsOn: ['SomeResource', 'AnotherResource', 'LastResource'],
      },
    },
  };

  // WHEN
  const newTemplate = {
    Resources: {
      BucketResource: {
        Type: 'AWS::S3::Bucket',
        DependsOn: ['AnotherResource', 'SomeResource'],
      },
    },
  };

  let differences = diffTemplate(currentTemplate, newTemplate);
  expect(differences.resources.differenceCount).toBe(1);

  differences = diffTemplate(newTemplate, currentTemplate);
  expect(differences.resources.differenceCount).toBe(1);
});

Observations

Terraform, a competing product, implements a similar algorithm.

References

AWS CDK is licensed under the Apache License 2.0.

Copyright 2018-2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.