This is the 4th, final part our blogpost series about building Furnace. In the first part we talked about the AWS services used in the brief, the second part was about AWS Go SDK and we began dissecting the intricacies of Furnace. The 3rd part was about the experimental plugin system of Furnace. In this part we'll discuss Unit Testing Furnace and how to work some magic with AWS and Go.

Mock Stub Fake Dummy Canned

Unit testing in Go usually follows the Dependency Injection model of dealing with Mocks and Stubs.

DI

In short, Dependency Inject is one object supplying the dependencies of another object. Dependency Inject is ideally used for removing the lock on a third party library such as the AWS client. Imagine having a code which solely depends on the AWS client. How would you unit test that code without ACTUALLY having to connect to AWS? You can’t. Each time you try to test the code, it will run the live code connecting to AWS in order to perform the operations it’s design to run. The Ruby library with its metaprogramming allows you to set the client globally to stub responses, but alas!- this is not the world of Ruby.

Here is where DI comes to the rescue. The only thing we need to achieve is the ability to inject the AWS client into the code at the earliest. Once we do this, we will have the ability to change it to our own implementation in the tests. For example: we would like the CreateApplication call to fail; or we would like a DescribeStack call that returns an aws.Error(“StackAlreadyExists”).
For this, we’ll need the API of the AWS client, (which is provided by AWS).

AWS Client API

In order for DI to work, the injected object must be of a certain type. Luckily, AWS provides an Interface for all of its clients, i.e. we can implement our own version for all of the clients such as S3, CloudFormation, CodeDeploy, etc.
For each client we want to mock out, an *iface package should be present. For example:
 "github.com/aws/aws-sdk-go/service/cloudformation/cloudformationiface"
In this package, you’ll find and use the interface like this:

  1. type fakeCloudFormationClient struct {
  2.     cloudformationiface.CloudFormationAPI
  3.     err error
  4. }

And with this, we have our own CloudFormation client. The real code uses real clients as function parameters.  For example:

  1. // Execute defines what this command does.
  2. func (c *Create) Execute(opts *commander.CommandHelper) {
  3.     log.Println("Creating cloud formation session.")
  4.     sess := session.New(&aws.Config{Region: aws.String(config.REGION)})
  5.     cfClient := cloudformation.New(sess, nil)
  6.     client := CFClient{cfClient}
  7.     createExecute(opts, &client)
  8. }

We can’t test {{Execute}} itself since it’s using the real client, (if you are using a global from some library, you will be able to test it, even Execute in this case ). There’s little logic in this function for this very reason: all the logic is in small functions for which the main starting point and our testing opportunity is createExecute.

Stubbing Calls

Now that we have our own client, and with the power of Go’s interface embedding, (as seen above with CloudFormationAPI), we only have to stub the functions which we are actually using, (as opposed to every function of the given interface). This looks like this:

  1.     cfClient := new(CFClient)
  2.     cfClient.Client = &fakeCloudFormationClient{err: nil}

When cfClient is a struct like this:

  1. // CFClient abstraction for cloudFormation client.
  2. type CFClient struct {
  3.     Client cloudformationiface.CloudFormationAPI
  4. }

And a stubbed call can then be written as follows:

  1. func (fc *fakeCreateCFClient) WaitUntilStackCreateComplete(input *cloudformation.DescribeStacksInput) error {
  2.     return nil
  3. }

This can range from a very trivial example, such as the one above, to intricate examples, like this gem:

  1. func (fc *fakePushCFClient) ListStackResources(input *cloudformation.ListStackResourcesInput) (*cloudformation.ListStackResourcesOutput, error) {
  2.     if "NoASG" == *input.StackName {
  3.         return &cloudformation.ListStackResourcesOutput{
  4.             StackResourceSummaries: []*cloudformation.StackResourceSummary{
  5.                 {
  6.                     ResourceType:       aws.String("NoASG"),
  7.                     PhysicalResourceId: aws.String("arn::whatever"),
  8.                 },
  9.             },
  10.         }, fc.err
  11.     }
  12.     return &cloudformation.ListStackResourcesOutput{
  13.         StackResourceSummaries: []*cloudformation.StackResourceSummary{
  14.             {
  15.                 ResourceType:       aws.String("AWS::AutoScaling::AutoScalingGroup"),
  16.                 PhysicalResourceId: aws.String("arn::whatever"),
  17.             },
  18.         },
  19.     }, fc.err
  20. }

The ListStackResources stub lets us test two scenarios based on the stackname. If the test stackname is ‘NoASG’, it will return a result which equals to a result containing no AutoScaling Groups. Otherwise, it will return the correct ResourceType for an ASG.

It’s common practice to line up several scenarios based on stubbed responses in order to test the robustness of your code. Unfortunately, this also means that our tests will be a bit cluttered with stubs, mock structs and whatnots. To remedy this, Furnace uses a package wide struct file in which most of the mocked structs are defined. From hereon, the tests will only contain specific stubs for that particular file. This can be further fine grained by having defaults which will only be overwritten when we need something else.

Testing Fatals

Now, the other point which is not really AWS related, but still comes to mind when dealing with Furnace, is testing error scenarios.

Because Furnace is a CLI application, it uses Fatals to signify when something has gone wrong; it doesn’t continue or recover because frankly, it can’t. If AWS throws an error, that’s it. We can attempt to retry it, but in 90% of the cases, it’s an error which is not recoverable, such as a syntax error in the configuration file.

So, how do we test for a fatal or an os.Exit? If you do a quick search, you’ll find a number of points relating to this issue. You may end up on this talk: GoTalk 2014 Testing Slide #23. It calls the test binary in a separate process and tests the exit code.

Some may say that you have to have your own logger implemented, and to use a different logger / os.Exit in your test environment. Others will tell you not to test around os.Exit and fatal errors. Instead, it suggests that you return an error with only the main to pop a world ending event. I’ll leave it up to you which you prefer to use. Either is fine.

Furnace uses a global logger for handling errors. For example: 

  1. // HandleFatal handler fatal errors in Furnace.
  2. func HandleFatal(s string, err error) {
  3.     LogFatalf(s, err)
  4. }


LogFatalf is an exported variable var LogFatalf = log.Fatalf. During a test, just override this variable with a local anonymous function:

  1. func TestCreateExecuteEmptyStack(t *testing.T) {
  2.     failed := false
  3.     utils.LogFatalf = func(s string, a ...interface{}) {
  4.         failed = true
  5.     }
  6.     config.WAITFREQUENCY = 0
  7.     client := new(CFClient)
  8.     stackname := "EmptyStack"
  9.     client.Client = &fakeCreateCFClient{err: nil, stackname: stackname}
  10.     opts := &commander.CommandHelper{}
  11.     createExecute(opts, client)
  12.     if !failed {
  13.         t.Error("expected outcome to fail during create")
  14.     }
  15. }

It can get even more granular when testing for the error message. To ensure that it actually fails at the point we think we are testing, it should look like this:

  1. func TestCreateStackReturnsWithError(t *testing.T) {
  2.     failed := false
  3.     expectedMessage := "failed to create stack"
  4.     var message string
  5.     utils.LogFatalf = func(s string, a ...interface{}) {
  6.         failed = true
  7.         if err, ok := a[0].(error); ok {
  8.             message = err.Error()
  9.         }
  10.     }
  11.     config.WAITFREQUENCY = 0
  12.     client := new(CFClient)
  13.     stackname := "NotEmptyStack"
  14.     client.Client = &fakeCreateCFClient{err: errors.New(expectedMessage), stackname: stackname}
  15.     config := []byte("{}")
  16.     create(stackname, config, client)
  17.     if !failed {
  18.         t.Error("expected outcome to fail")
  19.     }
  20.     if message != expectedMessage {
  21.         t.Errorf("message did not equal expected message of '%s', was:%s", expectedMessage, message)
  22.     }
  23. }

Conclusion

That’s it. I hope you’ve enjoyed reading this as much as I’ve enjoyed writing all these thoughts down.
I hope you learn and find value from my experiences. Any comments are appreciated and welcomed. Also, PRs and Issues can be submitted via the GitHub page of Furnace.

Thank you for reading! Gergely.