Lab 8: Part 2. Testing with Optimism

A test case is a piece of code that we will run while observing what happens (including what it prints and/or what it evaluates to). Often, this will be a single function call, and we'll be interested in the result value.

An expectation is an idea about what the correct result value (or printed output) should be when we run a test case. We can simply have expectations in our head, and check them by observing a test case ourselves, but we can also dictate them to the computer and have it check them for us.

Using optimism

Back in lab 2 we covered the use of optimism (the optimism reference is here), but that was before we had even covered custom functions. We're now revisiting this library because it offers us a way to do automatic testing which can improve the quality of our code as well as reducing the time it takes us to solve problems.

The provided file tracing.py, which you've just used to test out the debugger, includes three broken definitions of hasCGBlock, in addition to the correct one. Your job in this part of the lab is to define test cases that distinguish the correct version from the broken versions: the correct version should pass all of your tests, while the broken versions should fail at least one of them.

To define a test case in optimism, we use the testCase function. First, we need to import the library, like this:

from optimism import *

Now, we can do something like:

testCase(hasCGBlock('CGAGGGCCUG'))

For every test case we establish, we also need one or more expectations about what it should do. For this, if we're interested in a function's return value (or the value of a test expression) we can use expectResult. If we're interested in the printed output instead, we can use expectOutputContains, although we'd also have to use captureOutput before the test, and we'd eventually use restoreOutput after the test. For this lab, only expectResult is needed. We can use it like this:

expectResult(True)

Note that when we define test cases, if they crash, or forget to restore output, or cause other problems, this will affect the entire file's correctness. To isolate them, we can put them in a test function, and just call that function from the shell. So we can test when we want to, but any issues with the tests won't cause problems with the normal operation of other things in the file. The tracing.py file already has a test function in it, and that's where you should add your tests. Putting that all together, we should have a test function that looks like this (the import could also go at the top of your file if you prefer):

from optimism import *

def test():
    """
    This function is designed to be used to set up and run tests. If you
    put them here, they won't interfere with anything else you might want
    to do in the rest of the file, until you call this function.
    """
    # Put your test cases and expectations here

    testCase(hasCGBlock('CGAGGGCCUG'))
    expectResult(True)

If you run the file, nothing should happen, although your functions will be defined. If you then call test in the shell, the results should look like this:

>>> %Run tracing.py
>>> test()
✓ tracing.py:100

The check-mark indicates that your test passed, and it reports the line number where the expectation was established. The red color is because it's part of the error log rather than normal printed output (although in this case, it's not actually an error).

More testing

Partner B

Now that you've got one test set up, create copies of it for the hasCGBlock1, hasCGBlock2, and hasCGBlock3 functions, with the same expectation for each test. Run your file, and check which of the three extra functions pass or fail this test. If things are set up correctly, hasCGBlock1 should fail the test, but hasCGBlock2 and hasCGBlock3 should pass it. One test is certainly not enough to catch all possible problems, after all.

With your partner, brainstorm at least 2 more tests to try, and apply them to each of the four functions.

Mass testing

Partner A

Copy-pasting these test cases can easily get to be a lot of typing, so let's use a loop to minimize that. First, we'll define a testCases variable holding a list of tuples, like this:

def test():
    ...
    testCases = [
        ('CGAGGGCCUG', True),
        ('CG', False)
    ]

Each tuple should contain an argument in the first position, and the expected value for the result of hasCGBlock in the second position. Now define a loop which goes through those test cases, and uses the testCase and expectResult functions to define four tests for each iteration, one for each of the variants of hasCGBlock we've provided. For the smoothest solution, use multiple loop variables to deal with the tuples in the list of test cases.

Refer back to the lecture notes on testing and debugging for an example of this list-of-tuples strategy.

Once this is set up, adding a new test case and testing it with all four functions is as simple as adding a new row to our testCases variable.

Finding counterexamples

Partner B

See if you can come up with a set of tests such that the correct hasCGBlock passes all of them, while each of the incorrect versions fails at least one of them. Any test where one of the incorrect versions fails is a counterexample for that version.

You should start this process without reading the details of the code for the broken functions, but if you're having trouble finding a counterexample for one of them, try looking at their code or using the debugger to step through it to figure out what's wrong, and design your counterexample based on that.

Once you've found at least one counterexample for each broken version of hasCGBlock, you're ready to move on to the next part of the lab.

Table of Contents