1. Overview

As you've learned so far in CS111, it can be challenging to create programs that work correctly in all cases.

Programs that don't behave correctly are said to be buggy because they contain bugs = reasons why they don't work. The process of finding and fixing these bugs is called debugging.

The first step in the debugging process is testing. You don't know whether a program might be incorrect until you have evidence that it's not working as expected. Today we will discuss how to develop test cases that help us determine cases in which programs misbehave.

Testing test cases interactively in Canopy is cumbersome, so we will develop testing functions to automate testing with test cases.

Until this point, we have provided you with test cases and testing functions both in Canopy and via Codder. But we want to wean you from these provided resources and teach you how to develop your own test cases and testing functions. This is an important skill you need to master as you mature as programmers.

What do you do when testing reveals cases in which your program doesn't work? Then you need to apply debugging techniques to determine why it misbehaves and how to fix it.

The goal of this lecture is to give you practice with (1) defining testing functions (2) developing effective test cases and (3) using debugging techniques.

2. Motivation for Testing Functions

The file functionsToTest.py contains several functions that will be used as examples in this lecture. Many are buggy, but some are correct.

Do not study these functions definition now. Instead, treat them as a black boxes = functions that you will call without understanding how they work. We will look at the details later in lecture.

In [1]:
from functionsToTest import * # Load "black-box" functions that we will test

One of the functions in functionsToTest.py is a a buggy version of the countChar function that was defined correctly in the Lec 08 Sequences and Loops notebook.

def countChar(char, word):
    '''Return the number of time char occurs in word, ignoring case.'''
    # Details omitted

Without studying the function definition, let's try some test cases to see in which situations the function works and doesn't work. Make some hypotheses about possible bugs in the function.

In [2]:
countChar('s', 'Mississippi')
Out[2]:
4
In [3]:
countChar('S', 'Mississippi')
Out[3]:
4
In [4]:
countChar('p', 'Mississippi')
Out[4]:
2
In [5]:
countChar('P', 'Mississippi')
Out[5]:
2
In [6]:
countChar('i', 'Mississippi')
Out[6]:
3
In [7]:
countChar('I', 'Mississippi')
Out[7]:
3
In [8]:
countChar('M', 'Mississippi')
Out[8]:
0
In [9]:
countChar('m', 'Mississippi')
Out[9]:
0
In [10]:
countChar('a', 'Mississippi')
Out[10]:
0
In [11]:
countChar('S', 'MISSISSIPPI')
Out[11]:
0
In [12]:
countChar('s', 'MISSISSIPPI')
Out[12]:
0
In [13]:
countChar('I', 'MISSISSIPPI')
Out[13]:
0
In [14]:
countChar('i', 'MISSISSIPPI')
Out[14]:
0

3. Towards Automated Testing: Printing Test Cases

It's tedious to interactively test calls to countChar one at a time. Can we do better?

The following doesn't work. Why not?

In [15]:
countChar('s', 'Mississippi')
countChar('S', 'Mississippi')
countChar('i', 'Mississippi')
countChar('I', 'MISSISSIPPI')
countChar('M', 'Mississippi')
countChar('m', 'Mississippi')
Out[15]:
0

To see the results of multiple test cases from a single notebook cell, we'll need to use print.

The following is better, but still not very helpful. Why?

In [16]:
print(countChar('s', 'Mississippi'))
print(countChar('S', 'Mississippi'))
print(countChar('i', 'Mississippi'))
print(countChar('I', 'MISSISSIPPI'))
print(countChar('M', 'Mississippi'))
print(countChar('m', 'Mississippi'))
4
4
3
0
0
0

Let's print out more context for each call:

In [17]:
print("countChar('s', 'Mississippi') =>", countChar('s', 'Mississippi'))
print("countChar('S', 'Mississippi') =>", countChar('S', 'Mississippi'))
print("countChar('i', 'Mississippi') =>", countChar('i', 'Mississippi'))
print("countChar('I', 'MISSISSIPPI') =>", countChar('I', 'MISSISSIPPI'))
print("countChar('M', 'Mississippi') =>", countChar('M', 'Mississippi'))
print("countChar('m', 'MISSISSIPPI') =>", countChar('m', 'MISSISSIPPI'))
countChar('s', 'Mississippi') => 4
countChar('S', 'Mississippi') => 4
countChar('i', 'Mississippi') => 3
countChar('I', 'MISSISSIPPI') => 0
countChar('M', 'Mississippi') => 0
countChar('m', 'MISSISSIPPI') => 0

Let's capture the pattern in each line with a printCountChar function:

In [18]:
def printCountChar(char, word):
    print("countChar('" + char # This is just one string
          + "', '" + word      # concatenated out of parts
          + "') =>", 
          countChar(char, word) # This is the number that 
         )                      # results from calling countChar
    
printCountChar('s', 'Mississippi')
printCountChar('S', 'Mississippi')
printCountChar('i', 'Mississippi')
printCountChar('I', 'MISSISSIPPI')
printCountChar('M', 'Mississippi')
printCountChar('m', 'Mississippi')
countChar('s', 'Mississippi') => 4
countChar('S', 'Mississippi') => 4
countChar('i', 'Mississippi') => 3
countChar('I', 'MISSISSIPPI') => 0
countChar('M', 'Mississippi') => 0
countChar('m', 'Mississippi') => 0

4. Digression: Creating complex strings with .format

The string concatenations in printCountChar are difficult to create and read.

You have encounter similar situations before. Remember the canvas title in PS05 showConcentricCircles and circleRow?

Is there a better way? Yes! The .format method fills a template string with n "holes" (represented by {}) with n values. The n values are automatically converted to strings, so str does not need to be called on them.

For example:

In [19]:
sumTemplate = '{} + {} => {}'
sumTemplate.format(3,5,3+5)
Out[19]:
'3 + 5 => 8'
In [20]:
print(sumTemplate.format(3,5,3+5))
print(sumTemplate.format(10,7,10+7))
3 + 5 => 8
10 + 7 => 17

.format makes it much easier to create the complex strings in printCountChar, showConcentricCircles etc. because it's not necessary to (1) concatenate strings with + or (2) convert nonstrings to strings with str.

Using .format simplifies the definition of printCountChar:

In [21]:
def printCountChar(char, word):
    print("countChar('{}','{}') => {}".format(char, word, countChar(char, word)))
    
printCountChar('s', 'Mississippi')
printCountChar('S', 'Mississippi')
printCountChar('i', 'Mississippi')
printCountChar('I', 'MISSISSIPPI')
printCountChar('M', 'Mississippi')
printCountChar('m', 'Mississippi')  
countChar('s','Mississippi') => 4
countChar('S','Mississippi') => 4
countChar('i','Mississippi') => 3
countChar('I','MISSISSIPPI') => 0
countChar('M','Mississippi') => 0
countChar('m','Mississippi') => 0

Note that within the parens of print(), it's KO to move the .format to the next line if the line with the string gets too long:

In [22]:
def printCountChar(char, word):
    print("countChar('{}','{}') => {}"
          .format(char, word, countChar(char, word)))
    
printCountChar('s', 'Mississippi')
printCountChar('S', 'Mississippi')
printCountChar('i', 'Mississippi')
printCountChar('I', 'MISSISSIPPI')
printCountChar('M', 'Mississippi')
printCountChar('m', 'Mississippi')
countChar('s','Mississippi') => 4
countChar('S','Mississippi') => 4
countChar('i','Mississippi') => 3
countChar('I','MISSISSIPPI') => 0
countChar('M','Mississippi') => 0
countChar('m','Mississippi') => 0

5. Digression: Iterating over Lists of Tuples

One way to iterate over lists of tuples (or lists of lists) is to access the tuples elements by index:

In [23]:
nameTuples = [
    ('Harry','Potter'),
    ('Hermione','Granger'),
    ('Ron','Weasley'),
    ('Luna','Lovegood')
]

for tup in nameTuples:
    print('First name is {} and last name is {}'.format(tup[0], tup[1]))
First name is Harry and last name is Potter
First name is Hermione and last name is Granger
First name is Ron and last name is Weasley
First name is Luna and last name is Lovegood

But if the length of all the tuples in the list is the same, we can instead name the n individual tuple elements by n comma-separated variable names between the for and in:

In [24]:
for first, last in nameTuples:
    print('First name is {} and last name is {}'.format(first, last))
First name is Harry and last name is Potter
First name is Hermione and last name is Granger
First name is Ron and last name is Weasley
First name is Luna and last name is Lovegood

This is basically a convenient abbreviation for the following:

In [25]:
for tup in nameTuples:
    first = tup[0]
    last = tup[1]
    print('First name is {} and last name is {}'.format(first, last))
First name is Harry and last name is Potter
First name is Hermione and last name is Granger
First name is Ron and last name is Weasley
First name is Luna and last name is Lovegood

6. Towards Automated Testing: Looping over Test Cases

We can put test inputs as tuples into a list testInputs and then test each pair of inputs:

In [26]:
testInputs = [
    ('s','Mississippi'),
    ('i','Mississippi'),
    ('p','Mississippi'),
    ('m','Mississippi'),
    ('a','Mississippi'),
]

for char, word in testInputs: # Example of tuple assignment
    printCountChar(char, word)
countChar('s','Mississippi') => 4
countChar('i','Mississippi') => 3
countChar('p','Mississippi') => 2
countChar('m','Mississippi') => 0
countChar('a','Mississippi') => 0

We can adapt the above to consider char and word with different cases:

In [27]:
for char, word in testInputs: 
    printCountChar(char.lower(), word)
    printCountChar(char.lower(), word.lower())
    printCountChar(char.lower(), word.upper())
    printCountChar(char.upper(), word.lower())
    printCountChar(char.upper(), word.upper())
countChar('s','Mississippi') => 4
countChar('s','mississippi') => 4
countChar('s','MISSISSIPPI') => 0
countChar('S','mississippi') => 4
countChar('S','MISSISSIPPI') => 0
countChar('i','Mississippi') => 3
countChar('i','mississippi') => 3
countChar('i','MISSISSIPPI') => 0
countChar('I','mississippi') => 3
countChar('I','MISSISSIPPI') => 0
countChar('p','Mississippi') => 2
countChar('p','mississippi') => 2
countChar('p','MISSISSIPPI') => 0
countChar('P','mississippi') => 2
countChar('P','MISSISSIPPI') => 0
countChar('m','Mississippi') => 0
countChar('m','mississippi') => 0
countChar('m','MISSISSIPPI') => 0
countChar('M','mississippi') => 0
countChar('M','MISSISSIPPI') => 0
countChar('a','Mississippi') => 0
countChar('a','mississippi') => 0
countChar('a','MISSISSIPPI') => 0
countChar('A','mississippi') => 0
countChar('A','MISSISSIPPI') => 0

7. Automated Testing with Expected Values

Above, we still need to carefully examine the results of each test case to determine when they're correct or incorrect. Can we do better?

Yes! By including the expected result value in the test tuple, we can highlight test cases that fail.

In [28]:
countCharTestCases = [
    ('s','Mississippi', 4),
    ('i','Mississippi', 4),
    ('p','Mississippi', 2),
    ('m','Mississippi', 1),
    ('a','Mississippi', 0),
]

def testCountChar(testCases):
    for char, word, expected in testCases:
        actual = countChar(char, word)
        if actual == expected:
            print(("   PASSED: countChar('{}', '{}') returned "
                   + "expected value {}")
                   .format(char, word, expected))
        else:
            print((  "***FAILED: countChar('{}', '{}')"
                   + " returned {} but expected {}")
                   .format(char, word, actual, expected))
            
testCountChar(countCharTestCases)
   PASSED: countChar('s', 'Mississippi') returned expected value 4
***FAILED: countChar('i', 'Mississippi') returned 3 but expected 4
   PASSED: countChar('p', 'Mississippi') returned expected value 2
***FAILED: countChar('m', 'Mississippi') returned 0 but expected 1
   PASSED: countChar('a', 'Mississippi') returned expected value 0

If we only care about highlighting the FAILED cases, we can (1) remove the printing of PASSED cases and (2) keep track of the number of FAILED cases in order display a summary at the end.

In [29]:
def testCountChar2(testCases):
    numFails = 0 # Counter for number of failed tests
    for char, word, expected in testCases:
        actual = countChar(char, word)
        if actual != expected:
            numFails += 1
            print((  "***FAILED: countChar('{}', '{}')"
                   + " returned '{}' but expected '{}'")
                   .format(char, word, actual, expected))
    # Summarize the tests 
    numTests = len(testCases)
    numPasses = numTests - numFails
    print('Passed {} and failed {} of {} test cases.'
          .format(numPasses, numFails, numTests))
            
testCountChar2(countCharTestCases)
***FAILED: countChar('i', 'Mississippi') returned '3' but expected '4'
***FAILED: countChar('m', 'Mississippi') returned '0' but expected '1'
Passed 3 and failed 2 of 5 test cases.

Also, in this particular case, we can leverage list comprehensions to give us more combinations of lower/upper case values in our test cases. This is not generally useful, but is helpful in this specific problem.

In [30]:
moreCountCharTestCases = \
    (countCharTestCases
     + [(char.lower(), word, expected)
        for char, word, expected in countCharTestCases]
     + [(char.lower(), word.lower(), expected)
        for char, word, expected in countCharTestCases]
     + [(char.lower(), word.upper(), expected)
        for char, word, expected in countCharTestCases]
     + [(char.upper(), word.lower(), expected)
        for char, word, expected in countCharTestCases]
     + [(char.upper(), word.upper(), expected)
        for char, word, expected in countCharTestCases])
    
testCountChar2(moreCountCharTestCases)
***FAILED: countChar('i', 'Mississippi') returned '3' but expected '4'
***FAILED: countChar('m', 'Mississippi') returned '0' but expected '1'
***FAILED: countChar('i', 'Mississippi') returned '3' but expected '4'
***FAILED: countChar('m', 'Mississippi') returned '0' but expected '1'
***FAILED: countChar('i', 'mississippi') returned '3' but expected '4'
***FAILED: countChar('m', 'mississippi') returned '0' but expected '1'
***FAILED: countChar('s', 'MISSISSIPPI') returned '0' but expected '4'
***FAILED: countChar('i', 'MISSISSIPPI') returned '0' but expected '4'
***FAILED: countChar('p', 'MISSISSIPPI') returned '0' but expected '2'
***FAILED: countChar('m', 'MISSISSIPPI') returned '0' but expected '1'
***FAILED: countChar('I', 'mississippi') returned '3' but expected '4'
***FAILED: countChar('M', 'mississippi') returned '0' but expected '1'
***FAILED: countChar('S', 'MISSISSIPPI') returned '0' but expected '4'
***FAILED: countChar('I', 'MISSISSIPPI') returned '0' but expected '4'
***FAILED: countChar('P', 'MISSISSIPPI') returned '0' but expected '2'
***FAILED: countChar('M', 'MISSISSIPPI') returned '0' but expected '1'
Passed 14 and failed 16 of 30 test cases.

8. Designing Black-box Test Cases

Testing a function like countChar without seeing its definition is called black-box testing, because the testing is purely based on its input/output behavior according to its contract without being able to see the code implementing the function. It's as if it's a mechanical contraption whose internal workings are hidden inside a black box and cannot be viewed.

8.1 Categories of Black-box Test Cases

When designing black-box tests, you must imagine ways in which the function might be implemented and how such implementations could go wrong. Some classes of test cases:

  1. Regular cases: These are "normal" cases that check basic advertised input/output functionality, like tests of counting different letters in "Mississippi" for countChar.

  2. Implied conditional cases: When the contract mention different categories of an input (e.g., positive or negative numbers, vowels vs. nonvowels), it implies that these categories will be checked by conditionals in the function body. Since those conditionals could be wrong, testing all combinations values from input categories is prudent.

  3. Edge cases: These are tests of extreme or special cases that the function might not handle properly. For example

    • For numeric inputs, extreme inputs can include 0, large numbers, negative numbers, and floats vs. ints.

    • Fencepost errors are off-by-one errors, which are common in programs. E.g n elements in a list are separated by n-1 commas, not n.

    • For inputs that are indices of sequences, test indices near the ends of the sequence, e.g., indices like 0, 1, -1 and len(seq), len(seq)-1, len(seq)+1.
      Since Python allows negative indices, you should also test -len(seq), -len(seq)-1, -len(seq)+1.

    • For functions involving elements of sequences, test elements in the first and last positions of the sequences, e.g. characters at the beginning and end of a string.

    • For inputs that are sequences, empty and small sequences are often not handled correctly, so you should always test empty and singleton strings/lists/tuples. When specific numbers are mentioned in the contract (e.g. isBeauteous tests for 3 consecutive vowels) it's important to test strings of length <= 3 as edge cases.

    • For inputs expected to be booleans, what happens if other Truthy/Falsey values are supplied? Is it OK that to treat other Truthy/Falsey values as True/False?

8.2 Example: Designing Black-box Tests for countChar

In the case of testing countChar, how confident are we that our tests with testing different characters in different capitalizations of "Mississippi" effectively tests countChar?

Rather than testing a long string like "Mississippi", it may be more effective to carefully test a combination of shorter strings and characters in those strings.

Some things to keep in mind:

  • Although the parameter to countChar is named word, it can be any string, so don't get hung up on making it an actual word.

  • Because the contract looks for a particular character in the word, tests really only need to distinguish between that character and other characters. So we can make the character we're looking for 'a' and use 'b' for all other characters. (This assumes the code doesn't do something crazy like handle particular characters or classes of characters --- like vowels --- specially.)

  • The empty string should be tested as an edge case.

  • It's important to test characters in the first and last positions of the string.

  • Because the contract mentions upper and lower case, testing different combinations of case in the character and word is essential.

Based on the above considerations, here is a set of black-box test cases for countChar:

In [31]:
blackBoxCountCharTestCases = [
    # Test the empty string
    ('a','', 0),
    # Test "negative" singleton string
    ('a','b', 0),
    # Test all capitalizations of "positive" singleton string
    ('a','a', 1), ('a','A', 1), ('A','a', 1), ('A','A', 1), 
    # Test two-element strings (where char can be at beginning or end of word)
    ('a','Aa', 2), ('a','aA', 2), ('A','Aa', 2), ('A','aA', 2),
    # No need to repeat capitalization combinations here:
    ('a','ab', 1), ('a','ba', 1), ('a','bb', 0), 
    # Length-3 strings distinguish ends from middles
    ('a', 'aaA', 3), ('a', 'aAA', 3), ('A', 'aaA', 3), ('A', 'aAA', 3),
    ('a', 'aab', 2), ('a', 'aba', 2), ('A', 'baa', 2), ('A', 'aAA', 2),
    ('a','abb', 1), ('a', 'bab', 1), ('a','bba', 1), ('a', 'bbb', 0),
    # Try a few longer strings
    ('a','aAAaA', 5), ('A','aAAaA', 5), 
    ('a','abAbA', 3), ('A','abAbA', 3),
    ('a','babAb', 2), ('A','babAb', 2),
    ('a','bbbbb', 0), 
]

testCountChar2(blackBoxCountCharTestCases)
***FAILED: countChar('a', 'a') returned '0' but expected '1'
***FAILED: countChar('a', 'A') returned '0' but expected '1'
***FAILED: countChar('A', 'a') returned '0' but expected '1'
***FAILED: countChar('A', 'A') returned '0' but expected '1'
***FAILED: countChar('a', 'Aa') returned '0' but expected '2'
***FAILED: countChar('a', 'aA') returned '0' but expected '2'
***FAILED: countChar('A', 'Aa') returned '0' but expected '2'
***FAILED: countChar('A', 'aA') returned '0' but expected '2'
***FAILED: countChar('a', 'ab') returned '0' but expected '1'
***FAILED: countChar('a', 'ba') returned '0' but expected '1'
***FAILED: countChar('a', 'aaA') returned '1' but expected '3'
***FAILED: countChar('a', 'aAA') returned '0' but expected '3'
***FAILED: countChar('A', 'aaA') returned '1' but expected '3'
***FAILED: countChar('A', 'aAA') returned '0' but expected '3'
***FAILED: countChar('a', 'aab') returned '1' but expected '2'
***FAILED: countChar('a', 'aba') returned '0' but expected '2'
***FAILED: countChar('A', 'baa') returned '1' but expected '2'
***FAILED: countChar('A', 'aAA') returned '0' but expected '2'
***FAILED: countChar('a', 'abb') returned '0' but expected '1'
***FAILED: countChar('a', 'bba') returned '0' but expected '1'
***FAILED: countChar('a', 'aAAaA') returned '1' but expected '5'
***FAILED: countChar('A', 'aAAaA') returned '1' but expected '5'
***FAILED: countChar('a', 'abAbA') returned '0' but expected '3'
***FAILED: countChar('A', 'abAbA') returned '0' but expected '3'
***FAILED: countChar('a', 'babAb') returned '1' but expected '2'
***FAILED: countChar('A', 'babAb') returned '1' but expected '2'
Passed 6 and failed 26 of 32 test cases.

8.3 Exercise: Black-box Testing of isBeauteous

Recall that the isBeauteous function from PS05 Task 1c had this contract:

In [32]:
def isBeauteous(word):
    """Call a word "beauteous" if it contains at least three consecutive vowels.            
    'beauteous' itself is beauteous (both 'eau' and 'eou'), but so are.                     
    'delicous' (iou) and 'sequoia' ('uoia'). In contrast,                                   
    'aardvark', and 'nation' are not.                                                        
    This predicate returns True if word is beauteous and False otherwise. 
    """

Below, develop a list of black-box test cases for isBeateous.

  • Use aeiou for vowels. It's a good idea to use some different vowels in case the code happens to have equality tests involving the vowels.

  • Use b and maybe a few other consonants (e.g., c, d) for nonvowels

  • Since isBeateous involves the number 3, testing many strings with length <=3 and a few some strings whose length is >= 3 is a good idea.

In [33]:
blackBoxIsBeauteousTestCases = [
    # Test the empty string
    ('', False), 
    # Test singleton strings
    ('a', False), ('b', False),
    # Test strings of length 2 and 3 with different numbers of vowels
    ('ae', False), ('ab', False), ('ba', False), ('bb', False), 
    ('aei', True), ('bae', False), ('abe', False), ('aeb', False),
    # Test longer strings with and without three consecutive vowels.
    # Include cases where 
    # (1) there are 3 vowels, but not consecutive
    # (2) 2 (but not 3) vowels are consecutive
    # (3) 3 consecutive vowels are at beginning or end of string 
    ('aeib', True), ('abei', False), ('aebi', False), ('baei', True),
    ('aeibc', True), ('aebic', False), ('aebci', False), 
    ('abeci', False), ('abcei', False),('bcaei', True), ('baeic', True),
    ('abeicoua', True), ('abeiocua', True), ('aeibouca', True),
    ('aebiocua', False)
]

Below is a function testIsBeateous that is does much more than the versions of testCountChar above. You are not expected to understand all the details, but here's what it does:

  • In addition to taking a list of test case tuples, it takes a list of function names isBeauteousFunctionNames (in this case, the seven "isBeauteous" through "isBeauteous7").

  • It uses eval to convert function names like "isBeauteous1" into the associated function (e..g isBeauteous1) from functionsToTest.py, and tests each of the seven function versions on all the test cases in testCases.

  • It uses Python's try/except construct to assign the string ERROR to the variable actual in the case where applying the function to a test case raises an exception.

In [34]:
isBeauteousFunctionNames = [
    "isBeauteous1", "isBeauteous2", "isBeauteous3", "isBeauteous4", 
    "isBeauteous5", "isBeauteous6", "isBeauteous7" 
]


def testIsBeauteous(funcNames, testCases):
    for funcName in funcNames: 
        print('-'*60) # Print a line
        print('Testing', funcName) 
        isBeauteousFunc = eval(funcName) # Turn function name, like "isBeauteous1", 
                                         # into a function that can be called. 
    
        numFails = 0 # Counter for number of failed tests
        for word, expected in testCases:
            # Try/except is a way of handling cases where function call
            # raises an exception. You don't have to understand this. 
            try: 
                actual = isBeauteousFunc(word)
            except: 
                actual = 'ERROR'
            if actual == expected:
                print(("   PASSED: {}('{}') returned "
                   + "expected value {}")
                   .format(funcName, word, expected))
            else:
                numFails += 1
                print((  "***FAILED: {}('{}')"
                       + " returned {} but expected {}")
                       .format(funcName, word, actual, expected))

        # Summarize the tests 
        numTests = len(testCases)
        numPasses = numTests - numFails
        print('Passed {} and failed {} of {} test cases.'
              .format(numPasses, numFails, numTests))
            
testIsBeauteous(isBeauteousFunctionNames, blackBoxIsBeauteousTestCases)
------------------------------------------------------------
Testing isBeauteous1
   PASSED: isBeauteous1('') returned expected value False
   PASSED: isBeauteous1('a') returned expected value False
   PASSED: isBeauteous1('b') returned expected value False
   PASSED: isBeauteous1('ae') returned expected value False
   PASSED: isBeauteous1('ab') returned expected value False
   PASSED: isBeauteous1('ba') returned expected value False
   PASSED: isBeauteous1('bb') returned expected value False
   PASSED: isBeauteous1('aei') returned expected value True
   PASSED: isBeauteous1('bae') returned expected value False
   PASSED: isBeauteous1('abe') returned expected value False
   PASSED: isBeauteous1('aeb') returned expected value False
   PASSED: isBeauteous1('aeib') returned expected value True
***FAILED: isBeauteous1('abei') returned True but expected False
***FAILED: isBeauteous1('aebi') returned True but expected False
   PASSED: isBeauteous1('baei') returned expected value True
   PASSED: isBeauteous1('aeibc') returned expected value True
***FAILED: isBeauteous1('aebic') returned True but expected False
***FAILED: isBeauteous1('aebci') returned True but expected False
***FAILED: isBeauteous1('abeci') returned True but expected False
***FAILED: isBeauteous1('abcei') returned True but expected False
   PASSED: isBeauteous1('bcaei') returned expected value True
   PASSED: isBeauteous1('baeic') returned expected value True
   PASSED: isBeauteous1('abeicoua') returned expected value True
   PASSED: isBeauteous1('abeiocua') returned expected value True
   PASSED: isBeauteous1('aeibouca') returned expected value True
***FAILED: isBeauteous1('aebiocua') returned True but expected False
Passed 19 and failed 7 of 26 test cases.
------------------------------------------------------------
Testing isBeauteous2
   PASSED: isBeauteous2('') returned expected value False
   PASSED: isBeauteous2('a') returned expected value False
   PASSED: isBeauteous2('b') returned expected value False
   PASSED: isBeauteous2('ae') returned expected value False
   PASSED: isBeauteous2('ab') returned expected value False
   PASSED: isBeauteous2('ba') returned expected value False
   PASSED: isBeauteous2('bb') returned expected value False
***FAILED: isBeauteous2('aei') returned False but expected True
   PASSED: isBeauteous2('bae') returned expected value False
   PASSED: isBeauteous2('abe') returned expected value False
   PASSED: isBeauteous2('aeb') returned expected value False
   PASSED: isBeauteous2('aeib') returned expected value True
   PASSED: isBeauteous2('abei') returned expected value False
   PASSED: isBeauteous2('aebi') returned expected value False
***FAILED: isBeauteous2('baei') returned False but expected True
   PASSED: isBeauteous2('aeibc') returned expected value True
   PASSED: isBeauteous2('aebic') returned expected value False
   PASSED: isBeauteous2('aebci') returned expected value False
   PASSED: isBeauteous2('abeci') returned expected value False
   PASSED: isBeauteous2('abcei') returned expected value False
***FAILED: isBeauteous2('bcaei') returned False but expected True
   PASSED: isBeauteous2('baeic') returned expected value True
***FAILED: isBeauteous2('abeicoua') returned False but expected True
   PASSED: isBeauteous2('abeiocua') returned expected value True
   PASSED: isBeauteous2('aeibouca') returned expected value True
   PASSED: isBeauteous2('aebiocua') returned expected value False
Passed 22 and failed 4 of 26 test cases.
------------------------------------------------------------
Testing isBeauteous3
   PASSED: isBeauteous3('') returned expected value False
   PASSED: isBeauteous3('a') returned expected value False
   PASSED: isBeauteous3('b') returned expected value False
   PASSED: isBeauteous3('ae') returned expected value False
   PASSED: isBeauteous3('ab') returned expected value False
   PASSED: isBeauteous3('ba') returned expected value False
   PASSED: isBeauteous3('bb') returned expected value False
   PASSED: isBeauteous3('aei') returned expected value True
   PASSED: isBeauteous3('bae') returned expected value False
   PASSED: isBeauteous3('abe') returned expected value False
   PASSED: isBeauteous3('aeb') returned expected value False
   PASSED: isBeauteous3('aeib') returned expected value True
   PASSED: isBeauteous3('abei') returned expected value False
   PASSED: isBeauteous3('aebi') returned expected value False
   PASSED: isBeauteous3('baei') returned expected value True
   PASSED: isBeauteous3('aeibc') returned expected value True
   PASSED: isBeauteous3('aebic') returned expected value False
   PASSED: isBeauteous3('aebci') returned expected value False
   PASSED: isBeauteous3('abeci') returned expected value False
   PASSED: isBeauteous3('abcei') returned expected value False
   PASSED: isBeauteous3('bcaei') returned expected value True
   PASSED: isBeauteous3('baeic') returned expected value True
   PASSED: isBeauteous3('abeicoua') returned expected value True
   PASSED: isBeauteous3('abeiocua') returned expected value True
   PASSED: isBeauteous3('aeibouca') returned expected value True
   PASSED: isBeauteous3('aebiocua') returned expected value False
Passed 26 and failed 0 of 26 test cases.
------------------------------------------------------------
Testing isBeauteous4
   PASSED: isBeauteous4('') returned expected value False
   PASSED: isBeauteous4('a') returned expected value False
   PASSED: isBeauteous4('b') returned expected value False
   PASSED: isBeauteous4('ae') returned expected value False
   PASSED: isBeauteous4('ab') returned expected value False
   PASSED: isBeauteous4('ba') returned expected value False
   PASSED: isBeauteous4('bb') returned expected value False
   PASSED: isBeauteous4('aei') returned expected value True
   PASSED: isBeauteous4('bae') returned expected value False
   PASSED: isBeauteous4('abe') returned expected value False
   PASSED: isBeauteous4('aeb') returned expected value False
   PASSED: isBeauteous4('aeib') returned expected value True
   PASSED: isBeauteous4('abei') returned expected value False
   PASSED: isBeauteous4('aebi') returned expected value False
   PASSED: isBeauteous4('baei') returned expected value True
   PASSED: isBeauteous4('aeibc') returned expected value True
   PASSED: isBeauteous4('aebic') returned expected value False
   PASSED: isBeauteous4('aebci') returned expected value False
   PASSED: isBeauteous4('abeci') returned expected value False
   PASSED: isBeauteous4('abcei') returned expected value False
   PASSED: isBeauteous4('bcaei') returned expected value True
   PASSED: isBeauteous4('baeic') returned expected value True
***FAILED: isBeauteous4('abeicoua') returned False but expected True
   PASSED: isBeauteous4('abeiocua') returned expected value True
   PASSED: isBeauteous4('aeibouca') returned expected value True
   PASSED: isBeauteous4('aebiocua') returned expected value False
Passed 25 and failed 1 of 26 test cases.
------------------------------------------------------------
Testing isBeauteous5
   PASSED: isBeauteous5('') returned expected value False
   PASSED: isBeauteous5('a') returned expected value False
   PASSED: isBeauteous5('b') returned expected value False
   PASSED: isBeauteous5('ae') returned expected value False
   PASSED: isBeauteous5('ab') returned expected value False
   PASSED: isBeauteous5('ba') returned expected value False
   PASSED: isBeauteous5('bb') returned expected value False
   PASSED: isBeauteous5('aei') returned expected value True
   PASSED: isBeauteous5('bae') returned expected value False
   PASSED: isBeauteous5('abe') returned expected value False
   PASSED: isBeauteous5('aeb') returned expected value False
   PASSED: isBeauteous5('aeib') returned expected value True
   PASSED: isBeauteous5('abei') returned expected value False
   PASSED: isBeauteous5('aebi') returned expected value False
   PASSED: isBeauteous5('baei') returned expected value True
   PASSED: isBeauteous5('aeibc') returned expected value True
   PASSED: isBeauteous5('aebic') returned expected value False
   PASSED: isBeauteous5('aebci') returned expected value False
   PASSED: isBeauteous5('abeci') returned expected value False
   PASSED: isBeauteous5('abcei') returned expected value False
   PASSED: isBeauteous5('bcaei') returned expected value True
   PASSED: isBeauteous5('baeic') returned expected value True
   PASSED: isBeauteous5('abeicoua') returned expected value True
   PASSED: isBeauteous5('abeiocua') returned expected value True
   PASSED: isBeauteous5('aeibouca') returned expected value True
   PASSED: isBeauteous5('aebiocua') returned expected value False
Passed 26 and failed 0 of 26 test cases.
------------------------------------------------------------
Testing isBeauteous6
   PASSED: isBeauteous6('') returned expected value False
   PASSED: isBeauteous6('a') returned expected value False
   PASSED: isBeauteous6('b') returned expected value False
   PASSED: isBeauteous6('ae') returned expected value False
   PASSED: isBeauteous6('ab') returned expected value False
   PASSED: isBeauteous6('ba') returned expected value False
   PASSED: isBeauteous6('bb') returned expected value False
   PASSED: isBeauteous6('aei') returned expected value True
   PASSED: isBeauteous6('bae') returned expected value False
   PASSED: isBeauteous6('abe') returned expected value False
   PASSED: isBeauteous6('aeb') returned expected value False
   PASSED: isBeauteous6('aeib') returned expected value True
   PASSED: isBeauteous6('abei') returned expected value False
   PASSED: isBeauteous6('aebi') returned expected value False
   PASSED: isBeauteous6('baei') returned expected value True
   PASSED: isBeauteous6('aeibc') returned expected value True
   PASSED: isBeauteous6('aebic') returned expected value False
   PASSED: isBeauteous6('aebci') returned expected value False
   PASSED: isBeauteous6('abeci') returned expected value False
   PASSED: isBeauteous6('abcei') returned expected value False
   PASSED: isBeauteous6('bcaei') returned expected value True
   PASSED: isBeauteous6('baeic') returned expected value True
   PASSED: isBeauteous6('abeicoua') returned expected value True
   PASSED: isBeauteous6('abeiocua') returned expected value True
   PASSED: isBeauteous6('aeibouca') returned expected value True
   PASSED: isBeauteous6('aebiocua') returned expected value False
Passed 26 and failed 0 of 26 test cases.
------------------------------------------------------------
Testing isBeauteous7
   PASSED: isBeauteous7('') returned expected value False
***FAILED: isBeauteous7('a') returned ERROR but expected False
   PASSED: isBeauteous7('b') returned expected value False
***FAILED: isBeauteous7('ae') returned ERROR but expected False
   PASSED: isBeauteous7('ab') returned expected value False
***FAILED: isBeauteous7('ba') returned ERROR but expected False
   PASSED: isBeauteous7('bb') returned expected value False
   PASSED: isBeauteous7('aei') returned expected value True
***FAILED: isBeauteous7('bae') returned ERROR but expected False
***FAILED: isBeauteous7('abe') returned ERROR but expected False
   PASSED: isBeauteous7('aeb') returned expected value False
   PASSED: isBeauteous7('aeib') returned expected value True
***FAILED: isBeauteous7('abei') returned ERROR but expected False
***FAILED: isBeauteous7('aebi') returned ERROR but expected False
   PASSED: isBeauteous7('baei') returned expected value True
   PASSED: isBeauteous7('aeibc') returned expected value True
   PASSED: isBeauteous7('aebic') returned expected value False
***FAILED: isBeauteous7('aebci') returned ERROR but expected False
***FAILED: isBeauteous7('abeci') returned ERROR but expected False
***FAILED: isBeauteous7('abcei') returned ERROR but expected False
   PASSED: isBeauteous7('bcaei') returned expected value True
   PASSED: isBeauteous7('baeic') returned expected value True
   PASSED: isBeauteous7('abeicoua') returned expected value True
   PASSED: isBeauteous7('abeiocua') returned expected value True
   PASSED: isBeauteous7('aeibouca') returned expected value True
***FAILED: isBeauteous7('aebiocua') returned ERROR but expected False
Passed 15 and failed 11 of 26 test cases.

9. Designing Glass-box Test Cases and Minimal Counterexamples

Glass-box testing occurs when you are testing a function/program whose code you can inspect. Because you can see the implementation, you can focus on test cases that take advantage of implementation details in order to attempt to get the function to misbehave.

For example:

  • You should supply test inputs that force every conditional branch in the code to be executed at least once.

  • When loops are involved, you should supply inputs that cause the loop to be executed zero, one, and multiple times.

  • If a loop is executed over a sequence, you should test that it processes all elements of the sequence appropriately. In particular, it should avoid so-called fence-post errors in which it fails to appropriately process the first or last elements of the sequence.

  • When sequence indices are involved, you should supply test inputs that force these indices to be edge cases.

The main goal in glass-box testing is finding counterexamples = inputs that cause the function to misbehave. Particularly interesting counter examples are minimal counterexamples, which are the "shortest" counterexamples. E.g., in functions with string inputs, the shortest string that exhibits a bug is a minimal counterexample.

9.1 Example: Glass-Box Testing of isBeauteous1

Below is a buggy version of the isBeauteous function named isBeauteous1:

In [35]:
def isBeauteous1(word):
    counter = 0
    for letter in word:
        if isVowel(letter):
            counter += 1
        if counter == 3:
            return True
    return False

The problem with isBeauteous1 is that it just counts that the number of vowels in word is 3 without checking that they are consecutive. It will behave correctly on strings with fewer than 3 vowels or with 3 consecutive vowels, but will incorrectly return True for strings that have 3 vowels without having three consecutive vowels.

Below, give a minimal counterexample on which isBeauteous1 gives the incorrect answer:

In [36]:
isBeauteous1('abei') # Three vowels separated by one nonvowel
Out[36]:
True

9.2 Exercise: Minimal Counterexamples for Other Buggy isBeauteous Functions

Below are three other buggy versions of isBeauteous. Develop minimal counterexamples for each of them

In [37]:
def isBeauteous2(word):
    counter = 0
    for letter in word:
        if counter >= 3:
            return True
        elif isVowel(letter):
            counter += 1
        else:
            counter = 0
    return False

# Put your minimal counterexample for isBeauteous2 here: 
isBeauteous2('aei')
# isBeauteous2 returns True for 3 consecutive vowels on the character *after*
# the 3rd consecutive vowel. So it fails when no character follows 3 consecutive 
# vowels. So the minimal counterexample is just 3 consecutive vowels. 
Out[37]:
False
In [38]:
def isBeauteous4(word):
    letters = ''
    for letter in word:
        letters += letter
        if not (letter == word[0] or letter == word[1]):
            if (isVowel(letters[-1])
                and isVowel(letters[-2])
                and isVowel(letters[-3])):
                return True
    return False

# Put your minimal counterexample for isBeauteous4 here: 
isBeauteous4('aea')
# The test `not (letter == word[0] or letter == word[1])` is intended
# to prevent testing of indices -1, -2, -3 until at least three
# characters have been processed. But it accidentally leads to misbehavior
# when a vowel in a 3-consecutive vowel sequence whose index is >=2
# happens to be the same as the one of the first-two letters in the word,
# e.g. 'abaei', 'bacoua', 'aea'. The shortest of these has 3 characters.
Out[38]:
False
In [39]:
def isBeauteous7(word):
    for i in range(len(word)):
        if (isVowel(word[i])
            and isVowel(word[i+1])
            and isVowel(word[i+2])):
            return True
    return False

# Put your minimal counterexample for isBeauteous4 here: 
isBeauteous7('a')
# isBeauteous generates an IndexError when a vowel that is not part 
# of a 3-consecutive-vowel sequence is not followed by at least two more
# characters in a word, e.g., 'bcae', 'bcad', 'bae', 'bad', 'ad', 'a'. 
# The shortest of these is a single vowel. 
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-39-95b25207b0ef> in <module>()
      8 
      9 # Put your minimal counterexample for isBeauteous4 here:
---> 10 isBeauteous7('a')
     11 # isBeauteous generates an IndexError when a vowel that is not part
     12 # of a 3-consecutive-vowel sequence is not followed by at least two more

<ipython-input-39-95b25207b0ef> in isBeauteous7(word)
      2     for i in range(len(word)):
      3         if (isVowel(word[i])
----> 4             and isVowel(word[i+1])
      5             and isVowel(word[i+2])):
      6             return True

IndexError: string index out of range

10. Debugging Techniques

Test cases help us determine cases in which functions misbehave. But then how do we determine why they misbehave and how do fix them?

Here we study some debugging techniques for identifying and fixing bugs in programs. Most of these techniques involve adding print statements to a program.

You should also consult Peter Mawhorter's debugging poster, which is linked from the Reference menu of the CS111 web site.

Some of our examples will involve cs1graphics, so let's import that.

In [40]:
from cs1graphics import *

10.1 Pay Attention to Error Messages

Sometimes bugs lead to errors when running a program. In many cases, studying the error messages will help you to identify the location of the bug. For example, can you use the error message to find and fix the bug in the following code?

In [41]:
def makeCanvasWithCircle(canvasSize, circleColor):
    """Create and return a canvasSize x canvasSize white canvas
    containing a center circle whose diameter is canvasSize and
    whose color is circleColor."""
    canv = Canvas(canvasSize, canvasSize, "white")
    radius = size/2
    circ = Circle(radius, Point(radius, radius))
    canv.addCircle(circ)
    return canv
    
makeCanvasWithCircle(300, 'red')
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-41-bc5e61aeafc1> in <module>()
      9     return canv
     10 
---> 11 makeCanvasWithCircle(300, 'red')

<ipython-input-41-bc5e61aeafc1> in makeCanvasWithCircle(canvasSize, circleColor)
      4     whose color is circleColor."""
      5     canv = Canvas(canvasSize, canvasSize, "white")
----> 6     radius = size/2
      7     circ = Circle(radius, Point(radius, radius))
      8     canv.addCircle(circ)

NameError: name 'size' is not defined

In the case of Syntax Errors, Canopy often only notices these one or more lines after the actual syntax error, so you need to look for the error before the line being reported in the error message. For example, where is the bug in the following example?

In [ ]:
import random

def randomGesture():
    n = random.randint(1,3)
    if n == 1:
        # Still trying to figure out what to do here. 
        
def testRandomGesture():
    print('randomGesture() => {}'.format(randomGesture()))
    
testRandomGesture()

10.2 Use print to indicate a function has been called

It's often helpful to know when a function has been called. We can add a print to the beginning of the function to show that it's being called.

In [42]:
def concentricCircles(size, numCircles, color1, color2):
    """Correct function from PS05 Task 2"""
    print('concentric circles(...)') #*** DEBUGGING PRINT that ignores argument values
    
    allCircles = Layer() # Create layer to hold all circles                                
    largestRadius = size/2
    thickness = largestRadius/numCircles # Note that this is a float                       

    # create as many circles as specified by 'numCircles'                                  
    for i in range(numCircles):

        # In this approach, make circles from smallest to largest                          
        radius = thickness*(i+1) # i+1 because i starts at 0                               
        oneCircle = Circle(radius)

        # Choose color based on whether i is even or odd                                   
        if i % 2 == 0:
            oneCircle.setFillColor(color1) # use color1 for even i                         
        else:
            oneCircle.setFillColor(color2) # use color2 for odd i                          

        # In smallest to largest approach, must set depth to see                           
        # smaller circles. Lower depths appear in front of higher depths,                  
        # so can naturally use i for the depth.                                            
        oneCircle.setDepth(i) # could also use radius                                      

        allCircles.add(oneCircle) # add current circle to layer                            

    return allCircles # return layer   

Below are two buggy versions of circleRow from PS05 Task 2. Both display a single circle. Based on the printed feedback from concentricCircles, you can tell whether they are creating only one concentric circle layer or multiple layers that happen to be at the same position.

In [43]:
def circleRow1(numCirclesInRow, size, numCircles, color1, color2):
                        
    canv = Canvas(numCirclesInRow*size, size, title='ignore title for now')
                
    for i in range(numCirclesInRow):
        circ = concentricCircles(size, numCircles, color1, color2)
        circ.move(size/2, size/2)                                                
        canv.add(circ)

    return canv  

circleRow1(3, 300, 5, 'yellow', 'green')
concentric circles(...)
concentric circles(...)
concentric circles(...)
Out[43]:
<cs1graphics.Canvas at 0x106eb5240>
In [44]:
def circleRow2(numCirclesInRow, size, numCircles, color1, color2):
                        
    canv = Canvas(numCirclesInRow*size, size, title='ignore title for now')
    circ = concentricCircles(size, numCircles, color1, color2)   
    canv.add(circ)
                
    for i in range(numCirclesInRow):
        circ.moveTo(size/2 + i*size, size/2)                                                

    return canv 

circleRow2(3, 300, 5, 'magenta', 'cyan')
concentric circles(...)
Out[44]:
<cs1graphics.Canvas at 0x106f8bbe0>

10.3 Use print to show a function call with its arguments

Rather than just knowing a function has been called, it's generally helpful to know what arguments it has been called with.

Study what's printed by beats and play to help figure out why play is not correct.

In [45]:
def beats(gest1, gest2):
    #*** DEBUGGING PRINT: Print call with args 
    print('beats({}, {})'.format(gest1, gest2))  
    return (gest1 == 'rock' and gest2 == 'scissors'
            or gest1 == 'scissors' and gest2 == 'paper'
            or gest1 == 'paper' and gest2 == 'rock')
            
def play(you, opponent):
    #*** DEBUGGING PRINT: Print call with args 
    print('play({}, {})'.format(you, opponent)) 
    # Ignore invalid gestures for now. 
    if beats(you, opponent):
        print('You win')
    elif not beats(you, opponent):
        print('Opponent wins')
    else:
        print('Game is a tie')
            
play('scissors', 'paper')
play('paper', 'scissors')
play('paper', 'paper')
play(scissors, paper)
beats(scissors, paper)
You win
play(paper, scissors)
beats(paper, scissors)
beats(paper, scissors)
Opponent wins
play(paper, paper)
beats(paper, paper)
beats(paper, paper)
Opponent wins

10.4 Use print to show the return value of a function

In addition to showing the arguments to a function when a function is called, it's often a good idea to show both the arguments and the return value when it returns.

In order to do this, it is often necessary to introduce a variable (such as result) to first name the returned value so that it can be printed before it is returned (without recalculating it).

Here are example of code before/after adding the debugging prints:

In [46]:
import math

def squareBefore(n):
    return n*n

def squareAfter(n):
    result = n*n
    #*** DEBUGGING PRINT: Print call with args and return value
    print('square({}) => {}'.format(n, result))
    return result

def hypotenuseBefore(a,b):
    return math.sqrt(squareBefore(a) + squareBefore(b))

def hypotenuseAfter(a,b):
    result = math.sqrt(squareAfter(a) + squareAfter(b))
    #*** DEBUGGING PRINT: Print call with args and return value
    print('hypotenuse({}, {}) => {}'.format(a, b, result))
    return result

hypotenuseAfter(3,4)
square(3) => 9
square(4) => 16
hypotenuse(3, 4) => 5.0
Out[46]:
5.0

When returns are performed in conditional branches, you should:

  1. Initialize result to None before the conditionals.
  2. Replace each return Expr by result = Expr
  3. End the function body with return result

Below are examples of some buggy code before/after adding the debugging prints. Use the printed output to help you find and fix the bugs:

In [47]:
def isEvenBefore(n):
    return n%2

def isEvenAfter(n):
    result = n%2
    #*** DEBUGGING PRINT: Print call with args and return value
    print('isEven({}) => {}'.format(n, result))
    return result
          
def chooseColorBefore(index, color1, color2):
    '''If index is even, return color1; otherwise return color2'''
    if isEvenBefore(index):
          return color1
    else:
          return color2
          
def chooseColorAfter(index, color1, color2):
    '''If index is even, return color1; otherwise return color2'''
    result = None
    if isEvenAfter(index):
          result = color1
    else:
          result = color2
    print('chooseColor({}, {}, {}) => {}'
          .format(index, color1, color2, result))
          
for i in range(4):
    chooseColorAfter(i, 'blue', 'green')
isEven(0) => 0
chooseColor(0, blue, green) => green
isEven(1) => 1
chooseColor(1, blue, green) => blue
isEven(2) => 0
chooseColor(2, blue, green) => green
isEven(3) => 1
chooseColor(3, blue, green) => blue
In [48]:
import random

def randomGestureBefore():
    n = random.randont(1,4)
    if n == 1:
        return 'rock'
    if n == 2:
        return 'paper'
    if n == 3:
        return 'scissors'
    
def randomGestureAfter():
    n = random.randint(1,4)
    result = None
    if n == 1:
        result = 'rock'
    if n == 2:
        result = 'paper'
    if n == 3:
        result = 'scissors'
    #*** DEBUGGING PRINT: Print call with args and return value
    print('randomGesture() => {}'.format(result))
    return result

# Test randomGestureAfter 10 times
for i in range(10):
    randomGestureAfter()
randomGesture() => scissors
randomGesture() => scissors
randomGesture() => scissors
randomGesture() => rock
randomGesture() => rock
randomGesture() => paper
randomGesture() => rock
randomGesture() => paper
randomGesture() => scissors
randomGesture() => rock

10.5 Use print to show both calling and returning from a function

When a function is giving an error, it's a good idea to use print to show both when the function is called and when it returns.

Here's an example; use the printed information to find and fix the bug.

In [49]:
def isBookendsBefore(word):
    '''Returns True if word begins and ends with the same character;
    otherwise returns False'''
    return word[0] == word[-1]

def isBookendsAfter(word):
    '''Returns True if word begins and ends with the same character;
    otherwise returns False'''
    #*** DEBUGGING PRINT: Print call with args
    print("Entering isBookends('{}')".format(word))
    result = word[0] == word[-1]
    #*** DEBUGGING PRINT: Print call with args and return value
    print("Exiting isBookends('{}') => {}".format(word, result))

for w in ['mom', 'cat', 'I', '', 'ee']:
    isBookendsAfter(w)
Entering isBookends('mom')
Exiting isBookends('mom') => True
Entering isBookends('cat')
Exiting isBookends('cat') => False
Entering isBookends('I')
Exiting isBookends('I') => True
Entering isBookends('')
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-49-00e08ac93bb9> in <module>()
     14 
     15 for w in ['mom', 'cat', 'I', '', 'ee']:
---> 16     isBookendsAfter(w)

<ipython-input-49-00e08ac93bb9> in isBookendsAfter(word)
      9     #*** DEBUGGING PRINT: Print call with args
     10     print("Entering isBookends('{}')".format(word))
---> 11     result = word[0] == word[-1]
     12     #*** DEBUGGING PRINT: Print call with args and return value
     13     print("Exiting isBookends('{}') => {}".format(word, result))

IndexError: string index out of range

10.6 Using print to display iteration tables

We saw in Lec 09 Iteration 1 that a function adding two print statments to a loop (one right before the loop and one at the end of the loop body) can display an iteration table for the state variables of the loop.

Let's review that technique here in the context of debugging the definition of countChar given at the beginning of this notebook.

In countCharTable below, in addition to displaying the state variables i and counter, we also display word[i], since this is important for debugging.

For completeness, we might also want to print when the function is called and when it returns. But to avoid too much clutter, we will include only the iteration table prints in these examples.

In [50]:
def countCharTable(char, word):
    '''Return the number of time char occurs in word, ignoring case.'''
    counter = 0 
    #*** DEBUGGING PRINT: Print first row of iteration table
    print("countChar loop: | i: N/A | word[i]: N/A | counter: {} |".format(counter))
    for i in range(1, len(word)-1):
        if word[i] == char.lower():
            counter += 1
        #*** DEBUGGING PRINT: Print remaining rows of iteration table
        print("countChar loop: | i: {} | word[i]: '{}' | counter: {} |"
              .format(i, word[i], counter))
    return counter

We know from testing that countChar('m', 'mississippi') returns 0 when 1 is expected. Why is that? Let's see:

In [51]:
countCharTable('m', 'mississippi')
countChar loop: | i: N/A | word[i]: N/A | counter: 0 |
countChar loop: | i: 1 | word[i]: 'i' | counter: 0 |
countChar loop: | i: 2 | word[i]: 's' | counter: 0 |
countChar loop: | i: 3 | word[i]: 's' | counter: 0 |
countChar loop: | i: 4 | word[i]: 'i' | counter: 0 |
countChar loop: | i: 5 | word[i]: 's' | counter: 0 |
countChar loop: | i: 6 | word[i]: 's' | counter: 0 |
countChar loop: | i: 7 | word[i]: 'i' | counter: 0 |
countChar loop: | i: 8 | word[i]: 'p' | counter: 0 |
countChar loop: | i: 9 | word[i]: 'p' | counter: 0 |
Out[51]:
0

Ah, because i starts at 1 rather than 0, the letter starting the word is never counted. Let's fix that:

In [52]:
def countCharTableFix1(char, word):
    counter = 0
    #*** DEBUGGING PRINT: Print first row of iteration table
    print("countChar loop: | i: N/A | word[i]: N/A | counter: {} |".format(counter))
    for i in range(0, len(word)-1): #*** Bug Fix #1: start index should be 0, not 1
        if word[i] == char.lower():
            counter += 1
        #*** DEBUGGING PRINT: Print remaining rows of iteration table
        print("countChar loop: | i: {} | word[i]: '{}' | counter: {} | "
              .format(i, word[i], counter))
    return counter

Now countCharTableFix1('m', 'mississippi') works as expected:

In [53]:
countCharTableFix1('m', 'mississippi')
countChar loop: | i: N/A | word[i]: N/A | counter: 0 |
countChar loop: | i: 0 | word[i]: 'm' | counter: 1 | 
countChar loop: | i: 1 | word[i]: 'i' | counter: 1 | 
countChar loop: | i: 2 | word[i]: 's' | counter: 1 | 
countChar loop: | i: 3 | word[i]: 's' | counter: 1 | 
countChar loop: | i: 4 | word[i]: 'i' | counter: 1 | 
countChar loop: | i: 5 | word[i]: 's' | counter: 1 | 
countChar loop: | i: 6 | word[i]: 's' | counter: 1 | 
countChar loop: | i: 7 | word[i]: 'i' | counter: 1 | 
countChar loop: | i: 8 | word[i]: 'p' | counter: 1 | 
countChar loop: | i: 9 | word[i]: 'p' | counter: 1 | 
Out[53]:
1

countCharTableFix1('i', 'mississippi') still returns 3 rather than the expected 4. Why?

In [54]:
countCharTableFix1('i', 'mississippi')
countChar loop: | i: N/A | word[i]: N/A | counter: 0 |
countChar loop: | i: 0 | word[i]: 'm' | counter: 0 | 
countChar loop: | i: 1 | word[i]: 'i' | counter: 1 | 
countChar loop: | i: 2 | word[i]: 's' | counter: 1 | 
countChar loop: | i: 3 | word[i]: 's' | counter: 1 | 
countChar loop: | i: 4 | word[i]: 'i' | counter: 2 | 
countChar loop: | i: 5 | word[i]: 's' | counter: 2 | 
countChar loop: | i: 6 | word[i]: 's' | counter: 2 | 
countChar loop: | i: 7 | word[i]: 'i' | counter: 3 | 
countChar loop: | i: 8 | word[i]: 'p' | counter: 3 | 
countChar loop: | i: 9 | word[i]: 'p' | counter: 3 | 
Out[54]:
3

Oh, the loop never processes the last letter i at word[10] because the second argument to range is len(word)-1 rather than len(word). Let's fix this second bug:

In [55]:
def countCharTableFix2(char, word):
    '''Return the number of time char occurs in word, ignoring case.'''
    counter = 0
    #*** DEBUGGING PRINT: Print first row of iteration table
    print("countChar loop: | i: N/A | word[i]: N/A | counter: {} |".format(counter))
    for i in range(0, len(word)): #*** Bug Fix #1: start index should be 0, not 1
                                  #*** Bug Fix #2: start index should be len(word), 
                                  #      not len(word)-1
        if word[i] == char.lower():
            counter += 1
        #*** DEBUGGING PRINT: Print remaining rows of iteration table
        print("countChar loop: | i: {} | word[i]: '{}' | counter: {} |"
              .format(i, word[i], counter))
    return counter

With this second fix, countCharTableFix2('i', 'mississippi') now works as expected:

In [56]:
countCharTableFix2('i', 'mississippi')
countChar loop: | i: N/A | word[i]: N/A | counter: 0 |
countChar loop: | i: 0 | word[i]: 'm' | counter: 0 |
countChar loop: | i: 1 | word[i]: 'i' | counter: 1 |
countChar loop: | i: 2 | word[i]: 's' | counter: 1 |
countChar loop: | i: 3 | word[i]: 's' | counter: 1 |
countChar loop: | i: 4 | word[i]: 'i' | counter: 2 |
countChar loop: | i: 5 | word[i]: 's' | counter: 2 |
countChar loop: | i: 6 | word[i]: 's' | counter: 2 |
countChar loop: | i: 7 | word[i]: 'i' | counter: 3 |
countChar loop: | i: 8 | word[i]: 'p' | counter: 3 |
countChar loop: | i: 9 | word[i]: 'p' | counter: 3 |
countChar loop: | i: 10 | word[i]: 'i' | counter: 4 |
Out[56]:
4

countCharTableFix2('I', 'MISSISSIPPI') still returns 0 rather than the expected 4. Why is that?

In [57]:
countCharTableFix2('I', 'MISSISSIPPI')
countChar loop: | i: N/A | word[i]: N/A | counter: 0 |
countChar loop: | i: 0 | word[i]: 'M' | counter: 0 |
countChar loop: | i: 1 | word[i]: 'I' | counter: 0 |
countChar loop: | i: 2 | word[i]: 'S' | counter: 0 |
countChar loop: | i: 3 | word[i]: 'S' | counter: 0 |
countChar loop: | i: 4 | word[i]: 'I' | counter: 0 |
countChar loop: | i: 5 | word[i]: 'S' | counter: 0 |
countChar loop: | i: 6 | word[i]: 'S' | counter: 0 |
countChar loop: | i: 7 | word[i]: 'I' | counter: 0 |
countChar loop: | i: 8 | word[i]: 'P' | counter: 0 |
countChar loop: | i: 9 | word[i]: 'P' | counter: 0 |
countChar loop: | i: 10 | word[i]: 'I' | counter: 0 |
Out[57]:
0

The reason isn't obvious from the iteration table, but it does give a hint. Why is counter not being incremented when word[i] is the letter I? It's because we're comparing word[i] with char.lower() when we should be using word[i].lower() instead. Let's fix that third bug:

In [58]:
def countCharTableFix3(char, word):
    '''Return the number of time char occurs in word, ignoring case.'''
    counter = 0
    #*** DEBUGGING PRINT: Print first row of iteration table
    print("countChar loop: | i: N/A | word[i]: N/A | counter: {} |".format(counter))
    for i in range(0, len(word)): #*** Bug Fix #1: start index should be 0, not 1
                                  #*** Bug Fix #2: start index should be len(word), 
                                  #      not len(word)-1
        if word[i].lower() == char.lower(): #*** Bug Fix #3: add .lower() to word[i]
            counter += 1
        #*** DEBUGGING PRINT: Print remaining rows of iteration table
        print("countChar loop: | i: {} | word[i]: '{}' | counter: {} |"
              .format(i, word[i], counter))
    return counter

Now countCharTableFix3('I', 'MISSISSIPPI') works as expected.

In [59]:
countCharTableFix3('I', 'MISSISSIPPI')
countChar loop: | i: N/A | word[i]: N/A | counter: 0 |
countChar loop: | i: 0 | word[i]: 'M' | counter: 0 |
countChar loop: | i: 1 | word[i]: 'I' | counter: 1 |
countChar loop: | i: 2 | word[i]: 'S' | counter: 1 |
countChar loop: | i: 3 | word[i]: 'S' | counter: 1 |
countChar loop: | i: 4 | word[i]: 'I' | counter: 2 |
countChar loop: | i: 5 | word[i]: 'S' | counter: 2 |
countChar loop: | i: 6 | word[i]: 'S' | counter: 2 |
countChar loop: | i: 7 | word[i]: 'I' | counter: 3 |
countChar loop: | i: 8 | word[i]: 'P' | counter: 3 |
countChar loop: | i: 9 | word[i]: 'P' | counter: 3 |
countChar loop: | i: 10 | word[i]: 'I' | counter: 4 |
Out[59]:
4

If we fix the all three bugs in countChar, do we resolve all the test case failures?

Below, note that we comment out the debugging prints so they do not interfere with the testing. But we do not delete the debugging prints, since we may want to uncomment them for debugging purposes in the future!

In [60]:
def countChar(char, word): # FIXED VERSION, WITH PRINTS COMMENTED OUT
    '''Return the number of time char occurs in word, ignoring case.'''
    counter = 0
    #*** DEBUGGING PRINT: Print first row of iteration table
    # print("countChar loop: | i: N/A | word[i]: N/A | counter: {} |".format(counter))
    for i in range(0, len(word)): #*** Bug Fix #1: start index should be 0, not 1
                                  #*** Bug Fix #2: start index should be len(word), 
                                  #      not len(word)-1
        if word[i].lower() == char.lower(): #*** Bug Fix #3: add .lower() to word[i]
            counter += 1
        #*** DEBUGGING PRINT: Print remaining rows of iteration table
        # print("countChar loop: | i: {} | word[i]: '{}' | counter: {} |"
        #       .format(i, word[i], counter))
    return counter

testCountChar(moreCountCharTestCases)
   PASSED: countChar('s', 'Mississippi') returned expected value 4
   PASSED: countChar('i', 'Mississippi') returned expected value 4
   PASSED: countChar('p', 'Mississippi') returned expected value 2
   PASSED: countChar('m', 'Mississippi') returned expected value 1
   PASSED: countChar('a', 'Mississippi') returned expected value 0
   PASSED: countChar('s', 'Mississippi') returned expected value 4
   PASSED: countChar('i', 'Mississippi') returned expected value 4
   PASSED: countChar('p', 'Mississippi') returned expected value 2
   PASSED: countChar('m', 'Mississippi') returned expected value 1
   PASSED: countChar('a', 'Mississippi') returned expected value 0
   PASSED: countChar('s', 'mississippi') returned expected value 4
   PASSED: countChar('i', 'mississippi') returned expected value 4
   PASSED: countChar('p', 'mississippi') returned expected value 2
   PASSED: countChar('m', 'mississippi') returned expected value 1
   PASSED: countChar('a', 'mississippi') returned expected value 0
   PASSED: countChar('s', 'MISSISSIPPI') returned expected value 4
   PASSED: countChar('i', 'MISSISSIPPI') returned expected value 4
   PASSED: countChar('p', 'MISSISSIPPI') returned expected value 2
   PASSED: countChar('m', 'MISSISSIPPI') returned expected value 1
   PASSED: countChar('a', 'MISSISSIPPI') returned expected value 0
   PASSED: countChar('S', 'mississippi') returned expected value 4
   PASSED: countChar('I', 'mississippi') returned expected value 4
   PASSED: countChar('P', 'mississippi') returned expected value 2
   PASSED: countChar('M', 'mississippi') returned expected value 1
   PASSED: countChar('A', 'mississippi') returned expected value 0
   PASSED: countChar('S', 'MISSISSIPPI') returned expected value 4
   PASSED: countChar('I', 'MISSISSIPPI') returned expected value 4
   PASSED: countChar('P', 'MISSISSIPPI') returned expected value 2
   PASSED: countChar('M', 'MISSISSIPPI') returned expected value 1
   PASSED: countChar('A', 'MISSISSIPPI') returned expected value 0

Great! We now pass all the test cases. Does that mean our function is completely correct?

Not necessarily! Maybe there are some cases in which the function still doesn't work, but they're not in our list of test cases. So it may be too early to declare victory, but we've increased our confidence in the correctness of the countChar function definition.

11. Debugging Exercise with isBeauteous

Use the debugging techniques above, particularly printing iteration tables, to identify (but not necessarily fix) the bugs in the following buggy versions of isBeauteous. Test the print-augmented versions on potential counterexamples to identify bugs.

In [61]:
def isBeauteous1(word):
    #*** DEBUGGING PRINT: Print call with args
    print("Entering isBeauteous1('{}')".format(word))
    counter = 0
    #*** DEBUGGING PRINT: Print first row of iteration table
    print("isBeauteous1 loop: | letter: N/A | counter: {} |".format(counter))
    for letter in word:
        if isVowel(letter):
            counter += 1
        #*** DEBUGGING PRINT: Print remaining rows of iteration table
        print("isBeauteous1 loop: | letter: '{}' | counter: {} | ".format(letter, counter))
        if counter == 3:
            #*** DEBUGGING PRINT: Print call with args and return value
            print("Exiting isBeauteous1('{}') => {}".format(word, True))
            return True
    #*** DEBUGGING PRINT: Print call with args and return value
    print("Exiting isBeauteous1('{}') => {}".format(word, False))        
    return False

# Try the debugging version on potential counterexamples
isBeauteous1('abeci')
isBeauteous1('abei')
Entering isBeauteous1('abeci')
isBeauteous1 loop: | letter: N/A | counter: 0 |
isBeauteous1 loop: | letter: 'a' | counter: 1 | 
isBeauteous1 loop: | letter: 'b' | counter: 1 | 
isBeauteous1 loop: | letter: 'e' | counter: 2 | 
isBeauteous1 loop: | letter: 'c' | counter: 2 | 
isBeauteous1 loop: | letter: 'i' | counter: 3 | 
Exiting isBeauteous1('abeci') => True
Entering isBeauteous1('abei')
isBeauteous1 loop: | letter: N/A | counter: 0 |
isBeauteous1 loop: | letter: 'a' | counter: 1 | 
isBeauteous1 loop: | letter: 'b' | counter: 1 | 
isBeauteous1 loop: | letter: 'e' | counter: 2 | 
isBeauteous1 loop: | letter: 'i' | counter: 3 | 
Exiting isBeauteous1('abei') => True
Out[61]:
True
In [62]:
def isBeauteous2(word):
    #*** DEBUGGING PRINT: Print call with args
    print("Entering isBeauteous2('{}')".format(word))
    counter = 0
    #*** DEBUGGING PRINT: Print first row of iteration table
    print("isBeauteous2 loop: | letter: N/A | counter: {} |".format(counter))
    for letter in word:
        if counter >= 3:
            #*** DEBUGGING PRINT: Print call with args and return value
            print("Exiting isBeauteous2('{}') => {}".format(word, True))
            return True
        elif isVowel(letter):
            counter += 1
        else:
            counter = 0
        #*** DEBUGGING PRINT: Print remaining rows of iteration table
        print("isBeauteous1 loop: | letter: '{}' | counter: {} | ".format(letter, counter))
    #*** DEBUGGING PRINT: Print call with args and return value
    print("Exiting isBeauteous2('{}') => {}".format(word, False))       
    return False

# Try the debugging version on potential counterexamples
isBeauteous2('baei')
isBeauteous2('aei')
Entering isBeauteous2('baei')
isBeauteous2 loop: | letter: N/A | counter: 0 |
isBeauteous1 loop: | letter: 'b' | counter: 0 | 
isBeauteous1 loop: | letter: 'a' | counter: 1 | 
isBeauteous1 loop: | letter: 'e' | counter: 2 | 
isBeauteous1 loop: | letter: 'i' | counter: 3 | 
Exiting isBeauteous2('baei') => False
Entering isBeauteous2('aei')
isBeauteous2 loop: | letter: N/A | counter: 0 |
isBeauteous1 loop: | letter: 'a' | counter: 1 | 
isBeauteous1 loop: | letter: 'e' | counter: 2 | 
isBeauteous1 loop: | letter: 'i' | counter: 3 | 
Exiting isBeauteous2('aei') => False
Out[62]:
False
In [63]:
def isBeauteous4(word):
    #*** DEBUGGING PRINT: Print call with args
    print("Entering isBeauteous4('{}')".format(word))
    letters = ''
    #*** DEBUGGING PRINT: Print first row of iteration table
    print("isBeauteous4 loop: | letter: N/A | letters: '{}' |".format(letters))
    for letter in word:
        letters += letter
        if not (letter == word[0] or letter == word[1]):
            if (isVowel(letters[-1])
                and isVowel(letters[-2])
                and isVowel(letters[-3])):
                #*** DEBUGGING PRINT: Print call with args and return value
                print("Exiting isBeauteous4('{}') => {}".format(word, True))
                return True
        #*** DEBUGGING PRINT: Print remaining rows of iteration table
        print("isBeauteous4 loop: | letter: '{}' | letters: '{}' | ".format(letter, letters))
    #*** DEBUGGING PRINT: Print call with args and return value
    print("Exiting isBeauteous4('{}') => {}".format(word, False))
    return False

# Try the debugging version on potential counterexamples
isBeauteous2('baei')
isBeauteous2('acoua')
isBeauteous2('aea')
Entering isBeauteous2('baei')
isBeauteous2 loop: | letter: N/A | counter: 0 |
isBeauteous1 loop: | letter: 'b' | counter: 0 | 
isBeauteous1 loop: | letter: 'a' | counter: 1 | 
isBeauteous1 loop: | letter: 'e' | counter: 2 | 
isBeauteous1 loop: | letter: 'i' | counter: 3 | 
Exiting isBeauteous2('baei') => False
Entering isBeauteous2('acoua')
isBeauteous2 loop: | letter: N/A | counter: 0 |
isBeauteous1 loop: | letter: 'a' | counter: 1 | 
isBeauteous1 loop: | letter: 'c' | counter: 0 | 
isBeauteous1 loop: | letter: 'o' | counter: 1 | 
isBeauteous1 loop: | letter: 'u' | counter: 2 | 
isBeauteous1 loop: | letter: 'a' | counter: 3 | 
Exiting isBeauteous2('acoua') => False
Entering isBeauteous2('aea')
isBeauteous2 loop: | letter: N/A | counter: 0 |
isBeauteous1 loop: | letter: 'a' | counter: 1 | 
isBeauteous1 loop: | letter: 'e' | counter: 2 | 
isBeauteous1 loop: | letter: 'a' | counter: 3 | 
Exiting isBeauteous2('aea') => False
Out[63]:
False
In [64]:
def isBeauteous7(word):
    #*** DEBUGGING PRINT: Print call with args
    print("Entering isBeauteous7('{}')".format(word))
    #*** DEBUGGING PRINT: Print first row of iteration table
    print("isBeauteous7 loop: | i: N/A | word[i]: N/A |")
    for i in range(len(word)):
        if (isVowel(word[i])
            and isVowel(word[i+1])
            and isVowel(word[i+2])):
            #*** DEBUGGING PRINT: Print call with args and return value
            print("Exiting isBeauteous7('{}') => {}".format(word, True))
            return True
        #*** DEBUGGING PRINT: Print remaining rows of iteration table
        print("isBeauteous4 loop: | i: {} | word[i]: {} | ".format(i, word[i]))
    #*** DEBUGGING PRINT: Print call with args and return value
    print("Exiting isBeauteous7('{}') => {}".format(word, False))
    return False

# Try the debugging version on potential counterexamples
isBeauteous7('bcae')
isBeauteous7('a')
Entering isBeauteous7('bcae')
isBeauteous7 loop: | i: N/A | word[i]: N/A |
isBeauteous4 loop: | i: 0 | word[i]: b | 
isBeauteous4 loop: | i: 1 | word[i]: c | 
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-64-6bd53ca46a86> in <module>()
     18 
     19 # Try the debugging version on potential counterexamples
---> 20 isBeauteous7('bcae')
     21 isBeauteous7('a')

<ipython-input-64-6bd53ca46a86> in isBeauteous7(word)
      7         if (isVowel(word[i])
      8             and isVowel(word[i+1])
----> 9             and isVowel(word[i+2])):
     10             #*** DEBUGGING PRINT: Print call with args and return value
     11             print("Exiting isBeauteous7('{}') => {}".format(word, True))

IndexError: string index out of range
In [ ]: