{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Testing" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[Original notebook](https://github.com/profjsb/python-bootcamp/blob/master/Lectures/12_Testing/12_testing.ipynb) by Jarrod Millman, part of the Python-bootcamp.\n", "\n", "Modifications Hans Fangohr, Sept 2013:\n", "\n", "- Add py.test example\n", "- minor edits\n", "\n", "Move to Python 3, Sept 2016." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Motivation\n", "\n", "### Computing is error prone\n", "\n", "> In ordinary computational practice by hand or by desk machines, it is the\n", "> custom to check every step of the computation and, when an error is found,\n", "> to localize it by a backward process starting from the first point where the\n", "> error is noted.\n", ">\n", "> - Norbert Wiener (1948)\n", "\n", "### More computing, more problems\n", "\n", "> The major cause of the **software crisis** is that the machines have become\n", "> several orders of magnitude more powerful! To put it quite bluntly: as long\n", "> as there were no machines, programming was no problem at all; when we had a\n", "> few weak computers, programming became a mild problem, and now we have\n", "> gigantic computers, programming has become an equally gigantic problem.\n", ">\n", "> - Edsger W. Dijkstra (1972)\n", "\n", "## What testing is and is not...\n", "\n", "### Testing and debugging\n", "\n", "- debugging is what you do when you know a program is broken\n", "- testing is a determined, systematic attempt to break a program\n", "- writing tests is more interesting than debugging\n", "\n", "### Program correctness\n", "\n", "> Program testing can be used to show the presence of bugs, but never to show\n", "> their absence!\n", ">\n", "> - Edsger W. Dijkstra (1969)\n", "\n", "### In the imperfect world ...\n", "\n", "- avoid writing code if possible\n", "- write code as simple as possible\n", "- avoid cleverness\n", "- use code to generate code\n", "\n", "### Program languages play an important role\n", "\n", "> Programmers are always surrounded by complexity; we cannot avoid it. Our\n", "> applications are complex because we are ambitious to use our computers in\n", "> ever more sophisticated ways. Programming is complex because of the large\n", "> number of conflicting objectives for each of our programming projects. **If\n", "> our basic tool, the language in which we design and code our programs, is\n", "> also complicated, the language itself becomes part of the problem rather than\n", "> part of its solution.**\n", ">\n", "> --- C.A.R. Hoare - The Emperor's Old Clothes - Turing Award Lecture (1980)\n", "\n", "### Testing and reproducibility\n", "\n", "> In the good old days physicists repeated each other's experiments, just to\n", "> be sure. Today they stick to FORTRAN, so that they can share each other's\n", "> programs, bugs included.\n", ">\n", "> - Edsger W. Dijkstra (1975)\n", "\n", "### Pre- and post-condition tests\n", "\n", "- what must be true *before* a method is invoked\n", "- what must be true *after* a method is invoked\n", "- use assertions\n", "\n", "### Program defensively\n", "\n", "- out-of-range index\n", "- division by zero\n", "- error returns\n", "\n", "### Be systematic\n", "\n", "- incremental\n", "- simple things first\n", "- know what to expect\n", "- compare independent implementations\n", "\n", "### Automate it\n", "\n", "- **regression tests** ensure that changes don't break existing functionality\n", "- verify conservation\n", "- **unit tests** (white box testing)\n", "- measure test coverage\n", "\n", "### Interface and implementation\n", "\n", "- an **interface** is how something is used\n", "- an **implementation** is how it is written\n", "\n", "## Testing in Python\n", "\n", "### Landscape\n", "\n", "- errors, exceptions, and debugging\n", "- `assert`, `doctest`, and unit tests\n", "- `logging`, `unittest`, and `nose`\n", "\n", "### Errors & Exceptions\n", "\n", "#### Syntax Errors\n", "\n", "- Caught by Python parser, prior to execution\n", "- arrow marks the last parsed command / syntax, which gave an error" ] }, { "cell_type": "code", "execution_count": 52, "metadata": { "collapsed": false }, "outputs": [ { "ename": "NameError", "evalue": "name 'true' is not defined", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32mwhile\u001b[0m \u001b[0mtrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Hello world'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mNameError\u001b[0m: name 'true' is not defined" ] } ], "source": [ "while true:\n", " print('Hello world')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Exceptions\n", "\n", "- Caught during runtime" ] }, { "cell_type": "code", "execution_count": 53, "metadata": { "collapsed": false }, "outputs": [ { "ename": "ZeroDivisionError", "evalue": "division by zero", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mZeroDivisionError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;36m1\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;31mZeroDivisionError\u001b[0m: division by zero" ] } ], "source": [ "1/0" ] }, { "cell_type": "code", "execution_count": 54, "metadata": { "collapsed": false }, "outputs": [ { "ename": "NameError", "evalue": "name 'factorial' is not defined", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mfactorial\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;31mNameError\u001b[0m: name 'factorial' is not defined" ] } ], "source": [ "factorial" ] }, { "cell_type": "code", "execution_count": 55, "metadata": { "collapsed": false }, "outputs": [ { "ename": "TypeError", "evalue": "Can't convert 'int' object to str implicitly", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;34m'1'\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;31mTypeError\u001b[0m: Can't convert 'int' object to str implicitly" ] } ], "source": [ "'1' + 1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Exception handling" ] }, { "cell_type": "code", "execution_count": 56, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "No such file\n" ] } ], "source": [ "try:\n", " file = open('filenamethatdoesnotexist.txt')\n", "except FileNotFoundError:\n", " print('No such file')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Raising exceptions" ] }, { "cell_type": "code", "execution_count": 57, "metadata": { "collapsed": false }, "outputs": [ { "ename": "NotImplementedError", "evalue": "Still need to write this code", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mNotImplementedError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;32mraise\u001b[0m \u001b[0mNotImplementedError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Still need to write this code\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mnewfunction\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;32m\u001b[0m in \u001b[0;36mnewfunction\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mnewfunction\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mNotImplementedError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Still need to write this code\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0mnewfunction\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mNotImplementedError\u001b[0m: Still need to write this code" ] } ], "source": [ "def newfunction():\n", " raise NotImplementedError(\"Still need to write this code\")\n", "\n", "newfunction()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Debugging" ] }, { "cell_type": "code", "execution_count": 60, "metadata": { "collapsed": false }, "outputs": [ { "ename": "ZeroDivisionError", "evalue": "division by zero", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mZeroDivisionError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 7\u001b[0;31m \u001b[0mbar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;32m\u001b[0m in \u001b[0;36mbar\u001b[0;34m(y)\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mbar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 7\u001b[0m \u001b[0mbar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m\u001b[0m in \u001b[0;36mfoo\u001b[0;34m(x)\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mbar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mZeroDivisionError\u001b[0m: division by zero" ] } ], "source": [ "def foo(x):\n", " return 1/x\n", "\n", "def bar(y):\n", " return foo(1-y)\n", "\n", "bar(1)" ] }, { "cell_type": "code", "execution_count": 61, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "> \u001b[0;32m\u001b[0m(2)\u001b[0;36mfoo\u001b[0;34m()\u001b[0m\n", "\u001b[0;32m 1 \u001b[0;31m\u001b[0;32mdef\u001b[0m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m----> 2 \u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m 3 \u001b[0;31m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m 4 \u001b[0;31m\u001b[0;32mdef\u001b[0m \u001b[0mbar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m 5 \u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\n", "ipdb> p x # can x really be zero?\n", "0\n", "ipdb> up\n", "> \u001b[0;32m\u001b[0m(5)\u001b[0;36mbar\u001b[0;34m()\u001b[0m\n", "\u001b[0;32m 3 \u001b[0;31m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m 4 \u001b[0;31m\u001b[0;32mdef\u001b[0m \u001b[0mbar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m----> 5 \u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfoo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m 6 \u001b[0;31m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\u001b[0;32m 7 \u001b[0;31m\u001b[0mbar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0m\n", "ipdb> p y # what is y (one function call UP)\n", "1\n", "ipdb> exit \n" ] } ], "source": [ "%debug" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Fixing bugs " ] }, { "cell_type": "code", "execution_count": 62, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "inf" ] }, "execution_count": 62, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def foo(x):\n", " if x==0:\n", " return float('Inf')\n", " else:\n", " return 1/x\n", "\n", "bar(1)" ] }, { "cell_type": "code", "execution_count": 63, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "inf" ] }, "execution_count": 63, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def foo(x):\n", " try:\n", " return 1/x\n", " except ZeroDivisionError:\n", " return float('Inf')\n", "\n", "bar(1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Test as you code\n", "\n", "### Type checking " ] }, { "cell_type": "code", "execution_count": 64, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Please enter an integer: 5\n", "Casting 5 to integer.\n" ] } ], "source": [ "s = input(\"Please enter an integer: \") # s is a string\n", "if not isinstance(s, int):\n", " print(\"Casting \", s, \" to integer.\")\n", " i = int(s)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Assert invariants" ] }, { "cell_type": "code", "execution_count": 65, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "3\n" ] } ], "source": [ "if i % 3 == 0:\n", " print(1)\n", "elif i % 3 == 1:\n", " print(2)\n", "else:\n", " assert i % 3 == 2\n", " print(3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Example\n", "\n", "Let's make a factorial function." ] }, { "cell_type": "code", "execution_count": 66, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Overwriting myfactorial.py\n" ] } ], "source": [ "%%file myfactorial.py\n", "\n", "def factorial2(n):\n", " \"\"\" Details to come ...\n", " \"\"\"\n", "\n", " raise NotImplementedError\n", "\n", "def test():\n", " from math import factorial\n", " for x in range(10):\n", " print(\".\", end=\"\")\n", " assert factorial2(x) == factorial(x), \\\n", " \"My factorial function is incorrect for n = %i\" % x" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Let's test it ..." ] }, { "cell_type": "code", "execution_count": 67, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import myfactorial\n", "myfactorial.test()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Looks like we will have to implement our function, if we want to make any progress..." ] }, { "cell_type": "code", "execution_count": 68, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Overwriting myfactorial.py\n" ] } ], "source": [ "%%file myfactorial.py\n", "\n", "def factorial2(n):\n", " \"\"\" Details to come ...\n", " \"\"\"\n", "\n", " if n == 0:\n", " return 1\n", " else:\n", " return n * factorial2(n-1)\n", "\n", "def test():\n", " from math import factorial\n", " for x in range(10):\n", " assert factorial2(x) == factorial(x), \\\n", " \"My factorial function is incorrect for n = %i\" % x" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Let's test it ..." ] }, { "cell_type": "code", "execution_count": 69, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import importlib\n", "importlib.reload(myfactorial)\n", "myfactorial.test()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Seems to be okay so far. However, calling ``factorial2`` with a negative number, say, will result in infinite loop. Thus:\n", "\n", "### What about preconditions\n", "\n", "What happens if we call `factorial2` with a negative integer? Or something that's not an integer?" ] }, { "cell_type": "code", "execution_count": 70, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Overwriting myfactorial.py\n" ] } ], "source": [ "%%file myfactorial.py\n", "def factorial2(n):\n", " \"\"\" Find n!. Raise an AssertionError if n is negative or non-integral.\n", " \"\"\"\n", "\n", " assert n >= 0 and type(n) is int, \"Unrecognized input\"\n", "\n", " if n == 0:\n", " return 1\n", " else:\n", " return n * factorial2(n - 1)\n", "\n", "def test():\n", " from math import factorial\n", " for x in range(10):\n", " assert factorial2(x) == factorial(x), \\\n", " \"My factorial function is incorrect for n = %i\" % x" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `doctests` -- executable examples" ] }, { "cell_type": "code", "execution_count": 71, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "[1, 1, 2, 6, 24]" ] }, "execution_count": 71, "metadata": {}, "output_type": "execute_result" } ], "source": [ "importlib.reload(myfactorial)\n", "from myfactorial import factorial2\n", "[factorial2(n) for n in range(5)]" ] }, { "cell_type": "code", "execution_count": 72, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Overwriting myfactorial.py\n" ] } ], "source": [ "%%file myfactorial.py\n", "def factorial2(n):\n", " \"\"\" Find n!. Raise an AssertionError if n is negative or non-integral.\n", "\n", " >>> from myfactorial import factorial2\n", " >>> [factorial2(n) for n in range(5)]\n", " [1, 1, 2, 6, 24]\n", " \"\"\"\n", "\n", " assert n >= 0. and type(n) is int, \"Unrecognized input\"\n", "\n", " if n == 0:\n", " return 1\n", " else:\n", " return n * factorial2(n - 1)\n", "\n", "def test():\n", " from math import factorial\n", " for x in range(10):\n", " assert factorial2(x) == factorial(x), \\\n", " \"My factorial function is incorrect for n = %i\" % x" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Running doctests" ] }, { "cell_type": "code", "execution_count": 73, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Trying:\r\n", " from myfactorial import factorial2\r\n", "Expecting nothing\r\n", "ok\r\n", "Trying:\r\n", " [factorial2(n) for n in range(5)]\r\n", "Expecting:\r\n", " [1, 1, 2, 6, 24]\r\n", "ok\r\n", "2 items had no tests:\r\n", " myfactorial\r\n", " myfactorial.test\r\n", "1 items passed all tests:\r\n", " 2 tests in myfactorial.factorial2\r\n", "2 tests in 3 items.\r\n", "2 passed and 0 failed.\r\n", "Test passed.\r\n" ] } ], "source": [ "!python -m doctest -v myfactorial.py" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Real world testing and continuous integration\n", "\n", "### `unittest` and `nose`\n", "\n", "#### Test fixtures (Unittest)\n", "\n", "- create self-contained tests\n", "- setup: open file, connect to a DB, create datastructures\n", "- teardown: tidy up afterward\n", "\n", "#### Test runner (nose, pytest)\n", "\n", "- `nosetests`, `py.test`\n", "- test discovery: any callable beginning with `test` in a module\n", " beginning with `test`\n", "\n", "#### Testing scientific computing libraries\n", "\n", "Such libraries have often testing routines, for example:" ] }, { "cell_type": "code", "execution_count": 74, "metadata": { "collapsed": false }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "..................................................................." ] }, { "name": "stdout", "output_type": "stream", "text": [ "Running unit tests for scipy.integrate\n", "NumPy version 1.10.4\n", "NumPy relaxed strides checking option: False\n", "NumPy is installed in //anaconda/lib/python3.5/site-packages/numpy\n", "SciPy version 0.17.0\n", "SciPy is installed in //anaconda/lib/python3.5/site-packages/scipy\n", "Python version 3.5.1 |Anaconda 4.0.0 (x86_64)| (default, Dec 7 2015, 11:24:55) [GCC 4.2.1 (Apple Inc. build 5577)]\n", "nose version 1.3.7\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "........................................................................................................................................K.........................................\n", "----------------------------------------------------------------------\n", "Ran 245 tests in 3.009s\n", "\n", "OK (KNOWNFAIL=1)\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 74, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import scipy.integrate\n", "scipy.integrate.test()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Assertions revisited - numerical mathematics" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Mathematically\n", "\n", "$ x = (\\sqrt(x))^2$.\n", "\n", "So what is happening here:" ] }, { "cell_type": "code", "execution_count": 75, "metadata": { "collapsed": false }, "outputs": [ { "ename": "AssertionError", "evalue": "", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mAssertionError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mmath\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0;32massert\u001b[0m \u001b[0;36m2\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mmath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msqrt\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;31mAssertionError\u001b[0m: " ] } ], "source": [ "import math\n", "assert 2 == math.sqrt(2)**2" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "math.sqrt(2)**2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### NumPy Testing" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "What if we consider x and y almost equal? Can we modify our assertion?" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import numpy as np\n", "np.testing.assert_almost_equal(2, math.sqrt(2) ** 2)" ] }, { "cell_type": "code", "execution_count": 76, "metadata": { "collapsed": false }, "outputs": [], "source": [ "x=1.000001\n", "y=1.000002\n", "np.testing.assert_almost_equal(x, y, decimal=5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Testing with py.test" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Going beyond ``doctest`` and Unittest, there are two frameworks widely spread for regression testing: \n", "\n", "* nose (http://nose.readthedocs.org/en/latest/)\n", "\n", "* pytest (http://pytest.org)\n", "\n", "Here, we focus on pytest.\n", "\n", "The example we use is the ``myfactorial.py`` file created earlier:" ] }, { "cell_type": "code", "execution_count": 77, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# %load myfactorial.py\n", "def factorial2(n):\n", " \"\"\" Find n!. Raise an AssertionError if n is negative or non-integral.\n", "\n", " >>> from myfactorial import factorial2\n", " >>> [factorial2(n) for n in range(5)]\n", " [1, 1, 2, 6, 24]\n", " \"\"\"\n", "\n", " assert n >= 0. and type(n) is int, \"Unrecognized input\"\n", "\n", " if n == 0:\n", " return 1\n", " else:\n", " return n * factorial2(n - 1)\n", "\n", "def test():\n", " from math import factorial\n", " for x in range(10):\n", " assert factorial2(x) == factorial(x), \\\n", " \"My factorial function is incorrect for n = %i\" % x" ] }, { "cell_type": "code", "execution_count": 78, "metadata": { "collapsed": false }, "outputs": [], "source": [ "def factorial2(n):\n", " \"\"\" Find n!. Raise an AssertionError if n is negative or non-integral.\n", "\n", " >>> from myfactorial import factorial2\n", " >>> [factorial2(n) for n in range(5)]\n", " [1, 1, 2, 6, 24]\n", " \"\"\"\n", "\n", " assert n >= 0. and type(n) is int, \"Unrecognized input\"\n", "\n", " if n == 0:\n", " return 1\n", " else:\n", " return n * factorial2(n - 1)\n", "\n", "def test():\n", " from math import factorial\n", " for x in range(10):\n", " assert factorial2(x) == factorial(x), \\\n", " \"My factorial function is incorrect for n = %i\" % x" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Providing test functions\n", "\n", "(Addition to original notebook, Hans Fangohr, 21 Sep 2013)\n", "\n", "py.test is an executable that will search through a given file and find all functions that start with ``test``, and execute those. Any failed assertions are reported as errors.\n", "\n", "For example, ``py.test`` can run the ``test()`` function that has been defined already in ``myfactorial``:" ] }, { "cell_type": "code", "execution_count": 79, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[1m============================= test session starts ==============================\u001b[0m\r\n", "platform darwin -- Python 3.5.1, pytest-2.8.5, py-1.4.31, pluggy-0.3.1\r\n", "rootdir: /Users/fangohr/hg/teaching-python/notebook, inifile: \r\n", "plugins: nbval-0.3.1, cov-2.2.1\r\n", "\u001b[1m\r", "collecting 0 items\u001b[0m\u001b[1m\r", "collecting 1 items\u001b[0m\u001b[1m\r", "collected 1 items \r\n", "\u001b[0m\r\n", "myfactorial.py .\r\n", "\r\n", "\u001b[1m\u001b[32m=========================== 1 passed in 0.01 seconds ===========================\u001b[0m\r\n" ] } ], "source": [ "!py.test myfactorial.py" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This output (the '.' after ``myfactorial.py``) indicates success. We can get a more detailed output using the ``-v`` switch for extra verbosity:" ] }, { "cell_type": "code", "execution_count": 80, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[1m============================= test session starts ==============================\u001b[0m\r\n", "platform darwin -- Python 3.5.1, pytest-2.8.5, py-1.4.31, pluggy-0.3.1 -- //anaconda/bin/python\r\n", "cachedir: .cache\r\n", "rootdir: /Users/fangohr/hg/teaching-python/notebook, inifile: \r\n", "plugins: nbval-0.3.1, cov-2.2.1\r\n", "\u001b[1m\r", "collecting 0 items\u001b[0m\u001b[1m\r", "collecting 1 items\u001b[0m\u001b[1m\r", "collected 1 items \r\n", "\u001b[0m\r\n", "myfactorial.py::test \u001b[32mPASSED\u001b[0m\r\n", "\r\n", "\u001b[1m\u001b[32m=========================== 1 passed in 0.00 seconds ===========================\u001b[0m\r\n" ] } ], "source": [ "!py.test -v myfactorial.py" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Sometimes, we like having the tests for ``myfactorial.py`` gathered in a separate file, for example in ``test_myfactorial.py``. We create such a file, and within the file we create a number of test functions, each with a name starting with ``test``:" ] }, { "cell_type": "code", "execution_count": 81, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Overwriting test_myfactorial.py\n" ] } ], "source": [ "%%file test_myfactorial.py\n", "\n", "from myfactorial import factorial2 \n", "\n", "def test_basics():\n", " assert factorial2(0) == 1\n", " assert factorial2(1) == 1\n", " assert factorial2(3) == 6\n", " \n", "def test_against_standard_lib():\n", " import math\n", " for i in range(20):\n", " assert math.factorial(i) == factorial2(i)\n", " \n", "def test_negative_number_raises_error():\n", " import pytest\n", "\n", " with pytest.raises(AssertionError): # this will pass if \n", " factorial2(-1) # factorial2(-1) raises \n", " # an AssertionError\n", " \n", " with pytest.raises(AssertionError):\n", " factorial2(-10)\n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can now run the tests in this file using" ] }, { "cell_type": "code", "execution_count": 82, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[1m============================= test session starts ==============================\u001b[0m\r\n", "platform darwin -- Python 3.5.1, pytest-2.8.5, py-1.4.31, pluggy-0.3.1 -- //anaconda/bin/python\r\n", "cachedir: .cache\r\n", "rootdir: /Users/fangohr/hg/teaching-python/notebook, inifile: \r\n", "plugins: nbval-0.3.1, cov-2.2.1\r\n", "\u001b[1m\r", "collecting 0 items\u001b[0m\u001b[1m\r", "collecting 3 items\u001b[0m\u001b[1m\r", "collected 3 items \r\n", "\u001b[0m\r\n", "test_myfactorial.py::test_basics \u001b[32mPASSED\u001b[0m\r\n", "test_myfactorial.py::test_against_standard_lib \u001b[32mPASSED\u001b[0m\r\n", "test_myfactorial.py::test_negative_number_raises_error \u001b[32mPASSED\u001b[0m\r\n", "\r\n", "\u001b[1m\u001b[32m=========================== 3 passed in 0.01 seconds ===========================\u001b[0m\r\n" ] } ], "source": [ "!py.test -v test_myfactorial.py " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The ``py.test`` command can also be given a directory, and it will search all files and files in subdirectories for files starting with ``test``, and will attempt to run all the tests in those. \n", "\n", "Or we can provide a list of test files to work through:\n" ] }, { "cell_type": "code", "execution_count": 83, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[1m============================= test session starts ==============================\u001b[0m\r\n", "platform darwin -- Python 3.5.1, pytest-2.8.5, py-1.4.31, pluggy-0.3.1 -- //anaconda/bin/python\r\n", "cachedir: .cache\r\n", "rootdir: /Users/fangohr/hg/teaching-python/notebook, inifile: \r\n", "plugins: nbval-0.3.1, cov-2.2.1\r\n", "\u001b[1m\r", "collecting 0 items\u001b[0m\u001b[1m\r", "collecting 3 items\u001b[0m\u001b[1m\r", "collecting 4 items\u001b[0m\u001b[1m\r", "collected 4 items \r\n", "\u001b[0m\r\n", "test_myfactorial.py::test_basics \u001b[32mPASSED\u001b[0m\r\n", "test_myfactorial.py::test_against_standard_lib \u001b[32mPASSED\u001b[0m\r\n", "test_myfactorial.py::test_negative_number_raises_error \u001b[32mPASSED\u001b[0m\r\n", "myfactorial.py::test \u001b[32mPASSED\u001b[0m\r\n", "\r\n", "\u001b[32m\u001b[1m=========================== 4 passed in 0.01 seconds ===========================\u001b[0m\r\n" ] } ], "source": [ "!py.test -v test_myfactorial.py myfactorial.py" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Final thoughts\n", "\n", "- [Continuous integration (SymPy)](https://github.com/sympy/sympy/pull/2399)\n", "- [Logging](http://docs.python.org/2/library/logging.html)\n", "- [Python debugger](http://docs.python.org/2/library/pdb.html)\n", "\n", "#### Learn more\n", "\n", "* [http://software-carpentry.org](http://software-carpentry.org)\n", "* [http://docs.python.org/library/exceptions.html](http://docs.python.org/library/exceptions.html)\n", "* [http://docs.python.org/library/doctest.html](http://docs.python.org/library/doctest.html)\n", "* [http://docs.python.org/library/unittest.html](http://docs.python.org/library/unittest.html)\n", "* [http://docs.scipy.org/doc/numpy/reference/routines.testing.html](http://docs.scipy.org/doc/numpy/reference/routines.testing.html)\n", "* [http://nedbatchelder.com/code/coverage](http://nedbatchelder.com/code/coverage)\n", "* [http://somethingaboutorange.com/mrl/projects/nose](http://somethingaboutorange.com/mrl/projects/nose)" ] }, { "cell_type": "code", "execution_count": 84, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[0;33mchangeset: 616:a8ee6aafe5a7\u001b[0m\r\n", "tag: tip\r\n", "user: Hans Fangohr [bin] \r\n", "date: Fri Sep 16 19:57:15 2016 +0100\r\n", "summary: update to Python3, minor improvements\r\n", "\r\n" ] } ], "source": [ "!hg tip" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.4.4" } }, "nbformat": 4, "nbformat_minor": 0 }