Software Testing with Python

magnus lyckå www.thinkware.se

started as an electronics tester

{ SQA { Software Verification { Software Testing } } }

"V model" with more levels in practice:

system specification        ->      acceptance test
        |                             ^
        v                             |
 API Specification           ->    integration test
        |                             ^
        v                             |
 detailed design           ->     unit test

XP version:

system specified through | customer tests
           v                   ^
api validated through | continuous integration
           v                   ^
design by writing | programmer tests

Consequences of the XP approach: automated test -> more software to maintain?

unit testing: unit tests are about programmer intent, not specs

test fixture/case/suite/runner

trick in setUp: avoid "global" by doing RESULTS[:] = []

sys.exit(0) problem?

doctest - just mentioned, what about scaling?

Integration testing

http://staf.sourceforge/net/ - IBM framework with python API and other language support

TextTest - tables in web pages?

wxPython testing stub?

ask later - doesn't doctest put pressure against lots of tests?

Unit testing with mock code

Stefan Schwarzer

"mock code" is substitute code for production code, which is used by the code under test.

"mock object" is mock code in form of an object

"difficult tests" have hairy test setups like

• filesystem full
• net connection dropped
• mail server can't be reached

typical usage:

• test code prepares the mock object
• calls the code under test, passing in the mock object (ie. "connection object")
• checks the mock object for evidence, signs of usage

example: Logger object, which just appends args to history (and then, just dump the trace for comparison)

StringIO are natural mock objects

Keep mock code simple - "universal" mock objects get difficult to too complicated to maintain (they need their own unittests!)

Try not to depend on the implementation, only the interface (ie. the mock object might pick up operations from lower layers)

Use this when testing is needed, but convention tests are difficult, and a mock object works well; if it does require modifications to production code to support it, make sure they aren't bad for maintainability.

Magnus Lyckå comment: this shows why you should write tests first - so the code is inherently testable...

Towards a Better Test Runner

Marius Gedminas marius@pov.lt - Programmers of Vilnius

Yay - green-on-black slides!

PyUnit

SchoolTool test runner, inspired by the one in Zope 3; test runners get complicated as the tree gets complicated

Goals:

• find all modules automatically
• allow you to run a subset
• show progress indication

Also, automate various kinds of analysis to make life easier.

Conventions: tests.py or tests/test*.py in every package.

test_suite function, returns a TestSuite.

Helper - warn about TestFoo classes that aren't listed

Filtering - you want to be able to run them quickly to check if a failure was fixed.

• regex on filenames (avoids some slow imports)
• regex on class names (more granular but doesn't avoid the imports)
• test levels?

Analysis:

• can do code coverage analysis (using builtin python trace)
• detect common mistakes (leftover prints, global changes)
• find slowest tests (zope3, not SchoolTool)
• find memory leaks (zope3, not SchoolTool)

http://source.schooltool.org/svn/trunk/schooltool/test.py

[eichin:20040607T1100+02]

std: a complementary second standard library

Holger Krekel

Much of the standard lib, in particular unittest, are java-like, not pythonic

Standard lib exposes too much in terms of implementation details, leading to things like urllib vs. urllib2

"The best API is one that doesn't exist" (riffing on Raskin's "best UI" comments)

Innumerable incompatible extensions around unittest.py to do the bookkeeping of testing, many of which are larger than the module itself.

No standard way to run the tests for an arbitrary app.

std.utest

• use existing assert statement
• but magically reinterpret it to get more data than just "Failed"
• shows the source of the test, exactly the line of failure, and a reinterpretation of the statement! (std.magic)
• raises(WhatToCatch, "expr")

Does the last (.pop()) case in demo.py actually show the wrong thing, namely the context after the test?

Looks for test_*.py, test_* functions, in the module.

"restartable iterators", called "collectors" - for walking file/directory trees... collect functions and classes within them... (Do I want to use this for tree walking for stufflog?)

Collector/Runner/Reporter model; tests just start.

utest.conf simply in the namespace for options

std has a uniform access method for subversion repositories (svn-url and svn-working, not just filesystems.

Also has more explicit namespace control (std.initpkg)

(Armin Rigo is the other developer)

1.0 release real soon

svn co http://codespeak.net/svn/std/trunk/src/std

TextTest - a test framework

Johan Andersson, Geoff Bache - Carmen Systems AB - http://www.carmensystems.com

Acceptance Testing

Carmen does "development planning systems" for airline and railway industries, 190 employees; TextTest regression tests a number of their products.

Tests drive development, absolutely.

Focussed on "customer approval" tests (vs. developer intent)

Started out as a highly product-specific shell script using grep; became generic python, and eventually went on sourceforge.

Has both interactive and batch reporting tools

output, errors, and performance files - explicitly measures expected CPU time consumed!

PyGTK GUIs -- GUI selftest being presented at XP2004

"diagnostic debugging" practice (from XP?) is handled by additional tools, that go beyond the "it failed" defaults.

The actual end users don't deal with the text themselves, they have viewer apps and such.

http://www.carmensystems.com/research_development/programming.htm

Testing Panel Discussion

Magnus Lyckå Stefan Schwarzer Marius Gedminas Johan Andersson Holger Krekel plus Stefan Holek - zope2 testing tool author

Johan Andersson: "Not writing unit test code saves lots of time" - because they could automate the acceptance tests externally. It loses some of the benefits of unit tests, but they get back some of the benefit by additional practices: 'usage first', you write the caller first (sort of like test-first), and 'diagnostic debugging' - write more detailed trace statements for suspicious areas. (40 hour tests are performance tests from real customer demands.)

Holger Krekel: we have lots of different demands for testing, PyPy sees it very differently. Maybe there's a way to come up with a common ground so we can share tools?

Mock objects: sqlite "memory", zodb (zope2) - DemoStorage, cancel transactions; avoids having to duplicate all of Zope2 as mock objects, which need their own unit tests...

Audience question: how do you test the Infinite Filing cabinet...

Why is there so much more testing interest? More systems are around, have aged more, become complex, and need self test...

Quixote - web applications

[eichin:20040607T1340+02]

Harald Armin Masson (did the "why not to tell people about python" humour talk last year)

• training and process development - EU600/day + expenses

"program based"

• 3 of them "CherryPy" style (ie. embed python in HTML text)

Quixote - done for Mems Exchange

• logic, not templates
• no magic
• only 7000 lines of code, implemented in a week or two
• PTL, but mostly URLs == module paths
• can do XML-RPC too
• Triggered IE bugs by being to fast (but a service pack fixed them)
• subsumed medusa/asyncore; works with twisted
• preferred use: mod_scgi + apache
• mod_python works, but doesn't help (is a pain to configure)
• CGIHTTPserver.py is slow, but it reloads everything always, which is a win for development

audience: suggest cheetah, or at worst ZPT for designers

response: they wanted to use Word (not even DreamWeaver) so he fed Word's HTML output to mxTidy and extracted it from there.

Real World Email

[eichin:20040607T1340+02]

Anders Hamerquist ah@strakt.com

"CAPS" system that Alex Martelli talks about. Handles 100k messages (over what timescale/)

Problems all boil down to MIME - charset, but other things too.

Anders coped well with not having a working projector, writing mime header encoding examples on the chalkboard...

RFC2231 - encoding in structured fields, like mime-content-filename (outlook botches this, but the hack of setting the obsolete Content-type name= field, in addition to content-disposition, seems to work.)

PyKI - Public Key Infrstructure Development Framework

[eichin:20040607T1340+02]

Richard Zoni r.zoni@kelyansmc.it Piero Ribichini p.ribichini@kelyansmc.it

Context: X.509 certs, sigs, PKI, ASN.1, PKCS, LDAP

Goals: framework for PKI and PKI-enabled apps

Python: for rapid prototyping, but ended up in the final product, since performance from libs was sufficient (and the slow part is the smartcard)

Bignum Ext on GMP, crypto EXT, CryptoKI pkcs#11 drivers, PyQT

ASN.1 engine - Maps ASN.1 objs to Python Classes

"Crypto Pipes" - Buffer/Feeder/Eater, [rsa.rsa, pkcs1_5.pad]

timestamping, TLS, then their apps on top

ASN1 modules - manually generated from spec, since they need to be enriched with live methods too:

class Cert(ASN1.Sequence):
   def model(self):
      return [ foo("fooName"), ... ]

Signo Professional - 20,000 users, mostly because it was adopted by Italian CA, bundled with a USB token.

Servers, Dev; also SIGNO SecureLogOn - smartcard logon for Linux.

Open Source release of the core tech in Q3-2004

SIGNO-ETSI support ("better" signatures?)

SchoolTool

Steve Alexander steva@z3u.com

Funded by The Shuttleworth Foundation (in South Africa; founded to do risky things, that benefit SA and countries like it, not first world.)

Focussed very narrowly on school administration (Mark Shuttleworth, fownder of Thawte; keynote is tomorrow) not on delivering courseware etc.

Surveys indicated that a GUI was important, but then schools have gone off and decided that web based apps were the big thing...

Real concrete project, test driven development - not theoretical or abstract, involves school people.

A couple of schools have tried it and are going further, but no major adoptions yet. (There is funding for this.) As an open source platform, it is intended to force interopration, in the first world.

Third world countries don't need software per se; they need management tools to let them know how better to use the resources they have.

Development has bounties for funding (google "here be bounties" :-)

Models classes, departments, groups; students, teachers; attendance; bookable resources (rooms, equipment); allows for changing roles.

Milestone 5 released; fairly raw demo.

(Think about using this for IAP scheduling?)

Needs usability testing, or rather, tight development with actual educators in the loop.

There are debian packages and an apt-source.

A Case Study from Bristol UK

[eichin:20040607T1601+02]

Chris Withers chris@simplistix.cok.uk (Noted UK Python/zope consultant)

X2Y Document Conversion Server (not ready yet)

University of Bristol print services department (thesis printing, etc.)

Lots of request forms - paper, incorrectly filled in forms, no audit trail, phone in status checked. Too much word-of-mouth; more effort on tracking than working.

Goal: paperless job management. ePrint Direct (client/server commercial system) looked good at first. It's actually a windows printer driver, for submission. Problems:

• windows only
• license per client!
• still end up with job-sheets inside the facility

Proposal: user - web/email, print services - web (zope), x2y - talks http to zope. uses reportlab tools. Standardizing on PDF helps this...

Live run against the real server! http://www.isc.bris.ac.uk/printservices/

(Common problem - file attachment gets reset on reload with errors.)

X2Y itself - currently a script that just polls a URL. Moving to Twisted, as an actual server. XML-RPC, not because it is any good, but it is popular for integration. Email-in as well.

Swapped in "EasyPDFCreator" COM object, instead of openoffice.

http://www.simplistix.co.uk (the workflow framework may be adaptable)

dLCMS - dynamic Learning Content Management System

Samuel Schluep schluep@iha.bepr.ethz.ch

(speaker sounds like a young Peter Lorre :-) Institute for Hygiene and Applied Physiology.

Very little reuse in classic "e-learning" content.

"Learning Objects" - small, modular, reusable, durable (10 years or longer lifetime)

Standardization - metadata, packaging

This project uses XML directly.

No existing LCMS was accessible (or affordable, even for ETHZ) so based one on Silva/Zope.

Example: "Gesundheitsgefahren bei Lärm"

Working towards standardized packaging techniques (XML in ZIP seems to work?)