TAP Compliant PHP Testing Harness

I have been doing a fair bit of thinking concerning testing over the last couple of days. This was partially driven by a couple of emails Paul and I have sent back and forth as well as a posting in his blog. I am of course a big proponent of TAP. I think it provides alot of possibilities for integrating the current libraries that offer TAP support (test-more.php, SimpleTest + TAP-Reporter, PHPUnit2.) However, having libraries that support TAP compliant output are honestly of little extra value if there isn't a way to read it and subsequently aggregate the data.

TAP Compliancy allows you to use any framework/harness that supports TAP. However, to date, the only harness I could find for reading TAP was perl's Test::Harness and the only framework was Apache-Test. Now, I will be the first to admit that there is a fairly steep learning curve when it comes to using Apache-Test. Once you get the hang of it, things get pretty simple. Getting the hang of it though tends to be a problem. Because of this I thought it was time for someone to create a harness in php that is capable of being used to build a framework.

So, I spent my free time the last couple of days working on such an item and this is what I came up with: test-harness.php

This is a very rough first draft, if you will. I have successfully used it to run a 1,000 file testing suite that I use for my firm's internal API. It should be stated immediatly that my goal is NOT to mimic Test::Harness. My goal is to simply provide another option for people to use in their testing.

It is fairly simple to use. All you need to do is place it in your tests' folder, make a few configuration changes (if necessary) and you will be ready to go. The script does depend on your test-files being self sustaining (to a point) and they must also output TAP compliant results. So basically if you can run your individual test files from the command line you should be alright.

There are three different configuration settings that need to be looked at. They can all be found at the top of the test-harness.php file. The first one is the TAP_PHP_CLI constant. This simple needs to be the path to your php-cli. The second one is TAP_PHP_CLI_ARGS. This can be used to pass command line arguments straight to php. I anticipate (as the default value shows) that this will be used most often to set the include_path for the testing scripts. The final setting is the $_EXCLUDE_FILES array. This is used to specify which files the harness should ignore when cycling through the directories. test-harness.php should of course always be set. I have also set test-more.php as I was using this library for most of my testing.

Here is a brief list of the current features:

• Executes each test file in its own environment. This is accomplished by executing each test seperatly in it's own php-cli process.
• Supports the full TAP protocol. Currently there is no PHP library (afaik) that fully supports TAP however I do know there are plans to enable this in at least test-more.php. So when that happens the harness will be ready.
• Supports Multiple levels of verbosity. -s will not output anything (this can be used in automated build processes when you only need to know if the tests as a whole pass or fail.) --detail will output the overall test results as well as the TAP output of each test. By default only the overall pass/fail result of each file and a summary of failed tests, skipped tests, and bonus tests are displayed.
• Uses a return value to indicate overall success or fail. The program will return 1 on success and 0 on fail.

This is the first version of this file and I plan on releasing many more in the very near future. Some features I have in mind for the future are:

• Better test filtering. Executing every file in the directory is just not always the best way to do things. I would like to set it so that only *.php (and maybe *.phpt) files are run as well as maybe introducing a default directory to store php files that aren't to be tested. Basically I want to get rid of the $_EXCLUDE_FILES array.
• Support for running the harness over http and recieving pretty output.
• Make the script more interactive (optionally for the cli)
• There is also a host of additional features that I scribbled down on a notepad. If anybody else has any good ideas you are more than welcome to post them.