PHP Testing

I have now just committed YAML data sets to the Database Extension for PHPUnit. So now all those that love the simplicity and straightforwardness of YAML can use it with your data sets. I have also created a persistor for YAML datasets so you can easily convert existing data sets or database data into YAML representations.

This is really the first I have dealt with YAML and it is definitely a viable alternative for data sets. Here is what they will look like.

The biggest issue I had was figuring out how YAML dealt with trailing line breaks. It appears that '|-' did the trick. It will strip the final line break. Another thing to take note is that indention is important. It doesn't HAVE to be spaces (it can be tabs) it just needs to be consistent. The last thing is that strings do not need to be quoted, but if they are you can use escape characters (like \n.)

This was committed to the trunk of PHPUnit and I anticipate Sebastian will be merging it back into 3.4 shortly.

I have finally gotten some time to do some long awaited work on PHPUnit. The first on my agenda was a small improvement to filtering. This was related to ticket #526. Essentially I added the ability to do both white list and black list filtering.

For instance, if you wanted to specify an include filter you can do the following:

I have also deprecated passing the filter to the constructor. The reason for this is that I would like to begin adding a few more features to the filter and the current method of passing the parameters on the constructor would be unintuitive for these planned features. Basically I think it was a mistake for me to allow the passing of filter data on the constructor to begin with. I anticipate the current method working up to PHPUnit 4.0 but it will not be expanded on and will be removed either in PHPUnit 4 or in a release soon after that.

So, for exclude filters, instead of specifying them in the constructor, the new method will be:

While you can't mix include and exclude filters for tables (wouldn't make sense right now) you CAN do this for table columns of separate tables:

Some future features I plan on adding include, global column includes/excludes and wild cards. Anyhow, hope this is useful for someone.

And here are the advanced PHPUnit slides!

As promised here are my automated unit testing slides for the 2008 DC PHP conference.

When I was first creating the Database Extension for PHPUnit I realized that there was a very high likelihood that several people would have tests that were already written that they would like to add additional database tests too. To accomplish this I actually wrote the PHPUnit_Extensions_Database_DefaultTester class. In fact, if you were to look at the source of the database test case you will see that all of it's operations are actually forwarded to this class which does all of the work.

Please continue reading to see how you can use composition to add database tests to your existing test cases.

I have completed the initial feature set for the Database extension to PHPUnit. This is a essentially a port of DBUnit to PHP.

For those that may not have read any of my previous postings on the subject the goal of this software is to extend the functionality of PHPUnit to allow using seed data to put a given database into a known state prior to executing each test. It also provides an easy mechanism to compare database contents with an expected dataset.

The database extension has recently been merged into the PHPUnit 3.2 branch and is scheduled to be released in that version. Sebastian Bergmann will be introducing the extension in his Advanced Testing with PHPUnit talk at PHP|Works 2007 in Atlanta September 13 - 14. If you would like to tinker around with the database extension prior to it's release you can always download the latest copy of PHPUnit 3.2 from svn: svn://svn.phpunit.de/phpunit/phpunit/branches/3.2. The source can also be browsed at http://www.phpunit.de/browser/phpunit/branches/3.2/PHPUnit/Extensions/Database.

Please continue reading for an example of how you can now use PHPUnit to even more effectively test data-centric applications.

I have been working quite a bit lately with the cvs version PHPUnit3. I have spent alot of time in testing using SimpleTest and I have also given some time to test-more.php. However, up until just recently I had honestly never used PHPUnit before. The big motivational factor for me (other than to try something new) was to see how the code coverage feature was coming along.

I would have to say that after a few weeks of tinkering with it I am impressed. Now, keep in mind that I am talking about software that is still in development, so a degree of lattitude is required. The featureset of PHPUnit 3 is great. Here are a few notable features (in my mind):

• Mock objects
• Code Coverage Analysis
• Multitude of test logging options
• Built well for testing >php5 applications

Using the framework is an absolute breeze and there is excellent documentation in the form of a free book (draft version) written by Sebastian Bergmann and published by O'Reilly called "PHPUnit Pocket Guide." The API is also documented quite nicely and serves as a good second-source for learning how to do specific things.

The most impressive part of this experience in my mind was how runnable the software is straight out of CVS. Most of the time when I am crazy enough to use CVS version I always seem to get my copy of the source at a point where it is very unstable and requires tweaking just to get the software to fulfill its basic requirements. PHPUnit 3 on the other hand was able to run in it's standard mode without a flaw. The only thing that was missing that I felt I needed was html reports for the pass/fail status of the unit tests. The report templates and code was already to a point however that it was actually fairly trivial for me to make the changes to it that I thought would be needed to make it work. I have since passed those changes along to Sebastian and they have been added to CVS, so html reports are now in the cvs version as well. It felt good to be able to contribute a little bit to a very worthwhile project.

Sebastian has posted about the new reporting features on his blog that include some screenshots so I won't bother to rehash over all of that. You can just go check them out on his site.

I have been doing a fair bit of thinking concerning testing over the last couple of days. This was partially driven by a couple of emails Paul and I have sent back and forth as well as a posting in his blog. I am of course a big proponent of TAP. I think it provides alot of possibilities for integrating the current libraries that offer TAP support (test-more.php, SimpleTest + TAP-Reporter, PHPUnit2.) However, having libraries that support TAP compliant output are honestly of little extra value if there isn't a way to read it and subsequently aggregate the data.

TAP Compliancy allows you to use any framework/harness that supports TAP. However, to date, the only harness I could find for reading TAP was perl's Test::Harness and the only framework was Apache-Test. Now, I will be the first to admit that there is a fairly steep learning curve when it comes to using Apache-Test. Once you get the hang of it, things get pretty simple. Getting the hang of it though tends to be a problem. Because of this I thought it was time for someone to create a harness in php that is capable of being used to build a framework.

So, I spent my free time the last couple of days working on such an item and this is what I came up with: test-harness.php

This is a very rough first draft, if you will. I have successfully used it to run a 1,000 file testing suite that I use for my firm's internal API. It should be stated immediatly that my goal is NOT to mimic Test::Harness. My goal is to simply provide another option for people to use in their testing.

It is fairly simple to use. All you need to do is place it in your tests' folder, make a few configuration changes (if necessary) and you will be ready to go. The script does depend on your test-files being self sustaining (to a point) and they must also output TAP compliant results. So basically if you can run your individual test files from the command line you should be alright.

There are three different configuration settings that need to be looked at. They can all be found at the top of the test-harness.php file. The first one is the TAP_PHP_CLI constant. This simple needs to be the path to your php-cli. The second one is TAP_PHP_CLI_ARGS. This can be used to pass command line arguments straight to php. I anticipate (as the default value shows) that this will be used most often to set the include_path for the testing scripts. The final setting is the $_EXCLUDE_FILES array. This is used to specify which files the harness should ignore when cycling through the directories. test-harness.php should of course always be set. I have also set test-more.php as I was using this library for most of my testing.

Here is a brief list of the current features:

• Executes each test file in its own environment. This is accomplished by executing each test seperatly in it's own php-cli process.
• Supports the full TAP protocol. Currently there is no PHP library (afaik) that fully supports TAP however I do know there are plans to enable this in at least test-more.php. So when that happens the harness will be ready.
• Supports Multiple levels of verbosity. -s will not output anything (this can be used in automated build processes when you only need to know if the tests as a whole pass or fail.) --detail will output the overall test results as well as the TAP output of each test. By default only the overall pass/fail result of each file and a summary of failed tests, skipped tests, and bonus tests are displayed.
• Uses a return value to indicate overall success or fail. The program will return 1 on success and 0 on fail.

This is the first version of this file and I plan on releasing many more in the very near future. Some features I have in mind for the future are:

• Better test filtering. Executing every file in the directory is just not always the best way to do things. I would like to set it so that only *.php (and maybe *.phpt) files are run as well as maybe introducing a default directory to store php files that aren't to be tested. Basically I want to get rid of the $_EXCLUDE_FILES array.
• Support for running the harness over http and recieving pretty output.
• Make the script more interactive (optionally for the cli)
• There is also a host of additional features that I scribbled down on a notepad. If anybody else has any good ideas you are more than welcome to post them.

A couple days ago I came across a post in Chris Shiflett's blog concerning a tutorial that Geoff Young and he were giving at JApacheCon. Part of this post was giving some information on a small extension I wrote for SimpleTest called TAP Reporter.

I never realized anybody was actively using the extension and as a result I failed to develop it or even update the version I have on my site. Chris, Geoff, Marcus Baker, and I have since opened discussion on possible mods to Simple Test as well as a little discussion on how to make PHP testing an overall nicer experience. So it looks like there will be some changes to the TAP Reporter in the very near future.

That all being said, I do have an updated version of the tarball that fixes some bugs in tap-reporter.php and also enables a few more TAP featuers. The new features are support for SKIP and TODO directives as well as support for 'bail out!.' The code is very well documented (yay me) so I don't think there's much of a need for me to describe how to use it.

If you want some general information on TAP, Apache-Test and how they relate to SimpleTest then take a look at my older posts on the subject. A link to the new tar ball can be found below.

simpletest_apache-test.tar.gz

I have successfully gotten Simple Test to test itself in Apache-Test. It wasn't that hard really. I have added very basic Apache-Test support by just subclassing the SimpleReporter class in SimpleTest. The new class name is TAPReporter. It basically just allows a test to create TAP compliant (see this post) output which is then read by Apache-Test. There are a few things which in my opinion could be changed in SimpleTest in order to make TAP complancy much better and it would actually improve some of the functionality of SimpleTest apart from Apache-Test. I am somewhat hesitant to make those changes right now however as the code is not mine and really should be discussed with the creator of SimpleTest. (Which I do plan on doing.) I will outline later in this articles the changes I think would be useful. For now I would just like to review what I did do and how to run the tests.

As some of you know I am currently working on a project to allow Simple Test to be Apache-Test compliant. The process to take in doing so is to enable Simple Test to produce TAP compliant output. To do so obviously requires a healthy knowledge of the different aspects of the TAP protocol. The best way for me to get a grasp of something is of course to write about it. I gleaned pretty much all of this information from CPAN's TAP format documentation.

I told Marcus Baker a couple of weeks ago that I would like to lend a hand in figuring out the best way to get SimpleTest working with Apache-Test. I actually was able to put quite a bit of time into this today and have achieved some results finally. Now, these results are far from a solution. All I have managed to do thus far really is to understand enough about the inner workings of Apache-Test so I can actually get a "self-created" test working inside of the Apache-Test environment.