Recently, I have been splitting my development time between the PHP extension Phar, the next-generation PEAR installer Pyrus, and related PEAR2 packages. I’ve spent a considerable amount of time developing unit tests. Over time, I’ve noticed that testing is becoming increasingly onerous. Sometimes, just running them can take up to 75% of the development time. Because I don’t like to commit changes before I know they work, my development has begun to stall, and this has resulted in longer periods without testing. Worse, others have stopped running the tests prior to committing. Gradually, the code has gotten out of sync with the tests, and this is a bad thing.
When I began developing PHP extensions, I had to learn about PHPT, the standard way to test them. I had come to a personal impasse with PHPUnit. Among other things, fatal errors would bring down the entire test suite, and this made it difficult to find the source of the error. This was back in the early days when PHPUnit was still a PEAR package. It is also worth noting that continuous integration and solutions like CruiseControl — and the PHP implementation phpUnderControl — were not yet written. If you have full access to a CI server, then by all means, use it! The xUnit-PHPT combination can still be used on the client side, prior to commit, for extra reliability.
It was at this time that I designed the xUnit-PHPT combination that I have been using since. In this article, I’ll introduce this approach and show you how it could help.
What is xUnit?
xUnit is an abstraction that describes testing frameworks that adhere to similar testing patterns. The name comes from the first one, JUnit, and the x refers to the programming language. Many xUnit frameworks follow this naming convention: JUnit, PHPUnit, SUnit, FlexUnit, ASUnit, &c.
xUnit can be considered a design pattern for testing frameworks. These frameworks split up tests into test cases, fixtures, and test suites.
A test case is a single test. It contains assertions that test the correctness of whatever is being tested. The xUnit pattern does not specify what kind of test should be written, thus unit tests, acceptance tests, regression tests, and other forms of testing can all be represented using a test case.
A test fixture provides common setup and teardown procedures for a group of related tests.
A test suite is a group of related tests. Often, test suites combine several test fixtures.
What is PHPT?
PHPT is a dirt-simple PHP testing framework that uses simple markup to delineate test sections. Acquiring PHPT is simple; you can either use the run-tests.php tool included with PHP itself, use the PEAR installer’s run-tests command, or Pyrus’s run-phpt command. In addition, PHPUnit 3.3 supports running PHPT tests as a part of test fixtures, which also allows taking advantage of PHPUnit’s other advanced features. It is important to note that there are subtle but important differences in the support for PHPT. PHPUnit uses the same code as the PEAR installer, which is identical to that supported by Pyrus. The run-tests.php tool included with PHP supports some more arcane features needed by PHP developers, and the PEAR implementation supports a few sections such as --RETURNS-- for testing shell script return values that are not supported by run-tests.php.
A PHPT test is a file that ends in .phpt. Each test contains sections indicated by two dashes, an uppercase section name, and two more dashes. (For example, --TEST-- and --FILE--.) The online documentation provides more details about the format.
PHPT tests can lever more control over the environment that a test runs in, including setting environment variables with --ENV--, setting php.ini configuration directives with --INI--, setting $_GET, $_POST, stdin, and raw POST data. Even file uploads can be simulated with the --UPLOAD-- section if you’re using the PEAR installer or Pyrus implementation. It is incredibly powerful.
The test’s success or failure is verified by comparing expected output against actual output. Expected output is stored in the --EXPECT-- section of a test. There are also variants of --EXPECT-- that allow matching against a regular expression, or against simple replacement variable such as what sscanf() would expect.
Here is the simplest possible PHPT test:
--TEST--
simple test
--FILE--
<?php
echo 'this will be';
echo ' matched';
?>
--EXPECT--
this will be matched
Note the three sections. The --TEST-- section describes what the test is testing, the --FILE-- section contains the PHP script to run, and the --EXPECT-- section is used to match the output of the test.
When a test fails, several files are created. If the test is named test.phpt, then test.out, test.exp, test.diff, test.log, and test.php will be created in the same directory. The test.out file contains the actual output of the test, the test.exp file contains the expected output, the test.diff file contains the diff between the two, the test.log file contains the diff plus other failure information, and the test.php file contains the contents of the --FILE-- section to facilitate debugging.
Unlike PHPUnit, PHPT tests execute in their own separate PHP process, which makes it possible to test different configurations. PHPUnit 3.4 implements this ability to run test cases in separate PHP processes as an option, so this is something to look for if you need that feature and already use PHPUnit.
Combining xUnit and PHPT
This is the fun part. Before diving into how to implement test cases, test fixtures, and test suites with PHPT, it’s a good idea to see how PHPUnit does it. PHPUnit test cases are implemented as a method in a class extending PHPUnit_TestCase. Test cases call something like $this->assertEquals(), $this->assertTrue(), &c. Test fixtures are implemented as a class containing test cases and the special setup() and teardown() methods. These methods are called as bookends around each test case. Test suites are more abstract, and are implemented by writing a PHP script that defines which test fixtures to load and run.
To implement xUnit in PHPT, we need to figure out how to implement test fixtures, assertions, and test suites. Fixtures turn out to be relatively easy to model. Test cases are implemented in PHPT as files, and a fixture can be modeled quite easily using a directory. The setup and teardown can also be implemented with files that are then directly included in each test.
Assertions require us to create our own assertion library. This library needs to display failures. Although PHPT provides a simple way to detect errors using a diff, this is not really that useful for testing our PHP code. It’s better to stick to a simple principle:
“Don’t display anything unless there is an error.”
In addition, assertions should always be easy to track down in the test file, and should always have a simple description of what the assertion is testing.
The library I use adheres to these principles, and it’s flexible enough to be easily extended to add custom assertions. The test framework uses a very powerful feature of PHP to detect differences for most values: var_export(). This function converts a value into the actual PHP code it would take to create it. This allows us to easily compare values for equality, and also to cut and paste more complex values (such as large multi-dimensional arrays) into our tests.
Thus, a simple test using the xUnit principles would have a setup.php.inc file like this:
<?php
include dirname(__DIR__) . '/test_framework.php.inc';
$test = new PEAR2_PHPT;
?>
The test would look like this:
--TEST--
a simple xUnit-style phpt test
--FILE--
<?php
include __DIR__ . '/setup.php.inc';
$a = 'this will be matched';
$test->assertEquals('this will be matched', $a, 'verify string is the same');
?>
===DONE===
--CLEAN--
<?php
// optional cleanup
include __DIR__ . '/teardown.php.inc';
?>
--EXPECT--
===DONE===
Note that, unlike PHPUnit, the teardown in the --CLEAN-- section is also performed in a separate PHP process, and is guaranteed to run, even if the test contains a fatal error. This allows very fine-grained testing of filesystem issues with assurance that no mess will be left lying around, because directories and files can be created by the test in the current directory, and then erased in the cleanup section.
If the test were to fail, the .out file would contain something like this:
Test Failure: "verify string is the same"
in /home/greg/testit/test.php line 4
Expecting:
'this will be matched'
Received:
'this will not be matched'
===DONE===
]]>Additionally, if you have the Text_Diff package installed, you would see a text diff of the var_export() output:
Test Failure: "verify string is the same"
in /home/greg/testit/test.php line 4
-'this will be matched'
+'this will not be matched'
Expecting:
'this will be matched'
Received:
'this will not be matched'
===DONE===
The development process then becomes running tests, checking the .out files to see what happened, fixing the code or tests, rinse and repeat.
Code coverage reports and instantaneous integration with Pyrus
What if you want to see how your PHPT tests are actually testing the code? With the next-generation PEAR installer, Pyrus, and PHP 5.3 or newer, you can easily do this. Note that PHPUnit also has full support for code coverage reports. Both tools require Xdebug. Pyrus can be downloaded from the PEAR2 web site. To use the run-phpt command, you also need to install the developer tools; just type php pyrus.phar run-phpt, and Pyrus will walk you through the process.
Once you have acquired Pyrus, I recommend that you put all of your source code in a subdirectory named src, and all of your tests in a subdirectory named tests. If you do this, you can take full advantage of coverage reports simply by typing php pyrus.phar run-phpt -m.
If you have a slightly different layout, you can still run the coverage report, just check the options for the run-phpt command.
The -m switch to the run-phpt command tells Pyrus to only execute modified tests. A test is considered modified if it is new, changed, or if any of the files it includes have been altered. In other words, Pyrus will automatically discover and execute the minimal test suite needed to verify that a change you’ve made doesn’t cause any regressions. Pyrus will also tell you the percentage of code coverage without the need to look at a report.
The web-based coverage report is implemented as a Phar archive, and is distributed with the developer tools. You can also download the latest code from Subversion. Save this into a local directory (again, you must have PHP 5.3 or newer installed to use this viewer), point your web browser at the code, and enter the path to the pear2coverage.db file generated by Pyrus.
If your tests’ path is /home/myname/thing/tests, the pear2coverage.db file will be found in /home/myname/thing/tests/pear2coverage.db. The coverage viewer has several views of the data. Each source file can be viewed with coverage information for each line showing the number of tests that hit a particular line of code, as well as the percentage of lines covered versus lines not covered. In addition to a summary of coverage for the entire test suite, the report can also display the coverage of a single test, so that you can verify whether a line of code that you are expecting your test to execute is actually executed.
The run-phpt -m command is so efficient that I now find myself running unit tests after every code change without breaking up the flow of development. The first iteration is always the slowest (this is when all unit tests are executed), but subsequent runs tend to take seconds. The -m switch to the run-phpt command can also give you a feel for the complexity of the interdependencies of your unit tests. Generally speaking, you don’t want more than 4 to 5 tests to be executed when you make a minor change to the source code. If all of the tests are being executed, the file that you changed is extremely important and is used by everything, or your tests are loading too many files unnecessarily. This information can be helpful for refactoring before dependency hell begins to take over your code or your tests.
Why not use PHPUnit?
This is a fantastic question — and probably the most important one to ask. PHPUnit is a well-established testing tool with a large community of users as well as several nice external tools for code metrics in addition to unit testing. PHPUnit has extensive documentation — including a book — and is designed specifically for testing PHP code. PHPT was designed to test the C code underlying PHP, and the use of xUnit requires more effort on the part of the developer, initially, until the feel of xUnit in PHPT becomes second nature.
If I were in your shoes, I would first ask whether my project needs to use the current standard testing project (often the case in corporate environments), or whether any of the advantages of PHPT are worth the switch. If you need the ability to test expected fatal errors — which is nearly impossible with PHPUnit — PHPT may be a better choice. If you demand IDE support for testing, PHPUnit may be a better choice. If you need the ability to run a test using PHP as well as with the testing framework, or desire the ability to step through a test with a debugger, PHPT will be a better choice. If you’re new to unit testing, PHPUnit will definitely be a better place to begin. If you’re interested in trying a new development possibility, xUnit-PHPT might be good fit.
If you aren’t unit testing at all, then what are you waiting for? Get with the picture!
