WEB Advent 2010 / Mistrust and Verify

When working on a web app with coworkers or fellow volunteers, few things are more important within the group than trust. Trust among developers enables them to work confidently, eases communication between team members, and empowers each of them to make good decisions.

With the actual app, however, that trust should never come easily. With today’s web apps, you can’t trust anything. You can’t trust input from the user, you can’t trust your code, you can’t trust your processes, and you can’t trust your systems. So, while the Russian proverb says, “Доверяй, но проверяй,” meaning, “trust and verify,” these days you need to mistrust and verify.

The user

A user doesn’t need to be malicious to break your app, but malicious users do exist, so no input from any user can trusted. Any data received is immediately suspect, needs to be validated that it is the expected format (numbers are numbers, dates are dates, postal codes are valid, email addresses match RFC 822, and required data is the right length) and filtered to remove dangerous input. Plenty of articles exist on how to avoid SQL injection, but preventing cross-site scripting attacks and session hijacking is also important.

You should mistrust data coming into your app enough to read the articles in the PHP Security Consortium library, follow a post a day for a month of PHP security, or spend an hour reviewing the basics of PHP security.

A great resource (and must-read) for beginners — that is full of great reminders for experienced developers — is Essential PHP Security, which succinctly lists problem areas where all PHP developers should be cautious, along with solutions.

Fortunately for developers who prefer using frameworks, most PHP frameworks will handle the data checking and filtering, preventing common security holes. If you use a framework, embrace its way of handling data so as not to thwart its processes. Mistrust it just enough to verify it does what you expect, then sign up for its security mailing list. When a fix is reported, download the patch and review the changes; have you made the same error in your code?

The code

Input from users is only one part of the app; your code is another How do you trust your code, which, while brilliant at two in the morning, is perhaps less brilliant the next morning? You don’t. You write unit tests.

Unit tests let you automate the tedious job of verifying that your code works correctly and is returning the right data formats. They let you know when changes adversely affect the rest of the system.

SimpleTest and PHPUnit are two popular PHP testing frameworks. Creation of unit tests have been made relatively painless, but not error-free nor effortless.

After your unit tests are written, testing all of your functions and libraries, can you really trust them? Not really. Unit tests are white box testing; you know what your code is (supposed to be) doing, you know what input is going into the test, and you know what you’re expecting to be returned. With all of this knowledge, no, you can’t even trust your unit tests, as each step is known, but input from users is an unknown.

To be effective, unit tests need to be complete and need to be updated to handle the new cases you see during development and in production. Each time a bug is found, a problem discovered, or an exploit exposed, a unit test should be created for the issue. Brainstorm on edge cases; you might not put control characters in your form input fields, but people who cut and paste, or maybe use Emacs, just might. For multilingual applications, have you tried to break your tests with other languages?

A review of your unit tests from time to time is also a good idea. Ever look at a completed part of your project and find this?

/**
* Test validName function
* @param string input name
* @return boolean
*/
function testValidName($name) {
    // TODO: write validName test
    return true;
}

The deployment

So, user input is mistrusted, checked, and filtered. Code is mistrusted, reviewed, and tested. You’re ready to launch a new feature or an entire new project. Now is the time to mistrust your deployment processes if they are not documented, repeatable, and (preferably) non-destructive. Manual deployments are error-prone (“Was I on step 3 or 4?”), with undocumented processes being the worst (“What did you do?” “I don’t recall.”).

Documenting a launch process forces you to think through what steps need to happen, what requirements need to be met, and what problems you may encounter. Each step has assumptions that should be questioned. Are the development versions the same as the production versions for your OS, web server, PHP, and the database? Is any step resource limited? What happens if a step fails? Can you repeat a step without concern? What could possibly go wrong here?

Are there steps that require manual intervention? Can these steps be automated? Automating a deployment process makes it repeatable, and at that point, debuggable. Phing, Capistrano (yes, you can use it with PHP), shell scripts, your own PHP scripts, or even Ant can be used to build a deployment infrastructure tailored to your projects’ needs.

What error checks are necessary in that automation? How do you verify that the deployment is successful? For single-page web sites, answering “Hey, does the page display?” is sufficient. For larger apps, you can run unit tests against the live installation; log into an account, perform a series of common tasks, and verify the output.

The view

Once your app is launched, you can breathe a sigh of relief, then mistrust your view of your app. You may be close to your server, on a fast connection, using a browser with JavaScript enabled. Your users may be on the other side of the planet, with a slow connection, and JavaScript disabled.

You can mistrust your rosy view of your app and understand what your users are really seeing by installing Charles or another web debugging proxy. By increasing network latency and slowing down your connection, you can see your app from a different user’s point of view.

The server

Your app could be rock solid, perfectly secure, and blazingly fast, but your server could become a source of mischief. It’s best to mistrust that, too. Most web hosting these days is still shared hosting, with hundreds of users on one server. The wrong permissions or an outdated OS could lead to compromises that none of your code or unit tests could have prevented.

So, attack your application as a malicious user would. Use skipfish to attack your app, and Nmap to scan your server.

See what ports you have open, or what processes you have running. Figure out what you can shut down to remove potential threats. Having database access ports open on a server without limiting by IP address, for example, is a recipe for disaster.

Download rainbow tables, and try them against your system. Attempt to install a rootkit — both for the joy of trying to break into your system, and because, really, at this point, you mistrust everything about your server.

The backup

Of course, in the end, the fundamental worth of your app may be the data. Losing it could be catastrophic to the project or even your company. At this point, though, you know to mistrust your backups.

Your databases are backed up, but where are those backups? Are they on the database server? What happens when that hardware fails? Backed up to another server? What happens when your hosting provider goes out? Do you have a master-slave database server set up, so that you always have a hot backup? When is the last time you ran SHOW SLAVE STATUS to determine if the slave is actually replicating the master’s data? Do you have a heartbeat process going? Is someone listening to its responses? Monitoring the database status will work only if someone is paying attention.

The biggest question, and the ultimate test of any backup system, is have you successfully restored a system from one of your backups? Horror stories of empty backup files abound, so restore from one of those backups, and deploy a staging server — even if it’s only on a virtual machine. Restore from that backup before you really need it.

Fundamentally, programming defensively, mistrusting your assumptions, and verifying each part of your app, from the shortest line of code, to the latest deployment, creates a better, more secure application.

There are many ways for a project to fail; it’s best to eliminate the ones you can.

Mistrust and verify.

Other posts