Imagine getting a letter in the mail saying, "The computer has found you guilty. The fine is $250." You are not allowed to examine, or even question the computer. The only thing you can do is pay the money or go to jail.
Actually this is not fantasy, this is happening every day. Red light cameras are taking pictures of people, determining that they are guilty of a crime, and starting an automated process to punish them. But what happens when happens when people go into and challenge the computer? In all the cases I know about they have not been allowed to examine the software that decided they were guilty.
There are many different ways that software can cause problems. Let's take a look at a few. First there is something called binary round-off error. Everyone knows that 1+1 = 2. But did you know that 1/3 + 1/3 is not 2/3? 1/3 written in decimal is 1.33 and 2/3 is 2.67. So computing 1/3 + 1/3 we get
1.33
+ 1.33
============
2.66As you can see 2.66 is not 2.67. This is an example of rounding error. Errors like this can creep into every calculation done in a computer program. In some cases these error can get so large so as to overwhelm the original data. In other words the error can be larger than the original inputs. Without examining the source code and doing an numerical analysis of each operation involved in the production of the results, it is impossible to judge the accuracy of the results.
Yet these results are used every day to convict people of crimes.
Next let's take a look at resolution errors. For the propose of this discussion we're going to measure the speed of a car using it's position. At 12:00 we take a reading and our location is at mile marker 0.98. But our sensor only measure miles so at 12:00 we are mile 0. One minute later we are at mile marker 2.02. We've moved 1.04 miles in a minute, a little over 60MPH. But because our sensor only resolves things to the mile, it reports our distance as 2. Using the data we got from our sensor, we have moved 2-0 or 2 miles in a minute for a speed of 120MPH.
So depending on the resolution of the hardware being used, the results can be off by a tremendous amount. Yet questionable results like this are being used every day to convict people of crimes.
Most of the time when you ask the police how they know a computer oriented device such as a red light camera is accurate they answer "We tested it."
Testing can only reveal the presence of bugs not their absence. Testing can not assure that a program is accurate
Let me give you an example of testing. We have a computer program which can tell if a number is prime. Let's test it:
|
Input |
Output |
|
|
3 |
Prime |
Correct |
|
4 |
Not Prime |
Correct |
|
5 |
Prime |
Correct |
|
6 |
Not Prime |
Correct |
|
7 |
Prime |
Correct |
|
8 |
Not Prime |
Correct |
Now in real life it is not possible to test every input. Instead we can only test a sample of the data. So lets jump around a little:
|
10 |
Not Prime |
Correct |
|
11 |
Prime |
Correct |
|
12 |
Not Prime |
Correct |
|
13 |
Prime |
Correct |
|
14 |
Not Prime |
Correct |
|
17 |
Prime |
Correct |
|
18 |
Not Prime |
Correct |
|
19 |
Prime |
Correct |
|
20 |
Not Prime |
Correct |
There are twenty possible inputs in the range 1-20 and we've tested 15 of them. We've got 75% test coverage. That coverage is tremendous. With the millions of possible inputs that can occur with red light cameras, if they test even 0.1% it's a lot of tests.
Usually the police only run one or two tests. That not a series of tests, that's a quick sanity check.
But our program passes our test (with the 75% coverage) easily. So through high coverage testing we've "proved" the program works. Now let's take a look at the program:
If the number is odd, then it's prime.
As you can see this algorithm contains an obvious and glaring flaw. But this flaw is only revealed if you can inspect the code. Testing did not show it.
One of the problems with computer programs is that failures can be discreet. In other words the computer can perform the calculation thousands of times, fail once, then do it right for another thousand times.
One such example of this was a timing bug on the old TOPS-10 operating system. There was a problem which only occurred one minute each day. And the failure mode was different for the first second of that minute.
So unless the police tested the machine at the exact second you went through it, their tests are meaningless.
A good inspection system is key to making sure that the code is correct. It's interesting that the Fonding Fathers knew this is wrote it into the constitution.
--- amendment ---Far too often red light cameras and the programs inside them are treated as secret evidence. Let's contrast this with the evidence you can look at when you're caught speeding by RADAR. You can ask how the officer knows the RADAR gun is accurate. He'll tell you it was calibrated with a tuning fork. How do you know the tuning fork was accurate? He'll show you a certificate from the company that did the calibration. If you wish to probe further, you can get a paper trail that leads from the machine that verified the calibration of the tuning fork all the way back to the United States Bureau of Standards. You can also examine the manual, schematics, and repair history of the RADAR gun. In short everything is open.
If device is to be used as evidence against someone then it should be designed to a quality standard which can assure that the result is correct "beyond a reasonable doubt." Anyone who has a process in place which produces such high quality code, must as a part of that process produce a lot of documents: design specifications, code reviews, problem reports, software manufacturing procedures and so on in addition to the source code any programs in the device. All of these should be available to a defendant for review.
Here's a list of the questions I would be asking the police in court if I were fighting a red light camera ticket.
1) What quality standards were used to produce the code used in this machine?
If the officer does not know or says none (it has happened) then the code is no better than an random number generator.
2) If a ISO9000 quality standard was used, ask for the documentation generated by the inspection process.
3) What inspection process was used on the code?
Make sure that the person answering the questions knows the difference between an inspection process and a test process.
4) Please show me the numerical analysis done on the code to make sure that no significant round-off error have occurred? Also can I see the code to verify that analysis?
5) Describe the code production process and the steps used to make sure that the version of the code which underwent inspection is the one that is the same one that is installed in the camera.
6) Were any patches or bug fixes made to the software after inspection? Were all the program files re-inspected after the patch to make sure that the patch did not interfere with their operation?
7) What is the current patch level or software revision loaded into the camera? How can that be verified?
8) What is the resolution of the timing devices in the device and what is their allowed drift?
9) What are the specified operating temperatures for the devices that make up the computer?
10) What temperature control system is used inside the enclosure to assure that the temperature is kept within limits?
11) Are there any sensors in the device that make sure that temperature is kept within the allowed range? How were they tested?
12) How does temperature changes affect the performance of the timing devices?
13) How was the device installed and how was the installation documented?
14) How was the device configured and how was this configuration documented?
15) Has the device been physically altered since it was installed? (In San Diego one sensor was moved because the camera was not catching enough people. The configuration should have been changed but was not.)
16) Has the configuration parameters of the device been changed since installation? Why? How? Can you show me the documentation for the procedure?
17) What is the system for reporting bugs to the manufacturer?
18) What bugs have been reported? For what software version? Have the been fixed in the system in question? What system do you use for bug tracking?
19) If the code was only tested and no inspected, how do you know that there is not code in the camera that detects testing and generates correct results, but when production mode is detected changes the operation of the camera to increase revenue?
Here in San Diego, the company who made the red light cameras used a out of country contractor to write the code. Because of that, they told the court, they had no source code, no idea what quality standards were used to produce the code, and no inspection records. That level of documentation is not enough to assure anyone that the device works at all, much works so that the results can be proven accurate "beyond a reasonable doubt."
Can you imagine a police officer going into court and saying "I arrested him because I though he looked guilty. I won't tell you why I thought this, I won't let you look my track record to gage the validity of my judgment, and I won't tell you anything else. I say he looked guilty, so he must be." He'd be laughed out of court. Substitute "red light camera" for "police officer" and you have what some people consider valid evidence.
The trend of using closed source (secret) evidence against someone is extremely dangerous and must be stopped. Besides it's obviously unconstitutional.
In closing, I would like to leave you with a little letter written by someone who got one of those red light camera tickets. She wrote:
I swear that I was not driving the car in the
picture you sent me. I
I swear that your camera can not tell the difference
between my SUV and a two seat sports car. I swear
that I am a forty year old woman and not the twenty
something boy in the picture. I swear
that I do not know who owns the car in the photograph
and I swear I was across town shopping when this person
ran the red light.
Does that take care of this of do I need swear at
you some more?
I thought I has written everything I could about what could go wrong with a red light camera. Turns out that some of the public found a new weakness. It seems some students made a paper copy of the license plate of a teacher they didn't like, taped it over their real license and then started running red lights with cameras on them.