Introduction

Accuracy assessment is the procedure used to quantify the reliability of a classified image.

The standard accuracy assessment procedure is to construct an "error matrix." This is a square matrix in which the rows and columns represent the land cover classes from the classified image. So, if you have 5 classess, you will have an error matrix with five rows and five columns. The matrix is filled with counts taken from a sample. For every point in the sample you determine the "ground truth" and you know the image classification. In many situations the "ground truth" is determined from higher resolution air photos. Because the sample is not always acquired from the ground, it is often called "reference data" and it is assumed to be correct. Even if the reference data points are found by actual field visits, it is rarely done at the same same time the satellite is flying overhead. So, even data from the field should be called reference data.

For each point in the sample you have that point's reference data and its image classification. If the reference data shows class "i" and the image classification shows class "j", that sample point will increase the count in the "i,jth" element of the error matrix.

Filling the error matrix and then deriving some estimates from the matrix will become more clear as we work through an example. You should also check out Lillesand and Kiefer's section 7.14 and/or Jensen's Chapter 8, starting on page 247.

In this lab we will use a classified TM scene.

The classification is a rather simple six class system:

Cloudy Water (class 1)

Developed Land (class 2)

Hardwood Forest (class 3)

Evergreen Forest (class 4)

Agriculture/Grass (class 5)
Clear Water (class 6)

The reference data will be derived from 10 meter SPOT panchromatic data. (Why is this not an ideal reference data source?)

We will use the Imagine "Accuracy Assessment" module to randomly select the sample points. Imagine will automatically record the image class for each point in this sample. We will then use the viewer to determine the reference class for each point. Once we enter the reference class for each sample point we can use Imagine to construct the error matrix and derive some accuracy estimates.

Imagine Accuracy Assessement Procedures

The first step is to open the classified image that has been prepared for this lab. It is saved as: tm_data/ral94_a_sup.img
(which stands for Raleigh, 1994, albers projection, supervised classification).

The image should look like this:

where the colors represent the following:

Cloudy Water = light blue
Clear Water = dark blue
Developed Land = magenta
Hardwood Forest = dark green
Evergreen Forest = light green
Agriculture/Grass = yellow      

Once you have the classified image file open in a viewer, select Classifier from the main Imagine toolbar. Then select Accuracy Assessment.

This will give you the "Accuracy Assessment" Dialog box:

This will be the main window used throughout the accuracy assessment process. In this window we will specify a sample to be placed on the classified image, link a viewer containing the classified image, type in the reference data, and produce an accuracy assessment report. So, lets get started.

First select View/Select Viewer and click in the viewer containing the classified scene. Now the accuracy assessment window is linked to the viewer showing the classified scene. Then you need to select File / Open to open (withing the accuracy assessment window) the classified image file for which you want to conduct and accuracy assessment. Select File / Open and choose the tm_data / ral94_a_sup.img file.

The next step is to generate a set of sample points randomly placed over the classified image. This is done by selecting Edit / Create/Add Random Points . To create/add random points you will need to specify how the points will be collected:

The "Distribution Parameters" refer to how the sample point are placed over the classified scene. For this lab we have selected 70 points, distributed by a "Stratified Random" sample with Minimum of 10 points per class. This is a somewhat small number of points per class. In practice, it is recommended that you have a minimum of 50 points per class. Once these parameters are specified, you click on OK to conduct the sample. You may get a message window that looks something like this:

This means that Imagine was not able to collect at least 10 points in each class out of the maximum number of random points you specified. You can click Yes to let it search again. It may take a few tries before you have 70 points. If you have trouble, feel free to stop with fewer than 70 points.

Use File / Save Table to save the points so that you can come back to them later. Be sure to save the table in your own directory.

You should notice that the Accuracy Assessment dialog box has information in the first three columns. Select Edit / Show Class Values to see the classified values for each point. Do not leave these showing during the reference data interpretation.

Note that Imagine will put some points outside the boundary of the image and call them class zero. You should go in and delete those points (there aren't that many) because they will bias your accuracy results. Just select each row that has a class of zero (you can select multiple rows using the shift key and clicking), then right click and chose "delete selection."

To see more clearly the sample points you can select View / Change Color and specify a color other than white for "points with no reference."

The "Reference" column in the accuracy assessment table needs to be filled in by the analyst (you). As mentioned earlier, the reference data will be collected using SPOT panchromatic data which match the TM image data area.

Collecting the Reference Data

To collect the reference data, open spot_data/raleigh_pan96_alb.img in another viewer. Then open a Utility / Inquire Cursor in the SPOT window and select View / Link Viewer / Geographic and click in the viewer containing the classified image.

In the classifed image viewer there should be a cursor and a box. The box indicates the areas shown in the SPOT window relative to the TM image. Now examine, one by one, each reference point on the SPOT image using the Inquire Cursor tool. When you have determined which of the six classes in this classification is appropriate for each point, enter that class number (1-6) in the "Reference" column in the accuracy assessment window. Note that looking at the point's classified values before determining the reference class can bias your interpretation of the reference data. That is, since you probably want your classification to be correct, you may be inclined to interprete the reference data so that it matches the classified data. For this reason, to have a statistically defensible accuracy assessment, you must select Edit / Hide Class Values in the accuracy assessment window.

Producing an Accuracy Assessment Report

Once all of the reference data are collected, Imagine can generate an accuracy assessment report. From the accuracy assessment window, select Report / Options to see the different items you can select to be included in the accuracy assessment report. To actually produce the report select Report / Accuracy Report This will open a text editor window with an accuracy assessment report. This will be in the form of a standard error matrix and other measures that we have discussed in the lab lecture.

Review Questions:

1. Produce an accuracy assessment report for the ral94_a_sup image. Discuss your interpretation of the error matrix. In particular: which class seems to be classified the best, which is classified the worst, are any classes "confused"?

2. After analyzing the error matrix, describe one use of this classified image with which you would be comfortable. Also describe one use of this classified image with which you would NOT be comfortable.

3. We used the SPOT data as reference data. Comment on possible problems with using the SPOT data as reference. (That is, describe ways in which the SPOT data are not "ground truth".)

4. What is the Kappa statistic, and what does it tell you?

5. What complications are involved when you are doing accuracy assessment of change images such as those produced in the last lab?