For RGB colors, the easiest way to compare colors is to treat them as three-dimensional vectors. So to calculate similarity, you use a standard distance formula, e.g.:

Code:
Similarity = Sqr( (R2 - R1) ^ 2 + (G2 - G1) ^ 2 + (B2 - B1) ^ 2)
If you only need relative distances, you can omit the Sqr() for a performance boost. Detecting similar colors is very simple:

Code:
If Similarity < Threshold Then 
  (colors are similar)
Else
  (colors are not similar)
End If
The best Threshold value will depend greatly on the source images. Low-quality JPEGs will need a higher threshold than lossless formats like BMP or PNG. It will probably take some experimentation to find the best number. I'd start at 10 (if not using Sqr()) and work your way up from there.

<Begin potentially pedantic comments>

One of the problems with the RGB color space is that color distance is not uniformly perceptual. This is a fancy way of saying that color distance is not a very meaningful measurement for RGB colors.

For example, using the formula above, two dark colors with a similarity of 10 could look pretty much identical. However, two bright colors with a similarity of 10 may look very different.

Scientists have developed a number of color models that are perceptually uniform. These color models are the result of thousands of tests, where participants of all ages and genders would try to match differing colors, and scientists would use the resulting data to figure out exactly how humans perceive color. (Pretty mind-numbing work!) These experiments made it possible to develop color models that correlate very well with human perception.

CIELAB is probably the most famous perceptually uniform color space. If you need very accurate results, I'd recommend using it instead of RGB. Converting between RGB and CIELAB is complicated, but you can find well-known formulas here.

<End pedantry>