Theory of Sensitivity Testing

I have a program that creates a pool of equations with some variety of variables. I want to test the sensitivity of each equation to fluctuations in any of the variables. Basically, I haven't gone looking for some standard approach, but have been playing with different measures of sensitivity, and am looking for suggestions.

The simple sensitivity measure that I have does this:

1) Set all variables except the one being tested to their average values. (this works because I have average values for them, otherwise, any fixed value could be used).

2) Set the test variable to 1, and get the result of the equation. This is the OneValue.

3) Set the test variable to 100, and get the result of the equation. This is the HundredValue.

Now, there are many ways to get a single sensitivity out of this:

a) HundredValue/OneValue
b) OneValue/HundredValue * 100
c) (HundredValue-OneValue)/100
d) 100/(HundredValue-OneValue)

Each of these produces a different range of results. I'm not sure that one is better than any other, but a, b, and d have the problems that they can be undefined for certain results. Therefore, I am using C.

In theory, an equation such as mX + b would have X being perfectly sensitive to fluctuations. In other words, X100 would be 100 * X1. This is shown best with test A. In test C, you get almost 1. Values less than 1 would be less sensitive, values greater than 1 would be excessively sensitive (for instance x^2), negative values would indicate variables with inverse fluctuations ( such as 1/x, as x gets bigger, 1/x gets smaller).

Does anybody have any thoughts or suggestions on better ways to do this?

A bit of a long-winded way, perhaps, but I guess it depends on your needs:

Generate a gaussian distribution of values with a mean set to be the average of the variable you are testing and a standard deviation of 1. Run all these through the equation and then test the standard deviation of the results. You can then compare the s.d. before and after to see how they've changed. Effectively you are exactly looking at how small fluctuations in the input data distribution affect the output data distribution.

zaza

So I would be creating effective 'noise' around the mean for the variable in question, and seeing if the SD of the noise is the same as the SD of the result?

That's a good idea, but is it likely to provide a more valuable or different bit of information to the technique I am using? I briefly considered adding in the noise to the equation itself, but decided that the number of iterations would become insanely large (this is part of a modeling program, and the model I am currently running has a time to completion of about 60 hours). I don't have that concern to the same extent when looking at sensitivity, as the model is not running when I am testing sensitivity on the output. A few seconds delay would be burdensome, but tractable.

It's an interesting idea, what's the advantage?

Actually, just doing it theoretically probably wouldn't tell you much new - you can work out the sd of the result based on theoretical sd's of your input variables. If, however, you want to be actually modelling real data then you could use something along these lines as a "picker", to see just what sort of data combinations you'll end up with.
Effectively you seem to me to be saying "for a small change in x, what change in y do I get". This is effectively saying "if I input a variable with fluctuations (a distribution of width dx), what resulting distribution width dy do I get"? And the measure of a distribution width is the sd. You can easily whip something up in Excel that will show you how it works out.

zaza

Why 1 and 100? These specific values might not make any sense. Whenever I have to check the sensitivity of any output variable to variations of any input variable – and I do it quite often – I always apply percentages against percentages the way illustrated in the example that can be seen in the attached Excel file (with macros operated by buttons), which refers to a simple investment project. One is seeking what is the magnitude (in percentage) of variations in PW (present worth) when income or costs or the tax rate or the useful life vary x%, (x + delta)%, (x + 2.delta)%, and so on.

I hope this helps.

I think that's a good point. The thing about 1 and 100 is that 100 is 100 times 1....duh! I have also been thinking of switching to percentages, because, if a variable tends to fall into a range of 0.01-0.05, a value of 100 is HUGE.

Quote:

---

Originally Posted by zaza

Actually, just doing it theoretically probably wouldn't tell you much new - you can work out the sd of the result based on theoretical sd's of your input variables. If, however, you want to be actually modelling real data then you could use something along these lines as a "picker", to see just what sort of data combinations you'll end up with.
Effectively you seem to me to be saying "for a small change in x, what change in y do I get". This is effectively saying "if I input a variable with fluctuations (a distribution of width dx), what resulting distribution width dy do I get"? And the measure of a distribution width is the sd. You can easily whip something up in Excel that will show you how it works out.

zaza

---

I'll have to think about that one. There is some value to doing that, the SD of the result for a fluctuation of the input is probably more telling in some regards than comparing single point values. However, if the fluctuation is small, it may not give me direction (y gets smaller as x gets larger, and vice versa).

Quote:

---

Originally Posted by Shaggy Hiker

I'll have to think about that one. There is some value to doing that, the SD of the result for a fluctuation of the input is probably more telling in some regards than comparing single point values. However, if the fluctuation is small, it may not give me direction (y gets smaller as x gets larger, and vice versa).

---

When it comes to sensitivity of an output variable to each one of the input variables, only (de)increments make sense – in what magnitude and direction are concerned.

On the other hand, if you wish to find out the probability of an output variable assuming a certain minimum (or maximum value) – the so improperly called “risk” – then, the usual procedure is to build a simulation model, where each input random variable (the ones that you don’t control) is represented by a probability distribution (the one known as being the most representative). Variables which are deterministic in nature, assume a unique value (you control them). After hundreds or even thousands of repetitions are carried out (in each repetition, variables assume values picked randomly from their distributions and interact among them another), you calculate frequencies of the output data or check which theoretical distribution fits them the best. Only in this case statistical parameters such as a measure of variability (standard deviation, variance, coefficient of variability…) may adequately apply.

These two figures illustrate what I meant in my previous post.

http://img375.imageshack.us/img375/9480/forum7ji.th.jpg

I see what you mean. I'm not really looking to do what you are saying....or at least, I wasn't, but the more I think about it, the more it appears that there could be some useful information in there. I'll have to play around with that.

One of the problems I am having is that the values for some of my variables are better than guesses, but not a HUGE amount better. Some variables we can measure quite closely. In other cases, when it comes to fish (which I work with), we have solid, individual, measurements, but we have very little real confidence that the value we have is correct. If that problem is bad enough, I can't use the data, but I could find a function, and vary the inputs by a reasonable amount, and figure out where the output is most likely to lie. That may be of value to some people.....or maybe not.

I deduce from your words that your problem might perhaps be solved by fuzzylogic. If you are not acquainted with this technique, this link might be useful as a beginning.

http://documents.wolfram.com/applica...roduction.html

Hope it helps.

It actually sounds quite similar to a project I recently worked on, although there was nothing aquatic involved at all.

What I did was the following:

1) Figure out how to construct a "data-point" in terms of the set of variables.

2) Give each variable a mean and standard deviation.

3) Use a bootstrap Monte-Carlo (simulation) method to generate a dataset consisting of X samples (where X is large).

Then you can: Minimize the difference between the real dataset and the simulated dataset. Assume that both datasets are samples from a larger population and work out the probability that both samples come from the same population. Examine the sensitivities of the different variables by tweaking the standard deviations. Examine the effect of offsets in the mean. Various other things.

Maybe of interest?

The bootstrap Monte-Carlo (and half-decent Gaussian-distributed random number generator) I used were from "Numerical Recipes in C", adapted by me into VBA. I can post the code if anybody is interested.

zaza

Zaza,

It would be interesting to see the code you refer. I am very curious at it as I use Monte Carlo simulation a lot.

I only perform sensitivity analysis the way you describe when it addresses a random variable: What is the “impact” in the variance of an outcome when the variance of a random input variable “varies” plus or minus #%? When the input variables are deterministic, one should follow the steps I illustrated with examples in my previous posts in this thread.

I'd be interested in seeing it to, but I have to confess that I am even more interested in why you chose to port it to VBA of all things. I realize that VBA and VB6 are VERY close cousins, but something like that seems more appropriate as a stand-alone.

Evening,

As you say, Rassis, in this form it's mainly of use for random variables because they (are assumed to) be of Gaussian distribution. The random picker just generates the appropriate distribution.

I put it in VBA because that tends to be the easiest thing for a lot of people to use. Everybody in the office has Excel on their computer, so anybody can just pick it up and it'll run. I've put an Excel front-end on it, y'see, to do all the data input and automatic graphing. We don't have site licensing for VB in any other form and although there are a few copies knocking about they're confined to the IT boys.
It isn't the fastest code around, but it does the job. I'm at home now, so I'll get it tomorrow. It'll be dead easy to put in VB6 (cut/paste), and only marginally more work for .Net.
If you're interested, the NR books are actually free online here - there's tons of really useful scientific programming therein. There's C and Fortran versions, but it's pretty easy to work out what each line means and translate to VB.

HTH

zaza