The Great Ideas Thread

Have an idea you think is great? Post it here and you just might get some valuable feedback.

Here's an idea I think is great: a search engine that gives a lot of weight to Webpages that people say are relevant to what they're searching for. If the user finds what he or she is looking for, he or she can tell the search engine so with the click of a button.

Here's another one: a program that can interpret text as a human would. So it could deduce information about dogs from the following passage: "The dog (canis) is an animal of family X. This animal is usually domestic. The average height is 4 '," or "This animal, averaging 4' in height, belongs to family X and is usually domestic."

It could list this information like this:
The dog
-belongs to family X.
-is usually found in homes
-has an average height of 4 inches.

Google has a large index, but for reasons known only to Google, its "best guess" feature only works for certain queries, even if the same question's been asked and answered on places like Yahoo! Answers.

OK, yet another idea: a computer program that can use a set of data to make mathematical models that could be used as forecasts. For example, I could have that program analyze stock data and give me a forecast.

It's simple, we kill the moonman :afrog:

Quote:

---

Originally Posted by moonman239

OK, yet another idea: a computer program that can use a set of data to make mathematical models that could be used as forecasts. For example, I could have that program analyze stock data and give me a forecast.

---

Will it predict a crisis?

Quote:

---

Originally Posted by namrekka

Will it predict a crisis?

---

:lol: That would be a human's job.

I can think of other applications for such a program. Put it in a car and hook up some IR sensors and you could have yourself an automated car. Actually, Google has been testing car prototypes. Nevada said they'd give the cars a driver's license if they passed a driving test.

Your ideas are pretty great. The first one sounds like what Google is trying to do with variable success (there is a competing pressure on the parts of various people to game the system to produce certain results, which will always be a problem if the product is valuable). The second one sounds a bit like Wolfram Alpha. I think it is within reach, but not here quite yet. As for the third idea, I wrote such a program a few years back, and still have it. Technically, it is that kind of thing that is developed by those called "quants" that are somewhat responsible for the recession, and are largely responsible for the flash crash of a few years back. As a program, the one I wrote is kind of interesting in that it creastes a population of models that are considerably better than random. The best of those is not a safe model, though, because that falls under the category of "data dredging", which is risky to rely on (though lots do). However, because there is a population of independently derived, excellent, models, you can study the population to look for similarities. The similarities between the various models shows something, at the very least, about the data used to form the models. It may also show something about the system the model is used for.

The basic plan behind the program is that all math equations are nothing more than strings of characters that code for a certain pattern of result (the model curve itself) when given certain inputs. Therefore, I took all the variables that seemed to impact the system and built random equations using them. I then tested each model against the observed pattern and evaluated how well they predicted the observed pattern. This allowed me to rank the equations based on how well they did, which is all you need for a genetic algorithm. So, the program was using evolution to evolve the best equation it could to predict the observed pattern based on the variables input. There's more to it than that, as I have improved it in various ways, but it works pretty well.

What about a "super-high-power-WiFi-router-thing". So we can connect the alien internet to ours. A sort of universe internet. Then I can add new friends to my Facebook and dating all arround the universe. Wow........

Quote:

---

Originally Posted by moonman239

...
Google has a large index, but for reasons known only to Google, its "best guess" feature only works for certain queries, even if the same question's been asked and answered on places like Yahoo! Answers.

...

---

Because a question has been asked, and that it has been asked, does not reveal anything useful about the question and the answer.

In other words, there are stupid questions, and there are stupid answers, which for some bizarre reason, seem to be an 'acceptable answer'. It's not called Yahoo! for nothing. As much as I hate stackoverflow, at least their philosophy is to push the correct answer (and, yes, there are most often times a 'correct' answer).

With regards to 'interpreting text as a human would', i know it's not what you are mentioning, but I thing the iPhone app, WordLens is the freaking coolest translation app ever - it translates text using the live camera, replacing the text with translated text.

As far as ideas go, well, I have a bunch, but I'm too busy fixing all the broken 'ideas'...

Quote:

---

Originally Posted by SJWhiteley

As much as I hate stackoverflow....

---

This is about the umpteenth time I've heard someone here throw a jab at StackOverflow. Is there some history between their site and VBForums I'm not aware of ? I am getting the impression that StackOverflow is to VBForums what Christianity is to Atheism.

Quote:

---

Originally Posted by Niya

This is about the umpteenth time I've heard someone here throw a jab at StackOverflow. Is there some history between their site and VBForums I'm not aware of ? I am getting the impression that StackOverflow is to VBForums what Christianity is to Atheism.

---

I don't think it is between VBForums and StackOverflow. I do think that StackOverflow comes off as zealots, who often appear more interested in form than content. One thing that angers me is to see how many C# answers are given for VB questions, but if you give a VB answer to a C# question you get hammered.

Sounds like a rough culture. While I don't participate there as I do here, I do find about 80% of the answers I need on the more in-depth and highly technical issues regarding programming over there via Google. I never actually noticed the nuances of their online culture.

Quote:

---

Originally Posted by Shaggy Hiker

As for the third idea, I wrote such a program a few years back, and still have it. Technically, it is that kind of thing that is developed by those called "quants" that are somewhat responsible for the recession, and are largely responsible for the flash crash of a few years back.

---

I have a hard time reconciling those allegations with
the notion that for every seller, there has to be a buyer.

What allegations?

The stock market is based not so much on willing sellers and willing buyers, but the price the seller is willing to sell at and the price the buyer is willing to buy at. If you take the housing crash, there were plenty of willing buyers, they just weren't willing to buy at the price the seller was willing to sell at. In an automated system, if you say, "when x happens, sell Y.", there is always a chance of a feedback loop. You need to be saying, "when x happens, sell Y unless the price is below z.", but that's a prospect with its own dangers.

In any case, the program I wrote created models, but it did it through a process of selective data dredging. The resulting models always did a good job of fitting a curve to whatever data was available. Most people don't want to know anything about models other than their curves, but there's more to them than that. The problem is that every model has a set of assumptions for which it is true, and a set of assumptions for which it is false. All I was doing was creating a good models for a particular set of assumptions. Not only would those models be unpredictable if the assumptions changed, it also wasn't possible to be entirely certain what the assumptions behind the models were, so it wouldn't be possible to predict when the models were no longer viable.

Allegation 1 - "quants" are somewhat responsible for the recession
Allegation 2 - "quants" are largely responsible for the flash crash

"Allegation" may have been a tad aggressive, but the implications were
certainly present. I don't know the answer, but I'm just a little leary with
those as explanations.

In any case, the approach you described regarding a system consisting of "a population
of models" is, indeed, interesting.

While it may not be the same as your approach, I've heard of something referred to as "training"
the a system on "recent" data. "Curve fitting" to prior historical data is indeed a pitfall, and from
what I understand, this sort of "training" is aimed at minimizing the curve fitting dilemma.

As for no longer knowing if the model is viable ,,, what if the model was, say, optimized for a given
time-frame ,, also known as "in-sample". The model is then tested on "new" data ,, aka "out-of-sample."
Would that possibly answer the question?

It would improve our understanding of the problem, but it would never answer the question. Every model has a set of assumptions, we just don't always know whether we know what all the assumptions are. I would certainly use that approach, and see what happened to models in different time segments. However, we can never be sure what all the conditions were for any one time segment, and we can never be sure what will happen tomorrow. As an extreme example: If a sizeable asteroid took out either the US or China tomorrow, it would almost certainly impact our stock portfolios in ways that no model would accurately predict...unless we were in the impact zone, cause I can predict what THAT would do to my portfolio pretty well.

As for the quants being responsible for the flash crash, it may be a question of what is a quant. I was counting those who are creating advanced statistical models to guide computerized, automated, buying and selling of stocks. The flash crash was too fast to have been implemented by humans, though we did do it ultimately, since humans wrote the programs that performed the actual selling. Whether such quants had a pivotal role in the recession is a bit harder to say, but I think they did. I believe they drove purchasing based on models that didn't take into account the impact of a sudden softening of the housing market. I believe that they used models that were based on an assumption of housing market dynamics that had held true for most of the previous three to seven decades, and the models encouraged people to take overly exposed positions which turned devastating when the assumptions of the model were violated. Not all models got it wrong, though, just some of the larger ones, but that was enough to cause consternation throughout the top of the economy, and consternation flows downhill faster than good times.

I'm a little puzzled by the notion that "we don't know what all the assumptions are".
Could you elaborate?

And just to muddy the waters a bit, why the emphasis on "prediction"? True, some
notion of prediction is involved in devising and/or accepting a signal to open a trade,
but couldn't a statistical analysis of such a signal be sufficient to accept/reject it?
Seemingly, one would want at least 30 instances for it to have any significance.

Further, I agree that one could not predict extremes like the one you noted. But, wouldn't
it be sufficient to have a plan of how to "react"? Translated, that would be either to
hold or close the position.

The problem with real-world models like that is that the assumptions are very hard to fathom. One trivial assumption for the housing models was that house prices would always rise. Obviously, people knew that trivial statement was wrong, but they assumed that when house prices dropped it was just a short-term fluctuation. On a more fundamental level, the assumption would be: People want to own houses. If that assumption ever changed, which it could, then most of the underlying principles of building houses would shift in unpredictable ways. When it comes to houses, that's a bit hard to see, but consider any models that were based on horse usage back in the late 1800s. People could make economic decisions based on the increasing number of people, the demand of people for horses for such things as transportation, and based on that you could predict the need for certain industry and the growth in a few sectors. Then, in a relatively short time frame, a new invention wrecked all that planning as people moved rapidly to a totally different means of transportation. That's the bigger problem with these models: We base them on what has happened in the past and how we expect the future to follow from the past and the present, but there are inventions and events that can take the future in radically different directions than the past. Since we can't know what those events will be, we can't model them, and all the models we currently have could be wrecked by a change we couldn't possibly have forseen.

Well, one thing is certain: a Bad Ideas Thread would have been far more entertaining.

Shaggy

True, dat.

But, I would suggest that your "parameters" are qualitative (eg, desire for housing, possibility of a new invention)
What if your "parameters" were quantitative (eg, price, time)

Quote:

---

Originally Posted by Niya

Sounds like a rough culture. While I don't participate there as I do here, I do find about 80% of the answers I need on the more in-depth and highly technical issues regarding programming over there via Google. I never actually noticed the nuances of their online culture.

---

That's one of the reasons I think of Stackoverflow as a club for people who actually bother to read the manual.

Quote:

---

Originally Posted by abhijit

That's one of the reasons I think of Stackoverflow as a club for people who actually bother to read the manual.

---

Well thank God that some people take the time to do so ;). Their site layout is really confusing and not very intuitive but the information is really quite good. I found quite a number of very interesting insights and links over there that have helped me solved a wide range of issues.

Quote:

---

Originally Posted by Niya

Their site layout is really confusing and not very intuitive

---

That is an understatement, if ever I saw one.

lol...Well I don't wanna be too mean :mad:

Quote:

---

Originally Posted by Niya

Their site layout is really confusing and not very intuitive but the information is really quite good.

---

Really? What don't find intuitive about it?

Quote:

---

Originally Posted by TheBigB

Really? What don't find intuitive about it?

---

Well for one thing, when I came here, I didn't have to actually think about how to sign up. I have an account over there and I distinctly remember being very confused about how to sign up or even if they require it. I remember looking all over the place to find where you had to sign up. vBulletin and every other forums software makes that very easy to find.

The layout of the site itself reminds me of what happens sometimes in FireFox when the CSS for a site fails to load. It looks unorganized. It makes it pain in the ass to navigate. I can never figure out where anything is. Its layout seems hastily designed and usability suffers, at least for me. They should have used a forum format or something like it.

Shaggy

If you hadn't already heard, this should warm the cockles of your heart ...

Just after 1:00 PM Eastern, the DOW dropped 150 pts and then fully recovered
in 5 minutes as a result of a fake Twitter post on AP that a bomb had gone off
at the White House .. the AP account apparently was hacked.

So, even quantitative approaches have to deal with Black Swan events .. ;)

Quote:

---

But, I would suggest that your "parameters" are qualitative (eg, desire for housing, possibility of a new invention)
What if your "parameters" were quantitative (eg, price, time)

---

You don't really have a choice if your dealing with "soft" subjects like market economics. The truth is that many of the parameters ARE qualitative and you need to find a way to deal with them if you're going to accurately model a system to the point where you can make useful predictions. At that point you've basically got two broad choices:-
1. Try to find quantitatives ways of expressing your qualitative parameters (but that's really just introducing a new layer of modeling)
2. Ignore the qualitative parameters entirely and only deal with those which can be quantified

From what I've heard in the media, poor understanding of and/or choosing to just ignore the qualitative parameters that were feeeding into the housing market and stock exchange were a large part of why so many of the models failed. They tended to just mine raw data looking for patterns and assumed those patterns would repeat. That's a flawed aproach because it carries the implicit assumption that people and the groups they form are rational and will take the same action when faced with the same circumstances twice. On the whole they are rational but occasionally they're not and when they get irrational enough it collapses the model... and the economy in this case.

The other major contributor (again, only from what I've read) was that they didn't draw their dataset from far enough back. In hindsight you'd have thought that someone datamining to find patterns in market behaviour would have included data from 1900 onwards so that they had a decent sized crash or two in the mix but from what I understand, most datasets only went back a decade or so. I don't know the reasons for that but suspect it was to do with data availability. Either way, it meant that they were making prediction based on models that had been built on assumptions drawn from a period of continuous growth. The models weren't capable of predicting a bubble bursting because the models had never "seen" a bubble bursting. They couldn't predict the outcome of a crash because they'd never "seen" a crash.

You can take that further as well. Even if the data to include a crash had been included there's a very good chance that the models would have failed to predict this crash because they'd never seen this crash. Every crash is different. Indeed, a major exacerbating factor of the 1920s crash was not that the banks were too big to fail, they were too small to cope. There were hundreds of small banks so a failure of one had less impact, but each was individually far less able to soak a loss than todays giants, making runs far more likely. Actually, the banking crisis ended up being FAR worse in the 20s because the banks were smaller than they are today, which makes me raise an eyebrow when I hear politicians and the media demanding that we break them up. A model that looked at the 20s data and said "this is how people and their banks behave in a crisis" would have drawn a multitude of false conclusions.

Funky

What you suggest rings true.

But I can't help but return to a precept that I recall from a Stats course I
took in grad school .. in a normal distribution, one needs at least 30 samples
(ie, examples, instances, etc) for a "prediction" to have any statistical significance.

One or two examples is clearly too few, so models based on such a low number
would seem to be doomed to failure.

I think that depends on what you're modelling but it sounds like a pretty reasonable place to start if you're looking at just one or two parameters. But as you increase the number of parameters you need a bigger dataset and when you're dealing with something as complex as an economy the number of possible parameters is huge so the dataset has to run into the millions to be reliable.

That wasn't really the problem though. The guys doing the datamining were probably running tests across billions of transactions. The problem was that the transactions were all drawn from an unrepresentative sub set. They were taken across a period of unremitting growth. It's a bit like sticking ginger kids in the sun and using the data to predict how long it will take brunettes to get a sun tan.

Personally I think it's impossible to accurately predict something as complex as an economy. You hit on an interesting point in post 15 when you asked "how would you react?" That's the bit that was missing really. Economists fell so in love with their statistical models that they began to believe they could never be wrong and banks and investment houses bought into that. As a consequence they failed to put contingency plans in place for what seems like, in hindsight, to be an inevitable outcome. I.e. Continued growth above inflation ALWAYS means a bubble and ALL bubbles burst. That's just common sense, really, but it's an unpleasant bit of common sense to acknowledge if you're bonus at the end of the month depends on the growth carrying on.

Quote:

---

That wasn't really the problem though. The guys doing the datamining were probably running tests across billions of transactions. The problem was that the transactions were all drawn from an unrepresentative sub set. They were taken across a period of unremitting growth.

---

Fair enough.
And to make matters worse, they were probably overly leveraged.
Put up $1, borrow $25.
If you have a small win, the return is magnified by a factor of 25.
But, if you have an unexpectly large loss, you are wiped out.

Quote:

---

they failed to put contingency plans in place for what seems like, in hindsight, to be an inevitable outcome.

---

Spot on. I've heard it referred to as an Exit Strategy.

Quote:

---

And to make matters worse, they were probably overly leveraged.
Put up $1, borrow $25.
If you have a small win, the return is magnified by a factor of 25.
But, if you have an unexpectly large loss, you are wiped out.

---

I think you're probably right but this one's a tricky balancing act. If you allow high leverage you enable more lending to business which helps to grow your economy... but it does leave you more exposed if things go wrong. Restrict the allowable leverage too much and you have business collapsing left right and centre because they can't get the credit they need. I've heard lots of people saying that banks should only be able to leverage the actual deposits and capital they hold on a 1 for 1 basis but that's insane, the economy would collapse.

I think the real problam on leverage rules is that they're to simplistic. Saying to a bank "you can lend x times your assets" doesn't take any account of the riskiness of the loans they're making. It actually encourages the banks to make riskier loans and investments which offer a higher return because that means they're effectively leveraging more of their assets. A better piece of legislation would somehow take account of risk. Of course, that's difficult because risk is qualitative rather than quantitative... and round and round we go.

I think the number of values has to be greater than 30 for there to be real meaning in some of these models. It has to be hundreds or thousands, and they have to come from the same baseline. The crash in the 20s wasn't just about the banks being too small to cope, there also wasn't FDIC (the US version of deposit insurance that FD mentioned in his fretting about banking thread). Gone was gone at that time, so if you feared that your bank would fail you pretty much HAD to react, because if you were right then you'd lose everything. FDIC changed that, and models of behavior in bad times that existed before FDIC are no longer valid.

I thought they had already determined that the entire problem was caused by black holes????

Quote:

---

a program that can interpret text as a human would

---

So, wildly inaccurately and with a bias entirely in support of a predetermined view even when it clearly argues for the opposite?

Would you have it any other way?

Quote:

---

Originally Posted by Shaggy Hiker

The crash in the 20s wasn't just about the banks being too small to cope, there also wasn't FDIC (the US version of deposit insurance that FD mentioned in his fretting about banking thread). Gone was gone at that time, so if you feared that your bank would fail you pretty much HAD to react, because if you were right then you'd lose everything. FDIC changed that, and models of behavior in bad times that existed before FDIC are no longer valid.

---

I'm fuzzy on this, but I think another feature of the 20s was buying on margin,
excessive margin, at that (ie, leverage). My sense is that in the aftermath, this
was no longer allowed. Even today, I think, the most one can leverage a stock
transaction is 2-to-1.

Derivatives, however, are another matter. LTCM, in the late 90s, was leveraging
at 25-to-1, but I believe they were at least doing instruments that were listed
on exchanges. The type of CDOs that were being traded in 2005 and after, I think,
weren't even listed instruments .. no way to know what the "market values" were.

Quote:

---

Originally Posted by Spoo

Derivatives, however, are another matter. LTCM, in the late 90s, was leveraging
at 25-to-1, but I believe they were at least doing instruments that were listed
on exchanges. The type of CDOs that were being traded in 2005 and after, I think,
weren't even listed instruments .. no way to know what the "market values" were.

---

Although that was the primary reason:
http://www.youtube.com/watch?v=c8FqG8ss3e8

Just to be clear, I wasn't saying the size of the banks was the only reason for the crash, just that it was one of the factors that deepened the ensuing depression and that it's not a factor that would particularly apply today. Shaggy's sugestion of the lack of FDIC is another example and the buying on credit that Spoo identified was definitely one of the contributing factors that created the bubble in the first place. Witis, the site you linked to (I'm guessing YouTube) is blocked for me at work so I can't comment on that. There were a bunch of other factors too. Manipulation of the stock markets by industry insiders, a lack of legislation, a dip in the gold prices... all sorts. Change any one of those factors or introduce a new one and you'd get a different crash... or maybe no crash at all.

And that's the point really, there are a load of factors that come into play in how the stock market will behave and to create a perfect model you would have to somehow predict and model every single one of those factors perfectly. That's an impossible task, of course, so instead we accept that our models are imprefect. They're useful because they can make predictions that will mostly be true... or at least mostly true. But when we fall too in love with them and start to view them as infallible, which is what started to happen with the financial models that were so popular in the last decade or so, we put ourselves on a course for some disaster or other.

Quote:

---

Originally Posted by FunkyDexter

Witis, the site you linked to (I'm guessing YouTube) is blocked for me at work so I can't comment on that.

---

Click on the link in post 32 and wait till you get home for post 36!

Quote:

---

Click on the link in post 32 and wait till you get home for post 36!

---

Ah, yep that's part of the current crash, certainly (though not the 20s one). One thing that article stresses is that it wasn't the model as such that was at fault (though it did identify some failing in subsequent models). Rather it was the blind adherance to the model and a lack of understanding of it by the people who were using it. I think I'd agree with that.

As an engineer freind of mine is fond of saying: Simulation is like masturbation. Convenient and fun but no substitute for the real thing.

Quote:

---

Originally Posted by FunkyDexter

Ah, yep that's part of the current crash, certainly (though not the 20s one). One thing that article stresses is that it wasn't the model as such that was at fault (though it did identify some failing in subsequent models). Rather it was the blind adherance to the model and a lack of understanding of it by the people who were using it. I think I'd agree with that.

As an engineer freind of mine is fond of saying: Simulation is like masturbation. Convenient and fun but no substitute for the real thing.

---

It really did create a giant financial black hole, it didn't matter how much funding went in, none of it ever came out!