Results 1 to 40 of 40

Thread: The Great Ideas Thread

  1. #1

    Thread Starter
    Addicted Member
    Join Date
    Mar 2007
    Posts
    183

    The Great Ideas Thread

    Have an idea you think is great? Post it here and you just might get some valuable feedback.

    Here's an idea I think is great: a search engine that gives a lot of weight to Webpages that people say are relevant to what they're searching for. If the user finds what he or she is looking for, he or she can tell the search engine so with the click of a button.

    Here's another one: a program that can interpret text as a human would. So it could deduce information about dogs from the following passage: "The dog (canis) is an animal of family X. This animal is usually domestic. The average height is 4 '," or "This animal, averaging 4' in height, belongs to family X and is usually domestic."

    It could list this information like this:
    The dog
    -belongs to family X.
    -is usually found in homes
    -has an average height of 4 inches.

    Google has a large index, but for reasons known only to Google, its "best guess" feature only works for certain queries, even if the same question's been asked and answered on places like Yahoo! Answers.

    OK, yet another idea: a computer program that can use a set of data to make mathematical models that could be used as forecasts. For example, I could have that program analyze stock data and give me a forecast.
    Do not read this sentence.
    You read that last one, didn't you?
    Darn. You now read the previous two.

    Check out my pins on Pinterest!

  2. #2
    Frenzied Member TheBigB's Avatar
    Join Date
    Mar 2006
    Location
    *Stack Trace*
    Posts
    1,511

    Re: The Great Ideas Thread

    It's simple, we kill the moonman
    Delete it. They just clutter threads anyway.

  3. #3
    Fanatic Member namrekka's Avatar
    Join Date
    Feb 2005
    Location
    Netherlands
    Posts
    639

    Re: The Great Ideas Thread

    Quote Originally Posted by moonman239 View Post
    OK, yet another idea: a computer program that can use a set of data to make mathematical models that could be used as forecasts. For example, I could have that program analyze stock data and give me a forecast.
    Will it predict a crisis?

  4. #4

    Thread Starter
    Addicted Member
    Join Date
    Mar 2007
    Posts
    183

    Re: The Great Ideas Thread

    Quote Originally Posted by namrekka View Post
    Will it predict a crisis?
    That would be a human's job.

    I can think of other applications for such a program. Put it in a car and hook up some IR sensors and you could have yourself an automated car. Actually, Google has been testing car prototypes. Nevada said they'd give the cars a driver's license if they passed a driving test.
    Do not read this sentence.
    You read that last one, didn't you?
    Darn. You now read the previous two.

    Check out my pins on Pinterest!

  5. #5
    Super Moderator Shaggy Hiker's Avatar
    Join Date
    Aug 2002
    Location
    Idaho
    Posts
    38,988

    Re: The Great Ideas Thread

    Your ideas are pretty great. The first one sounds like what Google is trying to do with variable success (there is a competing pressure on the parts of various people to game the system to produce certain results, which will always be a problem if the product is valuable). The second one sounds a bit like Wolfram Alpha. I think it is within reach, but not here quite yet. As for the third idea, I wrote such a program a few years back, and still have it. Technically, it is that kind of thing that is developed by those called "quants" that are somewhat responsible for the recession, and are largely responsible for the flash crash of a few years back. As a program, the one I wrote is kind of interesting in that it creastes a population of models that are considerably better than random. The best of those is not a safe model, though, because that falls under the category of "data dredging", which is risky to rely on (though lots do). However, because there is a population of independently derived, excellent, models, you can study the population to look for similarities. The similarities between the various models shows something, at the very least, about the data used to form the models. It may also show something about the system the model is used for.

    The basic plan behind the program is that all math equations are nothing more than strings of characters that code for a certain pattern of result (the model curve itself) when given certain inputs. Therefore, I took all the variables that seemed to impact the system and built random equations using them. I then tested each model against the observed pattern and evaluated how well they predicted the observed pattern. This allowed me to rank the equations based on how well they did, which is all you need for a genetic algorithm. So, the program was using evolution to evolve the best equation it could to predict the observed pattern based on the variables input. There's more to it than that, as I have improved it in various ways, but it works pretty well.
    My usual boring signature: Nothing

  6. #6
    Fanatic Member namrekka's Avatar
    Join Date
    Feb 2005
    Location
    Netherlands
    Posts
    639

    Re: The Great Ideas Thread

    What about a "super-high-power-WiFi-router-thing". So we can connect the alien internet to ours. A sort of universe internet. Then I can add new friends to my Facebook and dating all arround the universe. Wow........

  7. #7
    PowerPoster SJWhiteley's Avatar
    Join Date
    Feb 2009
    Location
    South of the Mason-Dixon Line
    Posts
    2,256

    Re: The Great Ideas Thread

    Quote Originally Posted by moonman239 View Post
    ...
    Google has a large index, but for reasons known only to Google, its "best guess" feature only works for certain queries, even if the same question's been asked and answered on places like Yahoo! Answers.

    ...
    Because a question has been asked, and that it has been asked, does not reveal anything useful about the question and the answer.

    In other words, there are stupid questions, and there are stupid answers, which for some bizarre reason, seem to be an 'acceptable answer'. It's not called Yahoo! for nothing. As much as I hate stackoverflow, at least their philosophy is to push the correct answer (and, yes, there are most often times a 'correct' answer).

    With regards to 'interpreting text as a human would', i know it's not what you are mentioning, but I thing the iPhone app, WordLens is the freaking coolest translation app ever - it translates text using the live camera, replacing the text with translated text.

    As far as ideas go, well, I have a bunch, but I'm too busy fixing all the broken 'ideas'...
    "Ok, my response to that is pending a Google search" - Bucky Katt.
    "There are two types of people in the world: Those who can extrapolate from incomplete data sets." - Unk.
    "Before you can 'think outside the box' you need to understand where the box is."

  8. #8
    Angel of Code Niya's Avatar
    Join Date
    Nov 2011
    Posts
    8,598

    Re: The Great Ideas Thread

    Quote Originally Posted by SJWhiteley View Post
    As much as I hate stackoverflow....
    This is about the umpteenth time I've heard someone here throw a jab at StackOverflow. Is there some history between their site and VBForums I'm not aware of ? I am getting the impression that StackOverflow is to VBForums what Christianity is to Atheism.
    Treeview with NodeAdded/NodesRemoved events | BlinkLabel control | Calculate Permutations | Object Enums | ComboBox with centered items | .Net Internals article(not mine) | Wizard Control | Understanding Multi-Threading | Simple file compression | Demon Arena

    Copy/move files using Windows Shell | I'm not wanted

    C++ programmers will dismiss you as a cretinous simpleton for your inability to keep track of pointers chained 6 levels deep and Java programmers will pillory you for buying into the evils of Microsoft. Meanwhile C# programmers will get paid just a little bit more than you for writing exactly the same code and VB6 programmers will continue to whitter on about "footprints". - FunkyDexter

    There's just no reason to use garbage like InputBox. - jmcilhinney

    The threads I start are Niya and Olaf free zones. No arguing about the benefits of VB6 over .NET here please. Happiness must reign. - yereverluvinuncleber

  9. #9
    Powered By Medtronic dbasnett's Avatar
    Join Date
    Dec 2007
    Location
    Jefferson City, MO
    Posts
    9,754

    Re: The Great Ideas Thread

    Quote Originally Posted by Niya View Post
    This is about the umpteenth time I've heard someone here throw a jab at StackOverflow. Is there some history between their site and VBForums I'm not aware of ? I am getting the impression that StackOverflow is to VBForums what Christianity is to Atheism.
    I don't think it is between VBForums and StackOverflow. I do think that StackOverflow comes off as zealots, who often appear more interested in form than content. One thing that angers me is to see how many C# answers are given for VB questions, but if you give a VB answer to a C# question you get hammered.
    My First Computer -- Documentation Link (RT?M) -- Using the Debugger -- Prime Number Sieve
    Counting Bits -- Subnet Calculator -- UI Guidelines -- >> SerialPort Answer <<

    "Those who use Application.DoEvents have no idea what it does and those who know what it does never use it." John Wein

  10. #10
    Angel of Code Niya's Avatar
    Join Date
    Nov 2011
    Posts
    8,598

    Re: The Great Ideas Thread

    Sounds like a rough culture. While I don't participate there as I do here, I do find about 80% of the answers I need on the more in-depth and highly technical issues regarding programming over there via Google. I never actually noticed the nuances of their online culture.
    Treeview with NodeAdded/NodesRemoved events | BlinkLabel control | Calculate Permutations | Object Enums | ComboBox with centered items | .Net Internals article(not mine) | Wizard Control | Understanding Multi-Threading | Simple file compression | Demon Arena

    Copy/move files using Windows Shell | I'm not wanted

    C++ programmers will dismiss you as a cretinous simpleton for your inability to keep track of pointers chained 6 levels deep and Java programmers will pillory you for buying into the evils of Microsoft. Meanwhile C# programmers will get paid just a little bit more than you for writing exactly the same code and VB6 programmers will continue to whitter on about "footprints". - FunkyDexter

    There's just no reason to use garbage like InputBox. - jmcilhinney

    The threads I start are Niya and Olaf free zones. No arguing about the benefits of VB6 over .NET here please. Happiness must reign. - yereverluvinuncleber

  11. #11
    PowerPoster Spoo's Avatar
    Join Date
    Nov 2008
    Location
    Right Coast
    Posts
    2,656

    Re: The Great Ideas Thread

    Quote Originally Posted by Shaggy Hiker View Post
    As for the third idea, I wrote such a program a few years back, and still have it. Technically, it is that kind of thing that is developed by those called "quants" that are somewhat responsible for the recession, and are largely responsible for the flash crash of a few years back.
    I have a hard time reconciling those allegations with
    the notion that for every seller, there has to be a buyer.

  12. #12
    Super Moderator Shaggy Hiker's Avatar
    Join Date
    Aug 2002
    Location
    Idaho
    Posts
    38,988

    Re: The Great Ideas Thread

    What allegations?

    The stock market is based not so much on willing sellers and willing buyers, but the price the seller is willing to sell at and the price the buyer is willing to buy at. If you take the housing crash, there were plenty of willing buyers, they just weren't willing to buy at the price the seller was willing to sell at. In an automated system, if you say, "when x happens, sell Y.", there is always a chance of a feedback loop. You need to be saying, "when x happens, sell Y unless the price is below z.", but that's a prospect with its own dangers.

    In any case, the program I wrote created models, but it did it through a process of selective data dredging. The resulting models always did a good job of fitting a curve to whatever data was available. Most people don't want to know anything about models other than their curves, but there's more to them than that. The problem is that every model has a set of assumptions for which it is true, and a set of assumptions for which it is false. All I was doing was creating a good models for a particular set of assumptions. Not only would those models be unpredictable if the assumptions changed, it also wasn't possible to be entirely certain what the assumptions behind the models were, so it wouldn't be possible to predict when the models were no longer viable.
    My usual boring signature: Nothing

  13. #13
    PowerPoster Spoo's Avatar
    Join Date
    Nov 2008
    Location
    Right Coast
    Posts
    2,656

    Re: The Great Ideas Thread

    Allegation 1 - "quants" are somewhat responsible for the recession
    Allegation 2 - "quants" are largely responsible for the flash crash

    "Allegation" may have been a tad aggressive, but the implications were
    certainly present. I don't know the answer, but I'm just a little leary with
    those as explanations.

    In any case, the approach you described regarding a system consisting of "a population
    of models" is, indeed, interesting.

    While it may not be the same as your approach, I've heard of something referred to as "training"
    the a system on "recent" data. "Curve fitting" to prior historical data is indeed a pitfall, and from
    what I understand, this sort of "training" is aimed at minimizing the curve fitting dilemma.

    As for no longer knowing if the model is viable ,,, what if the model was, say, optimized for a given
    time-frame ,, also known as "in-sample". The model is then tested on "new" data ,, aka "out-of-sample."
    Would that possibly answer the question?

  14. #14
    Super Moderator Shaggy Hiker's Avatar
    Join Date
    Aug 2002
    Location
    Idaho
    Posts
    38,988

    Re: The Great Ideas Thread

    It would improve our understanding of the problem, but it would never answer the question. Every model has a set of assumptions, we just don't always know whether we know what all the assumptions are. I would certainly use that approach, and see what happened to models in different time segments. However, we can never be sure what all the conditions were for any one time segment, and we can never be sure what will happen tomorrow. As an extreme example: If a sizeable asteroid took out either the US or China tomorrow, it would almost certainly impact our stock portfolios in ways that no model would accurately predict...unless we were in the impact zone, cause I can predict what THAT would do to my portfolio pretty well.

    As for the quants being responsible for the flash crash, it may be a question of what is a quant. I was counting those who are creating advanced statistical models to guide computerized, automated, buying and selling of stocks. The flash crash was too fast to have been implemented by humans, though we did do it ultimately, since humans wrote the programs that performed the actual selling. Whether such quants had a pivotal role in the recession is a bit harder to say, but I think they did. I believe they drove purchasing based on models that didn't take into account the impact of a sudden softening of the housing market. I believe that they used models that were based on an assumption of housing market dynamics that had held true for most of the previous three to seven decades, and the models encouraged people to take overly exposed positions which turned devastating when the assumptions of the model were violated. Not all models got it wrong, though, just some of the larger ones, but that was enough to cause consternation throughout the top of the economy, and consternation flows downhill faster than good times.
    My usual boring signature: Nothing

  15. #15
    PowerPoster Spoo's Avatar
    Join Date
    Nov 2008
    Location
    Right Coast
    Posts
    2,656

    Re: The Great Ideas Thread

    I'm a little puzzled by the notion that "we don't know what all the assumptions are".
    Could you elaborate?

    And just to muddy the waters a bit, why the emphasis on "prediction"? True, some
    notion of prediction is involved in devising and/or accepting a signal to open a trade,
    but couldn't a statistical analysis of such a signal be sufficient to accept/reject it?
    Seemingly, one would want at least 30 instances for it to have any significance.

    Further, I agree that one could not predict extremes like the one you noted. But, wouldn't
    it be sufficient to have a plan of how to "react"? Translated, that would be either to
    hold or close the position.

  16. #16
    Super Moderator Shaggy Hiker's Avatar
    Join Date
    Aug 2002
    Location
    Idaho
    Posts
    38,988

    Re: The Great Ideas Thread

    The problem with real-world models like that is that the assumptions are very hard to fathom. One trivial assumption for the housing models was that house prices would always rise. Obviously, people knew that trivial statement was wrong, but they assumed that when house prices dropped it was just a short-term fluctuation. On a more fundamental level, the assumption would be: People want to own houses. If that assumption ever changed, which it could, then most of the underlying principles of building houses would shift in unpredictable ways. When it comes to houses, that's a bit hard to see, but consider any models that were based on horse usage back in the late 1800s. People could make economic decisions based on the increasing number of people, the demand of people for horses for such things as transportation, and based on that you could predict the need for certain industry and the growth in a few sectors. Then, in a relatively short time frame, a new invention wrecked all that planning as people moved rapidly to a totally different means of transportation. That's the bigger problem with these models: We base them on what has happened in the past and how we expect the future to follow from the past and the present, but there are inventions and events that can take the future in radically different directions than the past. Since we can't know what those events will be, we can't model them, and all the models we currently have could be wrecked by a change we couldn't possibly have forseen.
    My usual boring signature: Nothing

  17. #17
    Lively Member homer13j's Avatar
    Join Date
    Nov 2003
    Location
    Where the dirt bikes and ATVs play
    Posts
    80

    Re: The Great Ideas Thread

    Well, one thing is certain: a Bad Ideas Thread would have been far more entertaining.

  18. #18
    PowerPoster Spoo's Avatar
    Join Date
    Nov 2008
    Location
    Right Coast
    Posts
    2,656

    Re: The Great Ideas Thread

    Shaggy

    True, dat.

    But, I would suggest that your "parameters" are qualitative (eg, desire for housing, possibility of a new invention)
    What if your "parameters" were quantitative (eg, price, time)

  19. #19
    PowerPoster abhijit's Avatar
    Join Date
    Jun 1999
    Location
    Chit Chat Forum.
    Posts
    3,226

    Re: The Great Ideas Thread

    Quote Originally Posted by Niya View Post
    Sounds like a rough culture. While I don't participate there as I do here, I do find about 80% of the answers I need on the more in-depth and highly technical issues regarding programming over there via Google. I never actually noticed the nuances of their online culture.

    That's one of the reasons I think of Stackoverflow as a club for people who actually bother to read the manual.
    Everything that has a computer in will fail. Everything in your life, from a watch to a car to, you know, a radio, to an iPhone, it will fail if it has a computer in it. They should kill the people who made those things.- 'Woz'
    save a blobFileStreamDataTable To Text Filemy blog

  20. #20
    Angel of Code Niya's Avatar
    Join Date
    Nov 2011
    Posts
    8,598

    Re: The Great Ideas Thread

    Quote Originally Posted by abhijit View Post
    That's one of the reasons I think of Stackoverflow as a club for people who actually bother to read the manual.
    Well thank God that some people take the time to do so . Their site layout is really confusing and not very intuitive but the information is really quite good. I found quite a number of very interesting insights and links over there that have helped me solved a wide range of issues.
    Treeview with NodeAdded/NodesRemoved events | BlinkLabel control | Calculate Permutations | Object Enums | ComboBox with centered items | .Net Internals article(not mine) | Wizard Control | Understanding Multi-Threading | Simple file compression | Demon Arena

    Copy/move files using Windows Shell | I'm not wanted

    C++ programmers will dismiss you as a cretinous simpleton for your inability to keep track of pointers chained 6 levels deep and Java programmers will pillory you for buying into the evils of Microsoft. Meanwhile C# programmers will get paid just a little bit more than you for writing exactly the same code and VB6 programmers will continue to whitter on about "footprints". - FunkyDexter

    There's just no reason to use garbage like InputBox. - jmcilhinney

    The threads I start are Niya and Olaf free zones. No arguing about the benefits of VB6 over .NET here please. Happiness must reign. - yereverluvinuncleber

  21. #21
    PowerPoster Spoo's Avatar
    Join Date
    Nov 2008
    Location
    Right Coast
    Posts
    2,656

    Re: The Great Ideas Thread

    Quote Originally Posted by Niya View Post
    Their site layout is really confusing and not very intuitive
    That is an understatement, if ever I saw one.

  22. #22
    Angel of Code Niya's Avatar
    Join Date
    Nov 2011
    Posts
    8,598

    Re: The Great Ideas Thread

    lol...Well I don't wanna be too mean
    Treeview with NodeAdded/NodesRemoved events | BlinkLabel control | Calculate Permutations | Object Enums | ComboBox with centered items | .Net Internals article(not mine) | Wizard Control | Understanding Multi-Threading | Simple file compression | Demon Arena

    Copy/move files using Windows Shell | I'm not wanted

    C++ programmers will dismiss you as a cretinous simpleton for your inability to keep track of pointers chained 6 levels deep and Java programmers will pillory you for buying into the evils of Microsoft. Meanwhile C# programmers will get paid just a little bit more than you for writing exactly the same code and VB6 programmers will continue to whitter on about "footprints". - FunkyDexter

    There's just no reason to use garbage like InputBox. - jmcilhinney

    The threads I start are Niya and Olaf free zones. No arguing about the benefits of VB6 over .NET here please. Happiness must reign. - yereverluvinuncleber

  23. #23
    Frenzied Member TheBigB's Avatar
    Join Date
    Mar 2006
    Location
    *Stack Trace*
    Posts
    1,511

    Re: The Great Ideas Thread

    Quote Originally Posted by Niya View Post
    Their site layout is really confusing and not very intuitive but the information is really quite good.
    Really? What don't find intuitive about it?
    Delete it. They just clutter threads anyway.

  24. #24
    Angel of Code Niya's Avatar
    Join Date
    Nov 2011
    Posts
    8,598

    Re: The Great Ideas Thread

    Quote Originally Posted by TheBigB View Post
    Really? What don't find intuitive about it?
    Well for one thing, when I came here, I didn't have to actually think about how to sign up. I have an account over there and I distinctly remember being very confused about how to sign up or even if they require it. I remember looking all over the place to find where you had to sign up. vBulletin and every other forums software makes that very easy to find.

    The layout of the site itself reminds me of what happens sometimes in FireFox when the CSS for a site fails to load. It looks unorganized. It makes it pain in the ass to navigate. I can never figure out where anything is. Its layout seems hastily designed and usability suffers, at least for me. They should have used a forum format or something like it.
    Treeview with NodeAdded/NodesRemoved events | BlinkLabel control | Calculate Permutations | Object Enums | ComboBox with centered items | .Net Internals article(not mine) | Wizard Control | Understanding Multi-Threading | Simple file compression | Demon Arena

    Copy/move files using Windows Shell | I'm not wanted

    C++ programmers will dismiss you as a cretinous simpleton for your inability to keep track of pointers chained 6 levels deep and Java programmers will pillory you for buying into the evils of Microsoft. Meanwhile C# programmers will get paid just a little bit more than you for writing exactly the same code and VB6 programmers will continue to whitter on about "footprints". - FunkyDexter

    There's just no reason to use garbage like InputBox. - jmcilhinney

    The threads I start are Niya and Olaf free zones. No arguing about the benefits of VB6 over .NET here please. Happiness must reign. - yereverluvinuncleber

  25. #25
    PowerPoster Spoo's Avatar
    Join Date
    Nov 2008
    Location
    Right Coast
    Posts
    2,656

    Re: The Great Ideas Thread

    Shaggy

    If you hadn't already heard, this should warm the cockles of your heart ...

    Just after 1:00 PM Eastern, the DOW dropped 150 pts and then fully recovered
    in 5 minutes as a result of a fake Twitter post on AP that a bomb had gone off
    at the White House .. the AP account apparently was hacked.

    So, even quantitative approaches have to deal with Black Swan events ..

  26. #26
    Super Moderator FunkyDexter's Avatar
    Join Date
    Apr 2005
    Location
    An obscure body in the SK system. The inhabitants call it Earth
    Posts
    7,900

    Re: The Great Ideas Thread

    But, I would suggest that your "parameters" are qualitative (eg, desire for housing, possibility of a new invention)
    What if your "parameters" were quantitative (eg, price, time)
    You don't really have a choice if your dealing with "soft" subjects like market economics. The truth is that many of the parameters ARE qualitative and you need to find a way to deal with them if you're going to accurately model a system to the point where you can make useful predictions. At that point you've basically got two broad choices:-
    1. Try to find quantitatives ways of expressing your qualitative parameters (but that's really just introducing a new layer of modeling)
    2. Ignore the qualitative parameters entirely and only deal with those which can be quantified

    From what I've heard in the media, poor understanding of and/or choosing to just ignore the qualitative parameters that were feeeding into the housing market and stock exchange were a large part of why so many of the models failed. They tended to just mine raw data looking for patterns and assumed those patterns would repeat. That's a flawed aproach because it carries the implicit assumption that people and the groups they form are rational and will take the same action when faced with the same circumstances twice. On the whole they are rational but occasionally they're not and when they get irrational enough it collapses the model... and the economy in this case.

    The other major contributor (again, only from what I've read) was that they didn't draw their dataset from far enough back. In hindsight you'd have thought that someone datamining to find patterns in market behaviour would have included data from 1900 onwards so that they had a decent sized crash or two in the mix but from what I understand, most datasets only went back a decade or so. I don't know the reasons for that but suspect it was to do with data availability. Either way, it meant that they were making prediction based on models that had been built on assumptions drawn from a period of continuous growth. The models weren't capable of predicting a bubble bursting because the models had never "seen" a bubble bursting. They couldn't predict the outcome of a crash because they'd never "seen" a crash.

    You can take that further as well. Even if the data to include a crash had been included there's a very good chance that the models would have failed to predict this crash because they'd never seen this crash. Every crash is different. Indeed, a major exacerbating factor of the 1920s crash was not that the banks were too big to fail, they were too small to cope. There were hundreds of small banks so a failure of one had less impact, but each was individually far less able to soak a loss than todays giants, making runs far more likely. Actually, the banking crisis ended up being FAR worse in the 20s because the banks were smaller than they are today, which makes me raise an eyebrow when I hear politicians and the media demanding that we break them up. A model that looked at the 20s data and said "this is how people and their banks behave in a crisis" would have drawn a multitude of false conclusions.
    The best argument against democracy is a five minute conversation with the average voter - Winston Churchill

    Hadoop actually sounds more like the way they greet each other in Yorkshire - Inferrd

  27. #27
    PowerPoster Spoo's Avatar
    Join Date
    Nov 2008
    Location
    Right Coast
    Posts
    2,656

    Re: The Great Ideas Thread

    Funky

    What you suggest rings true.

    But I can't help but return to a precept that I recall from a Stats course I
    took in grad school .. in a normal distribution, one needs at least 30 samples
    (ie, examples, instances, etc) for a "prediction" to have any statistical significance.

    One or two examples is clearly too few, so models based on such a low number
    would seem to be doomed to failure.

  28. #28
    Super Moderator FunkyDexter's Avatar
    Join Date
    Apr 2005
    Location
    An obscure body in the SK system. The inhabitants call it Earth
    Posts
    7,900

    Re: The Great Ideas Thread

    I think that depends on what you're modelling but it sounds like a pretty reasonable place to start if you're looking at just one or two parameters. But as you increase the number of parameters you need a bigger dataset and when you're dealing with something as complex as an economy the number of possible parameters is huge so the dataset has to run into the millions to be reliable.

    That wasn't really the problem though. The guys doing the datamining were probably running tests across billions of transactions. The problem was that the transactions were all drawn from an unrepresentative sub set. They were taken across a period of unremitting growth. It's a bit like sticking ginger kids in the sun and using the data to predict how long it will take brunettes to get a sun tan.

    Personally I think it's impossible to accurately predict something as complex as an economy. You hit on an interesting point in post 15 when you asked "how would you react?" That's the bit that was missing really. Economists fell so in love with their statistical models that they began to believe they could never be wrong and banks and investment houses bought into that. As a consequence they failed to put contingency plans in place for what seems like, in hindsight, to be an inevitable outcome. I.e. Continued growth above inflation ALWAYS means a bubble and ALL bubbles burst. That's just common sense, really, but it's an unpleasant bit of common sense to acknowledge if you're bonus at the end of the month depends on the growth carrying on.
    The best argument against democracy is a five minute conversation with the average voter - Winston Churchill

    Hadoop actually sounds more like the way they greet each other in Yorkshire - Inferrd

  29. #29
    PowerPoster Spoo's Avatar
    Join Date
    Nov 2008
    Location
    Right Coast
    Posts
    2,656

    Re: The Great Ideas Thread

    That wasn't really the problem though. The guys doing the datamining were probably running tests across billions of transactions. The problem was that the transactions were all drawn from an unrepresentative sub set. They were taken across a period of unremitting growth.
    Fair enough.
    And to make matters worse, they were probably overly leveraged.
    Put up $1, borrow $25.
    If you have a small win, the return is magnified by a factor of 25.
    But, if you have an unexpectly large loss, you are wiped out.

    they failed to put contingency plans in place for what seems like, in hindsight, to be an inevitable outcome.
    Spot on. I've heard it referred to as an Exit Strategy.

  30. #30
    Super Moderator FunkyDexter's Avatar
    Join Date
    Apr 2005
    Location
    An obscure body in the SK system. The inhabitants call it Earth
    Posts
    7,900

    Re: The Great Ideas Thread

    And to make matters worse, they were probably overly leveraged.
    Put up $1, borrow $25.
    If you have a small win, the return is magnified by a factor of 25.
    But, if you have an unexpectly large loss, you are wiped out.
    I think you're probably right but this one's a tricky balancing act. If you allow high leverage you enable more lending to business which helps to grow your economy... but it does leave you more exposed if things go wrong. Restrict the allowable leverage too much and you have business collapsing left right and centre because they can't get the credit they need. I've heard lots of people saying that banks should only be able to leverage the actual deposits and capital they hold on a 1 for 1 basis but that's insane, the economy would collapse.

    I think the real problam on leverage rules is that they're to simplistic. Saying to a bank "you can lend x times your assets" doesn't take any account of the riskiness of the loans they're making. It actually encourages the banks to make riskier loans and investments which offer a higher return because that means they're effectively leveraging more of their assets. A better piece of legislation would somehow take account of risk. Of course, that's difficult because risk is qualitative rather than quantitative... and round and round we go.
    The best argument against democracy is a five minute conversation with the average voter - Winston Churchill

    Hadoop actually sounds more like the way they greet each other in Yorkshire - Inferrd

  31. #31
    Super Moderator Shaggy Hiker's Avatar
    Join Date
    Aug 2002
    Location
    Idaho
    Posts
    38,988

    Re: The Great Ideas Thread

    I think the number of values has to be greater than 30 for there to be real meaning in some of these models. It has to be hundreds or thousands, and they have to come from the same baseline. The crash in the 20s wasn't just about the banks being too small to cope, there also wasn't FDIC (the US version of deposit insurance that FD mentioned in his fretting about banking thread). Gone was gone at that time, so if you feared that your bank would fail you pretty much HAD to react, because if you were right then you'd lose everything. FDIC changed that, and models of behavior in bad times that existed before FDIC are no longer valid.
    My usual boring signature: Nothing

  32. #32
    Addicted Member Witis's Avatar
    Join Date
    Jan 2011
    Location
    VB Forums Online Freedom Mode: Operational
    Posts
    213

    Re: The Great Ideas Thread

    I thought they had already determined that the entire problem was caused by black holes????
    All men have an inherent right to life, the right to self determination including freedom from forced or compulsory labour, a right to hold opinions and the freedom of expression, and the right to a fair trial and freedom from torture. Be aware that these rights are universal and inalienable (cannot be given, taken or otherwise transferred or removed) although you do risk losing the aforementioned rights should you fail to uphold them e.g Charles Taylor; United Nations sources: http://www.un.org/en/documents/udhr/, http://www.ohchr.org/EN/Professional...ages/CCPR.aspx. Also Charles I was beheaded on the 30th of January of 1649 for trying to replace parliamentary democracy with an absolute monarchy, the same should happen to Dr Phil and Stephen Fry; source: http://www.vbforums.com/showthread.p...ute-Monarchism.

    The plural of sun is stars you Catholic turkeys.

  33. #33
    PowerPoster dunfiddlin's Avatar
    Join Date
    Jun 2012
    Posts
    8,245

    Re: The Great Ideas Thread

    a program that can interpret text as a human would
    So, wildly inaccurately and with a bias entirely in support of a predetermined view even when it clearly argues for the opposite?
    As the 6-dimensional mathematics professor said to the brain surgeon, "It ain't Rocket Science!"

    Reviews: "dunfiddlin likes his DataTables" - jmcilhinney

    Please be aware that whilst I will read private messages (one day!) I am unlikely to reply to anything that does not contain offers of cash, fame or marriage!

  34. #34
    Super Moderator Shaggy Hiker's Avatar
    Join Date
    Aug 2002
    Location
    Idaho
    Posts
    38,988

    Re: The Great Ideas Thread

    Would you have it any other way?
    My usual boring signature: Nothing

  35. #35
    PowerPoster Spoo's Avatar
    Join Date
    Nov 2008
    Location
    Right Coast
    Posts
    2,656

    Re: The Great Ideas Thread

    Quote Originally Posted by Shaggy Hiker View Post
    The crash in the 20s wasn't just about the banks being too small to cope, there also wasn't FDIC (the US version of deposit insurance that FD mentioned in his fretting about banking thread). Gone was gone at that time, so if you feared that your bank would fail you pretty much HAD to react, because if you were right then you'd lose everything. FDIC changed that, and models of behavior in bad times that existed before FDIC are no longer valid.
    I'm fuzzy on this, but I think another feature of the 20s was buying on margin,
    excessive margin, at that (ie, leverage). My sense is that in the aftermath, this
    was no longer allowed. Even today, I think, the most one can leverage a stock
    transaction is 2-to-1.

    Derivatives, however, are another matter. LTCM, in the late 90s, was leveraging
    at 25-to-1, but I believe they were at least doing instruments that were listed
    on exchanges. The type of CDOs that were being traded in 2005 and after, I think,
    weren't even listed instruments .. no way to know what the "market values" were.

  36. #36
    Addicted Member Witis's Avatar
    Join Date
    Jan 2011
    Location
    VB Forums Online Freedom Mode: Operational
    Posts
    213

    Re: The Great Ideas Thread

    Quote Originally Posted by Spoo View Post
    Derivatives, however, are another matter. LTCM, in the late 90s, was leveraging
    at 25-to-1, but I believe they were at least doing instruments that were listed
    on exchanges. The type of CDOs that were being traded in 2005 and after, I think,
    weren't even listed instruments .. no way to know what the "market values" were.
    Although that was the primary reason:
    All men have an inherent right to life, the right to self determination including freedom from forced or compulsory labour, a right to hold opinions and the freedom of expression, and the right to a fair trial and freedom from torture. Be aware that these rights are universal and inalienable (cannot be given, taken or otherwise transferred or removed) although you do risk losing the aforementioned rights should you fail to uphold them e.g Charles Taylor; United Nations sources: http://www.un.org/en/documents/udhr/, http://www.ohchr.org/EN/Professional...ages/CCPR.aspx. Also Charles I was beheaded on the 30th of January of 1649 for trying to replace parliamentary democracy with an absolute monarchy, the same should happen to Dr Phil and Stephen Fry; source: http://www.vbforums.com/showthread.p...ute-Monarchism.

    The plural of sun is stars you Catholic turkeys.

  37. #37
    Super Moderator FunkyDexter's Avatar
    Join Date
    Apr 2005
    Location
    An obscure body in the SK system. The inhabitants call it Earth
    Posts
    7,900

    Re: The Great Ideas Thread

    Just to be clear, I wasn't saying the size of the banks was the only reason for the crash, just that it was one of the factors that deepened the ensuing depression and that it's not a factor that would particularly apply today. Shaggy's sugestion of the lack of FDIC is another example and the buying on credit that Spoo identified was definitely one of the contributing factors that created the bubble in the first place. Witis, the site you linked to (I'm guessing YouTube) is blocked for me at work so I can't comment on that. There were a bunch of other factors too. Manipulation of the stock markets by industry insiders, a lack of legislation, a dip in the gold prices... all sorts. Change any one of those factors or introduce a new one and you'd get a different crash... or maybe no crash at all.

    And that's the point really, there are a load of factors that come into play in how the stock market will behave and to create a perfect model you would have to somehow predict and model every single one of those factors perfectly. That's an impossible task, of course, so instead we accept that our models are imprefect. They're useful because they can make predictions that will mostly be true... or at least mostly true. But when we fall too in love with them and start to view them as infallible, which is what started to happen with the financial models that were so popular in the last decade or so, we put ourselves on a course for some disaster or other.
    The best argument against democracy is a five minute conversation with the average voter - Winston Churchill

    Hadoop actually sounds more like the way they greet each other in Yorkshire - Inferrd

  38. #38
    Addicted Member Witis's Avatar
    Join Date
    Jan 2011
    Location
    VB Forums Online Freedom Mode: Operational
    Posts
    213

    Re: The Great Ideas Thread

    Quote Originally Posted by FunkyDexter View Post
    Witis, the site you linked to (I'm guessing YouTube) is blocked for me at work so I can't comment on that.
    Click on the link in post 32 and wait till you get home for post 36!
    All men have an inherent right to life, the right to self determination including freedom from forced or compulsory labour, a right to hold opinions and the freedom of expression, and the right to a fair trial and freedom from torture. Be aware that these rights are universal and inalienable (cannot be given, taken or otherwise transferred or removed) although you do risk losing the aforementioned rights should you fail to uphold them e.g Charles Taylor; United Nations sources: http://www.un.org/en/documents/udhr/, http://www.ohchr.org/EN/Professional...ages/CCPR.aspx. Also Charles I was beheaded on the 30th of January of 1649 for trying to replace parliamentary democracy with an absolute monarchy, the same should happen to Dr Phil and Stephen Fry; source: http://www.vbforums.com/showthread.p...ute-Monarchism.

    The plural of sun is stars you Catholic turkeys.

  39. #39
    Super Moderator FunkyDexter's Avatar
    Join Date
    Apr 2005
    Location
    An obscure body in the SK system. The inhabitants call it Earth
    Posts
    7,900

    Re: The Great Ideas Thread

    Click on the link in post 32 and wait till you get home for post 36!
    Ah, yep that's part of the current crash, certainly (though not the 20s one). One thing that article stresses is that it wasn't the model as such that was at fault (though it did identify some failing in subsequent models). Rather it was the blind adherance to the model and a lack of understanding of it by the people who were using it. I think I'd agree with that.

    As an engineer freind of mine is fond of saying: Simulation is like masturbation. Convenient and fun but no substitute for the real thing.
    The best argument against democracy is a five minute conversation with the average voter - Winston Churchill

    Hadoop actually sounds more like the way they greet each other in Yorkshire - Inferrd

  40. #40
    Addicted Member Witis's Avatar
    Join Date
    Jan 2011
    Location
    VB Forums Online Freedom Mode: Operational
    Posts
    213

    Re: The Great Ideas Thread

    Quote Originally Posted by FunkyDexter View Post
    Ah, yep that's part of the current crash, certainly (though not the 20s one). One thing that article stresses is that it wasn't the model as such that was at fault (though it did identify some failing in subsequent models). Rather it was the blind adherance to the model and a lack of understanding of it by the people who were using it. I think I'd agree with that.

    As an engineer freind of mine is fond of saying: Simulation is like masturbation. Convenient and fun but no substitute for the real thing.
    It really did create a giant financial black hole, it didn't matter how much funding went in, none of it ever came out!
    All men have an inherent right to life, the right to self determination including freedom from forced or compulsory labour, a right to hold opinions and the freedom of expression, and the right to a fair trial and freedom from torture. Be aware that these rights are universal and inalienable (cannot be given, taken or otherwise transferred or removed) although you do risk losing the aforementioned rights should you fail to uphold them e.g Charles Taylor; United Nations sources: http://www.un.org/en/documents/udhr/, http://www.ohchr.org/EN/Professional...ages/CCPR.aspx. Also Charles I was beheaded on the 30th of January of 1649 for trying to replace parliamentary democracy with an absolute monarchy, the same should happen to Dr Phil and Stephen Fry; source: http://www.vbforums.com/showthread.p...ute-Monarchism.

    The plural of sun is stars you Catholic turkeys.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width