Reading template

[Satal Keto]

Hey all,

Someone asked me this question a while ago, I've been puzzling over how it would be possible to achieve it and have come to the conclusion that I wont be able to figure this out without getting some input from people.
We all know that there are many different types of forum (like vBulletin and IPB), now if you was to try and make an application that was to go through a forum and create a list of all the threads you would have to do some code that would parse the HTML of the page and this would have to be written for each of the different types of forum.
Can you guys think of any way of creating an application that could be generic enough that you could add a new config file and then the application could read a different format. Obviously it would be possible to use RSS but that sort of defeats the purpose of this intellectual exercise.

As I mentioned before this is more of an intellectual exercise than me actually trying to create something so if people say it can't be done then I'll accept that.

Thanks for any thoughts you guys may have on how the above scenario could be achieved.

Satal

[cicatrix]

You can create templates for a number of most widespread forums and update them as new versions appear, this will consume less time than trying to analyze the layout. I suppose that if you try hard enough you'll be able to achieve some results in this field (it's more akin to optical recognition though). Your app should guess whether this link is a subforum/thread title and not something else. If you throw out links that lead off-site (ads most probably) the rest can be considered more closely. Each link should be weighted against some criteria and calculate some quality function (i.e. the more properties of this link satisfy the criteria you define the better). For example there should be no less than 3-4 similar repetitions in the URL, they will most probably contain the word 'thread' in the link, etc. This algorithm will make mistakes but I don't think it's completely impossible, let's say it's incredibly difficult instead.
Still, if you're interested more in practical results than academic studies of AI it'll be more easy to provide a set of templates.

I wish they'd developed some Web 2.0 open standard for forums, but alas, forum owners won't appreciate that. Just think of the income from ads they will lose.

[Satal Keto]

Yeah I think the template route would be what I would have to look at as I don't think my current level of programming experience would allow me to do AI (although would definitely want to look at that some day).

Do you have any idea about how it might be possible to create a template? I could do it quite easily if there was a set number of threads per page, but that could be configurable so I'm not certain how that could be done.

I might actually have to try and create this to enhance my learning and test my programming abilities.

Well perhaps someone with some programming skills should do that cicatrix, that would be quite a USP if no other forum system is Web 2.0.

[cicatrix]

I'd started with PHPBB documentation. Or VBulletin (for this forum). Alternatively, simply view the HTML source of this forum (or some other, the more - the better). Try to find common distinctive features. I. e. links to forumdisplay.php will lead to forums while links to showthread.php will lead to individual threads. (for PHPBB 3.x it will be viewforum.php and viewtopic.php). But documentation is better.
A very basic template will consist of the names of two php scripts (for forum/topic).

[Satal Keto]

Haha brilliant, theres me thinking, ok I'll have to come up with some way of figuring out if that tr is a thread or just a random tr. Your ways much simpler

Thanks cicatrix

[UPDATE]
I would give you rep, but apparently I've given you too much