FORUM ISSUE: Speed

[brad jones]

Since the update, the forum has had times where the speed has been slow. We are a aware of this issue and are continuing to make tweaks and changes to the forum to fix the speed issues.

At this time, we've even brought in the vBulletin support folks to help identify what the issues are so as to get the forum back up to an acceptable level of speed.

Thank you everyone for the feedback on this (and other) issues. I wanted to make this post so that you would know we are still continuing to work on this issue.

I'll update this thread as I learn more on what is happening regarding the speed issue. Please use the other thread for any other forum issues.

Sincerely,

Brad!

[Doogle]

@Brad: From my perspective the issue is not one of absolute speed, rather, it's the inconsistancy of responses. I can be happily pottering away and then suddenly experience a 10-20 second delay often followed by a 'Server Timed Out' or 'Database Error'. More often than not the extended response time issue lasts for about 5 - 10 minutes and then returns to 'normal' (ie acceptable). Also, when replying to a Thread it appears to 'hang' but on re-opening the Thread the reply has actually been posted - I guess that's why some duplicates are happening, people may be clicking on the 'Post Reply' again after waiting a minute or so.

I am sure that those of us who have had to deal with these sorts of problems in our Professional lives understand just how difficult they are to sort out and unless it's something 'really silly' will take some time to resolve. The most recent one I had to deal with was tracked down to someone having connected 3 servers to an old 10 Base T Ethernet switch instead of the Gigabit switch (That went into the 'silliest of silly' folder!)

[Nightwalker83]

@Brad: From my perspective the issue is not one of absolute speed, rather, it's the inconsistancy of responses. I can be happily pottering away and then suddenly experience a 10-20 second delay often followed by a 'Server Timed Out' or 'Database Error'. More often than not the extended response time issue lasts for about 5 - 10 minutes and then returns to 'normal' (ie acceptable). Also, when replying to a Thread it appears to 'hang' but on re-opening the Thread the reply has actually been posted - I guess that's why some duplicates are happening, people may be clicking on the 'Post Reply' again after waiting a minute or so.

Yeah, I get that a lot! It annoys me because I don't know if my post has been posted and I don't want to double/triple post or wait to see if someone just posted before you.

[brad jones]

Thanks guys. I actually use this and our other forums too (specifically Codeguru), so I fully understand the "erratic" speed issue. That is what took a little longer to get the developers and techs onto this issue (because they'd see no issues in the logs, nor would they hit the issue when they would poke on the forum). That was at the beginning though and they now are fully aware of how things are manifesting.

As mentioned, we have our people as well as vBulletin looking to try to track this down to a long term solution.

For me, it seems the biggest slow downs happen when I have the most important things to say..... But that is just Murphy's Law and my luck coming together....

Brad!

[szlamany]

I joined Code Guru a few weeks ago to start asking C++ questions (ironically because I wanted to avoid slow VB code for some tasks!) - and I noticed how truely poor that site performed - and I could not wrap my arms around this. Was it my isp connection? Something in the office with a bad nic card?

Then over the next weekend - actually that same weekend (I think) because I was needy for assistance - was that our venerable vbforums went down for this upgrade.

And it came back looking like that Code Guru site I experienced just a few days earlier - both visually (I'll save the discussion on dull-gray website design for another day - did anyone look into how it works for people with disabilities??) - and unfortunately - in performance...

Have you measured the easy stuff yet? Like used the Net panel in Firebug to make sure the network data delivery is not hitting a bottleneck? Are the services that create the data for return taking too long - is the "code" to buld the data not efficient (ie - using StringBuilder like concepts?).

Where is the speed problem?

Is it a scaleability problem? Only able to handle x-number of open "searches"?

btw - searches were real bad a week ago - my "recent posts" was not finding all my activity - and I seem to think that you are using the SEARCH ability to serve lots of purposes.

Is the search the whole bottleneck?

Is it the constant GUEST visitors that are really just bots sucking today's meaningful VB headlines out of the forums?

I would be really upset if my "programming team" could not at least pinpoint some areas that might be suspect - or at least eliminate some areas that are not part of the problem.

Best practice was obviously not followed in some area of coding the backend...

[brad jones]

szlamany -

You have to consider that the forum software was not written by us, but rather is a third party product. The product has a great reputation and has not given us issues in the past. The fact that it is not our software does limit our knowledge of how it operates, although I'm guessing our developers know a lot more about that as they've spent time digging in.

The problem with trusting a third party vendor is that we assume they've managed the scalability issues and have done solid testing with the software. As a point of reference, VBForums was started by the vBulletin guys, so we know they are familar with large forums and with coding. We don't question that.

As to testing the obvious - that has been done. We've done a lot of that ourselves. Additionally, we've pulled in the vBulletin support. As to this site and Codeguru seeming to have similar issues - the sites are on the same server farm, are both big, and both have been upgraded to the same level. As such, it is not suprising that there are similar issues.

There are a couple of options that have been discussed on what to try next. Several changes have already been made and one or two big options are being considered. The forums are very important, so we will keep working on this until we come up with a viable solution.

As to search and its results -- that is addressed in the other thread. This site has a lot of posts (a lot). It took a crazy long time to index all those posts into the search database. That is done and search should be working smoothly now.

As to traffic - we only care about living creatures. Our internal guys are pretty good at filtering out the non-relevant bots and such. After all, we do a lot of advertising, which requires us to have clean pageviews.


As a final comment, let me add that we have both developers and IT pros looking at the issue.

Brad!

[RhinoBull]

The question however remains - is it ever going to be solved? Performance has been an issue for few weeks already; this is embarrassing...
Sorry for being so trivial.

[brad jones]

The question however remains - is it ever going to be solved?

Yes.

[Arve K.]

Unfortunately, the site still have those speed issues. I have noticed that on *************s, when I click on a thread, and go back again, the page doesn't reload. I think that is something you should apply to this site as well, if I wanted to update the page, well then I can just hit the refresh-button, instead of waiting ~5 seconds for the page to reload. Especially considering the speed issues we are experiencing now.

Oh.. ******** is another VB .net forum..

[DataMiser]

I too have been seeing a lot of issues, the site has been very slow and quite often the pages just do not load or take up to 30 or more seconds. Seems like about 1 in 5 pages fail to load on average since the upgrade and about 4 out of 5 are slow every now and then a page will pop right up like it should but that has been rare lately.

[westconn1]

more than 10 days since the last update of progress, or lack there of
the adage of no news is good news does not seem to apply here

[brad jones]

Sorry - I oversee more than one forum. I'll try to get better at posting updates to both key forums at the same time....

[dunfiddlin]

For the love of God would somebody please get this site working. There really are only so many times that I can count the cracks in my ceiling and trace the outline of the cobwebs in the corner!! It's getting worse not better bringing a whole new meaning to the word hangtime! Every navigation today has been like walking backward through a vat of glue in lead boots uphill! Make it stop!!!!!

[Maverickz]

Any chance of rolling back to the old version until this version can be fixed? The way it is now is painful to use.

[Arve K.]

Somedays are better than others, today it seem to be working as supposed. Last night it was painfully slow though...

[Maverickz]

I made that post after getting DB Errors 4 times in a row trying to read my new PM's.

[DataMiser]

I was on earlier today. At first I could not get anything to load then everything was fine. I read posts in maybe 8-10 threads then responded to one. When I hit the Post Quick Reply button it popped up the message about navigate away from page. I hit the ok and waited for a few moments to see what was going to happen. The progress bart was moving but slowly. I went to take a crap, came back, still loading, went to the fridge came back back still loading. Went to the store and came back to find it had finally loaded.

I do not think I have been able to make more than 3 posts in a session without seeing this type of thing at least once since the "upgrade?" I wonder if this new software was code named Vista?

[Maverickz]

LOL I agree DataMiser. I really think that as long as this has been going on we can say the upgrade is a failure and needs to be rolled backed. Let them work out the bugs then try it again later. If I were a new user coming here for help or advice for the first time and had these problems, I would leave and never come back.

[brad jones]

I asked about rolling back the forums previously. For a variety of reasons, that is really not an easy solution. On a more static site, it would be a real consideration. On a forum.... it would be tough and could introduce a number of additional issues....

Brad!

[Arve K.]

Today must have been one of the worst days since the update, plenty of timeouts and disconnections.

[FunkyDexter]

I asked about rolling back the forums previously. For a variety of reasons, that is really not an easy solution. On a more static site, it would be a real consideration. On a forum.... it would be tough and could introduce a number of additional issues....

From experience in similar (if not identical) situations I'd second that. You can't just go to an old backup because the data state has moved on trying to reverse a migration is usually much harder than the migration itself. I guess we're in it for better or worse and we've just got to hope that the issues get fixed soon.

[Edgemeal]

if I wanted to update the page, well then I can just hit the refresh-button,

Agree, that really is annoying no matter if the site is fast or not... When you click "go back" I expect to return to the page I was viewing at the scroll position!

I've been dreading the day this site would upgrade to this newer forum software, another forum I visit took 4 months to get back to normal.

[Niya]

Well I'm holding on to hope. I just been to vBulletin.com and they use the same major version as this upgrade and it performs superbly. It was almost strange since I have gotten used to vB 4 glitching out all the time here. I have full confidence the guys here would be able to match that performance eventually. Its actually a good upgrade.

[DataMiser]

Hope you're right and that it gets sorted soon

[Niya]

I think it would be more of a gradual process rather than a sudden boost to perfection. I remember the first week of the upgrade here, it was terrible. There's been a large improvement since then.

[Arve K.]

I agree, but I can't help being a little irritated when the site is at its slowest... After all, it has been over a month since the upgrade, and in my honest opinion the site should be running at nearly 100 percent now. It has never appeared super quick after the update, at least not compared to other sites using similar forum software. I know this is a big forum, but still... there are other big forums out there also...

But after all, I do like the look of the forum now, so I'm not saying I want the old version back, but seriously, something must be done soon. And I'm not so happy about the lack of feedback from the administrators either... What is being done? What is the plan??

[brad jones]

And I'm not so happy about the lack of feedback from the administrators either... What is being done? What is the plan??

My apologies. You are right. I've not responded since yesterday.l....

As to what is being done - I don't see giving a step-by-step, play-by-play detail of what is being done. That's just not going to happen. I stand by what I siad before in that we have our developers, our IT guys, and the help of vBulletin in looking at the speed issue. It is extermely important to us to fix as this is impacting our overall business in that we have a large number of forums. I personally work with our developer forums, which is seven different sizable forums. Trust me when I say getting this fixed is important to me. Unfortunately, we commited to a software (forum) product that didn't quite deliver what was expected. As such, the solutions don't seem to be easy, thus the fix hasn't happened quickly. I'm bugging people internally on a regular basis (to the point that if I moan any more than I am, they are liable to throw me out..... )

This will get figured out. The question is how long it is going to take. That's something I don't have an answer to yet - but I keep asking.

Brad!

[Arve K.]

I got a little carried away last night, sorry about that.

[Doogle]

this is impacting our overall business

There's an additional incentive (if one was needed)

Once the 'Powers That Be' realise there's an impact on the Bottom Line, even more Stops will be pulled out and the pressure on the Guys and Gals trying to fix it will be increased.

(Of course, once fixed it will then be a matter of punishing the innocent and rewarding the guilty)

Patience, being the only virute I posess, willout - I guess us moaning and groaning won't get it fixed any quicker, although it does act as a 'safety valve'

[Niya]

Well to be fair as a developer I can understand that software bugs can be quite frustrating to find much less fix in an elegant fashion. I believe that if the guys behind this knew exactly where the issues were, it would have been fixed by now. As such I can only assume that the guys behind the scenes are getting gray trying to fix the performance issues. I guess we can cut them some slack. I've had bugs that plagued me for months in my own projects which I'm sure were nowhere as large as vBulletin.

[brad jones]

Hey guys - I use this (and and a couple of our other forums) too. I feel the pain, and I understand the frustration.

I said this before, but it is worth repeating - one of the things that complicates fixing the issues is that the forum software is not software we wrote. It is a third party. As such, we are not in total control of optimizing and fixing the issues. We are leaning on them a bit as well as doing all the tweaks and changes we can on our side.

[szlamany]

I decided to use FF instead of IE so I could get some stats from firebug - took 6 seconds for the initial GET - what's up with that? That was just the FIRST GET in the CONSOLE window. Return 23KB - so that's not a size-bottleneck.

Took 8 seconds total.

Is that just to log me in? I'm suspect it might be since the second time only took 2.24 seconds for the INITIAL GET and a total of 4 seconds to load page.

But then I tried more in FF - and it was terrible. Couldn't get SUBSCRIBED threads to even click and look like it was moving - clicked NEW THREADS and just stalled...

So I'm back in IE now - logged in very fast (not the 8 or 4 seconds that FF showed).

But the forum is still really painfully slow.

I find I'm not using it casually to look around any more at all.

Have you run statistical reports of your most active users to see what their levels have dropped to? That might be the right thing to do and then show it to the people with the programming budget.

It stinks when I'm actually spending time debugging performance issues with a web site!

@niya - I've been at this business since 1980. If we ever let something like this linger this long with such apparent lack of progress towards a fix - well that's just about how you lose customers.

6 seconds just now to get a NEW POSTS screen. What takes 6 seconds? I can query medical claim tables with 3 million rows and get results in milliseconds.

What is the backend DB?

How is a search done? With a query?

Show your CODE - maybe we can help

[DataMiser]

Funny, I know we have became spoiled with speed but honestly 4-8 seconds would be and should be very tolerable. Where it becomes a problem and it happens a lot lately is when the page takes 30 seconds to 5 minutes to load or doesn't load at all. Anything under 10 seconds I can live with.

I also switched over to firefox when I was having problems and for about 15 minutes I thought maybe the problem was IE but after a few more page loads it is clear that it is not the fault of either browser just sometimes you click a link and the site responds right away and other times it is slow while others it just hangs in either browser.

[Niya]

@niya - I've been at this business since 1980. If we ever let something like this linger this long with such apparent lack of progress towards a fix - well that's just about how you lose customers.

That's why I believe that whatever is causing this terrible performance is very difficult for them to solve. I would think they are sensible enough to know just how serious this problem is and to have it linger on too long could cause the volume of traffic here to take a hit which I would hope is highly undesirable for them. There's just no way I can imagine the guys aren't making this a top priority. But still I think users should continue to stress this to make sure they understand the gravity of it....just in case.

[brad jones]

15 seconds is the unacceptable bar. less than 10 is what I consider bad, under 5 is what it needs to be and faster than 5 is what is wanted.

We have times right now where 15 seconds would be good, so we obviously are focused on this.

As to showing our code.....
Most of the code is vBulletin's. It's not our code. If it was, this would be easier to tackle. This is a risk of using a 3rd party product and going "all in" with it.

[nbrege]

What is the backend DB?

According to the vBulletin website:

GENERAL FEATURES
Scalable Solution - database server and web servers can reside on separate machines.
MySQL back-end database
vBulletin can be run on any machine that can support PHP and MySQL (Solaris, BSD, Linux, Windows, Mac)
Written in PHP which makes for a fast and efficient product

[brad jones]

Correct - it is MySQL

[Arve K.]

Hi, I've noticed that the site has been very quick and responsive for the last two days

Good job to everyone who have been working on this!!

[brad jones]

The dev's said they applied some fixes. I'm still seeing a little slowness, so I don't think things are completely solved yet. I'm sure you guys weill tell me as well if things are still not quite "up to speed".

Brad!

[Doogle]

Well, I've been on all day and haven't seen one 'Database Error' or 'Server Unavailable'. Still a bit 'sticky' with response times, but when it's fast it's really fast. I guess the Guys and Gals are getting there - well done ! - more power to their elbow(s).