Jump to content
  • Member Statistics

    17,609
    Total Members
    7,904
    Most Online
    NH8550
    Newest Member
    NH8550
    Joined

Server Performance...feedback needed


stormtracker

Recommended Posts

  • Replies 128
  • Created
  • Last Reply

As someone else somewhat experienced in PHP/MySQL development, and who runs a large message board myself, I cannot emphasize the above enough.

I'm not surprised to hear that it's not a problem with overall CPU/mem load. That's rarely the case. It's far more likely the maximum number of simultaneous SQL connections being exceeded. This is where careful examination of log files, and perhaps addition of some debugging/error-handling code to the forum software, would be helpful. It may be as simple as substantially increasing a software limit, or it may require some rewriting to eliminate pconnects.

Short answer - this is definitely fixable.

As I said, I'd be happy to volunteer some of my time to diagnose and/or fix this... If the logs don't show what is happening, a higher log level is in order.

Link to comment
Share on other sites

We have guest viewing turned off, thus you can't view the board if you aren't a member/signed in.

 

Also, from our tech guru scott, the problem isn't hardware.  He says the hardware isn't even sweating:  The problem is clearly software/db related and we're going to be trying some new performance saving moves (like turning off features, hooks, etc)

 

 

Database:

Server load     1.12 (6 CPUs)     
Memory Used     60.53% (3,197,988 of 5,283,704)     
Swap Used     4.54% (88,944 of 1,959,920)

 

Web:

Server load     4.61 (16 CPUs)     
Memory Used     23.55% (3,633,172 of 15,429,888)     
Swap Used     0.01% (196 of 1,959,920)

 

That's with 2631 users.  Servers are bored and yawning, yet still randomly sluggish with the occasional error even in performance mode.

 

 

 
Link to comment
Share on other sites

We have guest viewing turned off, thus you can't view the board if you aren't a member/signed in.

 

Also, from our tech guru scott, the problem isn't hardware.  He says the hardware isn't even sweating:  The problem is clearly software/db related and we're going to be trying some new performance saving moves (like turning off features, hooks, etc)

 

So the problem isn't hardware, but the solution is to lessen the load on the hardware? Why not just figure out what is causing the sql errors? Am I missing something?

Link to comment
Share on other sites

So the problem isn't hardware, but the solution is to lessen the load on the hardware? Why not just figure out what is causing the sql errors? Am I missing something?

Yes, The point is to lighten the load on the dbm/sql...software.  Thats the hangup.   That;s what I kinda explained.

Link to comment
Share on other sites

Yes, The point is to lighten the load on the dbm/sql...software.  Thats the hangup.   That;s what I kinda explained.

 

I get what you're saying, but given everything I've seen on the site and in the thread, the temporary degradation of features is a short-term band-aid at best. 

 

A few things stand out from those hardware stats and other things in this thread:

  1. Your servers aren't using all of their ram or cpu time. This suggests that the bottleneck is IO-related, ie: your hard drives are working as hard as they can, but can't read (or write) data fast enough to keep up with demand.
  2. If you were running on Windoze servers, I'd look at whether or not you're running an x86 version of the DBMS or a x64 one. Both servers seem to be using an amount of ram equivalent to only the 32-bit memory space. That seems obvious, but could be overlooked quite easily. I'd imagine they both run linux though.
  3. Your databases are probably exhibiting one or more common scalability or maintenance problems: index and table fragmentation, inefficient or missing indexes, infrequent analysis runs(mysql)... and more. Maintenance is key...
  4. Turn on slow query logging. That will point out which queries take the longest to run on the DB server.
  5. Look at the indexes and query execution plans of your slowest queries. Unfortunately mysql doesn't offer index statistics like oracle or SQL Server, but you can still figure out where your slow queries are doing table or index scans on large tables.
  6. What does the disk config look like? If the DB server has the hardware to support it, add more disks and set up a RAID 10 or RAID 100 configuration. Even better if you're on EC2 or some other cloud server provider you can just add storage volumes and configure striping with LVM.

If all that is not enough to help, consider setting up replication for your DB server. If read performance during large load events is a problem (it probably is), a read replica can help immensely with handling the read load. 

 

Other things you could do:

  • install memcache (or some other supported cache) on the webserver and utilize some of that free ram on your webserver to cache resources that don't change between pageviews. IPB supports this.
  • use a CDN (cloudflare is great) to deliver static resources like images, css files, javascript, etc. Those resources get cached by browsers, but with thousands hitting the site per day that's still a thimblefull in the bucket.

I understand why features get turned off, but if the problem is only highlighted by high load, and getting worse over time, eventually you'll run out of things to turn off. 

 

Hope you get things figured out... I don't know weather well enough to contribute on the weather threads, but I do know software and servers... I'll shut up now.

Link to comment
Share on other sites

I get what you're saying, but given everything I've seen on the site and in the thread, the temporary degradation of features is a short-term band-aid at best. 

 

A few things stand out from those hardware stats and other things in this thread:

  1. Your servers aren't using all of their ram or cpu time. This suggests that the bottleneck is IO-related, ie: your hard drives are working as hard as they can, but can't read (or write) data fast enough to keep up with demand.
  2. If you were running on Windoze servers, I'd look at whether or not you're running an x86 version of the DBMS or a x64 one. Both servers seem to be using an amount of ram equivalent to only the 32-bit memory space. That seems obvious, but could be overlooked quite easily. I'd imagine they both run linux though.
  3. Your databases are probably exhibiting one or more common scalability or maintenance problems: index and table fragmentation, inefficient or missing indexes, infrequent analysis runs(mysql)... and more. Maintenance is key...
  4. Turn on slow query logging. That will point out which queries take the longest to run on the DB server.
  5. Look at the indexes and query execution plans of your slowest queries. Unfortunately mysql doesn't offer index statistics like oracle or SQL Server, but you can still figure out where your slow queries are doing table or index scans on large tables.
  6. What does the disk config look like? If the DB server has the hardware to support it, add more disks and set up a RAID 10 or RAID 100 configuration. Even better if you're on EC2 or some other cloud server provider you can just add storage volumes and configure striping with LVM.

If all that is not enough to help, consider setting up replication for your DB server. If read performance during large load events is a problem (it probably is), a read replica can help immensely with handling the read load. 

 

Other things you could do:

  • install memcache (or some other supported cache) on the webserver and utilize some of that free ram on your webserver to cache resources that don't change between pageviews. IPB supports this.
  • use a CDN (cloudflare is great) to deliver static resources like images, css files, javascript, etc. Those resources get cached by browsers, but with thousands hitting the site per day that's still a thimblefull in the bucket.

I understand why features get turned off, but if the problem is only highlighted by high load, and getting worse over time, eventually you'll run out of things to turn off. 

 

Hope you get things figured out... I don't know weather well enough to contribute on the weather threads, but I do know software and servers... I'll shut up now.

This might be helpful.  Thanks!

Link to comment
Share on other sites

Randy: Whatever you guys did to limit some features was really screwy when it came to logging in and the main page. I had no way of getting to the threads themselves... had to have Andy email me the link.

 

May want to fix that.

Had the same issue and was able to get in via 'Topics of Interest' on the main page.

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

  • Recently Browsing   0 members

    • No registered users viewing this page.

×
×
  • Create New...