We got the Main AS back up and all looks good on that box.  The main systems will take some time to get completely back in order (say a couple of hours), but it's looking good so far.

The main AS is down.  The two consequences of this are that the secondary AS is handling all of the load and the main stats update have been suspended until this is resolved.  The points are being kept locally on the servers, so WU's are being credited, just not showing up on our web site.

Engineers at NVIDIA (notably Scott LeGrand) have come up with a theory for the EUE's seen in core 1.15 (and a few others in the 1.15 to 1.18 range) on certain hardware. They found that this core had code optimizations that drove the GPU so hard that it would draw a lot more electricity (one sign of this was running hotter). In some boxes, this was too much electricity and this lead to numerical instabilities. When the same machine was given a beefier power supply, the problem went away.

We've been told that 8800's require 600W power supplies, but we're finding that even a little bigger (eg at least 650W) is important to leave some room for error. We are working to see if there is some way to detect this issue in software, but for now, if you're getting EUE's on the NV GPU client, this is something to consider.

By the way, this will be very important for us to consider future code optimizations. NV core v1.19 removed some optimizations to solve this problem, but there are many cards which would run fine w/this more optimized code. If we can find a way to detect whether the card can draw enough power, we may be able to choose different code paths to allow for greater optimization for cards which can handle it.

We're still looking into this. For now, if you're seeing issues with your card, please consider trying out a bigger power supply. We will continue to look to see if this is indeed the problem and what we can do to help the situation such that the code runs stably on all machines.

The forum at foldingforum.org is down.  We are in contact with the hosting company and they are working on it.  We don't have an ETA from them, but I hope this will be resolved later today.

Folding@home servers are up and running and is not affected in any way from the forum hosting company.

We're happy to announce that the PS3 client for Folding@home has won a prestigious design award:

The PS3's Folding@home service has nabbed a prestigious Good Design Award, AKA "G-Mark." Since 1957, the Good Design Award has been instituted by the Japanese Ministry of International Trade and Industry to award outstanding design in industrial and consumer products.

We reset 171.64.65.83 and now the PS3 clients are looking better.  We'll keep an eye on this, but it is the weekend in California right now, so we are on a limited staff (and thus response will be a slower than on week days).

This will involve brief (3-5 minute) network interruptions during the time period.  Most of FAH will not be involved, but the 171.64.65.XX network will see some downtime.

The Stanford CS building was off line briefly today starting at about 1:30pm pacific time, lasting for about 3 hours.  This meant that all of our machines on the 171.64.65.xx subnet were off line.  It looks like everything is back on line and Stanford IT is looking into causes.  This will likely lead to a bit of an excess load for a while until things settle down.

We will have a brief (1 hour) server outage tomorrow, October 28 at 10am pacific time.  It is expected to last about an hour.  This will only affect machines on the 171.64.65.XX network, so most of FAH will be up.  However, we expect there to be a server backlog for a little bit after the machines come back up.  Also, it's likely that one server or so will have trouble coming back up and will be delayed.

We're continue to work to keep the PS3 servers stoked with jobs.  For those who are having problems and are not running the latest FAH software (based on Life With Playstation or "LWP"), please upgrade.  We need to concentrate our servers on the latest clients and cannot guarantee work for older clients.