PcPerf bot

Our server is back up and we're in the progress of doing a stats update. This update alone will take some time and so WU's will take longer to get through the system for about 24 hours. The good news is that all the key systems are back on line. Voir l'article complet

Most machines are now back up with a few exceptions. One key exception is a server used in the stats updates. So, while the points are being recorded on local servers, these logs are not being inputted into the stats at the moment, i.e. stats updates are on hold at the moment. We will give an update when the stats update is back on line. Please keep in mind that the points are not being lost, just not being entered into the db and once this machine is up, all the logs (and all the old points from last night, etc) will be entered into the stats db. Voir l'article complet

Here's the update from Stanford Power mostly restored to campus Updated 3 p.m pacific time. - Power has been restored to the majority of the Stanford campus. All normal power is expected to be restored by 5 p.m. or sooner. A major outage occurred today at 11:30 a.m., affecting P G & E customers in Stanford, Menlo Park, Atherton and Palo Alto. If you are still experiencing difficulties on the Stanford campus, call 723-2281. At Stanford Hospital, dial 286. This one was a major disaster at Stanford and Palo Alto (and nearby cities), probably the biggest outage in a while. However, there seems to be some major Stanford power outage once a year, which is a major problem. With this in mind, we have been distributing more of FAH to outside of Stanford (with servers at UCSF, Columbia, Cal State U Long Beach, and U. Pittsburgh). We hope to have a European site soon. Once those sites are a bit more established, we'll see about pushing an assignment server to a non-Stanford site and we should be much more safe to Stanford-related issues. Also, it's good that we have servers in 4 different server rooms on campus. One stayed up the whole time, two came up fast, and the fourth (VSPGxx) is coming up now. Some servers will be slow to come up, so we expect this may take at least a few hours. The stats update has been turned off until this gets sorted out. We hope to turn it back on later tonight, but it may have to wait until tomorrow morning. Voir l'article complet

Palo Alto is having a massive power outage which has brought down much of Stanford too. More info to come. We will try to get this up asap once we get power back but it will be rough for a while. Traduction par Thor: Palo Alto a une coupure électrique massive qui a aussi touché une bonne partie de Stanford.Plus d'information à venir. Nous allons essayer de tout remettre en route dès que la puissance sera revenue mais ça risque d'être instable pendant un moment. Voir l'article complet

Here's the update from Stanford Power mostly restored to campus Updated 3 p.m pacific time. - Power has been restored to the majority of the Stanford campus. All normal power is expected to be restored by 5 p.m. or sooner. A major outage occurred today at 11:30 a.m., affecting P G & E customers in Stanford, Menlo Park, Atherton and Palo Alto. If you are still experiencing difficulties on the Stanford campus, call 723-2281. At Stanford Hospital, dial 286. This one was a major disaster at Stanford and Palo Alto (and nearby cities), probably the biggest outage in a while. However, there seems to be some major Stanford power outage once a year, which is a major problem. With this in mind, we have been distributing more of FAH to outside of Stanford (with servers at UCSF, Columbia, Cal State U Long Beach, and U. Pittsburgh). We hope to have a European site soon. Once those sites are a bit more established, we'll see about pushing an assignment server to a non-Stanford site and we should be much more safe to Stanford-related issues. Also, it's good that we have servers in 4 different server rooms on campus. One stayed up the whole time, two came up fast, and the fourth (VSPGxx) is coming up now. Some servers will be slow to come up, so we expect this may take at least a few hours. The stats update has been turned off until this gets sorted out. We hope to turn it back on later tonight, but it may have to wait until tomorrow morning. Traduction Systran: Voici la mise à jour de Stanford Puissance la plupart du temps reconstituée au campus Temps Pacifique mis à jour de 3 P.M. - La puissance a été reconstituée à la majorité du campus de Stanford. Toute la puissance normale est prévue d'être reconstitué par 5 P.M. ou plus tôt. Une panne importante s'est produite aujourd'hui à 11:30 heure du matin, affectant des clients de P G et d'E Stanford, à Menlo Park, Atherton et à Palo Alto. Si vous éprouvez toujours des difficultés sur le campus de Stanford, l'appel 723-2281. À l'hôpital de Stanford, composez 286. Celui-ci était un désastre important chez Stanford et Palo Alto (et villes voisines), probablement la plus grande panne dans un moment. Cependant, il semble y avoir une certaine coupure électrique principale de Stanford une fois par an, qui est un problème majeur. À cet effet, nous avions diffusé plus de Fahrenheit à en dehors de de Stanford (avec des serveurs à UCSF, à Colombie, à état U Long Beach de calorie, et à U. Pittsburgh). Nous espérons avoir un emplacement européen bientôt. Une fois que ces emplacements sont un peu plus établis, nous verrons au sujet de pousser un serveur de tâche à un non-Stanford pour situer et nous devrions être beaucoup plus sûrs aux issues Stanford-connexes. En outre, il est bon que nous ayons des serveurs dans 4 salles différentes de serveur sur le campus. On est resté vers le haut le temps plein, deux ont monté rapidement, et le quart (VSPGxx) est soulevé maintenant. Quelques serveurs seront lents pour monter, ainsi nous prévoyons que ceci peut prendre au moins quelques heures. La mise à jour de stat a été arrêtée jusqu'à ce que ceci obtienne triée. Nous espérons la tourner en arrière dessus soir postérieur, mais il peut devoir attendre jusqu'à demain matin. Voir l'article complet

It looks like the down net is coming back up now. The servers will be a bit loaded while everyone tries to put results back, but it should work itself out soon. Voir l'article complet

We have 4 different server rooms for redundancy and it looks like one of the room's network is down. All machines on the 171.64.65.XX net (VSPG*.stanford.edu) are unreachable right now. [Note that this server room is in the Stanford Computer Science building and it looks like the whole Stanford CS net is down (eg http://www.cs.stanford.edu/ is unresponsive).] We have notified the network admins and the server admins and they are working on this right now. It's Saturday, so they're on a reduced staff, but they're on it. This is something beyond my group's control (much like an electrical outage or a major natural disaster) and we have to wait for the Stanford CS networking gurus to do their magic. The good news is that much of FAH is still up. You can see what parts are up vs down on our serverstat page (http://fah-web.stanford.edu/serverstat.html). However, the bad is is that with many of the key machines unreachable, the other machines are heavily loaded at the moment. Once the additional server room comes on line, then everything should ease up. It's very unfortunate timing that this comes shortly after a rough set of weeks with our complete server upgrade, which lead to several machines being down in a rotation for updates. That lead to servers being hit hard too. Note that we have been working to improve client/server performance under such high loads. That development lead to server code changes and a new client binary available that helps uploads to servers when the servers are loaded. Please see the previous posts here for updates on those clients. Voir l'article complet

If you're having any networking issues with the 6.20 GPU client, we suggest you try the 6.20r1 client. You can find installers here: http://foldingforum.org/viewtopic.php?f=50&t=4806 We will put this on our download site after it's gone through a little more testing. Voir l'article complet

As of last Monday, the GPU2 projects (both NVIDIA and ATI) have been in production mode, which means that we've moved on from testing to direct research on them. The ATI cards have bee in production for a while (more than one month before) and we're starting to write our first peer reviewed scientific paper deriving from the results. We're very excited about the GPU2 possibilities right now, since we are getting incredible production from GPU2 clients. Once the first paper has been formally accepted by peer review, I'll talk more about the results. Voir l'article complet

We see that the site is down. There's a problem with the main Stanford web server and the University IT is working on it. Note that this only affects our web site, the stats, main servers, AS, data servers, etc are all up (FAH is up). UPDATE from Stanford IT: At the moment we're having problems with certain AFS directories that are affecting the WWW servers. I'm not sure when things will come back up yet, but I'll mail you to mention when things are better. I'm very sorry about the inconvenience -- we're working hard on tracking this down. It's been our experience that the web/AFS team is very strong, so hopefully they will have this resolved soon (eg by the end of business today). UPDATE: This appears to now be fixed. Voir l'article complet

The new server code has been in for about a day, and so far so good. The netloads are very low on the servers and everything appears to be behaving well from our side. We have also made a new client (6.20r1) available to improve uploads. The client patch should be a greater aid than our server patches, so if you are having upload problems, please consider giving it a try: http://foldingforum.org/viewtopic.php?f=50&t=4806 Voir l'article complet

We've been looking into what could be the cause of the GPU2 upload issues people have seen. We have some ideas and that has resulted in new server code and a new client. The new server code has been tested on vsp07 and is now rolled out on the other GPU2 servers (vspg1, vspg4v2, vspg3v2) and we're monitoring it. The new client (dubbed 6.20r1) is now undergoing closed beta testing. If that checks out, we'll release it for open testing. The new client is the key part of this fix (new client + old server code is probably ok), but the new server code will help a little. Also, the new server code is useful for the stats change over (described below, where we separate out NVIDIA from ATI stats). Note that the osstats page will be screwy for a while during this switchover. Voir l'article complet

The Win/SMP 6.22 client has been showing several issues, namely: bad EUE behavior (quits rather than moves on) no ACK sent issues core 0x7b errors We've been working to take these issues one by one and have some progress to mention. For issue #1 (EUE), we've released a new client (6.22r2) at http://foldingforum.org/viewtopic.php?f=46&t=4764 . If this looks good, we'll roll this into 6.23. We also had a meeting to see what's up with the ACK and have an idea there too. That will take a client mod as well and that's been coded and is starting to enter Stanford QA, then alpha testing, then beta testing. This client mod may also be relevant for the new GPU clients as well. The third issue is still a little perplexing since it is a core issue and in principal shouldn't come with a client update. Now that we think we may have got the first 2 issues resolved (or at least a patch to test), we're moving to deal with this third issue. I should stress that it's by no means guaranteed that the first two are solved, but we will need more testing to know for sure. We'll give more news as we know it. Voir l'article complet

We're working to change how the GPU2 servers report machines to the OSstats ( http://fah-web.stanford.edu/cgi-bin/main.py?qtype=osstats) page. The goal is to list NVIDIA and ATI clients separately (both are being lumped into a GPU2 category). In the next few days, you may see some weird numbers (too high or too low) on this stats page while we switch servers over. I'll post when we're done (hopefully only a few days). Voir l'article complet

We've been working on improvements to the server code to better handle GPU2 work units better. That new code is running on 171.64.122.74 right now and (so far) appears to be an improvement. If it looks better, we'll roll it out more broadly to all 4 GPU2 servers. We've also fixed an issue with GPU2 WU's which was leading to very large uploads. This has greatly shrunk the GPU2 WU return size and this should help out with the load a great deal. Finally, we're looking into network issues. We currently suspect that there may be some packet shaping on FAH WUs with some ISPs. This issue should be improved by the smaller WU uploads and the server issues, even if the ISP's don't change anything. Voir l'article complet

With the 6.20 (classic & GPU) and 6.22 (SMP) clients out, we can start looking forward to the next steps in client development. We still have some last bits of work to completely unify the clients, but the hard part is already completed there for the most part. The 6.2x series introduces several new features for donors, but in time, the clients have been getting gradually more and more complex to use. The Win/SMP and multi-gpu setups are examples of very challenging setups. Our primary plans for the future are to make setup much easier, especially for the very challenging clients (Win/SMP, multi-gpu). Our dream client install would be a single installer that would install either the classic, SMP, or GPU client based on donor choices during the install process, and the installer would take care of complex install situations (such as multi-gpu). It may take a while to get to a client installer like that. In the mean time, we are looking to improve our installation instructions to make the process easier with existing clients. We have spent a lot of effort developing these new clients and putting all of our effort into getting them to run well (the GPU2 client/core has gotten a great deal of effort, which has paid off in the transition from GPU1 to GPU2). It's now time to concentrate on usability and ease of installation. Our hope is that with this completed, we will have a very advanced set of clients, but with very easy installation. That's easier said than done, but that's where we're headed. Voir l'article complet

We took down the vspg3v2 GPU2/ATI server to add more jobs earlier this afternoon. It's back up now with plenty of new jobs and cranking away. Voir l'article complet

We've released new clients (v 6.20). For those running GPU clients (or other clients with an expiration deadline of August 2), please check these clients out. Here's what's new in the 6.20 client. * Added Forums to systray menu. * Support cpu affinity by default, this is related to an nVidia problem and awaits core-side changes. * Add additional parameters field in advanced preferences - no more shortcut editing!!! * UNSTABLE_MACHINE count reset after a correctly finished WU, for long-running systems * Open configuration dialog box when invalid settings detected * Fixed core launch directory which caused auto-upgrades to fail if core was present in "Program Files" * Added additional advanced config options to console clients * Added MachineID selector in systray gui * Client shows PRCG on results upload * Passkey now validated for hex content * Moved service installation to advanced config to prevent inadvertant use * Log date and time when automatically attempting to upload results. * Correctly restore system tray icon when shell (explorer.exe) crashes * Significantly improved queueinfo * Multiple copies of the client can no longer be started with the same "start in" dir * Automatically increase packet limit to max size for results that are too large * Failed uploads will be attempted again using alternative upload port (8080->80 and 80->8080) * Removed spurious "benchmarking..." messages * Added extra_parms support to console clients * Extra parameters are validated at configuration and launch time. Invalid settings will force reconfiguration * Fixed and updated service installation to allow use of working directories * Added ATI FireStream 9170 detection * Compressed Vista icon to reduce size of executable (saves ~ 200kb) * Updated MyFolding.html to match folding.stanford.edu style Voir l'article complet

We've released new clients (v 6.20). For those running GPU clients (or other clients with an expiration deadline of August 2), please check these clients out. Known issues: * Service installation in Vista with the GPU client doesn't work (noted in FAQ-NVIDIA and FAQ-ATI2) * If you're updating a service from a pre 6.20 client, you need to remove the old service first. (Currently being edited into relevant FAQs) Here's what's new in the 6.20 client: * Added Forums to systray menu. * Support cpu affinity by default, this is related to an nVidia problem and awaits core-side changes. * Add additional parameters field in advanced preferences - no more shortcut editing!!! * UNSTABLE_MACHINE count reset after a correctly finished WU, for long-running systems * Open configuration dialog box when invalid settings detected * Fixed core launch directory which caused auto-upgrades to fail if core was present in "Program Files" * Added additional advanced config options to console clients * Added MachineID selector in systray gui * Client shows PRCG on results upload * Passkey now validated for hex content * Moved service installation to advanced config to prevent inadvertant use * Log date and time when automatically attempting to upload results. * Correctly restore system tray icon when shell (explorer.exe) crashes * Significantly improved queueinfo * Multiple copies of the client can no longer be started with the same "start in" dir * Automatically increase packet limit to max size for results that are too large * Failed uploads will be attempted again using alternative upload port (8080->80 and 80->8080) * Removed spurious "benchmarking..." messages * Added extra_parms support to console clients * Extra parameters are validated at configuration and launch time. Invalid settings will force reconfiguration * Fixed and updated service installation to allow use of working directories * Added ATI FireStream 9170 detection * Compressed Vista icon to reduce size of executable (saves ~ 200kb) * Updated MyFolding.html to match folding.stanford.edu style Voir l'article complet

There's some news on the SMP client front. First, we've moved our OSX and Linux clients to a non-beta designation, reflecting the maturity of those clients. We expect those clients to run reasonably well. The Windows/SMP client is not there yet, but we have been continuing to work on it and improve it. Right now, we are making available two Win/SMP clients. The primary one (6.22) is on our download page ( http://folding.stanford.edu/English/DownloadWinOther ). People have reported issues with 6.22, so we have made 5.91/5.92 available with an extended expiration as well ( http://foldingforum.org/viewtopic.php?f=8&t=4369 ). With the public beta release of the v6.22 SMP client, we encourage all our beta users to upgrade to that client. However, since the 5.91/5.92 beta expiration date is rapidly approaching, we have decided to extend those clients to allow users more time to upgrade (and to give us a little more time to work out the issues that invariably come up during public beta testing). It's worth reminding people that these Windows/SMP clients are still very experimental, hence the reason for the expiration and extra points one gets. If you would like a "set it and forget it" client, this client is not for you (please check out our classic client in that case), and one should expect that the Win/SMP client will take a lot of extra work to run. See the discussion at the top of our download page ( http://folding.stanford.edu/English/DownloadWinOther ) and SMP FAQ for more details. We're working to improve the Win/SMP client and get it released non-beta like the OSX and Linux clients and hopefully we'll get there soon. An important part of this open beta is the beta testing reports we've gotten from people. Please post them in our forum (http://foldingforum.org). PS Here's the info from our download page on the nature of these clients High Performance clients. In addition to the clients that run in the background on typical computers, we also offer high performance clients, such as the GPU and SMP clients. These clients use more system resources, but are also much more productive. Please consider the use of these clients carefully. See the respective GPU, GPU2, SMP, and PS3 FAQs for more information. Beta clients. We often release clients early for donors to beta test. These beta versions likely have some rough edges, but we expect that they should work reasonably well for all donors. See the respective installation instructions for more details of known bugs for each of the beta versions. As in the use of any beta software, please make sure to back up your hard drive before installing. DO NOT not run a beta client if you or your machines cannot tolerate even the slightest instability or problems. Beta clients and servers performance may vary significantly from standard FAH clients during the development process, including but not limited to work unit shortages, server downtime for upgrades, short notice for client upgrades, and Points Per Day that differs a little or a lot from the developmental benchmark level. Finally, note that while the points per day for these clients are higher than the classic client, they can require a lot more maintenance due to their experimental or beta nature. If you would prefer to have a client which runs as smoothly as possible, we suggest you run our main client, not a high performance client. If you run a high performance client, expect a much more complex experience and much more work to keep the client running (which is compensated by extra points per day). Voir l'article complet

We've updated the server code on 171.64.122.74 to make it more responsive. We're keeping an eye on it to see for sure. One of the major differences is how long the server waits for a connection (it was waiting way, way too long before and that filled up all of its threads). We've cut that down as well as significantly increased the number of threads. In combination with no new assigns for this server for a while, this should drain it and get the server I/O back to normal. Voir l'article complet

We've been looking over the server code to see what's going on and I think we have some news. What's happening is that the server is being a bit too generous in terms of when it times out connections, filling up all of the available server threads. For now, we've bumped up the # of threads (not unlike adding more cashiers at the grocery isle, so that a few slow people don't slow down everyone) from 200 to 500. That should help a lot in the short term. On Monday, we'll make some code modifications to make a more complete fix to this. It may take a little while to get this tested and implemented, but I expect that this should be in by Monday afternoon. That should greatly ease what we're seeing. (PS Note Monday = Monday pacific time) In some ways, this has crept up on us, as nothing has changed (all our servers are up and running) -- it's just that the extended down time built up a large # of WU's to return and the server is still trying to catch up. We've turned off new assigns from this machine to help it catch up (no new WU's will mean it spends all it's time receiving WU's). With this change, the threads tweak, and the code update coming later today we should have some more news (hopefully good news) later on Monday. Voir l'article complet

For a while, the collection servers (CS's) haven't been working well. We've been looking into why. A few weeks ago we overhauled server-CS interaction and added some more CS's to help the load. We have continued to look into CS issues and have some new ideas. With these new ideas, we've made an update to how the collection server works. We think this will help clients send back results significantly. This update code has been placed on 2 CS's right now (171.64.122.76 and 171.64.122.86) and we're going to watch to see if this improves the situation. We have a few other tweaks that might help as well, but we want to try this one first. Voir l'article complet

One of our servers (171.64.122.74) had some issues during maintenance which delayed it's coming back on line. Due to this, 171.64.122.74 was back on line on Saturday (PST) instead of Friday evening, due to an issue found during maintenance. Luckily this maintenance is a once a year thing, so it won't be needed for a while. It's back up now. Due to this server being off line, the backup server (171.64.65.20) took quite a hit. To avoid this, we're moving to longer WU's (8-24 hours) and looking to add an additional server so the servers take less of a hit. Longer WU's alone will make a HUGE difference, since the servers would get hit much less frequently and thus the load would be much smaller. That should greatly improve these issues. The current situation is that all servers are up and running, but there's a lot of clients trying to access them. We'll keep an eye on them throughout the weekend to make sure they're going well. Voir l'article complet

Starting today and lasting for about a week, we will be taking down servers every day for planned maintenance. Today, we have taken down servers with the following IP addresses 171.64.122.124 171.64.122.125 171.64.122.131 171.64.122.132 171.64.122.133 171.64.122.136 171.64.122.137 171.64.122.138 171.64.122.139 171.64.122.141 171.64.122.142 171.64.122.70 and they will be back up today (if all goes according to plan). These servers above are not heavily used, so their downtime should not highly affect FAH. In subsequent days, we will be taking down other machines (see below). The downtimes for these machines should be much less (~2-5 hours). Update date IP Machine Name Contact 24-Jul 171.64.65.56 vspg4 kasson 24-Jul 171.64.65.73 vspg4v vvoelz 24-Jul 171.64.65.106 vspg4v2 relly1 24-Jul 171.64.65.58 vspg5 densign 24-Jul 171.64.65.83 vspg5v luttmann 24-Jul 171.64.65.111 vspg5v2 vvoelz 25-Jul 171.64.122.73 VSP06 densign 25-Jul 171.64.122.83 VSP16 kasson 25-Jul 171.64.122.88 VSP21 kasson 25-Jul 171.64.122.74 vsp07 densign 25-Jul 171.64.122.84 vsp17 nick 25-Jul 171.64.122.42 vsp22 karsson 28-Jul 171.64.122.75 vsp08 kim.branson 28-Jul 171.64.122.76 vsp09 luttmann 28-Jul 171.64.122.86 VSP19 luttmann 29-Jul 171.64.122.77 VSP10 29-Jul 171.64.65.121 vspg6-vz7 pande 30-Jul 171.64.65.20 vspg1 densign 30-Jul 171.64.65.63 vspg1v densign 30-Jul 171.64.65.96 vspg1v2 densign 30-Jul 171.64.65.42 vspg2 luttmann 30-Jul 171.64.65.64 vspg2v kasson 30-Jul 171.64.65.102 vspg2v2 vvoelz 31-Jul 171.64.65.55 vspg3 jchodera 31-Jul 171.64.65.65 vspg3v kasson 31-Jul 171.64.65.103 vspg3v2 densign 31-Jul 171.64.122.72 VSP05 gbowman 31-Jul 171.64.122.78 VSP11 gbowman 31-Jul 171.64.122.82 VSP15 gbowman Voir l'article complet