BOINC Project Servers Down. EDIT: NOW UP AND RUNNING AS NORMAL

[V_R]

Original thread.

There is a problem with the BOINC servers that is preventing agents to connect to them. You'll see a 'Project is down' message, if you look at the messages tab.

We are working on the problem but do not have an estimated time when they'll be back up.

We'll keep you posted. Hopefully everyone can survive on their task queue for the duration.


Update #1

Our servers are located at a hosting facility in Boulder, CO (USA). Unfortunately, the problem affecting the BOINC servers requires someone to have physical access to the servers. Boulder is in the middle of a significant snow storm and no-one is able to travel at this time. Additionally due to the snow storm the hosting facility is operating with significantly reduced staff. As a result it may be awhile before we are able to resolve this problem.

We sincerely apologize for the trouble.

This problem only affects BOINC clients.

Please note that no work will be lost. Also for work returned after the deadline please note that there is a 2 day grace period so your work will still be accepted.

 

[V_R]

Typical, they have been down since about midnight last night. As soon as i post this they get fixed!

The outage has been resolved. The hosting facility was able to get someone to reboot the server. This allowed us to connect remotely and fix the problem. Work is being sent out now and results are being validated.

 

[Anon_F]

Cheers V_R
Are you sure it's not fog effecting the servers & not snow

 

[V_R]

BOINC servers are in Colorado, They is having a blizzard, apparently its way worse there than it is here, Frankly i am glad i am not there, its cold enough now....

 

[muckshifter]

Not been able to get a result through for over 24hrs ... all PCs are havin' a nice rest.

 

[V_R]

You just beat me to it mucks, they went down again last night. I am guessing its down to the adverse weather over there...

FYI, It is the BOINC servers that have gone down. If you run out of work and wish to continue crunching you could install the UD Agent as a stop gap until the servers are back up. you cant run two units at once but you can continue to crunch.

Or cache more work units.

 

[Me__2001]

the servers have been down since about 4 or 5 pm yesterday, unfortunately i dont have any WU's stored so i'm sitting idle

 

[V_R]

They did come back up at one point, i managed to upload my compleated units and reload the cache. But they did go down again this morning.

 

[V_R]

Here is an update from WCG.....

As of the time of this post, the BOINC server remains down. I can't provide a good estimate of when it will be back up.

As a result of the BOINC server being down, the web servers were also down causing the site to be unavailable for approximately 7 hours ending about 20 minutes ago.

 

[Anon_F]

Just finished a work unit & no new work avaliable will wait until new work given out, when ever that may be.
Going to be a big backload of those needing to report when the servers go back on.
Oh well off to run AV scan, a quick defrag & download some updates

 

[CaptHook]

I manages to see the results at 7 PM EST or 0100 Z yesterday. Seems most of the Team is on Boinc as I saw only 5000 points for the day. THere will be a whopping rush of points as soon as they manage to get the servers up.

 

[CaptHook]

Latest from IBM via email....



Hello Again Harvey,

Yes unfortunately we are.

We are working on a solution for the issue with the BOINC servers. The worst case scenario is that the BOINC servers will be down until Tuesday, December 26, 2006. The best case scenario is that the servers will be up later today. We will know more at 6:00 p.m. Cental Time (USA).

 

[V_R]

I've gone through my cache of WU's already!

Now crunching with UD Agent. But the WCG websites down so i cant change the CPU throttle to 100% from the 60% default! Not to mention the UD Agent doent support dual core CPU's......

I hope they fix it soon....

 

[V_R]

Good news is the websites back.

The bad news is BOINC Servers still down...

FYI,

BOINC Project Status can be found here.

 

[V_R]

Latest Update.

The servers have been repaired. The problem turned out to be a faulty hard-drive which has been replaced. Now comes the hard part of rebuilding the hard-drive. This is going to be very time consuming. Early estimates are that it will take from 8 to 72 hours because much of the data has to come from a tape backup system. We will provide an update at least once each day. Once again, this only affects BOINC users. We apologize for this inconvenience.

In addition, we want to make sure our members know the following:

1) Based on what we know right now, no data has been lost. The results returned by the members are stored on network attached storage.
2) The database is also in a good state and the database will be available once the restore is complete

In the meantime when members BOINC clients complete their existing work queue they will see messages such as this when they attempt to communicate with our servers:

12/22/2006 7:59:23 PM|World Community Grid|Sending scheduler request: Requested by user
12/22/2006 7:59:23 PM|World Community Grid|(not requesting new work or reporting completed tasks)
12/22/2006 7:59:27 PM|World Community Grid|Scheduler request failed: HTTP file not found
12/22/2006 7:59:27 PM|World Community Grid|Deferring scheduler requests for 1 minutes and 0 seconds

The clients will continue to attempt communications with our servers periodically. Once we are again back in service clients will automatically return the results and complete communications.

 

[V_R]

We're back people!

Well its 3.30am but the BOINC servers areback up and running!!

We just got the BOINC servers up and running and you should start seeing normal operation patterns at this time, Knreed is doing a few more tests and will be posting another update later. Happy Holidays!!

There were stats accumulated by users on BOINC that were not correctly imported into the website database. In a few minutes we are going to re-run the stats since the outage to correctly import these stats.*


*Depending on when our in house stats run they may not be correct, will have a look tomorow.

All results crunched during the down time should validate without a problem.