System Status at The WELL | ||
|
This page is updated only when the status or other content changes. Tue Jan 5 16:29:01 PST 2010 The new Zimbra email system (iris.well.com) is back! You may now use the new system for new mail again, via the Web or with your preferred email application. We are now recovering from the hardware and related problems. The queue of email messages from Monday and early Tuesday is flowing in too, in addition to new mail. You may notice duplicates generated from our emergency actions, if you read some of that mail with Squirrelmail as suggested below. More information will be available soon. See the welltech conference for discussion. ____ Tues Jan 5 6:24:58 PST 2010 We have not restored access to iris.well.com yet. Work continues. Our efforts at disk repair on Monday were to no avail, and another server was pulled from service, wiped and prepared for restoration of backups. This machine will become the new iris.well.com server soon. Unfortunately, some difficulties cropped up in the backup restoration process. The good news is that we have since gotten what we feel is a credible diagnosis and theory of what that problem is, and we are working on it. Late yesterday we extended the previously announced forward/copy scheme. This means that you may not only see and respond to copies of mail received since 9:00 am PST at the Squirrelmail web page, but you will also see older mail there, from the time of the hardware crash Sunday morning. There may be some duplicates. We will temporarily keep delivering copies of new email sent to all Zimbra-enabled accounts in the older email systems, until full recovery. To recap yesterday's instructions, this temporary use of that old Squirrelmail web mail at webmail.well.com is your simplest short term solution. Log in there, and if necessary, click "fetch" to pull copies of your mail to that page for now for reading and replying. (You may also use POP off of the older servers, or Pine at your well.com home directory via SSH.) The best place to get questions answered will be the conference for WELL technical help. The shortcut is "go welltech" when logged in to the conferences. We are simply appalled at this series of machine failures, holiday timing and bad coincidences. As soon as the backed-up email is back, we will work on strategies to make a scenario like this one impossible. We are very sorry for the inconvenience. ____ Mon Jan 4 10:12:58 PST 2010 The partial email outage continues. This is a hardware outage on a new machine, It only impacts Zimbra users, on the new mail system only. Many WELL members have not upgraded to the modern server yet, and will see no effects at all, so if your mail is working, disregard this message. This morning at 9:00 am Pacific Time we put into effect a forward/copy scheme that will temporarily keep copies of new email sent to all Zimbra-enabled accounts in BOTH email systems. This means if you are a new mail user you may pick up copies of mail received since 9:00 am by fetching them to the older Squirrelmail web page. (Log in at webmail.well.com) These copies started being delivered about an hour ago, and will be provided until we get back in ship-shape with repaired or replacement hardware, most likely sometime later today. We believe that Squirrelmail at webmail.well.com is your simplest short term solution. Log in there, and fetch copies of your mail to that page for now. You will be able to see copies of that email on the new system when we restore operations. (If, instead of using webmail.well.com, you decide to temporarily change back over to POP on the old servers, please note what settings were working on the new Zimbra system so you can restore them later, since we will not have any record of what's been working for you on your devices. For those familiar with Pine, we will restore fetching to your home directory via Pine temporarily too.) For discussion of the situation, log in and go to the welltech conference. ___ Sun Jan 3 12:37:20 PST 2010 The new Zimbra email system (iris.well.com) is currently down for what appears to be a hardware failure. We are working on restoring service as soon as possible, but at this time, it looks like service will not be restored until sometime on Monday. We apologize for the inconvenience caused by this outage. ___ Wed Dec 2 21:49:27 PST 2009 The scheduled maintenance is complete. All systems appear to be up and running normally. ___ Mon Nov 30 13:16:08 PST 2009 We will be applying a variety of security updates and other system patches to all well.com systems on Wednesday evening (Dec 2) between the hours of 7pm and midnight. As each system is updated, the updated system will need to be rebooted. As a result, short outages of between 5 and 15 minutes (and possibly as much as 2 hours in the case of email) can be expected on all systems at different times. While mail on both the old and new systems will be unavailable during the outages for the respective machines, mail should be delivered as usual when the system is restored to operation. We apologize for any inconvenience caused by this necessary maintenance. ___ Fri Oct 2 12:08:50 PDT 2009 A little after 11:15am PDT we had problems with our DNS that made the WELL disappear for approximately 16 minutes for anybody who was not logged in at the time. All of our machines were running smoothly, but due to the DNS problem, our services were unavailable to new external connections during that period. Most email sent to addresses in the well.com domain will be re-sent. However, retrying depends on how other services configure their email servers. There is a chance that some email could have bounced back to the sender saying it was a "fatal error". We are sorry for the inconvenience. ___ Thu Sep 10 09:07:33 PDT 2009 Mail delivery has resumed. It will take some time - several hours, at least - for the backlog of mail to be delivered. ___ Thurs Sep 10 8:16 PDT 2009 The WELL's mail servers were updated last night and rebooted. However, this morning reports from dozens of members confirmed that a further fix is needed. We are working on it right now, and delivery of mail will resume as soon as the fix is completed. We are sorry for the inconvenience. ___ Tue Sep 8 15:48 PDT 2009 The WELL's mail servers will be down for a system software update between 7:30 and 8:30pm Pacific time on Wednesday September 9th. No email will be delivered or sent during this time. The WELL will queue incoming mail, and it will be delivered when the update is completed. ___ Fri Aug 7 09:20 PDT 2009 PicoSpan is unreachable at this time. We're working on the problem. Access to the conferences is available via the Web. ___ Fri Jun 5 13:56:09 PDT 2009 The Web interface to WELL conferences (user.well.com) will be taken out of service Monday June 8th from 8pm till 1am US Pacific Time so we can move the database from one host machine to another. This should prevent certain slowdowns. Complete Plan members may use ssh for PicoSpan (command line) access to conferences. During the downtime, we will also reboot the mail server (mh.well.com). Mail service should be back up again in fewer than 30 minutes. While the mail server is down, you won't be able to get new mail. Incoming mail will be delivered after the mail server is back online. We apologize for any inconvenience these preventative moves may cause. ___ Tue Apr 7 14:10:58 PDT 2009 There will be a reboot of the mail server (mh.well.com), Thursday night at 10:00 PM Pacific Time. The system should be back up again in less than 30 minutes. During the downtime, you won't be able to get new mail. Incoming mail will queue up during that time for delivery after the mail server is back online. We apologize for the inconvenience. Tue Mar 17 16:12:01 PDT 2009 There will be a reboot of the mail server (mh.well.com), Thursday night at 9 Pacific Time. The system should be back up again in less than 30 minutes. During the downtime, you won't be able to get new mail. Incoming mail will queue up during that time for delivery after the mail server is back online. ___ Wed Jan 14 01:39:03 PST 2009 The mail server (mh.well.com) will be taken out of service briefly Sunday at 9pm US Pacific Time so we can add additional memory. This should help prevent some of the mail delays we've been seeing the past 2 weeks. The total downtime should be no more than 30 minutes. Incoming mail will be queued locally, and will be delivered after the reboot. We apologize for the inconvenience. ___ Wed Jan 14 01:39:03 PST 2009 There will be brief outage of well.com Wednesday night at 11pm Pacific Time to correct a system configuration problem. The total downtime should be no more than a few minutes. For Further Information... For information about The WELL or for WELL Customer Support, contact: WELL HelpDesk
Email: Write to helpdesk [at] well.com for more information about The WELL and WELL Customer Support. |
||
|
Please log in to search members-only conversations. |
||
|
Home |
Learn About |
Conferences |
Member Pages |
Mail |
Store |
Services & Help |
Join Us
|