linksunten Archiv

In the night from the 29th to the 30th of April Indymedia linksunten was offline for nearly 5 hours due to a power outage. All services are restored by now.

IMPORTANT NOTICE April 29, 2012 - Due to a power outage at our provider at Telehouse, we have lost power to our cabinet affecting about half of all May First/People Link members. This power outage is not within our control.

All time America/New_York.

• 5:58 pm: Due to a power outage at our provider at Telehouse, we have lost power to our cabinet affecting about half of all May First/People Link members. This power outage is not within our control. We are pressuring Telehouse to bring power back up as soon as they can and we are positioned to bring our servers back online as soon as they do.
• 6:29 pm: we are onsite, however, still have no power to our cabinet. We have not been given an ETA on the return of power. At least two of the facility's five UPS systems are down, so it may be hours before power is restored. We will report as soon as we get an ETA.
• 7:00 pm: still no update on ETA
• 7:30 pm: Additional Telehouse technicians scheduled to arrive at 8:15 pm America/New_York time.
• 8:30 pm: Additional technicians have still not arrived
• 8:59 pm: power is restored, all servers coming back online now. It may be another 30 minutes before service is completely restored and another hour before traffic/resource usage is back to normal
• 9:03 pm: power was just lost again. All power is down at the cabinet
• 9:10 pm: power is restored again. We have turned on robideau (providing serial console access) and negri (providing hay, malcolm, proudhon, ranciere, algernon, bouazizi, clyde, dorothy) in order to have a functional OpenID provider. We are waiting before starting the other servers to ensure that the power will stay on.
• 9:49 pm: The power has stayed up for 40 minutes now, however, the Telehouse technicians have requested 30 minutes before confirming that the power will stay on. To avoid corrupted data, we are waiting on confirmation that it will stay up before we start the rest of the servers.
• 10:06 pm: The UPS technician has confirmed that we are by-passing the UPS, and are being powered by the backup generator. The technician expects to switch back to UPS, possibly in the next 15 minutes, and advised that it might cause another power outage.
• 10:45 pm: Power has been restored. All servers should be online (or soon to be online).
• 11:18 pm - most servers are running and operational, but responding very slowly as they deal with incoming mail that was queued while the servers were down (we do not expect any lost mail). albizu is the only primary/mosh server that is not yet up. It is running a slow file system check and is expected to be up in about 1 hour.
• 1:32 am: - albizu is up and running. At this point all servers directly in use by members should be fully functional. The only remaining server not online is blanco, one of our two onsite backup servers. blanco is still running a file system integrity check. We expect it to come online in a few more hours.