Oct 16, 2002 - Ottawa routing problems
Between 1415 and 1445 there was significant congestion in our Ottawa network. One of our customers mistakenly advertised the whole internet to our Ottawa router, causing our router to attempt to send 10mbps of traffic over the customer's 3Mbps link. The situation has been corrected, and our routing configuration for other customers will be updated tonight to avoid the possibility of repeat.
Oct 16, 2002 - Toronto routing problems
There have been routing problems in our Toronto POP this am due to high router loads. We are attempting to rebalance traffic to reduce the load. In addition we expect to have a Cisco 7206 router installed by the end of the month which should provide a significant increase in routing capacity.
Oct 15, 2002 - Network maintenance
We will be performing network maintenance Oct 16 between 0200 and 0300 Eastern time. Users will possibly experience 2 outages during this time lasting approximately 5 minutes.
Oct 14, 2002 - Servers compromised
Early this morning 2 servers were compromised; the main istop.com web server and ns.istop.com. The unauthorized access appears to have been achieved through a security hole in ssh. We are in the process of securing the servers; we will likely need to reboot the servers a couple times through the day in this process. We should be able to keep one server up at all times so users should see minimal impact.
Oct 13, 2002 - Toronto routing
This moring we had routing problems in Toronto. The situation was corrected by 1300 Eastern.
Oct 12, 2002 - Toronto routing
From 0645 to 0945 Eastern time we were having intermittent routing problems in our Toronto POP. One of our upstream internet providers (Peer1) had enabled proxy-arp on their router, leading to connectivity problems for some of our other routers connected to the same switch. A call has been placed to their NOC and the connection to them has been temporarily disabled until their configuration is fixed.
Oct 10, 2002 - Hosting accounts data recovery
Our own attempts to recover the data from the failed hard drive have been unsuccessful, so we are sending the drive to a professional data recovery lab. We were hoping to do the work on our own as the cost of professional recovery is generally a minimum of $1000. From initial discussions with the recovery lab all the data is likely to be recovered. The expected timeframe is Wednesday Oct 16.
Oct 9, 2002 - istop.com disk replacement
At 0600 this morning the failed disk was replaced. Customer accounts were reconfigured on the new disk and as of 0900 customers can re-upload their content. For customers that did not keep copies of their site content, we are still working on recovering the old data. This process may take a few more days. The recovery is taking longer than expected due to problems with our backups. So far we have only been able to restore backups from several months ago.
Oct 8, 2002 - istop.com load
1500 Eastern time: The problem earlier this morning was misdiagnosed - it was actually caused by a failed hard drive. We are currently working to replace the hard drive and restore the data.
Oct 8, 2002 - istop.com load
Our main server for web and outgoing SMTP mail experienced a very high load shortly after 1100. Normal operation returned by 1130. One of our web hosting customers was consuming excessive resources. The server configuration will be updated to ensure such customers cannot exceed their limits.
Oct 6, 2002 - news.istop.com down
news.istop.com is down due to a hard drive failure. news2.istop.com is functioning properly and can be used while news.istop.com is being repaired.
Oct 3, 2002 - news2.istop.com down
The server news2.istop.com has crashed. The situation is being investigated and we expect to have it up and running again by 1030 eastern time.
Sep 30, 2002 - Ottawa network congestion
1000 Eastern time: Due to a colo customer using excessive bandwidth, cogestion may be experienced for traffic passing through our Ottawa POP. As of 1100 the customer has been contacted and the situation will be rectified shortly.
Sep 19, 2002 - New POPmail software
Today new high-performance POPmail server software has been installed. Users that set their POP client to leave messages on server will likely get duplicate messages. We don't recommend customers use this feature in their POP client as it is problematic and unsupported. Leaving messages on server is only supported when using the IMAP protocol instead of POP.
Sep 12, 2002 - Toronto network
Between 2300 and 2400 work will be done on our network in our Toronto POP. It is expected to have no impact, though if the work does not go according to plan a couple brief outages of less than 5 minutes are possible.
Sep 12, 2002 - external network failure
Shortly after 1400 our NewYork network connection dropped. Due to a bug in our Cisco routers traffic did not reroute through our backup provider until they were rebooted. Connectivity was restored by 1420. We will schedule a maintenance period next week to upgrade the Cisco router software.
Sep 11, 2002 - mail server
Early this am the mail server power supply failed. A temporary replacement has been installed and the server will be up and running again by 0900 EST. A permanant replacement will be installed between 0300 and 0600 Sep. 14th.
Sep 10, 2002 - news2 and proxy problems
news2.istop.com has been down again since yesterday. The problem should be fixed permanantly later this evg.
2130 update: the server configuration has been updated and is working properly.
Sep 10, 2002 - New connectivity in Manhattan
Traffic is being routed through Level3 and Genuity in Manhattan from our POP in 60 Hudson St. We expect this will give customers improved network performance vs the Cogent connection we were using. During the transition over the past 2 days there were several brief outages lasting between a few seconds and a few minutes.
