[Mail_supt] network events - September

[Thor Sage]

Good morning,
This month has been somewhat rocky with respect to our delivery of bandwidth services to schools and governments.  This is exceptionally frustrating given our determination to provide best-in-class services to our client organizations.  We want everyone to feel confident that MVECA solutions are solid and dependable and we are working to ensure that our network services continue to meet all of your needs.  Some of our client organizations have expressed concern about network events that have occurred in September, so I wanted to provide a recap and explanation for each of these:


  1.  On September 3rd, OARNet made a configuration change that allowed more traffic to be routed to their Internet2 peer without making a corresponding change to the bandwidth caps they had previously applied to our local Internet2 connections.   This did cause significant slowdowns but was resolved quickly when we shut down our Internet2 peer locally and pushed all traffic to the regular internet peer.  We escalated first to OARNet's network support team and then to their leadership and the problem was corrected.  We have since turned our Internet2 peer back on and are seeing improved overall performance.  We have been frustrated that OARNet issued no statement on this that we could utilize to help our client organizations understand what happened.  Please know that this was outside our control and upstream from MVECA.  Despite this, we do take full responsibility for all network services our clients consume and will continue to drive the support process even when working with outside support organizations.  We do appreciate the fact that OARNet leadership was quick to respond once we escalated above their regular network helpdesk.
  2.  On Thursday September 23rd and Saturday the 25th we had severe slowdowns and outages related to denial of service incidents.  The worst of these was on Thursday the 23rd at around 10:30 a.m.  Both of these outages were related to a network flooding event known as TCP or SYN Flood.  A SYN Flood is a common form of Denial-of-Service (DDoS) attack  that can target any system connected to the Internet and providing Transmission Control Protocol (TCP) services. A SYN flood is a type of TCP State-Exhaustion Attack  that attempts to consume the connection state tables present in many infrastructure components, such as load balancers, firewalls, Intrusion Prevention Systems (IPS), and application servers. This type of DDoS attack can take down even high-capacity devices capable of maintaining millions of connections (such as those at MVECA).  After the incident on the 23rd and with the help of our firewall manufacturer, MVECA was able to put in place additional scripts and protections that would help us identify and eliminate the source of these attacks.  When the additional incident occurred on Saturday, we were able to track and block the malicious traffic entirely.  We have also put in place additional detection scripts, more stringent buffer and connection limits, and other configuration changes that will ensure that this type of attack does not have the same impact again.

Again, we apologize that this has occurred and want to assure everyone that MVECA's team is continuously on the job and working to eliminate all potential threats and disruptions.  Unfortunately, all large networks are susceptible to these sorts of incidents.  With MVECA, however, you have one throat to choke with respect to network operations.  The MVECA team will work to mitigate and avoid disruptions in every instance and will continue to provide comprehensive support on all network services.
Please contact me directly with additional questions or concerns.
Thank you,
Thor


Thor Sage
Executive Director
Miami Valley Educational Computer Association
937-767-1468  x3101
[http://www.mveca.org/images/logo.gif]<http://www.mveca.org/>       [i] <https://www.linkedin.com/company/mveca/> [t] <https://twitter.com/mvecarcog>
Not-for-profit Technology Services for Education and Local Governments


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.mveca.org/pipermail/mail_supt/attachments/20210928/00ece45f/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 3184 bytes
Desc: image001.jpg
URL: <http://listserv.mveca.org/pipermail/mail_supt/attachments/20210928/00ece45f/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.jpg
Type: image/jpeg
Size: 1229 bytes
Desc: image002.jpg
URL: <http://listserv.mveca.org/pipermail/mail_supt/attachments/20210928/00ece45f/attachment-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.jpg
Type: image/jpeg
Size: 1197 bytes
Desc: image003.jpg
URL: <http://listserv.mveca.org/pipermail/mail_supt/attachments/20210928/00ece45f/attachment-0002.jpg>