[Tech-l] Information about outage on 12/3

[Thor Sage]

Good morning.
Yesterday MVECA suffered a severe outage that interrupted services for all client organizations consuming NGFW (next generation firewall) services.  While completing common maintenance activity on our firewall configurations, Cisco Systems Technical Assistance Center directed MVECA to complete a restart of a security module on the firewall.  Ordinarily, this activity occurs outside of regular business hours and takes only about ten minutes.  In this instance, the security module did not recover as expected causing downtime that extended into the school day.  We were forced to rebuild configurations entirely in order to recover from the outage.
At this time Cisco Systems is investigating and attempting to identify the cause of this problem and we continue to work with them on corrective actions.
Additionally, MVECA will be accelerating our work to procure additional firewall hardware and licensing in order to add additional redundancy to our network and avoid future instances in which any single system can cause extended outages.
We do fully understand the impact of the outage yesterday and the resulting interruptions to regular instruction and online testing.  Please know that this sort of issue receives the highest priority and we are taking action to ensure that this doesn't happen again.
I would personally like to extend my most sincere apologies to you and your school district for the disruption.  Please feel free to contact me directly with any questions or concerns.
Thank you,
Thor

Thor Sage
Executive Director
Miami Valley Educational Computer Association
937-767-1468  x3101
[http://www.mveca.org/images/logo.gif]<http://www.mveca.org/>       [i] <https://www.linkedin.com/company-beta/3947840/> [t] <https://twitter.com/mvecarcog> [f] <https://www.facebook.com/MVECA-707401659416692/>
Not-for-profit Technology Services for Education and Local Governments



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.mveca.org/pipermail/tech-l/attachments/20191204/c4ad1082/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 3184 bytes
Desc: image001.jpg
URL: <http://listserv.mveca.org/pipermail/tech-l/attachments/20191204/c4ad1082/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.jpg
Type: image/jpeg
Size: 1229 bytes
Desc: image002.jpg
URL: <http://listserv.mveca.org/pipermail/tech-l/attachments/20191204/c4ad1082/attachment-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.jpg
Type: image/jpeg
Size: 1197 bytes
Desc: image003.jpg
URL: <http://listserv.mveca.org/pipermail/tech-l/attachments/20191204/c4ad1082/attachment-0002.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.jpg
Type: image/jpeg
Size: 1194 bytes
Desc: image004.jpg
URL: <http://listserv.mveca.org/pipermail/tech-l/attachments/20191204/c4ad1082/attachment-0003.jpg>