Maintaining Power in an Outage Event

The experience of a power loss at the local school and seeing how the battery backup systems work is a good one. The experience is good from the point of view that as an IT administrator and front line worker, you can see how the process functions in real time. It also allows for reflection and adjusting the plan to better serve the users of the technology systems at the school.

The school system has two legs of electrical power coming into the school system. One of those legs of electrical power shutdown at 8:30 am on this Friday morning. Lighting and outlet power were affected to portions of the building. Interestingly enough, the electrical wiring in any given room might have a portion of the circuits functional, while other outlets were dead. I am not exactly sure why the electrical circuits of the school were designed and engineered this way, but it is what it is.

When the electricity in one leg went down this morning, the main elementary computer telecommunications closet lost partial power. The phone system was unaffected along with some amplifier equipment. However, the three racks of computer switching, telecommunications and server equipment were immediately pushed to battery backup. The beeping of battery power was regular as the power continued to flow from their lead acid cells.

Within the first minute, the building's occupants were notified and by minute 5, non essential servers were shutdown including video surveillance and some backup devices. The storage server units were shutdown after users were notified; they were given 2 - 5 minutes of lead time. Our 1:1 students were able to function for about 15 minutes until the main telecommunication switching rack lost its battery inside the UPS.

The center rack containing iBoss, router, firewall and Ruckus wireless controller stayed up for an hour on its new CyberPower UPS. Other UPS units which were now lightly loaded stayed functional 30+ minutes into the event.

The take away at 45 minutes into the power outage event was that in order to preserve the wireless system in the building for a longer period of time, the primary HP ProCurve switch and the Elementary primary PoE HP ProCurve switch need their own UPS that can provide power for their functions as they are far more critical. They connect the high school to the elementary over fiber and also connect the Ruckus controller to the wireless APs. Maintaining that service for a minimum of 1 hour seems like a future goal worth attaining.

Once the power came back online, most of the battery UPS units rebooted themselves. Those UPS units shutdown manually, were restarted manually and the servers rebooted and tested. The process of bringing the main systems back online was probably 3 minutes. Some UPS units in closets at the high school needed to be manually restarted to power PoE and standard switches.

Some older small desktop UPSes showed their age and now are being seriously considered for replacement. We also know now that PoE switches draw power based upon the number of devices drawing power. At the high school, 8 PoE access points draw far less juice than the elementary school PoE which has 14 access points drawing power. Those 24 port PoE switches can draw up to 490 watts which is sure to hammer a usually good UPS in short order.

Our elementary switch structure comprised of 6 switches running at near 850 watts of power on a single SmartUPS 1500 lasted about 20 minutes. As mentioned above, 2 of those 6 switches will get their own CyberPower 2200 UPS. We will also move our X1 (Mac Mini server) that maintains the DNS and DHCP services to that new UPS to keep Internet access up as long as possible for our 1:1 folks and teachers. We believe we can go from 20 minutes of access to about 1 hour of access with a power outage.

MacBook Troubleshooting - Beep Codes

North Mahaska is almost 6 weeks into the first year of its 1:1 laptop program and the hardware has been remarkably resilient and the students seem to be using reasonable care with the MacBook Pros. There have been some carrying case problems with straps and zippers, but that is another story. We have learned a few new troubleshooting processes and we enjoy learning new techniques to solve problems.

On a recent occasion, one student's MacBook Pro started up with three beeps and showed abnormal behavior. Using Apple support guidelines, we now know the meaning of the startup beep codes. On a MacBook Pro, the startup POST (power on startup test) checks hardware functionality. The following beep codes indicate a variety of issues. Here is the listing provided by Apple technical documentation.

On MBPro Startup: 

1 beep = no RAM installed

2 beeps = incompatible RAM types

3 beeps = no good banks

4 beeps = no good boot images in the boot ROM (and/or bad sys config block)

5 beeps = processor is not usable

One beep seems self explanatory in that there are no RAM SO-DIMMs installed in the machine. While this is not likely to happen in many cases, you will quickly know the issue.

Two beeps on a MacBook Pro indicate that the RAM SO-DIMMs are not compatible. For instance, if the memory specification requires PC3-10600S DDR3 1333 MHz RAM, that is what should be procured and installed in the MacBook Pro. Not all RAM modules are up to Apple's specification and it is wise to purchase RAM from vendors who explicitly state MacBook Pro compatibility.

Three beeps indicates a tricker predicament. The problem is most likely not RAM compatibility despite the results of Google searches. The possible problems are more likely to be a problem with the RAM seating in the module banks. It is also possible that the circuitry of the RAM banks in conjunction with badly seated RAM modules has cause a memory management issue. Reseat the RAM modules and also reset the PRAM (parameter memory) on the MacBook Pro.

A POST with four beeps is a bit more serious. The ROM instructions for booting your computer will not function to startup the computer. Perhaps the ROM was corrupted by some electrical damage involving the ROM chip and circuitry. Generally, this problem will require a motherboard replacement.

Finally, five beeps also indicates a fatal problem. This failure is found in the central processing unit of the laptop. Again, damage of some type has occurred with the CPU of the MacBook Pro, and will require a motherboard replacement in most cases.

In the case of student laptops at North Mahaska, the problems seen so far have been minor. One laptop did come in with the 3 beep POST and a RAM reseat and PRAM reset seems to have fixed the problem. Otherwise, one hard drive and one trackpad in 300 laptops has failed in six weeks. We continue to monitor the laptops with JAMF Casper and ARD software products. If the hardware continues to be solid, then we can turn ourselves to software issues, which consume more time due to complexity relationships in installation and usage.

First Student Macbook Pro Problem

North Mahaska had its first student Macbook Pro failure today. The machine came into the office at around noon and would not boot into either main partition or recovery partition. My initial assessment was a faulty hard drive. I booted a trusted USB OS X drive and sure enough the hard drive could not be repaired and dismounted itself, unhappy at my attempted fix. I then placed the machine in "target mode" and connected it via Thunderbolt cable to my Macbook Air. Again the hard drive mounted until Disk Utility tried to work its magic.

Meanwhile, my wife is getting her feet wet calling Apple with her first "bad hard drive" on a Macbook Pro. She is working via Apple to send the Macbook Pro in for repair, which given the machine is 5 months old, is not problem. The student appears to have taken good care of the machine this first 3 weeks as the Macbook Pro is clean. So that is nice to see.

Currently we are building an Apple Diagnostic bootable USB drive for the Macbook Pro to aid in a positive identification of the problem hard drive. Given this is the first issue in the three week deployment, we had yet to make this diagnostic USB drive. We have made USB boot drives for Lion and Mountain Lion with the ability to use Disk Utility and other 3rd party utilities to aid in troubleshooting, but have Apple's tool will be good as well.

This incident also will highlight the recent call to backup files to the student USB thumb drives they were issues with their Macbook Pro laptops. My best guess is this student did not backup anything and was operating in a care-free manner. Time will tell if that was the case. We may attempt some file recovery in this first case for the sake of knowledge and to satisfy our own curiosity.

Starting off

Getting started and looking back

With a tremendous undertaking like rolling out a laptop program to 300 students, there are bound to be stories to tell, reflections on the past work, and insights on future possibilities. My blog writing has been haphazard in the past couple years, but then again the material has been lacking prior to this 1:1 laptop program.

To be clear, my role is that of volunteer to my wife, who is the technology coordinator at North Mahaska Schools. However, I do provide some skills and abilities to compliment the technology endeavors at the school. So, I am often called upon to help with professional honey do items. My goal is to try to write a blog entry once a week in a consistent manner. With Julie's help, I hope to review the week's challenges and successes.

Let's start with the last two weeks. The Macbook Pro laptops were rolled out on August 27 - 28 to 285 students in grades 7 - 12. The rollout started with the upper grades and ended with the lower grades. The presentations to the students and the initial boot up and connection to the network and JSS management server went well. One observation for people who do this 1:1 work or will be doing it soon is this: allow time for the younger student groups because they do take longer to step through the process of initial Macbook setup.

As of today, September 7th, all of the Macbook Pros have performed flawlessly. In two weeks time, only two students have had login issues. One student had a problem during the rollout process which was corrected on the spot. The other student was a non traditional older senior whom thought her password was all lowercase. The password was simply reset and the senior was happy. The Macbook Pros did sit all summer, but we arranged to have some football boys run all 285 machines through a charging cycle in mid August prior to the rollout.

Both Julie and I were able to attend JAMF Casper training in Minneapolis on August 30-31. A couple small issues unrelated to the laptop 1:1 were handled remotely through a VPN connection to the school network. If you are a technology person in a public school and you do administration tasks for machines inside a firewalled network - do yourself a favor and learn to use VPNs. I use a software product called VPN Tracker for my MacBook Air. At home we have a Sonicwall that provides a "site-to-site" VPN to the school network.

With JAMF Casper training completed, we spent a fair amount of effort putting policies in place to accomplish software pushes to student and teacher laptops. In order to do Evernote training, we pushed the latest version to teachers a couple hours prior to professional development. A couple teachers were missed because their laptops were closed. We also pushed version 3.3 of Evernote to all the students in order to upgrade the old 3.0.6 version on their laptops since April 2012. In less than 24 hours, 285 student laptops had Evernote 3.3. It is an awesome feeling to have solid software that helps make management and deployment of software work well.

One observation on JAMF Casper and management that is interesting to note. When students are sick for a couple days, you notice quickly. This is because as managers, Julie and I run inventory reports to check on how the JSS server is progressing in delivering and deploying software packages. A couple two or three days time gives us a good idea when someone is missing school.

Lastly, we are testing LanSchool for deployment to the students and teachers. The student deployment is ready for next Monday. The teachers will be rolled out as they come by with their laptops and get the Teacher program with their unique channel ID assigned. There is a planned teacher development session for learning the basics of setting up class lists and using LanSchool.

So we finish the week pretty happy overall. The list of things done by Julie in one week of work would cover two type written pages and the variety of tasks would surprise most people. Between PowerSchool work, adjustments to the school security cameras, and fulfilling technology requests, we wonder if all the work will get done.