Maintaining Power in an Outage Event

The experience of a power loss at the local school and seeing how the battery backup systems work is a good one. The experience is good from the point of view that as an IT administrator and front line worker, you can see how the process functions in real time. It also allows for reflection and adjusting the plan to better serve the users of the technology systems at the school.

The school system has two legs of electrical power coming into the school system. One of those legs of electrical power shutdown at 8:30 am on this Friday morning. Lighting and outlet power were affected to portions of the building. Interestingly enough, the electrical wiring in any given room might have a portion of the circuits functional, while other outlets were dead. I am not exactly sure why the electrical circuits of the school were designed and engineered this way, but it is what it is.

When the electricity in one leg went down this morning, the main elementary computer telecommunications closet lost partial power. The phone system was unaffected along with some amplifier equipment. However, the three racks of computer switching, telecommunications and server equipment were immediately pushed to battery backup. The beeping of battery power was regular as the power continued to flow from their lead acid cells.

Within the first minute, the building's occupants were notified and by minute 5, non essential servers were shutdown including video surveillance and some backup devices. The storage server units were shutdown after users were notified; they were given 2 - 5 minutes of lead time. Our 1:1 students were able to function for about 15 minutes until the main telecommunication switching rack lost its battery inside the UPS.

The center rack containing iBoss, router, firewall and Ruckus wireless controller stayed up for an hour on its new CyberPower UPS. Other UPS units which were now lightly loaded stayed functional 30+ minutes into the event.

The take away at 45 minutes into the power outage event was that in order to preserve the wireless system in the building for a longer period of time, the primary HP ProCurve switch and the Elementary primary PoE HP ProCurve switch need their own UPS that can provide power for their functions as they are far more critical. They connect the high school to the elementary over fiber and also connect the Ruckus controller to the wireless APs. Maintaining that service for a minimum of 1 hour seems like a future goal worth attaining.

Once the power came back online, most of the battery UPS units rebooted themselves. Those UPS units shutdown manually, were restarted manually and the servers rebooted and tested. The process of bringing the main systems back online was probably 3 minutes. Some UPS units in closets at the high school needed to be manually restarted to power PoE and standard switches.

Some older small desktop UPSes showed their age and now are being seriously considered for replacement. We also know now that PoE switches draw power based upon the number of devices drawing power. At the high school, 8 PoE access points draw far less juice than the elementary school PoE which has 14 access points drawing power. Those 24 port PoE switches can draw up to 490 watts which is sure to hammer a usually good UPS in short order.

Our elementary switch structure comprised of 6 switches running at near 850 watts of power on a single SmartUPS 1500 lasted about 20 minutes. As mentioned above, 2 of those 6 switches will get their own CyberPower 2200 UPS. We will also move our X1 (Mac Mini server) that maintains the DNS and DHCP services to that new UPS to keep Internet access up as long as possible for our 1:1 folks and teachers. We believe we can go from 20 minutes of access to about 1 hour of access with a power outage.

First Student Macbook Pro Problem

North Mahaska had its first student Macbook Pro failure today. The machine came into the office at around noon and would not boot into either main partition or recovery partition. My initial assessment was a faulty hard drive. I booted a trusted USB OS X drive and sure enough the hard drive could not be repaired and dismounted itself, unhappy at my attempted fix. I then placed the machine in "target mode" and connected it via Thunderbolt cable to my Macbook Air. Again the hard drive mounted until Disk Utility tried to work its magic.

Meanwhile, my wife is getting her feet wet calling Apple with her first "bad hard drive" on a Macbook Pro. She is working via Apple to send the Macbook Pro in for repair, which given the machine is 5 months old, is not problem. The student appears to have taken good care of the machine this first 3 weeks as the Macbook Pro is clean. So that is nice to see.

Currently we are building an Apple Diagnostic bootable USB drive for the Macbook Pro to aid in a positive identification of the problem hard drive. Given this is the first issue in the three week deployment, we had yet to make this diagnostic USB drive. We have made USB boot drives for Lion and Mountain Lion with the ability to use Disk Utility and other 3rd party utilities to aid in troubleshooting, but have Apple's tool will be good as well.

This incident also will highlight the recent call to backup files to the student USB thumb drives they were issues with their Macbook Pro laptops. My best guess is this student did not backup anything and was operating in a care-free manner. Time will tell if that was the case. We may attempt some file recovery in this first case for the sake of knowledge and to satisfy our own curiosity.

Time Machine Backups to USB Thumb Drives

"You are one crazy dude."

I can hear and see the comments coming a mile and a half away on this shin-dig. Many technology savvy people will be questioning the idea of using a USB thumb drive for a regular backup device. I agree with those technology folks and for my own personal and professional use, I would not use any sized USB thumb drive. So why am I publishing a YouTube how to video on this topic?



The school and the Apple 1:1 support team thought this was a workable solution for the students at a reasonable cost. Perhaps the thinking was that since other schools had tried USB thumb drives with some success, it was a good idea. Perhaps the thumb drives were the school's solution to providing a escape clause if the primary drive in the Macbook Pro died and the student forgot to backup.

The reason the YouTube video was created was to give the students with a screencast to show them how to do a Time Machine backup. It just happens that the Lexar 8 GB thumb drives they received along with their Macbook Pro is an option for a Time Machine backup. At the beginning of the video, I make a subtle hint on the USB drive's fragility and small capacity. I also point out the low cost of 500 GB external hard drives that connect via USB.

To further promote the idea of moving toward the external hard drive idea, here are some reasons why the USB thumb drives are not the best media for a Time Machine (or any regular) backup.

• Thumb drives do not integrate well into backups because of versions and metadata.
• Thumb drives do not integrate well with pockets, bags, dogs and small children since they are small and very easy to lose or misplace.
• Thumb drives are physically flimsy and fragile. One local school in 2010-11 report a 50% + failure of thumb drives given to 7th & 8th graders over that school year - most due to physical damage.
• Thumb drives are subject to corruption. People tend to unplug a drive without unmounting it.
• Thumb drives have low read-write speeds relative to spinning disk media. Thumb drives are NOT solid state drives.
• Thumb drives are not ubiquitous and their data does not naturally integrate with net-based cloud storage.
• Thumb drives take up an extra USB slot on the Macbook Pro system.

My final thoughts here go towards the idea that Time Machine is a great built in backup utility that works very well - just not with USB thumb drives. Google yourself an appropriate sized external hard drive that works with either USB, Firewire or Thunderbolt and start backing up correctly.

Starting off

Getting started and looking back

With a tremendous undertaking like rolling out a laptop program to 300 students, there are bound to be stories to tell, reflections on the past work, and insights on future possibilities. My blog writing has been haphazard in the past couple years, but then again the material has been lacking prior to this 1:1 laptop program.

To be clear, my role is that of volunteer to my wife, who is the technology coordinator at North Mahaska Schools. However, I do provide some skills and abilities to compliment the technology endeavors at the school. So, I am often called upon to help with professional honey do items. My goal is to try to write a blog entry once a week in a consistent manner. With Julie's help, I hope to review the week's challenges and successes.

Let's start with the last two weeks. The Macbook Pro laptops were rolled out on August 27 - 28 to 285 students in grades 7 - 12. The rollout started with the upper grades and ended with the lower grades. The presentations to the students and the initial boot up and connection to the network and JSS management server went well. One observation for people who do this 1:1 work or will be doing it soon is this: allow time for the younger student groups because they do take longer to step through the process of initial Macbook setup.

As of today, September 7th, all of the Macbook Pros have performed flawlessly. In two weeks time, only two students have had login issues. One student had a problem during the rollout process which was corrected on the spot. The other student was a non traditional older senior whom thought her password was all lowercase. The password was simply reset and the senior was happy. The Macbook Pros did sit all summer, but we arranged to have some football boys run all 285 machines through a charging cycle in mid August prior to the rollout.

Both Julie and I were able to attend JAMF Casper training in Minneapolis on August 30-31. A couple small issues unrelated to the laptop 1:1 were handled remotely through a VPN connection to the school network. If you are a technology person in a public school and you do administration tasks for machines inside a firewalled network - do yourself a favor and learn to use VPNs. I use a software product called VPN Tracker for my MacBook Air. At home we have a Sonicwall that provides a "site-to-site" VPN to the school network.

With JAMF Casper training completed, we spent a fair amount of effort putting policies in place to accomplish software pushes to student and teacher laptops. In order to do Evernote training, we pushed the latest version to teachers a couple hours prior to professional development. A couple teachers were missed because their laptops were closed. We also pushed version 3.3 of Evernote to all the students in order to upgrade the old 3.0.6 version on their laptops since April 2012. In less than 24 hours, 285 student laptops had Evernote 3.3. It is an awesome feeling to have solid software that helps make management and deployment of software work well.

One observation on JAMF Casper and management that is interesting to note. When students are sick for a couple days, you notice quickly. This is because as managers, Julie and I run inventory reports to check on how the JSS server is progressing in delivering and deploying software packages. A couple two or three days time gives us a good idea when someone is missing school.

Lastly, we are testing LanSchool for deployment to the students and teachers. The student deployment is ready for next Monday. The teachers will be rolled out as they come by with their laptops and get the Teacher program with their unique channel ID assigned. There is a planned teacher development session for learning the basics of setting up class lists and using LanSchool.

So we finish the week pretty happy overall. The list of things done by Julie in one week of work would cover two type written pages and the variety of tasks would surprise most people. Between PowerSchool work, adjustments to the school security cameras, and fulfilling technology requests, we wonder if all the work will get done.