Maintaining Power in an Outage Event

The experience of a power loss at the local school and seeing how the battery backup systems work is a good one. The experience is good from the point of view that as an IT administrator and front line worker, you can see how the process functions in real time. It also allows for reflection and adjusting the plan to better serve the users of the technology systems at the school.

The school system has two legs of electrical power coming into the school system. One of those legs of electrical power shutdown at 8:30 am on this Friday morning. Lighting and outlet power were affected to portions of the building. Interestingly enough, the electrical wiring in any given room might have a portion of the circuits functional, while other outlets were dead. I am not exactly sure why the electrical circuits of the school were designed and engineered this way, but it is what it is.

When the electricity in one leg went down this morning, the main elementary computer telecommunications closet lost partial power. The phone system was unaffected along with some amplifier equipment. However, the three racks of computer switching, telecommunications and server equipment were immediately pushed to battery backup. The beeping of battery power was regular as the power continued to flow from their lead acid cells.

Within the first minute, the building's occupants were notified and by minute 5, non essential servers were shutdown including video surveillance and some backup devices. The storage server units were shutdown after users were notified; they were given 2 - 5 minutes of lead time. Our 1:1 students were able to function for about 15 minutes until the main telecommunication switching rack lost its battery inside the UPS.

The center rack containing iBoss, router, firewall and Ruckus wireless controller stayed up for an hour on its new CyberPower UPS. Other UPS units which were now lightly loaded stayed functional 30+ minutes into the event.

The take away at 45 minutes into the power outage event was that in order to preserve the wireless system in the building for a longer period of time, the primary HP ProCurve switch and the Elementary primary PoE HP ProCurve switch need their own UPS that can provide power for their functions as they are far more critical. They connect the high school to the elementary over fiber and also connect the Ruckus controller to the wireless APs. Maintaining that service for a minimum of 1 hour seems like a future goal worth attaining.

Once the power came back online, most of the battery UPS units rebooted themselves. Those UPS units shutdown manually, were restarted manually and the servers rebooted and tested. The process of bringing the main systems back online was probably 3 minutes. Some UPS units in closets at the high school needed to be manually restarted to power PoE and standard switches.

Some older small desktop UPSes showed their age and now are being seriously considered for replacement. We also know now that PoE switches draw power based upon the number of devices drawing power. At the high school, 8 PoE access points draw far less juice than the elementary school PoE which has 14 access points drawing power. Those 24 port PoE switches can draw up to 490 watts which is sure to hammer a usually good UPS in short order.

Our elementary switch structure comprised of 6 switches running at near 850 watts of power on a single SmartUPS 1500 lasted about 20 minutes. As mentioned above, 2 of those 6 switches will get their own CyberPower 2200 UPS. We will also move our X1 (Mac Mini server) that maintains the DNS and DHCP services to that new UPS to keep Internet access up as long as possible for our 1:1 folks and teachers. We believe we can go from 20 minutes of access to about 1 hour of access with a power outage.

FreeNAS Experiment - Part 1

One of the potential projects that I have wanted to experiment with is building a SAN (storage area network) box. After hearing about some positive experiences from friends like Retr0Rob (Twitter) and Bob Martens with FreeNAS over the past year, I found myself with the means to build a nice looking FrankenSAN box without any cash expenditures. Going forward, I think putting some money into the second project box will be helpful. But for this first experiment, I am in learning mode, so using existing hardware is fine.

The first decision to make for my experimental SAN box is to determine the purpose for which it shall be used in my home environment. Looking at how I currently use technology at home, there were immediately two processes that I knew I could move to a FreeNAS box. The Macbook Air I use was being backed up via a USB hard drive which meant a cable (ball and chain) attachment. The backup solutions I have in place are CrashPlan (off site backup) and Time Machine (local backup). CrashPlan works over the Internet wirelessly. FreeNAS would give me the opportunity to backup wirelessly inside my home. Secondly, I wanted to put CDs / DVDs I owned on a digital storage device that could play back through my PS/3. FreeNAS promised this functionality via its plugin architecture.

The hardware available was an HP xw4600 workstation with 4GB of DDR2 RAM, (2) 1 TB Hitachi SATA hard drives, and internal USB port connector and gigabit Ethernet. The software released this past Friday was FreeNAS 8.3 RC1. I guess timing is everything. I downloaded a FreeNAS guide and read some of the initial setup instructions. With a copy of FreeNAS downloaded, I initiated the terminal commands necessary to prepare a 4GB USB thumb drive with a bootable copy of FreeNAS. I plugged the 4GB stick into the internal USB port inside the HP xw4600, put the side cover back in place and pressed the power button.

Within 60 seconds, a text menu was showing on the monitor that allowed for refinements to the network interface settings (if needed) and the ability to reboot, reset and shutdown the FreeNAS box. Most importantly was the WebUI IP address needed for the administration of the FreeNAS box.

My first priority was to setup account passwords and add myself as a user / group for use later in assigning permissions to the shared dataset (volume) that I would create for storing Time Machine data. One of the cool features of FreeNAS is that you can use ZFS as a filing system. I used the volume management tools to create a ZFS mirrored volume of 931.5 GB out of the two 1 TB Hitachi hard drives.

Let me take some time to explain that when using ZFS to create mirrored RAID sets, there are some issues to be aware of in management. ZFS is a serious file system with safety protocols in place that make you consider how hard drive management should be done. For instance, if you create a ZFS mirror of two hard drives, then mess up your FreeNAS experiment and start from scratch, those hard drives are locked into a mirror that is not immediately usable. You must become comfortable with some command line work to fix or destroy old mirrors.

I became familiar with the gmirror and gpart commands in order to destroy the existing mirror from a previous failed FreeNAS experiment. ZFS is really pretty straightforward to learn, is logical in its command structure, and worth learning its commands to make it perform well. Here is a gmirror wiki page to learn about the command structure.

After created the ZFS mirrored volume, I created a ZFS dataset to be used as a Time Machine backup volume. For my experiment with just my Macbook Air doing backups, I carved out 250 GB of space. I then assigned permissions for this ZFS dataset to my user / group. As my friend Bob Martens pointed out, multiple Macs could use the same login credentials to use the Time Machine backup. Each Mac would identify itself by machine name.

Once the permissions were set and verified on the ZFS dataset for the backup, I next moved over to setup the Apple (APF) Share so that my Macbook Air could see the network store. I look at this process as connecting a volume with the ZFS file system to the Macintosh world which uses the Apple Filing Protocol. For those living in the Windows world, ZFS would need to be connected using the SMB (Simple Message Block) protocol.

Using the Apple Share settings of FreeNAS, I connected the ZFS dataset on the path shown in the image to the left here. /mnt/LundNAS/LundBackup

The Allow List and Read-write Access boxes are the places in which you specify the users / groups that will have Apple Share access to the ZFS dataset. To make things simple, the user / group permissions and these users for Apple Share should be aligned. I did try to use a group called Family with two users - my wife Julie and myself Perry. However, I found that it did not work well initially because of my bad ZFS dataset permissions. More future testing is required for multiple users with different login credentials.

Lastly, FreeNAS requires that the Disk Discovery checkbox should be enabled and the mode should be set to Time Machine in order for clients to backup.

My observations when configuring and using Time Machine on my Macbook Air showed me a few interesting tidbits. The Time Machine volume visible to select as my backup destination. Once selected, my Macbook Air showed it was "Looking for Backup Disk" and after finding it, mounted the Time Machine backup volume on the desktop and did the backup. Once the backup was done, the mounted desktop volume is unmounted. This is important since the Macbook is mobile and a mounted volume would cause problems for mobility.

So far the Time Machine backup server and service is working well.

ISP Upgrade & IP Changes == Moodle Fun

Pella Christian Grade School is an awesome place to visit and work. My friend Dave Te Grothenhuis is a terrific educator and technology fan. We on occasion still get to work on some projects related to the Mac installation and networking over at PCGS. Most of the things we tackle on the technology front are pretty reasonable. Today's work was a little more intense than the usual fair.

As the use of computers, iPads and other wireless and wired devices proliferates in schools, the broadband Internet connections need to be upgraded to increase the bandwidth. At PCGS, the local ISP is WindStream and they installed new equipment to get the bandwidth up to 12 Mbps symmetrically. The newly installed equipment will allow for further increases to speed in the future. The technical issue that Dave and I needed to solve was that of IP changes. The new broadband circuit would not carry over the external WAN IP addresses that PCGS had been using on their router and DMZ server. Changes were necessary and not simple changes.

After contacting WindStream to get our new IP scheme and two usable IP addresses for the router WAN interface and the DMZ server's interface, we developed our plan. We realized that the router WAN interface would change, but with firewall services and rules in place, a simple IP number change was not possible. The DMZ also had some rules and services that needed modified in the router software. The total time to modify the router was probably 15 minutes.

WindStream also provide new DNS values and those DNS IP numbers needed to be applied to our servers and other network devices. This included the router, primary OS X server running DNS and DHPC, the DMZ server, and the iBoss content filter device. DNS forwarders were changed and DHCP was adjusted to pass the new DNS numbers. Many of the DHCP leases are 2 days, so it may take awhile for client machines to function at 100% speed on DNS lookups.

The real tricky changes were saved for the last tasks. The DMZ server was given its new static IP address and DNS values. However there are multiple services running on this DMZ server including the Alexandria library software system, the JMC school information system, and the Moodle server. JMC and Alexandria were easily updated and simple redirection and updates to URL bookmarks solved issues with connecting to those services. Moodle was another matter.

Moodle is installed on the OS X Server where Apache, SQL and PHP are part of the standard Apple package install. There are a couple other ways to install Moodle including on top of MAMP or using a manual install of Apache, MySQL and PHP. We simply use the Apple packaged services as part of OS X Server (in our case Snow Leopard).

The MySQL and Apache configuration files did not need modification, however config.php for Moodle does need configured for the new IP address. There are a couple way to do this. The first way is to enter the IP manually. The second way is to use PHP code to insert the IP address in the configuration file.

• $CFG->wwwroot   = 'http://71.7.17.203/moodle20';
• $CFG->wwwroot   = "http://"._SERVER["HTTP_HOST"]."/moodle20";

However with all the setting seemingly correct, Moodle would not present the home screen. The blank white screen showed up on the browser whenever going to the URL. So, we tested Apache, MySQL and PHP component and found all were functioning. Apache was tested by bringing up the default OS X web server page. MySQL was tested by using PHPMyAdmin and reviewing the database settings, privileges and tables for Moodle. Of course PHP was running for the MySQL testing, but we pulled up the info.php page to look over the settings.

With all the basics covered, we turned to the web server (Apache) log files and started seeing that errors were showing up in the moodle20/index.php file at line 31. So using TextWrangler, we opened that php file and found that the path to the config.php file was correct. However the Moodle install was no happy and returned file not found errors. So, what is a person to do next?

We went to the Moodle forums and posted a message. Shortly thereafter, we received a response to upgrade from version 2.1 to a new version. Being conservative, I downloaded Moodle version 2.1.8 (stable) and prepared to install it after backing up the current moodle20 folder. Running through the installation process created a new config.php file and updated some of the MySQL tables (minor changes). After finishing the updates, sure enough the Moodle server started working normally again.

After 4 hours of working the solution to the issues of changing IP addresses for your WAN Internet connection, I can safely say it is not something you want to do on a regular basis. But the learning process was also a valuable one. May all your endeavors be as educational.

Law Firm Macintosh Upgrades - using the Cloud!

Taking a little time off from North Mahaska Schools work, here is a blog entry about working through upgrades to a long time 20 year client and their Macintosh setup.


Today I caught up on some work for a lawyer client who uses Macs. I probably visit the office 2 – 3 times a year for some software installs and updates. We have worked with this Mac using law firm for 20 years. They have moved from Mac Centris machines to PowerPC to Intel based Macintoshes. There last purchase was mid-2007 iMacs and recently a MacMini Server to replace an aged eMac as a file server.

We have been slowly working towards operating system updates after we made sure that production software would work with the latest OS X releases. Since the law firm as 4 iMacs, of which one is not in use daily, it was targeted as our test machine. Production software is updated as time, money and reality forces the process. Several years ago we moved away from Word Perfect to MS Word, but now run an Open Office variant called NeoOffice. Parallels 5 runs virtual machines with Windows XP and Child Support software. There is a Mac-based billing program as well.

One of the clinkers that prevented us in upgrading earlier was the calendar software program used in the olden days of the 1990s called DayTimer. The software company was purchased and the Mac version dropped from development. There is always a risk in picking any software product. We moved to SOHO Calendar for a few years but found they too were challenging us with their lack of development in the switch from PowerPC to Intel. SOHO Calendar used the OpenBase database engine which proved flakey at times. What is an IT consultant to do, right?

Apple had introduced MobileMe and it’s move to iCloud was known to me, so we bit the bullet and started using iCal (now Calendar in Mountain Lion) and iCloud to synch across multiple iMacs, the Mac Mini server, and added an iPad in April 2012 and this week a Samsung Galaxy SIII smart phone. The only purchase need to the Samsung was SmoothSync for Calendar at $2.65 from the Android app store. Since the iMac are still running Snow Leopard which is really MobileMe aware and not super conducive to iCloud, we are now proceeding with test Mountain Lion on a single iMac. If all goes well, the remaining iMacs will get the upgrade in the future.

iCloud also provides some cloud file storage so that the law firm does not have to use jump drives for file moves. Soon URL synching across Safari will also be possible. We are already synching the contacts in the Address Book (Contact in Mountain Lion). The lawyer takes his iPad 3G to court and on the road to see updates to his calendar and check his email. His Samsung phone also provides alerts in email and calendar events.

Lastly, the iCloud browser interface allows any office employee to check on their own PC or Mac when at home or on the road. So far the use of iCloud has been smooth and productive for the law firm’s needs. We are considering integration of DropBox for some file sharing and work away from the office as well and across PC, Mac, Android, iPad, etc… thought in Mountain Lion, I suspect iCloud will find some uses too. DropBox is still a better solution for non Apple software like NeoOffice.

Some people complain about iCloud’s expense, but things are changing and work well for the current client’s needs. I always try to encourage clients to use solutions that while not free, work painlessly for their needs and allow from growth that is painless as well. So far, iCloud has been a sound investment in that regard.

MacBook Troubleshooting - Beep Codes

North Mahaska is almost 6 weeks into the first year of its 1:1 laptop program and the hardware has been remarkably resilient and the students seem to be using reasonable care with the MacBook Pros. There have been some carrying case problems with straps and zippers, but that is another story. We have learned a few new troubleshooting processes and we enjoy learning new techniques to solve problems.

On a recent occasion, one student's MacBook Pro started up with three beeps and showed abnormal behavior. Using Apple support guidelines, we now know the meaning of the startup beep codes. On a MacBook Pro, the startup POST (power on startup test) checks hardware functionality. The following beep codes indicate a variety of issues. Here is the listing provided by Apple technical documentation.

On MBPro Startup: 

1 beep = no RAM installed

2 beeps = incompatible RAM types

3 beeps = no good banks

4 beeps = no good boot images in the boot ROM (and/or bad sys config block)

5 beeps = processor is not usable

One beep seems self explanatory in that there are no RAM SO-DIMMs installed in the machine. While this is not likely to happen in many cases, you will quickly know the issue.

Two beeps on a MacBook Pro indicate that the RAM SO-DIMMs are not compatible. For instance, if the memory specification requires PC3-10600S DDR3 1333 MHz RAM, that is what should be procured and installed in the MacBook Pro. Not all RAM modules are up to Apple's specification and it is wise to purchase RAM from vendors who explicitly state MacBook Pro compatibility.

Three beeps indicates a tricker predicament. The problem is most likely not RAM compatibility despite the results of Google searches. The possible problems are more likely to be a problem with the RAM seating in the module banks. It is also possible that the circuitry of the RAM banks in conjunction with badly seated RAM modules has cause a memory management issue. Reseat the RAM modules and also reset the PRAM (parameter memory) on the MacBook Pro.

A POST with four beeps is a bit more serious. The ROM instructions for booting your computer will not function to startup the computer. Perhaps the ROM was corrupted by some electrical damage involving the ROM chip and circuitry. Generally, this problem will require a motherboard replacement.

Finally, five beeps also indicates a fatal problem. This failure is found in the central processing unit of the laptop. Again, damage of some type has occurred with the CPU of the MacBook Pro, and will require a motherboard replacement in most cases.

In the case of student laptops at North Mahaska, the problems seen so far have been minor. One laptop did come in with the 3 beep POST and a RAM reseat and PRAM reset seems to have fixed the problem. Otherwise, one hard drive and one trackpad in 300 laptops has failed in six weeks. We continue to monitor the laptops with JAMF Casper and ARD software products. If the hardware continues to be solid, then we can turn ourselves to software issues, which consume more time due to complexity relationships in installation and usage.

OpenOffice to the Rescue

A curricular need became a small technical issue earlier this past week and today I believe a solution came forward in the form of a free software product called OpenOffice. OpenOffice, which is now being developed by Apache.org, is a free alternative to the every popular and over hyped Microsoft Office.

Many people use the components of Microsoft Office in the form of Word, Excel and Powerpoint. Many teachers and organizations build files on those software programs and distribute them to other teachers across the state of Iowa. Recently the FFA organization needed to use a set of files developed in Microsoft Excel with a series of embedded macros. North Mahaska is standardized on iWork software and usually Numbers (the spreadsheet program) could easily open Excel files. However, specialized macros and programming can make using Numbers impossible. The FFA files are a case where Numbers will not work as a replacement for Excel.

The first response is often to look at pushing out Microsoft Excel to a bunch of student laptops so the work can proceed and students can open the files. However, software licensing makes pushing out the software a problem. In fact, we simply are going to try to stay legal on software license when a situation like this one forces a decision on the technical staff and administration. So what other options can save money and fulfill the software requirement?

OpenOffice seems to be the answer in testing the Excel files with their macros enabled. So earlier today, I built the OpenOffice 3.4.1 package and made a JSS policy to push out OpenOffice to the 21 vocational agriculture students. That was actually pretty straightforward and within 30 minutes over half the students had OpenOffice ready to go in the background. However there are two additional issues that need to be addressed for the students to successfully use OpenOffice.

Issue 1 is that the security settings of OpenOffice will not allow automatic execution of macros. However, a simple instruction sheet provides instructions for modifying OpenOffice security for macros. Go to Preferences --> OpenOffice.org --> Security and here you can change the macro security settings to allow the user to enable macros on individual files. The other option is to trust all files with macros on the user's home directory.

Issue 2 is that OpenOffice likes to save files in .ODT format. The user can save their files as Microsoft Word documents directly within the "save dialog box". (see the diagram Dialog 1) For a more permanent solution to saving files as .DOC or .DOCX, a trip to the OpenOffice preferences will allow the user to select the file format to use in saving files. (see diagram Dialog 2)

Dialog 1

Dialog 2

Oddities come in pairs

A brand new issue surfaced with the high school teacher laptops today. Two separate MacBook Pro laptops from two high school teachers threw a little curve ball on the technology team here today. Teachers decided to logout and log back in mid morning. Both were running OS X 10.7.4 and a standard set of applications. Both also were set to launch the open applications when logging back into their mobility accounts. One laptop had Mail, Excel and Word running. The other laptop had Mail running.

After a quick look on Apple Remote Desktop, we decided to bring them back to the office. The English teacher was giving her laptop over to upgrade to a new Macbook Pro (Summer 2012 model) and 10.8.x. The other teacher was to keep his late Fall 2011 Macbook Pro. We performed some basic maintenance procedures like Repair Disk and DiskWarrior and Repair Permissions. This did NOT solve the issue.

The mobility accounts synched their accounts, but would not go to the user's desktop. Instead each Macbook Pro presented the desktop background (the universe), the Apple menu icon and the Spotlight icon. No menu or desktop icons were displayed. The favorite spinning multicolored beach ball had replaced the mouse pointer cursor. On a brighter note, the local administrative account worked flawlessly. The culprit had to be a bad preference file for Finder that was launching and restoring the application prior to showing the desktop. Bummer!

The Lion desktop with no OS X Desktop showing.

Now most people like to pull out their hair at this point. I admit that seeing this for no apparent reason is frustrating, but not from the point of view that no solution can be found. The problem is that of answering the question, "why does this happen?". In my 25+ years of experience of playing Q&A with the end user, little information is garnered to shed light on the "why" question.

In this series of events, we already knew that we were moving these teachers to Mountain Lion from their current Lion software. We were not sure if the two processes for upgrading to Mountain Lion would solve the problem, but we decided to try both.

Process number one was to login with the local administrative account, copy our Mountain Lion installer to the Application folder and start the upgrade process. The second process was to take a new June 2012 Macbook Pro and use the migration tool and the older Macbook Pro in target mode with a Thunderbolt cable to move the mobility account. We proceeded with both options on the respective laptops.

Both machines finished the processes and when we logged into them, both problems were solved and the laptops were working fine. The downside is that we really do not know what happened to corrupt the Finder preferences for restoring the desktop on login. So that is disconcerting to us until we read up on the discussion thread if this is a known issue.

If anyone reading this blog has run across this issue, please let me know. I would love to share the issue in more detail and what causes this behavior. Is it a user behavior or a software glitch or some combination of how the user interacts with the laptop's operating system.