A Pound of Cure

Experience is the hardest teacher. And the cruelest.
I had a blog worthy experience the other night while patching a BI4 production system.

Back in the good old XIr2 days, we used to have to take a total outage to apply a server patch to BusinessObjects Enterprise. On that version, the patcher needed the services to be down in order to update them, then you would start them back up manually once you were finished. In a cluster it was the same.

Along came XI 3.1 and it’s clever little Server Intelligence Agent, and took that need away. Now you had to have a CMS up in order to apply a patch.

Additionally, in XIr2, when building a cluster, you could build one node all the way up first, then do your expand install and add your second node, then patch that one all the way up. This changed with XI 3.1 as well. You have to keep both nodes at the same patch level, so the patch process is more like a see-saw, back and forth, back and forth, until you reach your desired patch level.

Now to the experience from the other night. First, while not in the official documentation, there was an SAP note that was given to us by SAP Support that states that when patching a cluster, you should only have the node up that you are patching. All of your other nodes should be down. We found this out the hard way, while trying to install Patch 14 on a BI4 Production box. It got all the way through to the BOBJBiarFile step, and there it sat for over 2 hours. Sitting. Sitting.
About an hour in is when I decided to open up a support ticket with SAP, and when this SAP note came into the conversation. And in the Support Engineer’s defense, as soon as I shut down the other node, the installer kicked right back in and finished within a few minutes.

Now I could get philosophical at this point, and ponder the meaning of a BOBJBiarFile deployment hanging, or what could be included in a BOBJBiarFile as a part of a patch anyway. I know from watching the setupengine.log file that it was Webi Sample Reports, and again one could fall down the rabbit hole and ask why. But that’s a discussion for another time. The fact is that your patches DO deploy a BIAR file that contains sample reports. So also unlike the good old XIr2 days, you can no longer reclaim your disk space by deleting those sample report BIAR files. Subsequent patchers will require them.

For reference, the SAP note that discusses how to patch a cluster one node at a time is SAP Note 1358403 – How To: Install Patches to a 3.x cluster (logon required). The SAP Support Engineer told me in writing that this also applies to BI4. The summary of this note is that you should bring all cluster nodes down but one. Patch the first one, then while leaving the others down patch the remaining nodes using the first node to authenticate the patcher. The long and short of it is that all CMS services that are members of the cluster must must must be at the same patch level. By leaving one up as a “master node” and patching that one first, the rest get patched while they are down, and when they do come back up will all be at the same patch level.

So to wrap up, if you are patching BI4 clusters, be sure to schedule an ample downtime. So far, the BI4 patches take about an average of 3 hours to run. Multiply that by the number of nodes you have in your cluster, then add 10% to it to account for a visit from Mr. Murphy. Get your outage window going, then shut down all but the node you intend to patch. Take your necessary backups ahead of time (especially those web applications), then get to patch pushing. Also make sure you have plenty of free disk space before you start. The patcher files alone are about 2GB, and require 8 or 9 GB of free space to install. You do NOT want to run out of disk space during a patch installation. You’ll be doing Disaster Recovery if you do. Benjamin Franklin obviously had SAP BusinessObjects Enterprise in mind when he said “An ounce of prevention is worth a pound of cure.”