Windows Updates have their upsides. They keep us safe from bad stuff on our servers. Conversely, automated Windows Updates that go untested can surprise us with unexpected outages. Such is the case today when Tomcat running in an SAP BusinessObjects cluster started throwing a 404 error that oddly…didn’t actually look like a normal Tomcat error when viewed. Tomcat was unreachable. A quick look at stderror.out revealed the error:
java.net.BindException: Address already in use: JVM_Bind :80
Your mileage based on Tomcat version might vary, of course.
The first bit of investigation lead us down the path to ensure that Tomcat was running and the service was there. A quick check at Task Manager revealed that it was in fact started.
Next, we turned to a quick test of the service using telnet and observing that this Tomcat server had been reconfigured to listen on port 80 instead of the default 8080. Telnet was tested by typing ‘telnet localhost 80’ to see if Tomcat was listening properly. It failed to maintain a connection. This made it clear that despite the service running, it was not able to take a connection.
Next, we ran the command ‘netstat -nao | find “:80″‘ to look for any process listening on port 80. The interesting find here was that there was more than one process ID listening on port 80. This was our first indication that something in the Windows Update caused another Windows Service to begin to listen on port 80.
This discovery of process ID 4, the System Process, was causing Tomcat to choke. This service is apparently used by a number of potential Windows services. A quick turn to a thread that showed up on superuser.com revealed that a quick registry fix and a reboot would stop the System Process from trying to grab port 80 traffic.
Those steps are as follows:
- Open regedit
- Locate the key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP
- Back up your registry entry, unless you are feeling bold
- Set the value of “start” to 4 (effectively disabling)
- Reboot
Sure enough, after the reboot, Tomcat started and it accepted our connections. At the time of this post, I’ve not had the opportunity to dig into which Windows Update caused that behavior. If you spot it, feel free to leave it in the comments.