When the HeartBleed bug was unveiled, I checked all of my servers to see whether they were running vulnerable versions. They weren't, but once the patched versions were released it seemed a good juncture to test and roll out the update to one server.
What followed was something of a headache, initially with all the markings of a serious compromise.
Having now identified and resolved the root cause, I thought I'd write a post about it so that others seeing similar behaviour can get something of a headstart.
In response to threats such as CDorked, I run PHP Changed Binaries on all my servers, so any file in PATH is checked (daily) for changes, based on a cryptographic checksum. If any changes are detected, an alert is raised so that I can investigate the cause of the change.
The day after I updated OpenSSL, I started receiving alerts for a wide variety of files (I'd updated hashes following the update of OpenSSL)