A few days ago, Adobe released updates for CF 9, 10 and 11. I’ve been following comments raised about them since, and I’m disturbed to see some developers struggling to get their servers patched, especially when applying the update has taken the server down. I don’t know if these cases are isolated or might be more widespread, but I find this unacceptable.

I run several instances of ACF on CentOS. I’m subscribed to the CentOS-announce Digest mailing list, as I’m sure most CentOS server admins are. For many years now, whenever an update is released, perhaps once or twice a week, I ssh directly onto our servers, run yum update, and generally within less than a minute, the update completes. Never once has the process failed. Never once have I seen an incoming email on the CentOS mailing list that someone experienced a server crash, or any other issue for that matter, because of an update.

This is how ColdFusion updates should function, flawlessly.

In stark contrast, CF developers have become wary of applying updates. There is a risk that their server will compromised because of the security vulnerability the update is meant to patch, and there seems to be an equal if not greater risk that their server will be compromised, in one way or another, in the update process. CF 10 introduced automated updates, but as we see, the automated update process remains error prone.

It should not be like this. Rather than focusing on new features, the CF dev team needs to focus on getting the update process as flawless as it is on CentOS. What they seem to have lost track of is that, for a start, developers have families to support. Our clients expect us to keep their servers up and running 24/7. Our clients depend on the applications we build to run their businesses. As developers, we cannot afford to experiment on our clients. And yet, when Adobe releases a flawed update that takes servers down, effectively experimenting on us and our clients, oh, look, they are trapped between a security vulerability and a flawed update, the implicit statement received on our end is “We don’t care if you keep your clients. It does not matter to us at all.” Really?

What is even more disturbing is the way the Adobe dev team now uses us as their QA department, as if ColdFusion server was loosely organized open source project among wannabe hackers. It is not disturbing because the CF devs are actively reaching out trying to solve problems that are occurring, asking for logs, error message screenshots, suggesting the possible need to copy and paste jar files from here to there if some functionality is broken, instructing us to manually install the updates one at a time if the automated updater doesn’t work, etc. It is because these interchanges should never occur in the first place regarding an update. The update process needs to be flawless, as near to it as possible, so that we can rely on it. Like it is on CentOS. If they can do it, the CF dev team can do it.

The worst, however, is this, linked for reference but replicated below, an Adobe engineer asking a developer, a paying customer, to effectively crash his server “in off peak hours” so that he can get at the log files that might reveal what the problem is.

Hi Mark,

Of all the things connector logs with debug_level=debug are the most important one’s. Till now with all the information you have shared I can tell that there is some problem with shared memory access between worker threads. But to exactly pin point the problem I will be needing the connector logs. I understand it will be difficult for you to do something on production server but it is a must for this problem to be resolved.

If possible in off-peak hours, apply the update14 and reconfigure the connector. Once the connector is installed go to connector directory c:\coldfusion10\config\wsconfig\magicnumber\ and open isapi_redirect.properties in notepad.
Edit the line “log_level= info” and change it to “log_level= debug” and save the changes. Restart IIS, hit any cfm page and wait for app pool to crash. Once it is crashed take a backup of isapi_redirect.log(created in connector directory) in some other directory like c:\connector_logs\ directory. After the backup you can revert back the changes.
Lastly you have to share this backed up logs with us.

Thanks,
Milan.

When I say this is unacceptable, I mean that it is not acceptable for Adobe to use CF developers and their clients as their QA team, especially in that Milan says “I understand it will be difficult for you to do something on production server but it is a must for this problem to be resolved.” No, it is not a must to take down our servers, possibly endangering our client relationships, to fix your bugs.

Yes, this is very complex stuff. But in today’s world, that simply means Adobe needs to employ a vastly improved QA strategy than the one they have in place today for ColdFusion releases, especially for the update process itself.