My last post on Adobe Certification
As a follow up to my open letter on Adobe Certification (which again, never garnered an official response), I have an update to share with my readers. Before I do so though I have a lot of preparatory text I need to get out of the way. The following blog entry is based on a meeting I attended (non-NDA) for Adobe User Group Managers and Community Experts. It was specifically on Certification. This meeting left me frustrated and - frankly - a little bit more angry. I should perhaps have waited before blogging - but I wanted to share the notes while they are still fresh in mind. Just keep in mind that this may not be my most rational response. I'm emotional now (scratch that - I'm angry) and if you want to stop reading, I won't mind.
Note that when I use quotes it should not be considered exact quotes. Rather it is my notes and I wanted to visually differentiate between the slides in the meeting compared to my thoughts.
Software is Adobe's core business, not certifications. Anything they do outside of software though is meant to help the core business.
Very fair point and I agree completely. I wouldn't fault Adobe for a poor grammar choice in their docs for example.
Certifications serve three main purposes: insure adequate technical knowledge for certified instructors
ditto for adobe solution providers
provide credentials to help developers get more business and better jobs
Makes sense to me.
Certification is expensive. Costs include development, validation, localization, publishing, and tracking. The cost is around $50K per exam and every change is a $2.5K cost. (Unless they just turn off a bad question. Adobe does not make money on certifications.
Ok, no big surprise here, but frankly, I don't care. Really, I don't. The whole point of this issue is the quality of the certification and the fact that Adobe is selling something they don't stand behind. When the presenter was pushed on this, he said some internal forces were pushing back against just pulling the exams. Frankly those internal forces are wrong, in my opinion. Writing certifications are hard too. But darnit - if you can't stand behind it - don't sell it.
The creation process involves writing objectives (which is started fresh for each product version). Then they find individuals (like me) both internal and external to help write them. They are then reviewed (again, by folks like me, both internal and external). Based on feedback, Adobe sets a passing score for each exam.
For my thoughts on the write/review process, see the old blog entry.
After publication, how does Adobe verify the certification? They check pass rate. They aim for 60% of folks passing. They check the amount of time it takes to pass the test. They also do a 'point by serial correlation' check where they can look at a question that folks missed even though they passed. That's a good indicator of a bad test. Unfortunately it costs for Adobe to look at these.
Nothing much to say here - but again - I don't care about the costs (admittedly, easy for me to say).
Now we come across the slide that made me see red. Just ask the guys on my IM buddy list. They can't print here what I was saying:
Bad things can happen, and did. The first bullet point of blame was on the writers and reviewers. The last bullet point was on the vendor. The vendor was outsourced and looked good on paper.
So. Um. Can you see where I got angry? Hint - it's the second sentence. To say I was a bit upset would be a major understatement. Myself, and the the two reviewers, were very vocal. I also emailed Adobe. One of my contacts on the CF team, who I will not name, basically said it was the cert divisions problem, not the CF teams problem. So when the blame was laid right down on me personally, I kinda flipped.
Now the presenter, at the end, came back and acknowledged that that wasn't what he meant to do. He personally apologized to me. (And I'm not going to name him because he is an old timer, an Allaire guy, someone I know, and he was in a very bad position here and I don't want to shoot the messenger.) And let's be frank - you guys have seen my code. You know I make mistakes. Hell, I love to blog my mistakes because I figure people can learn from them. (Just ask me about how I messed up this CF server today. It was a real bonehead mistake.)
That being said - that was one of the worst slides in history. The number one bullet needed to be that the blame was squarely Adobe's fault. Stop passing the buck!
When pushed on the "if you won't stand behind it, don't sell it" point, again, the buck was passed, this time to internal forces. Apparently these internal forces won't let them stop selling the certs.
Adobe repaired as much as possible, but did not rewrite the certifications. One test was pulled. One test was not required for instructors anymore (Dreamweaver). One was totally rewrote (InDesign). On another test, it had a lot of issues but Adobe left it be since folks tended to pass because they figured out the right answers.
Not much to say here. The last sentence is a bit hard to believe though. Folks shouldn't have to figure out a certification.
The tests will not be rewritten for the current versions.
Last note - there are no plans (as of yet) for a Spry certification.
I don't know how much I can summarize this. I'm trying to clear my head a bit. I can say it is extremely frustrating. Extremely. I try not to say I have a "big" blog, because I figure as soon as I do my traffic will drop (and frankly, that guy with the big arms is kicking my butt ;) but I did expect to hear something on the blog from my last entry.
It sounds like Adobe really gets that they have a problem. But the #1 thing I got from this meeting was one big giant buck passing. Blame the writers/reviewers (and again, I'm not taking this as personally as I was initially). Blame the vendor. It's the cert team fault. Etc. None of this sounds good to me. I'm more than willing to say I know more about My Pretty Pony than large multinational corporations and economics, but at the end of the day, I don't believe Adobe is doing the right thing here. I've been called an Adobe Fan Boy on more than one occasion but I got to speak up here.
Comments
It would have been great for him to properly respond to your concerns in the chat and the previous blog post. I am really surprised at that honestly. If there is good feedback sent back but it goes to a black hole, then that is indeed a real problem. Maybe that is something they can work on during the whole openness effort. Good quality format for feedback that goes to the right person and is measurable...
yikers.
@joshua: The presenter, at the end, made it VERY clear it wasn't a personal attack on myself. I leave it up here because I honestly feel its part of a bigger issue of not accepting responsibility for the cert.
I do agree though - I do wish Adobe had responded to my post. I do NOT want to sound like a diva here. I get a LOT from Adobe. Early sneak peaks. Software giveaways, etc. But again I've been surprised by the lack of response on the blog.
A smart business leader would see that though there may not be direct returns on investment, there are infinite benefits to the company overall.
I would bet that what it would cost for them to certify ME comes out to far less than what I would then continue to spend on dev tools - software that I might opt to get elsewhere in the future if I wasn't already invested in Adobe in kind.
Education in IT is important - period - and Adobe just doesn't get it. I tried to hire an entry to mid level CF developer at our company (based in Indianapolis) for 2 months and guess what - they don't exist. Why? Because there is no new blood when it comes to CF. Why? Because Java, PHP and .NET have a long tradition of getting into universities via giving shit away or just because they're free offerings to begin with. Adobe does not care about education or certification because they're still selling product (for now). ColdFusion may already be doomed because it is no longer a dominant player in the app server market and it is not taught anywhere, grads get jobs then they rise through the corporate ranks and never they never consider CF as a solution at their respective companies. How did you learn CF? Self taught or taught by others at work? Maybe you learned the hard way like I did - by building apps in it and looking through the CFWACK books to make them better. Maybe you were lucky enough to find this blog or some others right off the bat.
At any rate, Adobe doesn't care about certifications or finding new users. When was the last time you saw a CF add in Information Week? I spent an hour on their website trying to map out CF and FLEX certification details and came to the conclusion that they don't care. If they did care then I would argue that the site wouldn't have been such a mess. Go to Microsoft or ORACLE if you want to see a software company that understands that a developer's career is often tied to a software vendor and how much they can prove they know about that software vendor's products and overall technology platform. Recently I've developed serious worries about building a career around CF. I'm taking my personal certification/training budget and using it on C#.
CF is my favorite and it is such a productive environment but it may be the next Novell. Do any of you want to cross your fingers and stake your careers on a niche? Is Adobe really spending any money to recruit new developers or show the world their great product? No, their completely insulated and surrounded by folks who already love their stuff. They're not out there trying to get people to switch from other solutions to CF. Show me one shread of evidence that they are and I'll shut up and post a retraction to that statement.
When you say Adobe do your colleagues and clients immediately think Photoshop and Acrobat? Do they even know what CF is? Who's fault is that?
Thanks Ray!
This will be a good test - if Adobe really does listen to us they will fix this.
I'm well aware of this and in my opinion it is about 5 years too late. The blame can't be completely put on Adobe as Allaire and Macromedia had the same shot and blew it.
You might say that the culture and image of our product of choice has evolved into one of being a maverick. If you think that's pretty cool just ask Borland how that's going for them.
Anyway I'm a sore loser and I'm pissed off that my product of choice is losing. Losing market share, losing mind share and eventually it will start losing money and it will get dropped. And all of the truly smart people like yourself and Ray and everyone else will go off into the sunset and master something else.
Lately being a CF proponent seems probably a lot like being a Detroit Lions fan. The team keeps getting worse and the front office keeps f@@king it up even worse. I wouldn't know about that because I'm a Colts fan and we finally got our act together.
What's sad is that this is a terrific, feature rich product with an inventive group of users (dwindling in numbers) who would lay down in traffic for it. CF is way better than the competition in many ways. But the folks who have made it great over the years aren't that good at the business side of this. How in the hell do you lose the thrown when you're the first one there? (CF existed before ASP and PHP)
I would disagree with you on this. I think the folks inside Adobe (specifically the CF team) are "getting" it more and more and come with more business savvy at the same time. People like Mike Nimer, Tom Jordahl (and other older CF engineers) have given way to newer developers and with Jason Delmore and Adam Lehman (controversy aside) on the product side, I think we can see more and more out of the CF team.
If they're "getting it" and CF8 was released a year ago then why no marketing campaign that reaches for new users nor any mainstream advertising? Why did it take just now before it dawned on anyone that CF is never front and center at universities or training centers (the place where many developers are born) and that maybe if it were given to colleges they might use it and expose their students to it. Oh but wait, how could said colleges teach classes on it if there aren't any decent training materials or certification tracks that mean anything?
I live in the 12th largest city in the US and ran an ad for an entry to mid level CF developer for 4 months and had 2 qualified candidates - each of whom had more than 8 years of experience - not exactly "mid level" or new blood.
So what is it that we'll "see more and more of"? More people picking up .NET or PHP? It took me many many months to get this upset. Do you think that I want to abandon a world class product like CF in favor of other products that are not as productive? No way, but if I want to continue to grow my business I'll have to because I can't seem to hire anyone and Adobe doesn't seem to place too much emphasis on growing the user base, marketing, education, training or certification.
This isn't the place to get into this here, but I understand where you are coming from. I was teaching CF in 2001, so I understand where the education aspect lacks. I've been in talks with my university to bring RAD web tools like CF and RIA into the curriculum. Also, as a hiring manager myself, I've seen the lack of talent out there as well. I've interviewed hundreds over the years. Lots of horror stories.
At the same time, I've noticed some changes inside Adobe. I've been a beta tester since version 6, and I've never had more access to make a difference myself. So maybe what I meant by "see more and more of" is what I'm seeing on the inside, and hopefully most others will see on the outside.
Adobe is a very mature company, and at the same it doesn't react quickly sometimes. In the post-Google age, its becoming less of an option to think old school.
So what am I saying? Don't keep frustrated, have some patience: keep the faith. With Railo, and changes announced for Centaur, CF is going to be accessible and "complete" than ever before; someone is listening... finally!
Why? Well if nothing else it "could" give the client a little bit of a warm and fuzzy knowing the entire team is certified, but that assumes that the certification is actually worth something that its not easy to get.
I will probably end up just getting it anyway but these revelations bother me. Certifications should be hard to get and worth the effort in getting.
Developers generally* have a strong sense of honour, and a highly analytical mind. We are engineers; if something is broken, it must be fixed or torn down. I find that developers find middle-ground a painful place to be.
Adobe through certifications have poured away credibility. It takes a hundred bits of good news** to out-weigh one bit of bad.
Adobe - remove the certifications, rewrite them, and then re-release them.
Ray - I read your article - the anger was creditable.
Joel
* Apologies for generalising
** e.g. the bbc statement yesterday
A headhunter had me take it a few weeks ago... most of the questions were pretty obscure, but I did a lot better than I'd expected:
ColdFusion MX
Score: 4.69
Percentile: Scored higher than 99% of previous examinees
Account Percentile: Scored higher than 99% of 658 examinees within this account
Proficiency Level: Expert (Master)
Haven't taken the CF one, but it sounds like it's not worth bothering with. Is there another one out there that is? Most people I mentioned Brainbench to said they'd never heard of it....
I've been dreading this exam, but unfortunately, it's a necessary evil for me to advance in my job.
Thanks for being vocal about this Ray.
A process could be something like design/create an application to do foo. Requirements: must use transfer/MG(or whatever)
4/5 panel person reviews the submissions then awards points for each person.
So you pick up points by participating in challenges or submitting work or by prior experience points from graders.
Once you build up enought points, we get a shiny plastic badge and a piece of paper from the community that says you know or possess skils to develop apps the CFML.
again - just a thought I had. If it is close to one your ideas, I'm a bad reader so I overlooked yours. I apologize in advance to that.
I consider myself jr going on midlevel myself, and I can see what you're saying; I feel kinda unique, in that the majority of CF developers I know have been at it for 5 years minimum.
I've been at it fulltime for about 3. (and with that comes learning CF by myself, learning bad habits, breaking bad habits, trial, error, etc)
I too hope that somehow Adobe really makes a push for CF very soon. While I may still be so early in the game I probably still qualify as a 'fanboy', I'd rather not have to move to .Net.
Please, Adobe? Do something?
I had a headhunter ask me to take a test yesterday - and was I SADLY disappointed. First - their customer is running CF7 and migrating to 8 - let's start there.
So the "test" to "prove" my resume - was based on CF SERVER 4.5 and CF STUDIO 4.5. Not only were the questions SERIOUSLY dated - but some of it was out-and-out BROKEN. (ex. missing "=" signs between tag attributes and their values).
If this had been some hand written test - I might have been less shocked. But this was from a "testing company" that provides "certifications". NOT IMPRESSED to say the least.
However, if you do want to or have to (job promotion, training requirement, etc...) take the CF8 certification exam, you might want to consider getting CF8 Exam Buster which I just released (http://centrasoft.com). I won't plug any further. Thanks, Brian