Big Question for January - Quality vs. Speed

The Big Question on the LCB for January is:

What are the trade offs between quality learning programs and rapid e-learning and how do you decide?

There's a lot involved in this question and I plan to revisit it several times during the month. I'm hoping to see some contributions that will help thinking.

Let me start with some important words of wisdom that I like to use at meetings:

You rush a miracle man, you get rotten miracles.

Props to the first person to leave a comment that identifies the source of this quote.

Only slightly more seriously, consider the following graph that I often use in presentations to illustrate a point:

Expenditure on training has a limit on the performance gains you can achieve. After a certain point, the cost of the Training exceeds the value of the performance gains. And often there's an minimum level that you can do where anything below that level would cause too many problems. This graph is completely over-simplified to make a point. In reality, there are many different ways we could make our expenditure and each one would have a different cost and effect on performance.

So, the real challenge posed by the Big Question is knowing when its really worth it to spend dollars on what we might consider a higher quality solution than providing something simpler that we know won't be as effective at improving performance.

And this isn't a theoretical question - it's something we face all the time. We are pretty sure that people will learn less from a rapid eLearning piece that's basically just a PowerPoint + Audio as compared to an hours worth of fun, interactive courseware. But, if the cost is significantly higher, is it really worth it? In what cases?

And it's not just an ROI question. Often, there's a lot more to What Clients Really Want than business outcomes that can be quantified into dollars.

In my mind, there's clearly no easy answer to this question. I'll be curious to see some of the ways that people attack this over the next few days.

What Clients Want

Karyn Romeis created an interesting post: Assessments in elearning responding to an earlier post of mine: State of Assessment by E-Learning Developers. This simple exchange has sparked quite a few different posts that I'll work on over the next few days.

Let me start with something I consider to be almost misleading in our industry. We hear all the time about how we SHOULD be doing Level 3 & 4. Will just said in his post Assessment Mistakes by E-Learning:

Stunning: Even after all the hot air expelled, ink spilled, and electrons excited in the last 10 years regarding how we ought to be measuring business results, nobody is doing it !!!!!!!!!

In Approaches to Evaluation of Training: Theory & Practice the authors state.

Evaluation becomes more important when one considers that while American industries, for example, annually spend up to $100 billion on training and development, not more than “10 per cent of these expenditures actually result in transfer to the job” (Baldwin & Ford, 1988, p.63). This can be explained by reports that indicate that not all training programs are consistently evaluated (Carnevale & Shulz, 1990). The American Society for Training and Development (ASTD) found that 45 percent of surveyed organizations only gauged trainees’ reactions to courses (Bassi & van Buren, 1999). Overall, 93% of training courses are evaluated at Level One, 52% of the courses are evaluated at Level Two, 31% of the courses are evaluated at Level Three and 28% of the courses are evaluated at Level Four. These data clearly represent a bias in the area of evaluation for simple and superficial analysis.

Maybe the authors didn't mean to say this, but they clearly say that 10% transfer is because of inconsistent evaluation. And clearly we are all biased towards the "simple and superficial." Yikes, I feel so dirty not doing Level 3 & 4.

But wait, we've all had the experience of when we try to do Level 3 & 4, we face considerable push back from internal clients and line managers. It's not that expensive to get the data nor really that much work, but it does take commitment. Obviously, the level of effort isn't worth it to the line managers. And this will be a topic for a future post, i.e., why don't clients care? But for know recognize that when you start with a client, you need to do a quick assessment of what they really wants:


Do they really care about effectiveness: changing behavior and driving business results? Many clients don't really care about this. They have a particular product/project in mind and they want you to get that done. As Karyn said in her post, the client will tell you, "Don't worry, your job is safe." They don't care about business outcomes. You'll often find this out pretty quickly when you start asking questions about "What do you expect people to do differently after this intervention?" or "What numbers are we trying to hit?" A blank stare often indicates that they don't really care. I actually think a surprising number of "training" projects involve clients who are on the don't care end of the spectrum.

Do they care about the looks of the product/project? Many clients come in with expectations about what will be produced and likely around what a "good looking" project will look like. They may be looking for highly interactive, or fun, or engaging, or games, or simulations, or ... If you suggest that a checklist might actually have better results than an hour-long, interactive training course, you may get lots of push-back. Some clients honestly don't care about looks, but most do. In fact, its much more common for a client to care about looks than to care about personality (I mean business results). Good looks are important in that it often implies user engagement, but this post is purely about assessing what the client really wants.

Remember that beauty is in the eye of the beholder. You definitely need to understand what they believe will be a good looking project. It may include what kind of format, graphic design, level of interaction, etc.

Also keep in mind that your assessment of what the client wants may also need to include thinking about other stakeholders (end-users, the client's boss, etc.) and what they want. Clark Aldrich recently suggested that helping your client get a promotion might be the ultimate goal. Thus, the "wants assessment" in that case needed to include assessing what your client's boss really wants. Do they care about business results? Do they want something good looking?

Finally, please, please, please do not show this graphic to your client (or ask the implied questions directly). Every client will tell you that personality is much more important than looks. Of course, they want it to be effective! Of course, it doesn't matter how good looking it is! They know that's the right answer! That's what they will tell you if you ask. Your job is to find out what they really think.

Stay tuned for more ...

More on Blogging and Community Issues

I just saw that TechCrunch added a forum to their blog. This relates closely to the discussion going on around blogs, discussion forums, communities, etc.:

• Blogs and Community – launching a new paradigm for online community?
• eLearning Discussions - An Attempt at Better Discussions in the Blogosphere
• Move from Discussion Groups to World of Blogs?
• Top Ten Reasons To Blog and Top Ten Not to Blog
• I found it inside my blog reader!
• Blogs vs. Discussion Groups or Mis-Understanding Blog Reading and ...

Many of the people posting into the TechCrunch forum are asking some familiar sounding questions: Why would a blog need a forum? How does the forum relate to comments?

And this is a case where it's only one blog. In our world, there are many blogs. So, it's even more complicated of an issue.

Top Ten Suggested New Year's Resolutions for eLearning Professionals

I just saw a great post The Knowledge Worker's New Year's Resolutions. It suggested some Resolutions for the Knowledge Worker. In short, it's a great post and worth reading on its own, but in the meantime, I've stolen some of what I think are the better ideas and am suggesting this to everyone to consider putting on their list of New Year's Resolutions.

And, yes, I know I'm a week late, but since you've now violated half of your resolutions, you can replace those with a couple of these. :)

So, here are my suggested resolutions:

1. I will find a project even if I have to do it on top of my current work where I can make a difference in business outcomes that truly matter to the business, already get measured, and will do my best to really make an impact on these outcome.
2. For the rest of the projects, I'll find out up-front where on the spectrum we are from checking-the-box to really impacting performance and will make sure that I act accordingly. In other words, I'll be passionate about impacting performance when that's really what's being asked for. I'll try to minimize the amount of time I spend on check-the-box training.
3. The next time I'm asked to produce an eLearning course, I will figure out the least content that I could present up-front to get people started and move the rest of the content I will present as a Reference Hybrid.
4. For the little bit of content that I do create as an eLearning course, I will spend at least 30 minutes brainstorming with colleagues on how to make it interesting, relevant and fun. I'll consider games, simulations, compelling storylines. I'll think like a novelist, movie producer, someone who cares about engaging the audience.
5. I'll collect a list of resources such as Clark Quinn's - Seven Steps to Better eLearning that I will use to challenge my eLearning course design. (What else do people use?)
6. I'll help my boss understand that they aren't Captain Piccard and that the enterprise is not the Starship Enterprise. "Computer make it so" doesn't yet work in this day and age. Even with Rapid eLearning and Ruby on Rails.
7. I won't do shovelware unless its required, then I'll figure out how to get other people the tools to shovel it out.
8. I will propell my own learning forward by starting my own blog and participating in the Blogosphere.
9. I will avoid featuritis when selecting tools especially an LMS.
10. I will get smart in eLearning 2.0 tools for personal and group learning and will help at least one group use them during 2007.

These are only suggested resolutions. I'm sure we'd all like to hear your actual resolutions or suggestions for additional resolutions. Either that, or tell me I'm nuts.

And once you create you actual resolutions, I would highly suggest you share them with your boss, your colleagues at work, and, of course, all of us (your peers).

As a bonus Top 9, I reformatted and edited some of Basex's Email Resolutions so that I could circulate them internally.

1. I will refrain from combining multiple themes and requests in one single e-mail.
2. I will make sure that the subject of my e-mail message clearly reflects both the topic and urgency of the missive.
3. I will read my own e-mails before sending them to make sure they are comprehensible to others.
4. I will make sure that the first two sentences of the email make it clear what the subject and required actions are relative to the email.
5. I will make my emails as short as possible.
6. I will not overburden colleagues with unnecessary e-mail.
7. I will reply judiciously, avoiding one word responses such as "Thanks" or "Great!"
8. I will be more conscious of the fact that "Reply to All" sometimes really does go to all.
9. I will not use email when the content of the communication has emotional aspects that won't be effectively communicated via email.

State of Assessment by E-Learning Developers

Will Thalheimer posted a must-read Assessment Mistakes by E-Learning Developers which I think is mistitled. It should be the "State of Assessment." I'm not at all surprised by his findings in that they echo much of what I've heard over the years. Among them...

11% said they did NO evaluation
26% said they did Level 1 smile sheets
48% said they measured Level 2 learning
15% said they measured Level 3 on-the-job performance
0% said they measured Level 4 business results (or ROI)

His discussion further points out that doing Level 2 assessment immediately at the end of training (i.e., did a post course test ... likely multiple-choice) was not nearly as accurate as assessing later on.

The comments that follow are interesting, but I'm surprised that we aren't being more honest about this. I personally am a big proponent of using metrics throughout the solution (at analysis to define what it is we are really going after, as part of the intervention to provide direction, to assess the outcomes and make change), but often unless the metrics are already being collected by the organization (e.g., customer satisfaction surveys), then Level 4 will not happen. And, likely Level 3 - even though it can be approximated (cheaply) with a very simple survey won't happen unless those questions are already being asked other ways.

Why is it? Are we bad at our jobs? Do we not care? Hardly. But we must make choices about where we push. When you discuss a simple post performance intervention survey to assess how we did in changing performance with internal or external clients, the response you get is "they won't spend the time to do it" ... and they are generally right. Again, unless its part of the intervention itself (pre-post assessment) or part of the culture of the organization, this isn't a battle that's worth fighting for as compared to other battles.

2006 and 2007 - Dump Microsoft Now and Best Posts of 2006

Stephen Downes provided a list of year-end wrap-ups that had several interesting posts. Included were:

• Wesley Fryer article that says that in 5-years, school district says - "not a single desktop in this 52,000-student school system in metropolitan Dallas will carry the image of a proprietary school software program."

Wow - dump your Microsoft stock now!

• Doug Belshaw post of his "20 top blog posts of 2006"

Doug's perspective is a bit different than mine, but the list was interesting to see. I'm curious what people in eLearning felt were the top posts of 2006? What struck you? I'll have to see if I can figure that out for myself.

eLearning Discussions - An Attempt at Better Discussions in the Blogosphere

Happy New Year (with usual caveats for Chinese New Year)!

Let me ramble a bit here and then there's something really good at the end (at least I think so).

A really great comment came in during the holidays relative to the post: eLearning Technology: Move from Discussion Groups to World of Blogs? Craig said something that has troubled me for a while:

Blogs are not that great as a platform for discussion.

I partly agree with him in that I've felt for a long time that the Blogosphere is a loose conversation and lacks the single location effect that you find in discussion groups. Craig highlights this with an example:

Case in point: If I have a tech question and I post it on my blog, my chances of getting a response are fairly low. If I post it on a blog where they talk about the tech I am interested in, my chances are higher. However, if I post to a tech forum, I am almost guaranteed to get a good discussion going.

And, I would tend to agree. However, if I have an eLearning Question, I think I'm much more likely to get answers to it by posting in my blog right now than posting a question in a forum. Especially a question around something like eLearning 1.0 vs. 2.0 - Help Needed (and I got lots of great help). On the other hand, if I want to find out something more specific around ROI, I would definitely post on the ROI group. They may change the question entirely and answer something else, but I would be more likely to get a response.

Craig then comments:

I also think that the relatively low numbers of individuals in elearning who participate in blogs and discussion groups may be in part to the fact that:

a) They are so busy developing / launching their elearning programs they just don't have the time to participate outside of the job.

b) The number of individuals in the elearning industry embracing the newer technology is still very small.

c) The elearning industry as a whole is very competitive and they haven't warmed up to the idea of transparency like Scoble evangelizes. Sharing best practices may be frowned upon from a competitive / corporate standpoint.

Just sharing my thought. Am I completely off the mark? Would love to get a discussion going ;)

What's interesting about this is that its a case-in-point of what is wrong in the world of blogs. Craig may not have his own blog and if he did, likely he doesn't draw a large crowd (who in eLearning does?). How can Craig ask this question in the world of blogs and get a response? Also, there are all sorts of posts and prior conversation around this that relate, but how could this question and those poeple be included in the conversation?

Instead, what will happen is that because Craig asked his question on an older post (its at least a few days old now - gasp - so old), it will be buried only to be seen by the Blogger who may put in a response, but no one else will come in to discuss it. The only way it will surface is if the blogger (me in this case) highlights the post by creating a new post (oh I just did that, eh). If not, Craig will get a response from me only - so he won't get his questions answered - which proves the point - not a great discussion vehicle.

So, finally, to the good part. I've been a casual CoComment user for quite a while and have used it to track my own conversations, i.e., to see responses to comments that I make on other web sites. However, Dave Lee did an interesting thing with the LCB that uses CoComments to collect comments around the Monthly Big Questions so that we can keep track of it. That inspired me.

Starting today, I'm going to start to track lots of interesting conversations as I see them occur using CoComment (where CoComment works) so that a question like Craig's will at least surface to those who are tracking these conversations. In other words, as I find interesting conversations (IMHO) going on around eLearning topics that I'm interested in, I'll track the conversation. That way you can see a little bit of the conversation in the right column of my blog or you can visit my CoComment page. Better yet you can subscribe to the RSS Feed which should link you back to the blog and so you don't have to visit my CoComment page.

This is partly in response to comments in eLearning Technology: How Do People Interact with Blogs? where several people said that they visit the blog in order to see the comments. That makes sense, but it's not all that efficient. I hope this turns out to be useful.

A couple caveats - CoComment is not the cleaness or prettiest feed out there. It also doesn't really do a very good job of "threading" the discussions - you'll need to jump around to follow the discussion as it rambles through the blogosphere. Further, its not always easy to know where good conversations are happening, although normally people will post about them. So, it is what it is. Finally, this is still a mess. If someone else finds interesting conversations and links them via CoComment - how do you filter the overlap? But, its the best I can figure out for now.

Now a couple of questions (an assignment):

1. Is Craig right? Are blogs a bad vehicle for discussion? For asking questions?

2. Is Craig going to get his questions answered? Do you have any answers for Craig?

3. Any thoughts on better ways to surface the conversations that we all know are going on but that are happening in such as scattered way?