Jon Plummer

Today I Learned

Weekly wins for the week of 2023 03 27

  • It wasn't COVID-19, just a sinus infection that dripped into crevices and alarmed passers-by. I'm nearly 100% now.
  • Nothing caught on fire while I was out.
  • The hackathon a few weeks ago was inspiring enough that I've started to learn a little but about iOS app development to fulfill a personal project. Since my last real coding experience was a bunch of kooky javascript stuff around the turn of the century I'm well out of my depth, but it's both fun and frustrating; there's a point at which I'm familiar, I'm familiar, I'm familiar with the concepts in a tutorial and then whap I'm met with something totally baffling. As the ultramarathoners say, RFM.

Weekly wins for the week of 2023 03 20

It's spring break, that awkward nearly month-long period when employees with kids start disappearing for a week at a time. Every year the challenge is to make sure that people have prepared their teams for their absence; that enough is done and questions answered (and backup help secured) that whoever remains can proceed without much trouble, and especially that the person's absence doesn't come as a surprise.

When I asked each person this week "what do we need to do to prepare your team for your absence" each person's answer started with what they've already done. That's a big win.

Weekly wins for the week of 2023 03 13

Last week was all about an off-site meeting involving the product and engineering groups.

(What does off-site mean in today's remote-first environment? Never mind.)

  • It was a good set of sessions! in particular, the hackathon presentations were funny and inspiring. Enough so that I'm tempted to get back into coding a little. Maybe I could hack(athon) a bit one day.
  • My team really came together working on our "elevator-pitch-style" team charter. I have some homework to do to set us up for the last step, but it's a pleasure to collect the good thoughts of good people trying hard to improve.
  • A coworker made their displeasure with the past UX regime abundantly clear in a group setting, and I decided I would not let it bother me. Even so, it did, for a bit. But later when we talked about it and I told them I had decided not to let it bother me, it actually no longer bothered me. It worked!

ChatGPT is going to tempt me to be more skeptical of your work

Lately I've been seeing a lot of posts on LinkedIn and elsewhere crowing about how ChatGPT could be used to perform UX tasks. Examples:

The enthusiasm is great but this level of shortcutting worries me. It's okay to ask ChatGPT to find things to read about a topic if you're fine with some of the results not being appropriate or even not existing. But I don't think it is fine to ask it how to do something or to perform research on your behalf. ChatGPT's emphasis is on delivering something that looks sensible, nothing more.

ChatGPT is not a knowledge model, it's a language model. If you'd like to dive into just how ChatGPT works, Steven Wolfram has a great explanation in his article What is ChatGPT Doing…and Why Does It Work?

The core idea is that ChatGPT is very good at figuring out what a very likely next word might be based on the prior words it has chosen, the prompt that it was given, and word frequency and proximity data derived from a huge amount of copy scraped from the internet. Since it doesn't actually know anything it does a great job of making plausible-sounding English ((ChatGPT can produce reasonable looking Python and other languages; a co-worker successfully asked it to return JSON in response to a bit of copy where someone asked for an appointment on a specific date and time.)) of the sort you might find anywhere on the net. Since the internet is the training data, the quality of the output is only about as good as the average quality of internet writing, which is not fabulous.

It's important to remember that there's no attempt to make sure that what ChatGPT returns is factually accurate. Bloggers and reporters experimenting with ChatGPT have accused it of making things up or "hallucinating," but this complaint assumes that accuracy should be expected. It should not. ChatGPT is just trying to be plausible.

I'm not saying not to use ChatGPT. It's great as a memory jogger, or to avoid the tyranny of the blank page. It makes a perfectly shitty first draft that you can then do real work on. But if you just accept what it has to say you are choosing a below-average and likely nonsensical result. And if you use it as a substitute for doing the work that ChatGPT is simulating the output of, you are lying to yourself and others.

Since ChatGPT produces superficially plausible output, hiring managers are going to need to scrutinize a candidate's work more closely, and quiz a candidate more carefully. (Yes, we should already be doing this.)

On a Slack team I'm on there was a recent debate as to whether or not an engineering manager should accept ChatGPT output as the answer to a coding test, if during their regular duties a new hire would be allowed to use resources like StackOverflow, which often provides code snippets, Google Search, or even ChatGPT. What do you think, given the above?

Weekly wins for the week of 2023 03 06

I'm light on wins this week. The most challenging part of posting wins each week is being conscious of them!

  • I wrote two blog posts this week, and without waiting for the weekend. I find that I'll write a little explainer for someone in the normal course of business, mentoring, etc. and realize that it would make a pretty good short post. (This might be my main source of content.)
  • I've an idea for another quick post re the favorite topic these days: ChatGPT. People's un-ironic embrace of ChatGPT as a substitute for actually doing the work yourself is understandable but alarming. It's going to cause us to look at people's work a bit differently for a while. (Actually, an idea is only a win of you execute on it, because ideas are cheap.)