I'm also a technical writer, and I've been using ChatGPT to help write training materials. As you said, nothing ChatGPT (the leading LLM) does is great out of the box, but it reduces my workload substantially and is encouraged by my client. One very helpful thing is asking ChatGPT to help with programming examples. I write training courses for software developers. I would have to look up everything and spend a lot of time getting the code right. ChatGPT spits it out; I run it in a compiler. It works every time. I make adjustments to make the code easier to understand, but it makes me think that software developers are in trouble. My son in enrolled in a computer science degree program at a top university. I've advised him to focus on AI. You want to make and sell shovels, not buy and dig with them.
I find it interesting how often I see people writing and speaking about how AI can REPLACE people. I think that's the wrong way to look at this. Rather than replacing your consultants or your own work, it is ENHANCING your productivity and theirs, and saving either of you from doing what for many is the more boring aspects of knowledge work.
I don't agree. When I was in college (a millennium ago), I hired people to spell check, fix my grammar, and type up my papers. I don't need people to do that today, and no one makes money at it any longer. You may not notice it, but these things that make some lives easier make others obsolete.
Well, if you had enough $$$ even in college to hire those people, you come from a different world than most.
Most people went from doing their own typing, spell checking and grammar to gradually being more productive, thanks to MS Word and Google docs which have made those things automated for over 10 years and made their users more productive. Even when I've had executive assistants, I never relied on them to do these things. So, this is a productivity bump, not a reason to fire people.
I recently got Claude to act like a statistical reviewer on a blog post with terrible NHST style thinking in evaluating LLM safety. It picked out all the issues I identified and explained them like I asked it to. Compared to last year, I think the chatbots have come a long way in being helpful for speeding up technical work.
That said these agents also repeat a lot of non-sense and myths by default. There are too many flat out wrong or misleading blog posts and papers on statistical thinking out there. You do have to bias them in the right direction with prompts.
Yup. First time around with the statistics Claude took some shortcuts that led to quite wrong answers. I had to redirect it and then it did ok, but still gave slightly wrong but good enough answers. I'm not sure exactly where the errors occurred the second time, which would have been a problem if I hadn't had any expert reviewer.
I'm not a statistician but I know enough statistics to know that you really don't want to outsource your statistics work. subject matter experts are paid handsomely for a good reason on that one. Very easy to get very wrong.
True for pretty much everything I pay people for, to be honest. But yes, statistics in trials especially isn't just blindly applying formulae. Claude is ok at knowing what to do but I still would not trust it to do stats on its own for anything important.
Yeah it doesn't really know how to handle bespoke situations correctly. But with where automated data analysis is going, I expect that will start to change soon.
Even in the published literature there is a notable discrepancy between quality of statistical reviewing in top clinical journals vs. say a statistical reviewer of the same study for an FDA submission. The latter are way more accurate because they often go so far as to re-analyze the data from scratch! So you have to ask Claude not to do pervasive but wrong ways of doing trials and analyzing them.
This may be a dumb question, but if you're not reading the source material, how do you know when the results that Claude gives you are incorrect? This is one of my biggest fears about using AI - that I can't trust the results.
I've been lazy, paying for ChatGPT since the beginning when they were the only game in town. I assumed that, as the runaway market leader, they'd be better at new features and more able to access best resources. But my positive experience with Claude free tier makes me wonder if Anthropic's underdog status forces them to try harder.
I'm also a technical writer, and I've been using ChatGPT to help write training materials. As you said, nothing ChatGPT (the leading LLM) does is great out of the box, but it reduces my workload substantially and is encouraged by my client. One very helpful thing is asking ChatGPT to help with programming examples. I write training courses for software developers. I would have to look up everything and spend a lot of time getting the code right. ChatGPT spits it out; I run it in a compiler. It works every time. I make adjustments to make the code easier to understand, but it makes me think that software developers are in trouble. My son in enrolled in a computer science degree program at a top university. I've advised him to focus on AI. You want to make and sell shovels, not buy and dig with them.
I find it interesting how often I see people writing and speaking about how AI can REPLACE people. I think that's the wrong way to look at this. Rather than replacing your consultants or your own work, it is ENHANCING your productivity and theirs, and saving either of you from doing what for many is the more boring aspects of knowledge work.
I don't agree. When I was in college (a millennium ago), I hired people to spell check, fix my grammar, and type up my papers. I don't need people to do that today, and no one makes money at it any longer. You may not notice it, but these things that make some lives easier make others obsolete.
Well, if you had enough $$$ even in college to hire those people, you come from a different world than most.
Most people went from doing their own typing, spell checking and grammar to gradually being more productive, thanks to MS Word and Google docs which have made those things automated for over 10 years and made their users more productive. Even when I've had executive assistants, I never relied on them to do these things. So, this is a productivity bump, not a reason to fire people.
I'm so ridiculously excited for Opus 3.5 that I'm pre-emptively worried it can't live up to my (entirely self-generated) hype.
Claude is also my most frequent flyer for anything science. Gemini does ok too for Q&A on bench techniques
I recently got Claude to act like a statistical reviewer on a blog post with terrible NHST style thinking in evaluating LLM safety. It picked out all the issues I identified and explained them like I asked it to. Compared to last year, I think the chatbots have come a long way in being helpful for speeding up technical work.
That said these agents also repeat a lot of non-sense and myths by default. There are too many flat out wrong or misleading blog posts and papers on statistical thinking out there. You do have to bias them in the right direction with prompts.
Yup. First time around with the statistics Claude took some shortcuts that led to quite wrong answers. I had to redirect it and then it did ok, but still gave slightly wrong but good enough answers. I'm not sure exactly where the errors occurred the second time, which would have been a problem if I hadn't had any expert reviewer.
I'm not a statistician but I know enough statistics to know that you really don't want to outsource your statistics work. subject matter experts are paid handsomely for a good reason on that one. Very easy to get very wrong.
True for pretty much everything I pay people for, to be honest. But yes, statistics in trials especially isn't just blindly applying formulae. Claude is ok at knowing what to do but I still would not trust it to do stats on its own for anything important.
Yeah it doesn't really know how to handle bespoke situations correctly. But with where automated data analysis is going, I expect that will start to change soon.
Even in the published literature there is a notable discrepancy between quality of statistical reviewing in top clinical journals vs. say a statistical reviewer of the same study for an FDA submission. The latter are way more accurate because they often go so far as to re-analyze the data from scratch! So you have to ask Claude not to do pervasive but wrong ways of doing trials and analyzing them.
This may be a dumb question, but if you're not reading the source material, how do you know when the results that Claude gives you are incorrect? This is one of my biggest fears about using AI - that I can't trust the results.
I usually go back and check the sources.
Why Claude and not ChatGPT or Gemini (or its various other competitors? I've been using ChatGPT Pro and loving it, but thinking of switching.
Just by personal experience, Claude seems to be smarter and better at reasoning.
I've been lazy, paying for ChatGPT since the beginning when they were the only game in town. I assumed that, as the runaway market leader, they'd be better at new features and more able to access best resources. But my positive experience with Claude free tier makes me wonder if Anthropic's underdog status forces them to try harder.