Are AI generated docs "good enough"?
Does it matter if the product documentation is generated if you prompt correctly?
Remember Devin? The AI software engineer that was going to push human software engineers out of their jobs? Devin was a name that I hadn't heard in a long time. That was until I was searching for AI generated developer documentation and came across DeepWiki, powered by Devin. DeepWiki reads Github repositories and product documentation and rewrites it to be "more concise".
But outside of DeepWiki, the amount of AI generated user facing documentation suggests that the voice of Claude is good enough to keep the reader's interest. Surely, these clankers¹ are trained on documentation written by professional writers who have spent time mulling over the audience to provide concise and clear documentation - so it should be able to replicate something borderline legible - right? ....Right?
Human versus Machines: Overview Sections
Let's compare DeepWiki's Spegel documentation versus my version.
A disclaimer: The Spegel maintainer has approved this usage and none of this is in any way disparaging about the project.
As someone who always skips the Overview section and goes right to the “Getting Started”, rewriting this section was interesting. I found myself looking at Helm, Istio, and other open source documentation to figure out what the purpose of the Overview section is. It seems like the overview tries to follow: this is what it is, how it does it, and the problem it solves - similar to a hook for a blog post.
I'll let you, the reader, decide which you prefer. Which kept your attention longer, had better flow, or was easier to follow?
Human versus Machines: Cade
Now, most AI-die-hards would tell me that DeepWiki is a mass produced AI generated content and that if the AI is prompted correctly then AI generated documentation is good. I asked another friend who built Cade - a runtime for operating pinball machines. All of this website's content and user documentation is also generated.
Disclaimer: Thank you Dittmar for letting me use your website - and check out Cade because it is actually cool. This is in no way to disparage Dittmar's tool.
Again, the readers can decide which is your preference.
Human versus Machines: Diagrams
Technical diagrams are the bane of my existence, I have even purchased a book to get better at making them. The key issue with creating good technical diagrams is knowing what are the important parts to show, versus which will make the diagram look too complicated to understand. But now, we use AI - because it has all the context to make pictures.
Remember when AI was generating photos where humans were all thumbs? Now, the thumb machine is generating technical diagrams.
We'll go back to DeepWiki and look at Spegel's generated diagram, versus what Spegel itself published.
Unfortunately, in this case, DeepWiki's incorrect generation leaves me with many questions. What is "External Systems"? And, why is Containerd over there? Also, the state tracker doesn't talk to objects like the OCI client at all. The correctness of labels and arrows are incredibly important for diagrams. Portraying which pieces talk to which, and why, can be crucial to whether it is deemed secure enough to use.
Who wins?
So, where does this leave us? Cynthia Dunlop did a report on over 668 people on AI-generated technical blog posts. 78% of these participants stop reading if they think a tech blog is AI-assisted or AI-authored. That could be potential adopters, purchasers, or potential maintainers that are leaving based on a product's "Getting Started" alone.
For enterprise documentation this means an internal review of AI generated documentation is crucial. Sales teams lament about how the first impression is the most important one² - it leaves an impact and sets the tone for the relationship. The "Overview" and "Getting Started" sections are the first interaction developers will have with your product. What should developers take away from these sections, and what will keep them engaged?
For the rest of us, having AI generated documentation disempowers us from holding the source of truth. Nowadays, developers go to AI to debug issues and ask for help. When the long winded generated answer fails, the developers fall back on the product documentation. Product documentation that is developer native will also help limit the amount of incoming pull requests and issues, which are most likely also AI generated.
But most importantly, to me, writing documentation is a labor of love. Whether you have spent 7 trillion tokens or wrote it yourself, being able to write in your voice what the tool is supposed to do and the problem it solves is crucial. If you don't write how it works, the expected behavior, and its architecture - how can you be certain you even know what it is?
¹ Clankers is a fun nickname for AI
² There is published research on this, actually.