Discussion about this post

User's avatar
Kaj Sotala's avatar

One LLM prompt I've been using for overcoming author blindness is just "how would you analyze this story/essay" and then asking follow-up questions about anything I'm curious about. Not asking it for feedback, but just seeing what it thinks the story/essay is about and what the plot/argument is. If there's anything it seems to misunderstand, I consider clarifying.

Though one potential problem is that some LLMs may be _too_ good at understanding all your references and things that you are hinting at relative to most readers. I wrote one story that I asked Claude Opus to analyze and it spit out a sophisticated analysis that basically understood everything that I was trying to do with the worldbuilding, and correctly drew connections to all the theoretical frameworks etc. that I was alluding to and drawing from.

Then I sent that same story to a professionally published author who I respect quite a bit, and got back a list of notes that indicated that it wasn't clear at all to him what the story was trying to do and he got confused several times. And reading those notes I could see that oh yeah, things X, Y and Z actually _are_ confusing, Opus just managed to figure out my intent despite them. Or it didn't focus on them in the first place: its analysis focused on the worldbuilding and the ideas the setting was communicating, while the editor said things like "you haven't described this apartment at all, I have no idea of what it looks like" which an LLM didn't comment on since it wasn't trying to visualize the apartment at all.

So just because the LLM understands the thing isn't a guarantee that a human would. One possible fix is to give the story to a dumber/smaller model to analyze. Another is to give the story to the model one chapter a time rather than in one piece. A model that's fed a full story will reason about it as a whole, so if e.g. something becomes clear in chapter 4 that was confusing in chapter 1, the model will just comment based on the fuller understanding. A model that has only been given chapter 1 will only work based on the information it has.

I think that this would be an issue even with the prompt described in your post, that asks it to comment on something line by line. If you've given it a full story, its reply might _start_ from the first line of your story. But it has still read your full story before it starts composing a reply, so information from the end of the story will still influence its interpretation of the beginning of it.

3 more comments...

No posts

Ready for more?