Question: Is it advisable for an indie author to include a glossary of invented words at the end of a fantasy novel for readers to reference?
Answer: No matter if you do or not, someone is always going to complain.
I think of the masterpiece, A Clockwork Orange, by Anthony Burgess while writing this answer. Penned in 1962, the novel depicted a society and thugs who ran rampant in the near future. Not only did he not include a glossary of the terms, he used slang from the main character throughout.
One part that sticks out in my mind is sex being referred to as the “Old in and out.” Of course, while reading the work you can easily tell what the slang or terms used meant. At times the “droog” (main character reference) is hard to understand, but an educated reader could tell what the meaning of the word was without any issues.
Would the novel been more popular with a glossary of the terms and words? Maybe, but it would not have been known as a literary masterpiece. The use of the slang and words caused much debate in classrooms around the world. This caused the controversy of the novel to make it such a classic.
The best thing is to give your readers an idea of what is being said through the narrative or actions of the characters. In this case, Burgess does just that in a masterful way.
A series I recently read, Demon Cycle, by Peter V. Brett, came up with a curse that most of the society understands, “The core with you”, meaning the demons drag you to the core where they live. This is a very good slant on “To hell with you”, which is something used in society today. We don’t need to know the reference in a glossary, but we do need to see how it is used in the story, thus telling us what it means.
Depending on how obscure you make your language should tell you how to go about solving this dilemma. Get your test readers to tell you if they had any issues with the terms and language used. If so, then find out if it is the invented words or the way they are used. If it is the way they are used, adjust your narrative. When your readers say, “What does this mean?” you know your narrative needs adjusting so the scene tells the reader the meaning.
Take the following sentence:
John took out his flaberdo and pointed it at Phil.
The question is what is a flaberdo? So we adjust the narrative to show the reader what the item is:
John unholstered his flaberdo and pointed the small handgun at Phil.
Now we know it is a type of handgun. So, with this change we don’t need to make a glossary.