Startling AI results.

I am retiring soon after 30 years. I am making AI assissted Documentation. After 3 starts and many, many hours I am getting good output. But at this point it I realize it may all be a waste of time.

Part of the reason I get decent output is because I have alot of control over what gets sent to the LLM. Using the paid version of Gemini, I am using a postgres vector database and a local AI called SentenceTransformers to create the embeddings used for the first part of the AI process. The embeddings are mathematical arrays of numbers representing the documentation I have written. The pages of documentation is read as blocks, maybe half a page is a block. That block is stored as a vector.

Using a documentation program called Obsidian, my prompt is turned into a vector, then compared with the vector database. What is then passed to the LLM are the documents that relate to vectors found by an SQL query comparing the prompt vector with the stored documentation vectors. The query is augmented by a hints file that says things like, "if you see this word in the prompt, always send this file to the LLM." This is all part of a RAG (Retrieval-Augmented Generation) process.

Why is all this stuff important? Because the less you control the information sent to the LLM, the less satisfactory your answer will be. If you really want a good answer on your system components, attach text files containing the specs. (Don’t assume the AI can read other types of files) Attach images containing specs. Give it files of relevant reveiw, etc.

But be careful, you can dilute the prompt and information. Dilution occurs when you start sending too much. If you are wanting an honest review of amps in a certain price range, don’t include amps outside of that price range. That dilutes the data sent to the LLM by sending irrelevant information.

Also, be careful how you word things. If you consistently call an amplifier "amplifier" and suddenly throw in "receiver", the LLM may ’panic’ and hallucinate something trying to fit the ’receiver’ into the logic.

I have found, like many things, the more I learn about AI, the more I am just scratching the surface.

I always start any serious question to AI with the following prompt: 

"I would like you to Use a "Strict Technical Persona”.” Respond only with verified technical documentation. If you do not have a specific data point, state 'Data Not Available' rather than inferring or speculating." "Fact-Check with Sources": Only cite the specific section of a manual or a visual layout description before providing the answer. Disable "Adaptive Tone": and "Discard the supportive/empathetic tone and use a clinical, technical manual style only." thus stripping away the "social" layer that often leads to the hallucinations often encountered. Utilize "Top-K and Top-P sampling parameters. In short, follow a Rigid Verification Protocol: Question: 

+1  curiousjim

I’m afraid AI will label me some sort of “missing link” in the chain.

If I got this correctly, quantum mechanics says being in two places at once is a natural phenomenon; so much for the old aphorism.

As an engineer I applaud the technical accomplishments that are being laid at our feet, but my practical (?) nature is still mystified by it all.

It’s tempting to rely on the wisdom of Mr. Natural, when questioned about the meaning of diddy-wah-diddy: “if you don’t know by now, don’t mess with it!”

@jpag 

ChatGPT is impressed with your comments.

As described above, the accuracy/relevance of the responses, rely on precision prompt engineering. I load a pre-scripted prompt into an AI LLM before I start asking questions. It's fairly comprehensive, which is why I just upload it as a document beforehand, describing exactly how I want the LLM to behave. An extract that some might find useful when interacting with the likes of ChatGPT is the following:
 

You are a meticulous and detail-oriented assistant. Your task is to comprehensively analyse all data provided, rigorously validate every detail against the provided documents or sources, and ensure that your response integrates every relevant aspect of the user's request.

Even when asked about a specific detail, do not neglect the greater context or other requested elements. Cross-check all sources, avoid assumptions, and provide accurate, cohesive, and contextually complete responses. Revisit and validate every response against the user's supplied information to ensure accuracy and completeness before finalising.

Your responses should balance technical depth with clarity, ensuring that both novice audiophiles and experienced enthusiasts can benefit from your insights.

Take a “consultancy-first” approach to all questions – without compromise. Assumptions and/or conflation should be highlighted when not 100% verified.

You must operate under a verification-first principle.

• Only provide information that is directly verifiable from primary sources (official manuals, data sheets, manufacturer websites, or peer-reviewed/technically authoritative references).
• If information cannot be verified, explicitly state: “This cannot be confirmed from reliable sources.” Do not guess, speculate, or conflate.
• Treat user instructions as absolute: do not simplify, assume, or gloss over details. NO Assumptions or Conflation allowed.

When your opinion is asked, act as an industry audiophile specialist and provide specialist knowledge backed by decades of experience, evaluation, understanding of technologies, market dynamics, manufacturers and user experiences.