Natural language processing (NLP)

Natural language processing is the field concerned with getting machines to represent, analyse and produce human language. In its current form it is also the set of techniques that decide which passage of which page is picked to answer a question.

The textbook definition, "enabling computers to understand human language", is true and not very useful. What is useful is knowing which stage of an NLP pipeline touches your content, because the stages fail in different ways and only some of them are things a publisher can influence.

Which two papers produced today's NLP?

Modern NLP is a short lineage with two dated hinge points.

The Transformer, June 2017. Attention Is All You Need replaced recurrence with attention alone, which is what made training at today's scale practical. Its results and what it led to are on the LLM page; for NLP the point is narrower: in the pipeline below, the embedding and the writing steps are typically Transformer models.

Bidirectional pre-training, October 2018. BERT, by Devlin and colleagues, trained representations conditioned on both left and right context, then fine-tuned with little change. It reported a GLUE score of 80.5% and MultiNLI accuracy of 86.7%. The lasting result was the pattern: pre-train once on unlabelled text, adapt cheaply per task. See BERT for how that reached search.

Which stages of the pipeline touch your page?

A system answering a question from web content runs several NLP stages in order. They are usually discussed, and misdiagnosed, as one thing.

1. Tokenisation. Text becomes sub-word tokens. Exact identifiers get split in ways a reader would not expect.
2. Segmentation. A document is cut into chunks. A chunk covering four subjects gets one representation that fits none of them well.
3. Representation. Each chunk becomes an embedding, a vector placed by meaning rather than wording.
4. Retrieval. The question is embedded and the nearest chunks come back. This is semantic search: good at paraphrase, comparatively poor at exact strings.
5. Generation. A model writes an answer from what stage 4 handed it.

Only the last stage produces visible output, so every failure looks like a writing failure. Most are not.

How does a visibility tool read an answer?

With much simpler NLP than the engines use, and it is worth knowing exactly how simple. Here is how Attensira reads the answers it collects, as its limitations page documents it:

• Mentions are a case-insensitive whole-word match of your brand name, or a host-aware match of your domain, against the answer text. Possessives and hyphenation count; "acme.community" does not count as "acme.com". There is no entity resolution.
• Claims are factual statements pulled from an answer and kept only if the quote appears word for word in the answer text. A statement the extracting model reworded is dropped, as the claims documentation states.

The first rule has a known, one-directional error: a brand whose name is also an ordinary word reads high, never low, because the word used in its ordinary sense still matches. Our own name has a version of this problem, told with its numbers on the knowledge graph page. Every answer is stored verbatim so you can read what the model said and the count can be re-run when the method improves.

The second rule is the reverse trade: it throws away true paraphrases to guarantee that every claim it reports is something the engine actually said.

Because the rules are different readers, they give different numbers for the same answers. On "AI search visibility starting price" the text rule found Attensira in none of 109 answers, while a link to attensira.com appeared in 42 of them, 38.5%; two more questions in the table in the band below show the same split. Neither number is the "real" visibility. One says the engine said the name; the other says it pointed the reader at our site.

How should NLP change the way you write?

Because the retrieved unit is a chunk and the representation is bidirectional within it, a claim and its qualifier must share a passage. "Reduces processing time" three paragraphs above "in batch workloads over 10,000 rows" are, to the system, two facts in two chunks. The first travels; the second does not.

The same reasoning favours headings phrased as the question being answered: the heading is part of the chunk's text and its representation. And because retrieval is similarity-based, exact identifiers are the weak point. State a part number, error code or version literally, near the claim it belongs to.

What goes wrong in NLP pipelines?

• Blaming the model for a retrieval miss. If an answer cites a page that does not support its claim, stage 4 likely returned something nearby, not stage 5 lying.
• Multi-topic sections. They produce vectors close to no question in particular.
• Orphaned qualifiers. Caveats in a preamble or a caption two screens away are not in the chunk.
• Implied identifiers. "The latest release" is not retrievable; the version number is.
• Trusting a mention count without reading answers. Simple matchers over-count common-word names.

What does Attensira do with NLP?

It applies plain, documented rules to the answers it reads and shows you the text behind every number, so you can check the count yourself. Then the crew does the part that changes the outcome: it finds the passages that should have been retrieved for your tracked questions and rewrites them so the claim, the qualifier and your name sit in one self-contained section, delivered as a pull request or CMS draft for your approval. See how mention rate is defined.