# Natural language processing (NLP)

> For the complete documentation index, see [llms.txt](https://attensira.com/llms.txt).
> Every page on this site is also served as Markdown: append `.md` to any URL, or send `Accept: text/markdown`.

> NLP is the field of getting machines to read and write human language. The pipeline behind AI search, from tokens to retrieval, and where your page can fail.

- Canonical page: https://attensira.com/glossary/nlp
- Family: glossary
- Last updated: 2026-10-04
- Author: Karl-Gustav Kallasmaa (https://x.com/kallasmaa)
- Claims as data: https://attensira.com/claims.json?slug=nlp

**In short:** Natural language processing (NLP) is the field of computer science that gets machines to represent, analyse and produce human language. In AI search it is a pipeline: text is split into tokens, cut into chunks, turned into vectors, matched against a question, and only then used to write an answer. Most failures to be quoted happen before the writing step.

| Figure | What it counts | n |
| --- | --- | --- |
| 197 of 198 | answers naming Attensira on the 3 questions that contain the name |  |
| 0 of 4,326 | answers naming Attensira on the other 58 questions |  |
| 239 of 4,524 | answers citing an attensira.com page |  |

Window 2026-09-04 to 2026-10-04. attensira.com workspace, 61 tracked questions on ChatGPT and Perplexity, read over the Attensira MCP on 2026-10-04.

_Source: attensira.com workspace, 61 tracked questions on ChatGPT and Perplexity, read over the Attensira MCP, as of 2026-10-04._

## Key facts

- Also called: Natural language processing, computational linguistics
- Pipeline: Tokens, chunks, embeddings, retrieval, generation
- Key papers: [Transformer (2017), BERT (2018)](/glossary/bert-algorithm)
- Where quoting fails: Mostly before the writing step
- Mention rule in Attensira: [Whole-word, case-insensitive; no entity resolution](https://docs.attensira.com/reference/limitations)
- Claim rule in Attensira: [Kept only if quoted word for word](https://docs.attensira.com/measure/claims)

Natural language processing is the field concerned with getting machines to represent, analyse and produce human language. In its current form it is also the set of techniques that decide which passage of which page is picked to answer a question.

The textbook definition, "enabling computers to understand human language", is true and not very useful. What is useful is knowing which stage of an NLP pipeline touches your content, because the stages fail in different ways and only some of them are things a publisher can influence.

## Which two papers produced today's NLP?

Modern NLP is a short lineage with two dated hinge points.

**The Transformer, June 2017.** [Attention Is All You Need](https://arxiv.org/abs/1706.03762) replaced recurrence with attention alone, which is what made training at today's scale practical. Its results and what it led to are on the [LLM](/glossary/llm) page; for NLP the point is narrower: in the pipeline below, the embedding and the writing steps are typically Transformer models.

**Bidirectional pre-training, October 2018.** [BERT](https://arxiv.org/abs/1810.04805), by Devlin and colleagues, trained representations conditioned on both left and right context, then fine-tuned with little change. It reported a GLUE score of 80.5% and MultiNLI accuracy of 86.7%. The lasting result was the pattern: pre-train once on unlabelled text, adapt cheaply per task. See [BERT](/glossary/bert-algorithm) for how that reached search.

## Which stages of the pipeline touch your page?

A system answering a question from web content runs several NLP stages in order. They are usually discussed, and misdiagnosed, as one thing.

1. **Tokenisation.** Text becomes sub-word [tokens](/glossary/tokens). Exact identifiers get split in ways a reader would not expect.

1. **Segmentation.** A document is cut into [chunks](/glossary/chunking). A chunk covering four subjects gets one representation that fits none of them well.

1. **Representation.** Each chunk becomes an [embedding](/glossary/embeddings), a vector placed by meaning rather than wording.

1. **Retrieval.** The question is embedded and the nearest chunks come back. This is [semantic search](/glossary/semantic-search): good at paraphrase, comparatively poor at exact strings.

1. **Generation.** A model writes an answer from what stage 4 handed it.

Only the last stage produces visible output, so every failure looks like a writing failure. Most are not.

## How does a visibility tool read an answer?

With much simpler NLP than the engines use, and it is worth knowing exactly how simple. Here is how Attensira reads the answers it collects, as its [limitations page](https://docs.attensira.com/reference/limitations) documents it:

- **Mentions** are a case-insensitive whole-word match of your brand name, or a host-aware match of your domain, against the answer text. Possessives and hyphenation count; "acme.community" does not count as "acme.com". There is no entity resolution.

- **Claims** are factual statements pulled from an answer and kept only if the quote appears word for word in the answer text. A statement the extracting model reworded is dropped, as the [claims documentation](https://docs.attensira.com/measure/claims) states.

The first rule has a known, one-directional error: a brand whose name is also an ordinary word reads high, never low, because the word used in its ordinary sense still matches. Our own name has a version of this problem, told with its numbers on the [knowledge graph](/glossary/knowledge-graph) page. Every answer is stored verbatim so you can read what the model said and the count can be re-run when the method improves.

The second rule is the reverse trade: it throws away true paraphrases to guarantee that every claim it reports is something the engine actually said.

Because the rules are different readers, they give different numbers for the same answers. On "AI search visibility starting price" the text rule found Attensira in none of 109 answers, while a link to attensira.com appeared in 42 of them, 38.5%; two more questions in the table in the band below show the same split. Neither number is the "real" visibility. One says the engine said the name; the other says it pointed the reader at our site.

## Where do the two rules disagree most?

On questions that never name us. On "AI search visibility starting price" the text rule found Attensira in none of 109 answers, while 42 linked to attensira.com; two more questions show the same split.

**Two rules, two numbers: questions that never name us**

| Tracked question | Answers | Named (text rule) | Cited (link rule) |
| --- | --- | --- | --- |
| AI search visibility starting price | 109 | 0 | 42 (38.5%) |
| GEO tool that opens pull requests | 24 | 0 | 4 (16.7%) |
| what is generative engine optimization | 108 | 0 | 3 (2.8%) |

_Source: attensira.com workspace, ChatGPT and Perplexity, 30 days, read over the Attensira MCP, as of 2026-10-04 (241 answers)._

## How should NLP change the way you write?

Because the retrieved unit is a chunk and the representation is bidirectional within it, a claim and its qualifier must share a passage. "Reduces processing time" three paragraphs above "in batch workloads over 10,000 rows" are, to the system, two facts in two chunks. The first travels; the second does not.

The same reasoning favours headings phrased as the question being answered: the heading is part of the chunk's text and its representation. And because retrieval is similarity-based, exact identifiers are the weak point. State a part number, error code or version literally, near the claim it belongs to.

## What goes wrong in NLP pipelines?

- **Blaming the model for a retrieval miss.** If an answer cites a page that does not support its claim, stage 4 likely returned something nearby, not stage 5 lying.

- **Multi-topic sections.** They produce vectors close to no question in particular.

- **Orphaned qualifiers.** Caveats in a preamble or a caption two screens away are not in the chunk.

- **Implied identifiers.** "The latest release" is not retrievable; the version number is.

- **Trusting a mention count without reading answers.** Simple matchers over-count common-word names.

## What does Attensira do with NLP?

It applies plain, documented rules to the answers it reads and shows you the text behind every number, so you can check the count yourself. Then the crew does the part that changes the outcome: it finds the passages that should have been retrieved for your tracked questions and rewrites them so the claim, the qualifier and your name sit in one self-contained section, delivered as a pull request or CMS draft for your approval. See how [mention rate](/features/mention-rate) is defined.

## FAQ

### Is NLP the same thing as a large language model?

No. NLP is the field; a language model is one kind of method within it. The field also covers tokenisation, parsing, named entity recognition, coreference resolution and translation, several of which still run as narrower components inside systems people call an LLM.

### Which papers mark the shift to the current generation?

Two. Attention Is All You Need (Vaswani et al., submitted 12 June 2017) proposed the Transformer, the architecture behind current language models. BERT (Devlin et al., submitted 11 October 2018) showed that bidirectional pre-training plus light fine-tuning beat task-specific designs, reaching a GLUE score of 80.5%.

### Why does bidirectional matter for what gets cited?

Because the representation of a word is shaped by the text on both sides of it, within the chunk it sits in. A qualifier right after a claim is part of how that sentence is represented; a qualifier three paragraphs earlier, in a different chunk, is not there at all.

### What unit does a modern NLP system operate on?

Tokens, not words. Text is split into sub-word tokens before anything else happens, which is why product codes, version numbers and unusual brand names split in ways that make them harder to match exactly than a person would expect.

### How do AI visibility tools decide that an answer mentioned my brand?

Usually with a simple rule, not deep language understanding. Attensira documents its own method: a case-insensitive whole-word match of your brand name, or a host-aware match of your domain, with no entity resolution. It is transparent and fast, and it counts a common-word brand used in its ordinary sense as a mention.

### Does NLP still do anything besides generate text?

Yes, and the non-generative steps decide whether you are found. Segmentation, embedding and similarity search all run before a model writes a word; a miss there is invisible in the answer and is the more common reason a correct page is not quoted.