Natural Language Processing basics for marketers

What is Natural Language Processing?

Natural Language Processing, abbreviated NLP, is the field within artificial intelligence that deals with the interaction between computers and human language. It encompasses everything from recognizing individual words to understanding complete texts, generating summaries and answering questions. Every time you ask ChatGPT a question, have a text translated by Google Translate or a spam filter evaluates an email, NLP is at work.

For marketers, NLP is relevant because it is the technology that determines how AI models interpret your content. When Perplexity analyzes your blog post to determine whether it is relevant as a source for a user question, it uses NLP techniques to understand your text. The better your content aligns with the way NLP systems process language, the more effectively your content gets picked up and cited.

This is the technical foundation behind Answer Engine Optimization. AEO revolves around optimizing content for AI systems, and those AI systems are built on NLP technology. You do not need to become a data scientist, but a working understanding of the core concepts makes you a better content strategist.

The five core NLP processes

NLP encompasses dozens of techniques, but for marketers there are five core processes that are most relevant for content optimization. Each process determines an aspect of how AI models process your text.

Tokenization: splitting text into units

Tokenization is the first thing an NLP system does with your text: it splits the text into smaller units called tokens. A token can be a word, a subword or even a single character, depending on the model. GPT-4, for example, uses BPE (Byte Pair Encoding) tokenization, where common words are treated as a single token while uncommon words are split into multiple tokens.

# Example of tokenization (simplified)\n\nInput: "Schema.org markup improves your AI visibility"\n\nTokens: ["Schema", ".org", " markup", " improves", " your", " AI", " visibility"]\n\n# Common phrases are processed more efficiently\n# Jargon and neologisms cost more tokens\n\n# Practical consequence for marketers:\n# - Common terms are processed more efficiently\n# - Jargon and neologisms cost more tokens\n# - Clear, common language is better understood

The practical consequence for marketers is that common language is processed better than jargon or neologisms. When you write about "AI visibility optimization methodology," the NLP system has to work harder to understand this than when you write "improving your visibility in AI models." Both phrases mean the same thing, but the second is simpler for NLP systems to process.

Named Entity Recognition (NER): identifying entities

Named Entity Recognition is the NLP process that identifies proper names, organizations, places, dates and other named entities in text. When an AI model reads your text and encounters "Kobalt," NER must determine that this is an organization name, not two separate words.

NER is directly relevant to Schema.org markup and entity disambiguation. When you add Schema.org markup to your page, you give the NER system a confirmation: "Yes, 'Kobalt' is indeed an Organization." This strengthens the accuracy of entity recognition and increases the chance that your organization is correctly identified in AI-generated answers.

Sentiment analysis: determining tone and intent

Sentiment analysis is the NLP process that assesses the emotional tone of text: positive, negative or neutral. AI models use sentiment analysis to determine whether a text is informational, opinionated or promotional. This is relevant for AEO because AI models prefer informational, neutral content as a citation source over strongly promotional or emotionally charged content.

• Informational-neutral content is most frequently cited as a source in AI answers.
• Mildly positive content (enthusiastic but factual) is accepted, especially for reviews and experiences.
• Strongly promotional content ("the best solution ever!") is rarely cited because it is considered biased.
• Negative content is cited in comparative contexts, but less often as a primary source.

Dependency parsing: understanding sentence structure

Dependency parsing analyzes the grammatical structure of sentences to understand which word refers to which other word. In the sentence "The consultant who specializes in AEO advised the company on Schema.org markup," the NLP system must understand that "advised" belongs to "consultant" (not "AEO") and that "markup" is the object of the advising action.

For marketers, the practical lesson is clear: write sentences with a simple grammatical structure. Avoid long subordinate clauses, double negatives and ambiguous references. The simpler the sentence structure, the more accurately the NLP system understands your message.

Coreference resolution: linking references

Coreference resolution is the NLP process that determines when different words refer to the same entity. In the text "Kobalt is an AEO agency. The company was founded in 2015. They help clients with AI visibility," the system must understand that "the company" and "they" refer to "Kobalt."

# Example of coreference resolution\n\n# Clear (easy for NLP):\n"Kobalt is an AEO agency. Kobalt helps\n clients with AI visibility and Schema.org implementation."\n\n# Ambiguous (harder for NLP):\n"The agency works with the company. They have\n recently updated their approach. This has led to\n better results for them."\n# Who is "they"? Who is "them"? What is "this"?\n\n# Practical tip: repeat the entity name regularly\n# instead of exclusively using pronouns

Writing NLP-friendly content

Based on the five core processes, you can formulate concrete guidelines for writing content that NLP systems can effectively process. These are not rigid rules but principles that make your content more accessible to AI models.

Writing NLP-friendly content closely aligns with writing readable content for humans. The principles we discuss in our article on readability and Flesch scores are largely applicable to NLP optimization as well. Clear, structured text is easier to process for both people and machines.

1. Use common language and avoid unnecessary jargon. When you use technical terms, define them on first use.
2. Write sentences averaging 15 to 20 words. Longer sentences with complex clause structures are harder to parse.
3. Repeat entity names regularly instead of exclusively referring with pronouns like "it," "they" or "this."
4. Use an informational, neutral tone of voice. Avoid superlatives and strongly promotional formulations.
5. Structure your content with clear headings that summarize the content of each section.
6. Begin each section with the core message. NLP systems often weigh the first sentences of a paragraph more heavily.
7. Use bullet points for lists of items. NLP systems recognize and process lists more efficiently than the same information in continuous text.

NLP developments marketers should follow

The NLP field is evolving rapidly. There are several developments that will directly influence how AI models process content in the coming years and how marketers should respond.

• Longer context windows: models like GPT-4 and Claude can process increasingly longer documents at once. This increases the importance of consistency and coherence throughout your entire text.
• Multimodal NLP: models understand not only text but also images, charts and tables. Alt texts and captions thereby become part of the NLP process.
• Multilingual understanding: NLP models are becoming increasingly proficient in non-English languages. Dutch is better supported, but clear language remains important to prevent ambiguities.
• Retrieval-Augmented Generation (RAG): more and more AI systems combine their language model with real-time retrieved content. This increases the importance of content that can be quickly and correctly processed by NLP systems.

The relationship between NLP and E-E-A-T signals is becoming stronger. NLP systems are being trained not only to understand the content of text, but also to assess the authority and trustworthiness of the source. Author credentials, publication dates and source citations are being detected and weighted with increasing accuracy.

You do not need to become an NLP expert to write better content for AI. But those who understand how machines read naturally write texts that are better understood.

Key takeaways

• NLP is the technology behind how AI models read, understand and process your text into answers.
• The five core processes (tokenization, NER, sentiment analysis, dependency parsing and coreference resolution) determine how well your content is understood.
• Common language, short sentences, regular name repetitions and a neutral tone of voice make your content more NLP-friendly.
• Schema.org markup strengthens NLP processing by making entities and relationships explicit and machine-readable.
• Follow NLP developments such as longer context windows, multimodal understanding and RAG to keep your content strategy future-proof.

Frequently asked questions

Natural Language Processing is not a mysterious black box. It is a system with clear preferences, and those who understand those preferences write content that is valued by both humans and machines.