How to Analyze Word Frequency (and Why the Tokenizer Regex Is the One Thing That Matters)

The problem: you pasted a 10,000-word article into a spreadsheet and "the" is the most frequent word

You wrote a long article and want to know which words you overused. You paste it into a spreadsheet, split on spaces, count, sort. The top 10: "the," "and," "to," "of," "a," "in," "is," "that," "you," "it." Every English text produces the same top 10 because these are the structural words that hold sentences together — they tell you nothing about the content. The honest move is a word frequency analyzer that tokenizes with a Unicode-aware regex (not a naive whitespace split), excludes stop words by default, computes the frequency map in one pass, and then shows you the content words — the nouns, verbs, and adjectives that actually describe what the text is about — alongside bigrams, trigrams, and a diversity ratio that tells you whether your vocabulary is rich or repetitive.

Fastest path

Open the Word Frequency Analyzer, paste your text, click Analyze.

Input:    [paste 10,000-word article]

→ Total Words:     9,847  (after stop-word exclusion)
→ Unique Words:    1,832
→ Diversity Ratio: 18.6%  (type-token ratio)
→ Hapax Legomena:  1,204  (words appearing exactly once)

→ Top 5 by count:
  1. data       — 142 (1.44%)
  2. model      — 89  (0.90%)
  3. training   — 67  (0.68%)
  4. algorithm  — 54  (0.55%)
  5. accuracy   — 43  (0.44%)

→ Top 3 bigrams:
  1. machine learning     — 28
  2. neural network       — 21
  3. training data        — 17

The tool tokenized the text with a Unicode regex, excluded ~125 stop words, built a frequency Map in one pass, computed percentages, sorted by count descending (alphabetical tiebreaker), and extracted the top 20 bigrams and trigrams via a sliding window. The rest of this guide is why the tokenizer regex matters, why stop words are excluded by default, what hapax legomena measures, how n-grams work, and why the diversity ratio is a 150-year-old linguistic metric.

The substance: one regex, one Map, one sliding window

The tokenizer regex and why split-on-whitespace is wrong

The tool's tokenize function uses a Unicode property escape regex:

const matches = source.match(/[\p{L}\p{N}]+(?:'[\p{L}\p{N}]+)?/gu);

This matches sequences of Unicode letters (\p{L}) and numbers (\p{N}), with an optional internal apostrophe group ((?:'[\p{L}\p{N}]+)?). The u flag enables Unicode property escapes — \p{L} matches any letter in any script, not just ASCII. The g flag finds all matches.

This regex is the one thing that matters because every downstream computation depends on it. A naive text.split(/\s+/) would include punctuation attached to words — "data," and "data" would be two different tokens. A naive text.split(/[^a-zA-Z]/) would drop every non-ASCII letter — café, naïve, and 北京 would all be silently destroyed. The Unicode regex handles both: punctuation is a separator (not a letter), and Unicode letters are included.

The apostrophe group handles contractions: "don't" is one token, not two ("don" + "t"). The regex allows one apostrophe between letter sequences — "don't" matches, "o'clock" matches, but "'''" (three apostrophes) doesn't. Hyphens are not included in the character class, so "well-known" becomes two tokens: "well" and "known." This is a design choice — some tokenizers keep hyphens to preserve compound words. The tool's choice is to split on hyphens because they're ambiguous: "well-known" is a compound adjective, but "twenty-five" is a number, and "state-of-the-art" is a four-word phrase. Splitting is the simpler rule.

Stop words and why they're excluded by default

The tool ships a STOP_WORDS Set with ~125 common English words: articles (a, an, the), conjunctions (and, but, or), prepositions (in, on, at, of, to), pronouns (he, she, it, they, you, we), auxiliary verbs (is, are, was, were, have, has, had, do, does, did, can, could, should, would, will), and common adverbs (very, just, also, only, about).

The default is excludeStopWords: true because without exclusion, the top 10 of every English text is the same structural words. Try it: paste any 1,000-word English text, turn off stop-word exclusion, and the top 10 will be a permutation of "the and to of a in is that it." These words tell you the text is in English, but nothing about the content. With exclusion, the top 10 reflects the actual subject matter — "data," "model," "training" for a machine learning article; "patient," "treatment," "clinical" for a medical article.

The stop words are checked against the lowercase version of each token, so "The" and "the" are both excluded when case-insensitive (the default). The Set lookup is O(1) — the filter loop checks STOP_WORDS.has(lowerForStop[i]) for each token, which is a hash lookup, not a linear scan.

The frequency Map and one-pass counting

The buildFrequencyMap function is one loop:

const map = new Map<string, number>();
for (const token of tokens) {
  map.set(token, (map.get(token) ?? 0) + 1);
}
return map;

For each token, get the current count (or 0 if unseen), add 1, set it back. This is O(n) in the number of tokens and O(u) in space (u = unique words). The ?? 0 nullish coalescing handles the first occurrence — map.get(token) returns undefined for unseen tokens, and undefined ?? 0 evaluates to 0.

The percentage is computed after the map is built: count / totalWords * 100. The totalWords is the count of filtered tokens (after stop-word exclusion and min-length filter), not the raw token count. This means the percentages sum to 100% across all entries — each percentage represents the word's share of the analyzed text, not of the raw text.

Hapax legomena: the vocabulary richness metric

The tool counts hapaxCount = allEntries.filter(e => e.count === 1).length — the number of words that appear exactly once. The term is from Greek: "hapax" means "once," "legomenon" means "said thing." A hapax legomenon is a word that occurs only once in a given text.

This metric measures vocabulary richness. A text with 1,000 words where 600 are hapax (appear once) has a rich, varied vocabulary. A text with 1,000 words where only 50 are hapax has a repetitive, narrow vocabulary. The ratio of hapax to total words correlates with the type-token ratio (diversity) but is more sensitive to the long tail — the diversity ratio treats a word appearing twice the same as one appearing ten times, but hapax count distinguishes them.

In corpus linguistics, hapax legomena are also used to estimate vocabulary size: if you find H hapax words in a text of N words, the total vocabulary of the language is roughly N²/H (the Good-Turing estimate). The tool doesn't compute this — it just shows the raw count. But the count alone is useful: a high hapax count means the text is lexically diverse; a low count means it's repetitive.

The diversity ratio and the type-token ratio

The tool's diversity is uniqueWords / totalWords — the number of unique words divided by the total word count. In linguistics, this is the type-token ratio (TTR): types are unique word forms, tokens are total word occurrences. A TTR of 1.0 (100%) means every word is unique. A TTR of 0.1 (10%) means 10% of the words are unique — high repetition.

The TTR was introduced by the linguist George Zipf in the 1930s and is one of the oldest quantitative measures of vocabulary richness. It's sensitive to text length: longer texts tend to have lower TTR because the vocabulary saturates (you run out of new words and start repeating). This is why TTR is usually computed on standardized text lengths (e.g., the first 500 words) when comparing texts of different sizes. The tool computes it on the full text — for comparing texts of similar length, this is fine. For comparing a 100-word paragraph to a 10,000-word article, the shorter text will always have higher TTR.

N-grams: the sliding window

The tool's buildNgrams function extracts 2-word (bigram) and 3-word (trigram) phrases:

for (let i = 0; i <= tokens.length - n; i++) {
  const gram = tokens.slice(i, i + n).join(' ');
  map.set(gram, (map.get(gram) ?? 0) + 1);
}

A sliding window of size n moves across the token array. At position i, the window covers tokens[i] through tokens[i+n-1]. The window contents are joined with spaces to form the n-gram string, and the same Map-counting pattern as the word frequency counts occurrences. For n=2 on the tokens ["machine", "learning", "is", "fun"], the bigrams are "machine learning," "learning is," "is fun" — three bigrams from four tokens. The formula is tokens.length - n + 1 n-grams.

N-grams capture multi-word expressions that single-word frequency misses. "Machine" and "learning" might both appear in the top 20, but the bigram "machine learning" appearing 28 times tells you they're almost always used together. Trigrams go further: "training data quality" or "neural network architecture" are three-word phrases that single-word analysis can't find. The tool shows the top 20 bigrams and trigrams, sorted by count descending with alphabetical tiebreaker — the same sort logic as the word table.

The four sort modes and their tiebreakers

The tool's sortEntries function supports four modes, each with a tiebreaker:

• countDesc: b.count - a.count || a.word.localeCompare(b.word) — highest count first; ties broken alphabetically.
• countAsc: a.count - b.count || a.word.localeCompare(b.word) — lowest count first; ties broken alphabetically. Useful for finding hapax words (count = 1) at the top.
• alpha: a.word.localeCompare(b.word) — pure alphabetical. The localeCompare call handles Unicode sorting (accented characters, non-Latin scripts).
• length: b.word.length - a.word.length || b.count - a.count — longest word first; ties broken by count descending. Useful for finding the longest technical terms.

The tiebreakers matter because many words share the same count. Without a tiebreaker, the sort order of equal-count words is undefined (JavaScript's sort is not stable across engines). The localeCompare tiebreaker ensures deterministic output — the same input always produces the same order.

The word cloud and linear font scaling

The word cloud shows the top 10 words with font sizes scaled by count:

const t = (count - cloudMin) / (cloudMax - cloudMin);
return 0.9 + t * 1.6; // 0.9rem → 2.5rem

Linear interpolation: the most frequent word gets 2.5rem, the least frequent (of the top 10) gets 0.9rem. The interpolation is linear, not logarithmic — a word with count 100 is 10x bigger than count 10 in the cloud. Logarithmic scaling would compress the visual difference, making the cloud more uniform. The tool uses linear because the top 10 has a small range — the ratio between #1 and #10 is typically 3-5x, not 100x. Linear scaling in that range produces visible size differences without overwhelming the layout.

The if (cloudMax === cloudMin) return 1.5 guard handles the edge case where all top-10 words have the same count (e.g., a very short text where every word appears once). Without the guard, the division would produce NaN and every word would get font size NaN, which CSS would ignore.

Gotchas

• The tokenizer regex is the one thing that matters. [\p{L}\p{N}]+(?:'[\p{L}\p{N}]+)? with the u flag. A naive split(/\s+/) includes punctuation. A naive split(/[^a-zA-Z]/) destroys non-ASCII letters. The Unicode regex handles both. If you're building your own word frequency tool, start with this regex.
• Hyphens split words. "well-known" becomes ["well", "known"]. "state-of-the-art" becomes four tokens. If you need hyphenated compounds as single tokens, modify the regex to include hyphens in the character class: [\p{L}\p{N}-]+.
• Stop words are excluded by default. The top 10 without exclusion is always "the and to of a in is that it." Turn off stop-word exclusion only if you're analyzing structural word frequency, not content.
• The diversity ratio (TTR) is sensitive to text length. Longer texts have lower TTR because vocabulary saturates. Compare TTR only between texts of similar length. For a length-independent measure, compute TTR on the first 500 words of each text.
• Hapax count is a richness signal, not a quality verdict. A high hapax count means varied vocabulary, but a text full of rare words can be harder to read. A low hapax count means repetition, which can be deliberate (rhetorical emphasis) or accidental (lazy writing). Context matters.
• N-grams are computed on filtered tokens. If stop words are excluded (the default), the bigrams won't include "the model" or "is training" — the stop words are removed before the n-gram window slides. This changes the n-gram results compared to computing n-grams on raw text. If you need raw-text n-grams, turn off stop-word exclusion.
• The CSV export uses RFC 4180 quoting. Words with commas, quotes, or newlines are wrapped in quotes with doubled internal quotes — the same pattern as the CSV editor. If a word contains a newline (rare but possible in pasted text), the CSV stays valid.
• The word cloud is top-10 only. The cloud shows the 10 most frequent words, not the full frequency table. For the complete ranking, use the Table tab or download the CSV.
• The font scaling is linear, not logarithmic. A word with count 100 is 10x bigger than count 10 in the cloud. For top-10 displays with a small range, linear is fine. If the range is large (100x), the cloud would look unbalanced — switch to the table view for exact counts.
• Case sensitivity is off by default. "The" and "the" are counted as the same word. Turn on case sensitivity if you're analyzing capitalization patterns (e.g., "Apple" the company vs. "apple" the fruit).

Summary

• The tokenizer regex is the one thing that matters. [\p{L}\p{N}]+(?:'[\p{L}\p{N}]+)? with the u flag matches Unicode letters and numbers with optional internal apostrophes. Not split(/\s+/) (includes punctuation) and not split(/[^a-zA-Z]/) (destroys non-ASCII). Hyphens split words — "well-known" is two tokens.
• Stop words are excluded by default. ~125 common English words in a Set. Without exclusion, the top 10 is always structural words. With exclusion, the top 10 reflects the text's content. The Set lookup is O(1).
• The frequency Map is one pass. map.set(token, (map.get(token) ?? 0) + 1) for each token. O(n) time, O(unique words) space. Percentages are count / totalWords × 100 where totalWords is the filtered count, not the raw count.
• Hapax legomena measures vocabulary richness. Words appearing exactly once. High hapax count = rich vocabulary; low hapax count = repetitive. The term is from Greek ("said once"). The count is a signal, not a quality verdict — rare words can hurt readability.
• The diversity ratio is the type-token ratio. uniqueWords / totalWords — a 150-year-old linguistic measure from Zipf. Sensitive to text length: longer texts have lower TTR. Compare only between texts of similar length.
• N-grams are a sliding window. buildNgrams(tokens, n) slides a window of size n, joins the window with spaces, counts occurrences. Bigrams (n=2) capture two-word phrases; trigrams (n=3) capture three-word phrases. Computed on filtered tokens — stop-word exclusion changes the n-gram results.
• Four sort modes with deterministic tiebreakers. countDesc, countAsc, alpha (localeCompare), length. The tiebreaker (alphabetical or count) ensures the same input always produces the same order — JavaScript's sort is not stable across engines without one.
• Analyze at the Word Frequency Analyzer; for a simple word and character count use Word Counter, for extracting email addresses from text use Email Extractor, and for encoding text as binary use Text to Binary Converter.