This project was originally self-dogfooding. I went to a track meet to watch my old college team compete. I tried to track splits during races on my phone, but had trouble focusing on the race while also keeping splits, particularly when multiple competitors were involved. This seemed like a tractable problem, so I built a first version of a multi-stopwatch over the course of a couple weeks during the evening. It wasn’t perfect, but it made managing multiple stopwatches much easier than the other apps available at the time.

I kept idly coming up with feature my ideal stopwatch would have, but I realized I had made some fundamental mistakes in my haste to get a working version that would fundamentally prevent me from implementing them. I eventually decided to start over to build a better version to resolve those issues so I could add more meaningful features.

What Epochron Is🔗

Epochron runs as many stopwatches as you need, at once, in a browser. Each one can be named, assigned to a group, and record splits and laps against distances. A running timer can be copied to start tracking someone else mid-race, which is the situation that started this whole thing. Sessions can be archived and searched later.

It installs as a PWA, so it can live on a phone’s home screen and keeps working without a signal, which matters because cell service at many cross country/track meets is reliably terrible. Everything stays on the device: so no data leaves the browser. That’s partly a privacy stance and partly a practical one, since a tool that needs a server is a tool that fails in the field.

It’s built in Angular v22 and TypeScript, free, and the source is on GitHub. It can be used without sign-ups or downloads.

Stopwatch Structure🔗

A stopwatch is fundamentally just a set of labelled timestamps, starting with a start timestamp, and ending with a stop timestamp, with split and lap timestamps in between. This is fundamentally how this stopwatch is structured: as a

export interface UnitValue {
    value: number;
    unit?: string;
}

type OperationalStopWatchEventType = 'start' | 'stop' | 'resume';
type UserOperationalStopwatchEventType = 'user_start' | 'user_stop' | 'user_resume';
export type PerformanceMonitoringStopWatchEventType = 'split' | 'lap' | 'interval';
export type StopWatchEventType = OperationalStopWatchEventType | UserOperationalStopwatchEventType | PerformanceMonitoringStopWatchEventType;

export interface BaseStopwatchEvent<T extends StopWatchEventType> {
    metadata: CreationModificationDates;
    type: T;
    timestamp: Date;
    unit?: UnitValue;
}

type StartEvent = BaseStopwatchEvent<'start'>;
type StopEvent = BaseStopwatchEvent<'stop'>;
type SplitEvent = BaseStopwatchEvent<'split'>;
type LapEvent = BaseStopwatchEvent<'lap'>;

export interface StopwatchState {
    sequence: BaseStopwatchEvent[];
    lap: UnitValue | null;
}

By only storing timestamps, we can

• Dynamically derive our stopwatch view at runtime
• Simple adding/removing splits and modifying splits/laps (if necessary), as the corresponding changes to the stopwatch are automatically derived.
• Add more stopwatch event types as needed, which is a core aspect of adding more features

Groups🔗

In the original version of the stopwatch web app, users could only select multiple stopwatches temporarily to either start/stop them all at once or to add a split/lap. That functionality is preserved but also extends it to allow users to assign stopwatches to groups. This lets users only view/use specific stopwatches at a given time. More importantly, it allows users to designate the relationship between specific stopwatches, which I break into GroupTimingBehavior and GroupEvaluationBehavior:

export type GroupTimingBehavior = 
    | 'parallel'        // Stopwatches run simultaneously (e.g., team members working on the same task)
    | 'sequential'      // Stopwatches run in a defined order (e.g., relay race or assembly line)
    | 'independent'     // No timing constraints
    | 'synchronized'    // Start/stop together
    | 'overlapping';    // Partial temporal overlap (e.g., shift handoffs)

export type GroupEvaluationBehavior =
    | 'independent'     // No relation of evaluation
    | 'comparative'     // Ranked/compared against each other
    | 'cumulative'      // Summed for totals
    | 'threshold'       // Measured against targets/SLAs
    | 'proportional'    // Analyzed as percentages of whole
    | 'trending';       // Tracked for patterns over time

export interface GroupTraits {
    timing: GroupTimingBehavior;
    evaluation: GroupEvaluationBehavior[];
}

By default, a group is assigned independent timing behavior, and independent evaluation, meaning they are only in the group for convenience and will not be evaluated against each other. Users can assign a different timing behavior, and assign one-or-more evaluation behaviors.

Timing Behavior🔗

These represent how the user intends on using the stopwatches in a group. For example, if they are tracking concurrent race between individuals (individual or parallel), or are they tracking the members of a relay (sequential).

• Sequential: Useful for when stopwatch need to run in a defined order, like in a relay race or in an assembly line. This add a single start/stop button to the group, where when the start button starts the first stopwatch, and when the stop button is clicked, the next stopwatch is automatically started.
• Parallel: Useful for when the stopwatch are for the same concurrent task.
• Synchronized: Useful for when stopwatches must start and/or stop together. This adds a button to the UI to allow easy start/stop of all stopwatches in the group.

It is worth noting that timing behavior can be changed at any point, so a user can change them based on their given momentary task.

Evaluation Behavior🔗

These represent how the user intends on evaluating the stopwatches in a given group. In order for these evaluation behaviors to be useful, the user must be setting laps, and/or adding distances to the splits they record.

• Comparative: Stopwatches in the group are to be compared against one another. This results in rankings being shown at the group-level, with relative times between them
• Cumululative: The aggregated elapsed stopwatch time of all the stopwatches is meaningful. The group view will show the aggregated stopwatch time.
• Proportional: Shows stopwatch times at the group-level as percentages of the aggregate elapsed times.
• Trending: The viewer will be evaluating the stopwatches based on their trends while running. This will result in the group view identifying and displaying trends identified per-stopwatch while they are running.

As with timing behaviors, evaluation behaviors can be changed at any point, so a user can change them based on their given momentary task.

Time / Split Prediction🔗

During a race, coaches and enthusiasts are often projecting in their minds whether someone is on-pace or off-pace, or their projected time at given distances. I updated the multi-stopwatch to do split prediction based on previous lap and splits. It takes a rigorous approach. There were two pieces necessary to provide this functionality automatically:

• Time prediction: Can we make confident predictions?
• Split prediction: If we can make confident prediction predictions, which distances/splits should we predict?

Rather than coming up with a unified formula for predicting each, I implemented a variety of approaches for each problem.

Time prediction🔗

All five approaches share the same shape. They work on segments rather than cumulative splits, since the fifth 400 tells you more about the sixth than the total time through 2000m does. Each fits some model of pace to those segments, projects it forward a segment at a time until it reaches the target distance, and accumulates the variance of each projected segment as it goes. The output is never a bare number: it’s a point estimate paired with a confidence interval, which is what makes the first question answerable at all. The same machinery runs in reverse, so each can also answer where a runner will be at a given time rather than when they’ll reach a given distance.

What separates them is the assumption each makes about how pace changes over a race, and how much data that assumption costs.

• The Kalman filter assumes a true pace exists underneath the hand-timing noise, and also needs 2 splits.
• The regressions assume a linear trend and want 3.
• Generalized progression needs 4 splits before it will try to distinguish one pattern from another.

Those minimums are the practical answer to whether we can predict confidently: rather than one global threshold, each approach stays quiet until it has enough to work with, and falls back to something simpler when its own assumptions break down mid-race.

• Exponentially Weighted Moving Average (EWMA)
  • Assumption: assumes nothing beyond “recent is more relevant”
  • Minimum splits: 2
  • Simple exponential smoothing is applied to pace, which is a weighted average of segment paces with no trend term at all, where every prior split still counts but geometrically less. It makes the fewest assumptions of the currrent approaches and will lag a runner who is steadily changing pace, which is exactly the trade for being the one approach that works on almost no data.
• Kalman Filter
  • Assumption: assumes a true pace exists underneath the hand-timing noise
  • Minimum splits: 2
  • Models a latent “true pace” that evolves with process noise, observed with measurement noise. Hand timing is noisy in a way the other approaches treat as signal, and separating the two is the whole point. It can optionally carry a drift term to model systematic pace change. Process and measurement noise are estimated from the splits themselves rather than known ahead of time, so the estimates are principled but not optimal in the strict sense.
• Regressions: Assume a linear trend. Minimum splits: 3
  • Ordinary Least Squares (OLS) Linear Regression: Fits pace as ax + b, choosing the coefficients that minimize squared error across the observed segments, then integrates the fitted pace forward. x is the segment number by default, but can instead be cumulative distance or cumulative time, which separates distance-related fatigue from time-related fatigue. When no trend exists the slope goes to zero and the prediction collapses to the mean pace.
  • Weighted Least Squares (WLS) with Recency Decay: Similar to linear regression, but weighs the error of more recent splits higher. Weights decay exponentially with age down to a floor, so early splits lose influence without ever dropping out entirely.
• Generalized Progression: Changes in pace often follow a regular shape, so this fits several candidate shapes and keeps whichever explains the splits best. The three below are the same linear fit under a Box-Cox transform (λ = 1, 0, and -1 respectively), which means additional shapes cost nothing but a new λ. Candidates are scored by AICc, the small-sample correction to Akaike’s Information Criterion, so one always wins; the criterion ranks the shapes against each other rather than testing whether any progression exists at all.
  • Minimum splits: 4
  • Arithmetic: Splits are increasing/decreasing by a precise number (ex. Doug’s 400m splits are dropping 5 seconds each)
  • Geometric: Splits are increasing/decreasing by a percentage (ex. Doug is slowing down by 5% every 400m)
  • Harmonic: Speed is increasing/decreasing by a specific rate (ex. Doug’s speed is dropping by exactly 0.5 meters per second every 400m). The name is exact rather than loose: a harmonic progression is one whose reciprocals form an arithmetic progression, and speed is the reciprocal of pace.

Since no single approach is right for every race, the predictions can also be combined. An ensemble runs all of them and blends the results, weighting each by the tightness of its confidence interval, by its recent accuracy, or equally. An adaptive selector instead scores the candidates against the last few segments and hands the race to whichever is currently performing best.

Split prediction🔗

For split prediction, we account for the split distances that have already been recorded, since the application does not currently know what the target distance (if any) of the stopwatch is.

The approaches fall into two families: continuing the rhythm the user has already been recording, or working out which distances are meaningful for their sport.

Pattern continuation🔗

These only look at the recorded splits and the lap, and make no assumptions about the sport. Splits taken by hand are never exact, so each matches within a tolerance rather than on equality.

• Fixed Interval: Splits are being taken at a consistent distance (ex. 400/800/1200 predicts 1600). Deltas are clustered rather than compared directly, and a majority of them have to fall in the same cluster before an interval is claimed.
• Delta Sequence: Splits repeat on a cycle rather than a single interval (ex. 0/100/400/500/800 predicts 900, not 1200). A cycle is accepted while most of its positions still match, so one missed or late split doesn’t invalidate it.
• GCD Grid: Splits sit on a grid coarser than their pattern of use, which distinguishes every other grid position from every one (ex. 200/600/1000 predicts 1400, not 1200). Distances are rounded before the divisor is taken, since a single stray meter would otherwise collapse the grid.
• Lap Relative: Splits recur at the same position within a lap regardless of absolute distance (ex. always at the 200m mark of a 400m lap). Positions are grouped circularly, so a split taken just before the line stays with the ones taken just after it.
• Ratio / Fractional: Same as lap-relative, but in fractions of a lap, which matters when the lap doesn’t divide cleanly (ex. three-quarters of a 333.33m lap). Each position snaps to the nearest fraction people actually use, and anything that lands near none of them is ignored.

Milestone inference🔗

These assume people stop the watch at round numbers, so the question becomes which set of round numbers applies. Each sport gets a definition covering its base unit, conventional milestones, applicable distance range, expected pace range, and the lap distances that imply it. Every match is scored rather than accepted or rejected, so a near miss lowers confidence instead of eliminating a system.

• Lap-Based: Lap distance implies the sport (ex. 25m is swimming, 400m is track, 2000m is rowing).
• Round Number Affinity: Splits are scored against each system’s milestones, base unit, and distance range (ex. 1000/2000/3000 scores nearly perfectly for metric kilometers).
• Pace-Based: Median pace narrows the field on its own (ex. 0.9 sec/m is swimming and cannot be cycling), and the median is used specifically so a single bad split doesn’t move it.
• Composite: Weights the three inferences above, with lap highest and round number affinity lowest, and rewards agreement between them.
• Target Distance: Generates milestones up to a known or inferred finish, and appends the finish itself as the final split.

Try It🔗

Epochron is live at stopwatch.dougfenstermacher.com and the source is at github.com/dpfens/stopwatch, under the Unlicense. It’s at v0.2.0 and still moving; if you time things for a living or a hobby and something here does not fit how you actually work, I want to hear about it.

For that reason, I decided to take what I learned in my last post about hatching to see if I could use structural tensors to create more robust approach for turning images into decent sketches.

Below is an interactive demo of X-DoG/F-DoG (without extensions) for you to use. It has WebGL and WebGPU support (as a learning opportunity) so it’s very fast, even on large images. I have built some extensions, like multi-scale tensor/detail extraction, and color retention but they aren’t ready to be published yet.

If you want to read about the algorithm itself, you can read the original XDoG: An eXtended difference-of-Gaussians compendium including advanced image stylization paper.

UPDATE: I have converted this to use a Web Worker (run on a separate thread), after realizing FDoG can be a bit slower for larger images.

Drop an image here or click to upload

Supports JPG, PNG, WebP

• Original
• Result

Processing...

Processing...

Generate Line Drawing

Processing Mode

XDoG FDoG

Style Presets

✒️ Line Art ✏️ Pencil 🎨 Pastel 🖤 Charcoal 🪵 Woodcut

FDoG Style Presets

📐 Standard 🎨 Pastel 🪵 Woodcut

Core Parameters

σ (Sigma) - Edge Scale 1.4

k - Blur Ratio 1.6

p - Sharpening Strength 20

ε (Epsilon) - Threshold 0.78

φ (Phi) - Threshold Sharpness 100

FDoG Flow Parameters

σc - Structure Tensor Smoothing 2.28

σm - Flow-Aligned Smoothing 4.4

σa - Anti-Aliasing 1.0

Preprocessing

Preprocessing Mode None Light (clean images) Standard (most photos) Heavy (textured images) Nature (landscapes) Artistic (painterly)

Preprocessing smooths textures while preserving edges, reducing noise in the final result.

Tips

XDoG uses isotropic Gaussian blur - fast and great for most images. FDoG uses flow-guided blur along edge tangents for smoother, more coherent lines. Best for portraits and subjects where line continuity matters.

This is a guide to one particular technique called streamline hatching where hatching is a pen & paper approach for shading an image where strokes following the form of objects, density varies with tone, crosshatching building up in shadows, etc. More importantly, you’ll understand why each piece of the pipeline exists and what happens when you change it.

What We're Building Toward🔗

Hatching is a drawing technique where artists use parallel lines to create tone and texture. The direction of the strokes typically follows the form of what’s being drawn, like curving around a sphere, running along a cylinder, radiating from a corner etc. The spacing between strokes controls darkness, so closer strokes appear darker which means a darker shade like a shadow, and further apart strokes for highlights.

Cross-hatching adds a second layer of strokes at an angle to the first, building up darker tones in shadow regions.

When hatching is done well, hatching reveals what shape something is as well as how dark it is.

My goal was to replicate this. Given an input image, I wanted to:

1. Understand the directional structure of the image. where are the forms, and which way do they “flow”?
2. Generate strokes that follow this structure
3. Control density and placement so that dark areas get more strokes than light areas
4. Optionally add cross-hatching in the darkest regions

The output should look like something a human might have drawn.

The Pipeline at a Glance🔗

Here’s the full pipeline:

1. Input Image
2. Grayscale Conversion
3. Structure Tensor Computation (at multiple scales)
4. Orientation & Coherence Fields
5. Edge Tangent Flow (ETF) Refinement
6. Direction Propagation
7. Stroke Generation
8. Rendering

Here’s the working implementation. Try it with the preset shapes or upload your own image. Experiment with the parameters and watch how the output changes. The rest of this post explains what’s happening under the hood.

See the Pen Hatching - WebGPU by Douglas Fenstermacher (@dfens) on CodePen.

The Parameter Space🔗

The full system has many parameters. Here they are, grouped by the aspect of the outcome they effect:

Analysis parameters (affect how structure is perceived):

• sigma array: Scales for multi-scale analysis
• scaleCombination: How to combine scales (‘eigenvalue’ or ‘gradient’)
• fineScaleBias: Preference for fine vs. coarse detail
• minCoherenceThreshold: Below this, orientation is unreliable

Flow field parameters (affect smoothness and propagation):

• etfIterations: How many refinement passes
• etfKernelSize: Size of the refinement neighborhood
• propagationIterations: How far direction spreads into flat regions

Stroke parameters (affect individual strokes):

• strokeCount: How many strokes to attempt
• minLength, maxLength: Stroke length range
• strokeWidth, widthVariation: Appearance
• strokeJitter: Hand-drawn wobble

Placement parameters (affect overall coverage):

• toneInfluence: Weight darkness vs. uniform coverage
• minSpacing: Minimum distance between stroke seeds
• seedRandomness: Jitter in seed grid

Tone parameters (affect light/dark handling):

• backgroundBrightnessThreshold: What counts as “background”
• lowGradientThreshold: What counts as “no structure”
• crossHatch: Enable perpendicular strokes
• crossHatchThreshold: Darkness level for cross-hatching

Presets as Starting Points🔗

The implementation includes several presets that configure these parameters for different effects:

Sketch: Loose, gestural strokes

• Fewer, longer strokes
• Higher width variation
• Lower opacity

Engraving: Dense, controlled lines

• Many short strokes
• Low width variation
• High tone influence
• Cross-hatching enabled

Crosshatch: Emphasis on tonal building

• Medium density
• Strong cross-hatching in shadows

Simple Shapes: For geometric forms

• Single scale (coarse)
• High propagation to fill flat regions

Computational Considerations🔗

The implementation supports both CPU and WebGPU backends. The core algorithms are the same; only the tensor operations differ.

For CPU:

• Structure tensor computation: O(width × height × scales × kernel_size^2)
• ETF refinement: O(width × height × iterations × kernel_size^2)
• Stroke generation: O(stroke_count × max_length)

For interactive use on typical images (500×500), CPU processing takes 1-3 seconds on a modern browser. WebGPU reduces this substantially for the tensor operations but has overhead for the stroke generation phase which remains CPU-bound.

Further Reading🔗

For you want to look into the underlying techniques:

• Structure Tensor: Förstner, W., & Gülch, E. (1987). “A fast operator for detection and precise location of distinct points, corners and centres of circular features.” Search for “structure tensor image processing” for accessible tutorials.
• Edge Tangent Flow: Kang, H., Lee, S., & Chui, C. K. (2007). “Coherent Line Drawing.” This paper introduced the ETF refinement technique.
• Streamline Visualization: Search “streamline integration visualization” for the fluid dynamics perspective on field-following curves.
• Non-Photorealistic Rendering: The field of NPR has extensive literature on computational illustration techniques. Gooch & Gooch’s book “Non-Photorealistic Rendering” is a comprehensive introduction.

The complete implementation, including an interactive demo, is available on CodePen. Try it with your own images and experiment with the parameters. the best way to understand the system is to see how each setting affects the output.

Respecting User Privacy🔗

While embeddings have become a popular and powerful tool for analyzing text data, including user prompts, they come with significant privacy implications that warrant careful consideration. Embeddings capture the semantic essence of text, often in ways that can inadvertently reveal more information than intended. Embeddings can encode personal details, writing styles, or specific knowledge that, when analyzed in aggregate, could lead to the identification of individuals or the disclosure of private information.

For instance, embeddings of prompts related to medical conditions, financial situations, or personal relationships could, even unintentionally, create a detailed profile of a user’s life circumstances. In corporate environments, embeddings might capture proprietary information or strategic plans embedded within prompts. The risk of such unintended disclosures is particularly concerning given the often personal or sensitive nature of interactions with AI assistants.

Moreover, the high-dimensional nature of embeddings means they can capture subtle patterns that users themselves might not be aware they’re revealing. This “hidden” information could be exploited through advanced analysis techniques, potentially compromising user privacy in ways that are difficult to anticipate or mitigate.

Given these concerns, it’s crucial to adopt an approach that explicitly respects the privacy of end-users while still providing valuable insights for system improvement and user experience enhancement. This is where the analysis of orthogonal components comes into play. By focusing on abstract, high-level characteristics of prompts rather than their specific content, we can gain useful analytical insights while maintaining a strong commitment to user privacy.

In this scenario, we can think of orthogonal components as dimensions of interaction we are curious about, which do not influence each other. For example, the ambiguity of a prompt and the purpose of an LLM prompt can change independently. One can write a prompt with the same purpose with varying degrees of ambiguity. Changing the ambiguity of a prompt does not effect the purpose of the prompt, and vice versa.

The orthogonal component approach allows us to do a few things. We can add more dimensions and values within each dimension without invalidating our existing data, and every new dimension/value can provide context across all the other dimensions. And it provides a framework for analysis that is inherently more privacy-preserving, as it captures the nature of the interaction rather than the content itself.

In the following sections, I’ll explore how this privacy-respecting method can be implemented effectively, providing robust analytics capabilities without compromising the trust and confidentiality that users expect when interacting with LLMs.

Orthogonal Components🔗

The orthogonal components of a prompt provide a multidimensional framework for analyzing user interactions with large language models (LLMs). By breaking down prompts into these distinct elements, we can gain valuable insights into user behavior and LLM performance without compromising individual privacy. These components encompass a wide range of factors, from the basic intent of the prompt to the level of creativity expected in the response.

• Intent: The core purpose or goal of the prompt.
• Complexity: The level of difficulty or sophistication required in the response.
• Ambiguity: The degree of clarity or vagueness in the prompt’s wording.
• Scope: The breadth or narrowness of the topic or task at hand.
• Specificity: The level of detail requested in the response.
• Tone: The emotional or stylistic approach requested (e.g., formal, casual, humorous).
• Domain(s): The field or subject area the prompt relates to.
• Cognitive level: The depth of thought or analysis required (e.g., recall, application, evaluation).
• Persona: Any specific character or role the LLM is asked to assume.
• Abstraction level: How concrete or abstract the prompt and expected response should be.
• Creativity: The degree of originality or innovation expected in the response.
• Subjectivity vs. Objectivity: The degree of personal opinion versus impartial analysis required.

At the core of this framework is the intent, which identifies the fundamental purpose of the user’s query. This is complemented by the complexity and ambiguity components, which gauge the sophistication required in the response and the clarity of the prompt itself. The scope and specificity components work together to define the breadth of the topic and the level of detail requested, providing a clear picture of the user’s informational needs.

The tone and persona components offer insights into the stylistic and role-playing aspects of user interactions with the LLM. These elements can reveal patterns in how users engage with AI systems, whether they prefer formal interactions or more casual, character-driven exchanges. The domain component helps categorize prompts into specific fields or subject areas, allowing for a better understanding of the topics users are most interested in exploring.

Cognitive level and abstraction level components provide a deeper look into the type of thinking and conceptualization users expect from the LLM. These elements can help identify whether users are primarily seeking factual recall, complex analysis, or abstract reasoning. The creativity component adds another dimension, indicating the degree of originality users are looking for in AI-generated responses.

By analyzing the combinations of these orthogonal components across numerous user interactions, we can begin to create detailed personas of end-user conversations. These personas represent archetypal users based on their typical prompt patterns, without relying on any specific prompt content. For example, we might identify a “Detail-Oriented Analyst” persona who consistently uses prompts with high specificity, low ambiguity, and a preference for objective responses across various domains.

Another persona might be the “Creative Brainstormer,” characterized by prompts with high creativity, broad scope, and often involving persona-based interactions. By recognizing these patterns, product developers can gain valuable insights into how different user types engage with the LLM, informing decisions about feature development, interface design, and model fine-tuning.

These user personas can drive product development in several ways. For instance, if a significant portion of users fit the “Detail-Oriented Analyst” persona, developers might prioritize features that enhance the LLM’s ability to provide precise, well-structured responses. Conversely, for the “Creative Brainstormer” persona, the focus might be on expanding the model’s capacity for generating novel ideas and engaging in more dynamic, role-playing scenarios.

Importantly, this approach to analytics and persona creation maintains user privacy by focusing on aggregate patterns rather than private details of individual prompts. It allows organizations to make data-driven decisions about their LLM implementations while respecting user confidentiality. As AI technology continues to evolve, this balance between insightful analytics and robust privacy protection will be crucial in building and maintaining user trust in AI systems.

Conversations🔗

In addition to prompt-level orthogonal components, we can also identify and analyze components at the conversation level. These conversation-level components provide a broader perspective on user interactions with the LLM, capturing patterns and characteristics that emerge over the course of multiple exchanges. Here are some key conversation-level orthogonal components:

• Conversation Length: The number of turns or exchanges in the conversation.
• Topic Consistency: The degree to which the conversation stays focused on a single topic or ranges across multiple subjects.
• Depth Progression: How the conversation evolves in terms of complexity and detail from start to finish.
• Interaction Pattern: The rhythm of the conversation, such as rapid back-and-forth exchanges or longer, more deliberate responses.
• Goal Orientation: Whether the conversation is task-oriented, exploratory, or social in nature.
• Emotional Trajectory: How the tone or sentiment of the conversation changes over time.
• Creativity Fluctuation: The variation in creative elements throughout the conversation.
• Query Refinement: How user prompts become more or less specific or focused as the conversation progresses.
• AI Reliance: The extent to which the user relies on the AI’s responses versus contributing their own knowledge.
• Coherence: The logical flow and connectedness between different parts of the conversation.
• Learning Curve: Evidence of the user adapting their interaction style or understanding of the AI’s capabilities over time.
• Multi-modal Interaction: The use of different types of inputs or requests (e.g., text, code, data analysis) within a single conversation.

These conversation-level components provide valuable insights into how users engage with the LLM over extended interactions. By analyzing these components, we can identify patterns in user behavior, preferences in conversation style, and the effectiveness of the LLM in sustaining meaningful dialogues.

For example, by examining the Depth Progression and Query Refinement components, we might observe that successful problem-solving conversations often start with broad, exploratory queries and gradually narrow down to specific, detailed exchanges. This insight could inform improvements in the LLM’s ability to guide users through complex problem-solving processes.

Similarly, analyzing the Emotional Trajectory and Interaction Pattern could reveal how different conversation rhythms correlate with user satisfaction or productive outcomes. This information could be used to train the LLM to adapt its response style to better match user preferences and optimize engagement.

Importantly, these conversation-level components maintain user privacy by focusing on abstract characteristics of the interaction rather than specific content. They allow for rich analysis of user engagement patterns without risking the exposure of sensitive information contained within individual prompts or responses.

By combining these conversation-level insights with the prompt-level orthogonal components discussed earlier, we can build a comprehensive, privacy-respecting framework for understanding and improving LLM interactions.

Implementation🔗

Implementing a system to analyze prompts based on orthogonal components requires careful consideration of both technical and practical aspects. While the concept is promising, the execution needs to be efficient and cost-effective. Let’s explore a viable implementation strategy that balances accuracy, privacy, and resource utilization.

As noted, training a dedicated model to classify prompts based on the orthogonal components would indeed be a resource-intensive process. The extensive data labeling, cleaning, and refinement required for such an approach could make it prohibitively expensive and time-consuming for many organizations. This method, while potentially very accurate, may not be feasible for teams working with limited budgets or tight timelines.

The alternative approach of using the LLM itself to classify user prompts is more promising, especially when considering the capabilities of advanced models like those offered by OpenAI and Anthropic. The concern about doubling the token usage (and thus the cost) by submitting each prompt twice is valid. However, as suggested, OpenAI’s Structured Output responses provide an easy solution to this problem.

Custom schema responses allow us to define a specific output format that the LLM should follow. By carefully crafting this schema, we can instruct the LLM to not only provide the requested response to the user’s prompt but also to classify the prompt according to our orthogonal components. This approach effectively kills two birds with one stone, obtaining both the user’s desired output and our analytical data in a single API call.

To implement this, we would need to design a JSON schema that includes fields for each of the orthogonal components we want to track. For example, our schema might look something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
{
  "response": "string",
  "components": {
    "intent": ["string"],
    "complexity": "integer",
    "ambiguity": "integer",
    "scope": "string",
    "specificity": "integer",
    "tone": "string",
    "domain": ["string"],
    "cognitiveLevel": "string",
    "persona": ["string"],
    "abstractionLevel": "integer",
    "creativity": "integer",
    "subjectivity": "integer"
  }
}

With this schema in place, we would then modify our API calls to the LLM to request both the standard response and the prompt analysis. The LLM would process the user’s prompt, generate the appropriate response, and simultaneously classify the prompt according to our specified orthogonal components.

One challenge with this approach is ensuring that the LLM’s classification is consistent and accurate. To address this, we might need to provide clear guidelines or examples for each component as part of our system prompt. We could also implement a calibration process, where we periodically submit known prompts and compare the LLM’s classifications to predetermined benchmarks.

On the backend, we would need to set up a system to store and analyze the classification data. This could involve a database to record the orthogonal component values for each prompt, along with analytics tools to identify patterns and trends over time. This could be stored efficiently in a set of normalized SQL tables. The key is storing only the component classifications without any of the original prompt text. This sort of access could be controlled by providing external teams with permissioned VIEWS that exclude embeddings and original text.

To further enhance privacy, we could implement additional safeguards such as data aggregation and anonymization techniques. For instance, we might only analyze data in batches of a certain size, or use differential privacy methods to add controlled noise to the data, making it impossible to reverse-engineer individual prompts from the stored classifications.

As we accumulate data, we can begin to create user personas based on common patterns in the orthogonal components. This process might involve clustering algorithms to identify groups of users with similar prompt characteristics. These personas can then inform product development decisions, helping to tailor the LLM’s capabilities to the needs and preferences of different user types.

It’s important to note that while this implementation strategy is efficient in terms of API usage, it does rely on the accuracy and consistency of the LLM’s self-classification. Regular auditing and refinement of the classification process may be necessary to ensure the ongoing reliability of the data. Additionally, as LLM capabilities evolve, the classification schema and process may need to be updated to take advantage of new features or to address any emerging biases or inaccuracies.

Analysis🔗

Analyzing the orthogonal components of user prompts over time can reveal fascinating trends and patterns in LLM usage. This analysis could provide valuable insights into user behavior, evolving needs, and the overall effectiveness of the LLM. Let’s explore some potential avenues for analysis and the trends they might uncover.

Sequence Analysis🔗

By examining sequences of orthogonal component combinations, we could identify common user journeys or task flows. For instance, we might observe that users often start with low-complexity, high-ambiguity prompts in a particular domain, gradually increasing in specificity and complexity as they refine their queries. This could indicate a learning curve as users become more familiar with the LLM’s capabilities.

Temporal Patterns🔗

Analyzing how component combinations change over time (e.g., daily, weekly, or seasonally) could reveal interesting usage patterns. We might find that creativity and abstraction levels peak during certain times of day, possibly correlating with when users are engaged in brainstorming or creative tasks. Conversely, we might see a trend towards high-specificity, low-ambiguity prompts during typical working hours, suggesting task-oriented usage.

Domain Clustering🔗

By clustering prompts based on their domain component, we could identify which subject areas are most popular among users. This analysis might reveal unexpected connections between domains, such as users frequently transitioning between seemingly unrelated fields. Such insights could inform interdisciplinary applications of the LLM.

Cognitive Progression🔗

Tracking the cognitive level component over time for individual users or user groups could show how interaction with the LLM impacts cognitive engagement. We might observe a trend towards higher cognitive levels as users become more adept at formulating complex queries, indicating that the LLM is effectively supporting intellectual growth.

Persona Evolution🔗

By analyzing how the persona component changes across user sessions, we could gain insights into how users perceive and interact with AI. A trend towards more frequent use of specific personas might suggest that users are becoming more comfortable with role-playing scenarios or are finding particular personas especially helpful for certain tasks.

Ambiguity and Specificity Correlation🔗

Examining the relationship between ambiguity and specificity over time could reveal how users learn to formulate more effective prompts. We might see a general trend towards lower ambiguity and higher specificity as users become more experienced, potentially indicating improved communication between humans and AI.

Creativity Cycles🔗

Analyzing the creativity component across user bases could uncover cycles or triggers for creative use of the LLM. This might reveal specific combinations of other components that tend to precede or accompany high-creativity prompts, offering insights into the conditions that foster innovative thinking.

Tone Adaptation🔗

Tracking changes in the tone component could show how users adapt their communication style with the LLM over time. We might observe users converging on particular tones that they find most effective, or adapting their tone based on the complexity or domain of their queries.

By analyzing these trends and patterns, organizations could gain a deeper understanding of how the user’s approach to using the LLM changes over time. This information could drive decisions about model updates, feature development, and user education initiatives. For example, if analysis reveals that users struggle to reduce ambiguity in certain domains, this could prompt the development of more targeted help features or refinements to the LLM’s ability to handle ambiguous queries in those areas.

Wisdom of the Crowd🔗

We can extend our focus to compare trajectories across user conversations and between different users. This comparative analysis opens up new possibilities for clustering, similarity measurement, and pattern recognition on a broader scale.

Cross-Conversation Analysis🔗

By examining the trajectories of orthogonal component combinations across multiple conversations for a single user, we can identify consistent patterns in their LLM interaction style. For instance, we might discover that a user typically begins conversations with high-ambiguity, low-specificity prompts, gradually increasing in complexity and specificity as the conversation progresses. This could indicate a preference for exploratory dialogue followed by focused inquiry.

User Similarity Clustering🔗

Comparing these conversation trajectories between users allows us to cluster individuals with similar interaction patterns. We could employ techniques like Dynamic Time Warping (DTW) to measure the similarity between conversation sequences, accounting for variations in conversation length and pacing. This clustering could reveal distinct user archetypes, such as “Methodical Researchers” who consistently progress from broad to narrow queries, or “Creative Explorers” who maintain high levels of ambiguity and creativity throughout their interactions.

Divergence Analysis🔗

By measuring how quickly user trajectories diverge from one another, we can gain insights into the diversity of user needs and interaction styles. Rapid divergence might indicate a user base with varied expertise levels or diverse use cases, while slower divergence could suggest more homogeneous user behavior. This information could be valuable for tailoring the LLM’s capabilities to support a wide range of interaction styles or for identifying underserved user segments.

Temporal Cohort Comparison🔗

Grouping users into cohorts based on when they first started using the LLM allows us to compare trajectory evolution over time. We might find that newer users show more erratic patterns in their orthogonal component combinations, while long-term users demonstrate more stable and efficient trajectories. This could provide insights into the learning curve associated with LLM use and inform onboarding strategies.

Cross-Domain Transfer🔗

Analyzing how users’ trajectories change when they switch between domains can reveal interesting patterns of knowledge transfer. We might observe that users who exhibit high complexity and low ambiguity in one domain quickly achieve similar levels in new domains, suggesting effective cross-domain learning. Conversely, consistent drops in complexity when switching domains could indicate areas where additional user support is needed.

Collaborative Pattern Analysis🔗

For LLMs used in collaborative settings, we can analyze how the trajectories of multiple users interacting on the same project align or diverge. This could uncover effective collaboration patterns, such as complementary query styles that lead to more comprehensive project outcomes.

Anomaly Detection🔗

By establishing baseline trajectories for different user types, we can implement anomaly detection to identify unusual interaction patterns. This could be useful for detecting potential misuse of the LLM, identifying users who might benefit from additional support, or discovering innovative use cases that deviate from the norm.

Predictive Modeling🔗

Using historical trajectory data, we can build predictive models to anticipate future user needs based on the early stages of their conversation trajectories. This could enable proactive adjustments to the LLM’s behavior or the suggestion of relevant resources to support the user’s likely trajectory.

By leveraging these comparative analyses, teams/organizations can better understand their user base and how individuals interact with their LLM over time. This can contribute to live personalization strategies and inform feature development.

Next Steps🔗

The field of LLM analytics is rapidly evolving, and the approach we’ve discussed using orthogonal components is just the beginning. As we look to the future, several intriguing possibilities emerge for advancing LLM analytics while maintaining user privacy. Here are some speculative next steps:

1. Dynamic Component Analysis: Future systems might adaptively modify the orthogonal components based on observed patterns. This could involve automatically identifying new relevant components or refining existing ones to better capture emerging user behaviors and LLM capabilities.
2. Multi-Modal Analytics: As LLMs increasingly integrate with other AI systems (like image recognition or speech processing), analytics might expand to cover these multi-modal interactions. This could involve developing new orthogonal components specific to visual or auditory prompts.
3. Temporal Pattern Recognition: Advanced analytics might focus on identifying patterns in user behavior over time. This could involve tracking how individual users or user groups evolve in their interaction styles, potentially revealing learning curves or changing preferences in AI engagement.
4. Federated Learning for Analytics: To further enhance privacy, federated learning techniques could be applied to LLM analytics. This would allow multiple organizations to collaboratively improve their analytical models without sharing raw data, only aggregated insights.
5. Prompt Optimization Feedback Loops: Analytics data could be used to create systems that automatically suggest improvements to user prompts. This could help users more effectively engage with LLMs while also providing valuable data on prompt refinement patterns.
6. Emotional Intelligence Analytics: Future systems might attempt to gauge the emotional state or satisfaction level of users based on their prompts and interactions. This could involve developing new components related to user sentiment or engagement levels.
7. Cross-Platform Behavior Analysis: As LLMs become more ubiquitous, analytics might expand to track how users interact differently with various LLM implementations across different platforms or use cases.
8. Predictive Analytics for LLM Interactions: Advanced systems might use historical data to predict future user behaviors or needs, allowing for proactive improvements to LLM capabilities or user interfaces.

These potential advancements in LLM analytics could deepen our understanding of how humans interact with AI systems, driving improvements in LLM technology while maintaining the trust of end-users. As with any technology that is not yet mature, ethical considerations and transparency are crucial.

At their core, NVDMs are auto-encoders, consisting of an encoder-decoder architecture where the encoder maps the input documents to a continuous latent space, and the decoder reconstructs the original documents from the latent representations. The encoder is typically a neural network, such as a Convolutional Neural Network (CNN) or a Recurrent Neural Network (RNN), which learns to compress the input documents into a lower-dimensional latent space. The decoder, on the other hand, is responsible for generating the words in the documents based on the latent representations.

The key innovation in NVDMs is the incorporation of variational inference, which allows the model to learn a probabilistic distribution over the latent space. By introducing a variational posterior distribution, NVDMs can capture the uncertainty associated with the latent representations and enable more robust and interpretable topic modeling. The objective of training an NVDM is to maximize the evidence lower bound (ELBO), which consists of a reconstruction term and a regularization term that encourages the variational posterior to be close to a prior distribution, typically a multivariate Gaussian.

NVDMs have found numerous applications in natural language processing tasks, such as topic modeling, document clustering, text generation, and sentiment analysis. By learning rich and interpretable representations of documents, NVDMs enable users to gain insights into the underlying themes and semantic structures within their text data. Moreover, the ability to generate new documents from the learned latent space opens up possibilities for content creation and data augmentation.

Fundamentals of Neural Variational Document Models🔗

Neural Variational Document Models (NVDMs) are built upon three key concepts: latent variable models, variational inference, and the encoder-decoder architecture.

Latent Variable Models🔗

Latent variable models are probabilistic models that introduce unobserved (latent) variables to capture the underlying structure and dependencies in the observed data. In the context of document modeling, latent variables can represent abstract concepts, such as topics or themes, that influence the generation of words in a document. By learning the latent variables, NVDMs can uncover the hidden semantic structure within a collection of documents.

The joint probability distribution of an NVDM can be expressed as:

p(x, z) = p(z) * p(x|z)

where x represents the observed document, z represents the latent variables, p(z) is the prior distribution over the latent variables, and p(x|z) is the conditional probability of the document given the latent variables.

Variational Inference: Variational inference is a technique used to approximate intractable posterior distributions in probabilistic models. In NVDMs, the true posterior distribution p(z|x) is often computationally intractable due to the complexity of the model. Variational inference introduces a variational posterior distribution q(z|x), which is a simpler and more tractable distribution that approximates the true posterior.

The goal of variational inference is to minimize the Kullback-Leibler (KL) divergence between the variational posterior and the true posterior:

KL(q(z|x) || p(z|x))

By minimizing the KL divergence, the variational posterior is encouraged to be close to the true posterior, enabling efficient inference and learning in NVDMs.

Encoder-Decoder Architecture🔗

NVDMs employ an encoder-decoder architecture to learn the latent representations and generate documents. The encoder is a neural network that takes the input document x and maps it to a continuous latent space, producing the parameters of the variational posterior distribution q(z|x). The encoder can be implemented using various neural network architectures, such as Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs), depending on the nature of the text data.

The decoder, on the other hand, is responsible for generating the words in the document based on the latent representation z sampled from the variational posterior. The decoder is typically a neural network that models the conditional probability distribution p(x|z). Common choices for the decoder include RNNs or feed-forward neural networks with a softmax output layer to generate the probability distribution over the vocabulary.

During training, the encoder and decoder are jointly optimized to maximize the evidence lower bound (ELBO), which consists of two terms:

ELBO = E[log p(x|z)] - KL(q(z|x) || p(z))

The first term is the expected log-likelihood of the document given the latent variables, encouraging the decoder to generate documents similar to the input. The second term is the KL divergence between the variational posterior and the prior distribution, acting as a regularizer to prevent overfitting and promote a smooth latent space. By leveraging the power of latent variable models, variational inference, and the encoder-decoder architecture, NVDMs can learn meaningful and interpretable representations of documents, enabling various downstream tasks such as topic modeling, document clustering, and text generation.

Reparameterization trick🔗

In the context of NVDMs, the goal is to learn a variational posterior distribution q(z|x) that approximates the true posterior distribution p(z|x) over the latent variables z given the input document x. However, directly sampling from the variational posterior during the forward pass of the model would block the gradients, as the sampling operation is not differentiable.

The reparameterization trick addresses this issue by separating the sampling process into two parts: a deterministic part and a stochastic part. Instead of directly sampling from the variational posterior q(z|x), which is typically parameterized as a Gaussian distribution with mean μ and standard deviation σ, the trick introduces an auxiliary noise variable ε drawn from a standard Gaussian distribution N(0, 1).

The reparameterization is as follows:

z = μ + σ * ε, where ε ~ N(0, 1)

By expressing the sampling process in this way, the gradients can flow through the deterministic parts (μ and σ) of the reparameterization, while the stochastic part (ε) remains independent of the model parameters.

By using the reparameterization trick, the NVDM can backpropagate gradients through the sampling process and optimize the variational parameters (μ and σ) using standard gradient descent methods. This allows the model to learn a meaningful latent representation of the input documents while effectively approximating the true posterior distribution.

The reparameterization trick has been widely adopted in variational inference and generative models, as it enables efficient training and inference in models with continuous latent variables. It has been instrumental in the success of VAEs and their variants, including NVDMs, in various domains such as computer vision, natural language processing, and recommendation systems.

Advantages of NVDMs over Traditional Topic Modeling Techniques🔗

Neural Variational Document Models (NVDMs) offer several advantages over traditional topic modeling techniques, such as Latent Dirichlet Allocation (LDA) and Probabilistic Latent Semantic Analysis (PLSA). These advantages stem from the ability of NVDMs to leverage deep learning architectures and variational inference, enabling them to capture complex semantic relationships, handle large-scale datasets, and provide flexibility in model design. Let’s explore each of these advantages in more detail.

Capturing Complex Semantic Relationships: Traditional topic modeling techniques, such as LDA and PLSA, rely on bag-of-words representations and assume a simple generative process for document generation. They often struggle to capture the complex semantic relationships and dependencies between words and topics. In contrast, NVDMs utilize deep learning architectures, such as Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs), which can learn rich and expressive representations of text data.

By employing neural networks, NVDMs can automatically learn intricate patterns and dependencies in the text, capturing non-linear relationships between words and topics. The encoder in an NVDM can extract meaningful features and contextual information from the input documents, allowing the model to uncover more nuanced and semantically coherent topics. This ability to capture complex semantic relationships enables NVDMs to generate more interpretable and meaningful topic representations compared to traditional techniques.

Traditional topic modeling techniques often face scalability challenges when dealing with large-scale datasets. As the number of documents and vocabulary size increase, the computational complexity of these techniques grows exponentially, making them impractical for handling massive text corpora. NVDMs, on the other hand, can efficiently handle large-scale datasets thanks to their integration with deep learning frameworks like PyTorch.

NVDMs can leverage the parallel processing capabilities of modern hardware, such as GPUs, to speed up training and inference. The mini-batch training paradigm used in deep learning allows NVDMs to process documents in parallel, enabling efficient learning even on large datasets. Moreover, the stochastic nature of variational inference in NVDMs enables them to approximate the posterior distribution using subsamples of the data, further improving scalability.

PyTorch, with its optimized tensor operations and built-in support for GPU acceleration, provides a powerful platform for training NVDMs on large-scale datasets. The framework’s efficient memory management and ability to handle sparse computations make it well-suited for processing text data, which often exhibits high dimensionality and sparsity.

NVDMs offer a high degree of flexibility in model architecture, allowing researchers and practitioners to design models tailored to their specific needs and domains. The encoder-decoder architecture of NVDMs can be easily customized and extended to incorporate various neural network components and techniques.

For example, the encoder can be implemented using different neural network architectures, such as CNNs for capturing local patterns or RNNs for modeling sequential dependencies. The choice of architecture can be based on the characteristics of the text data and the desired level of complexity. Similarly, the decoder can be designed to generate words based on different probability distributions or incorporate additional information, such as document metadata or linguistic features.

PyTorch’s modular and flexible design makes it easy to experiment with different architectures and components in NVDMs. The framework provides a wide range of pre-built neural network modules, loss functions, and optimization algorithms, allowing users to quickly prototype and iterate on their models. The dynamic computational graph in PyTorch enables dynamic model construction and modification, facilitating the development of custom NVDM architectures.

Furthermore, NVDMs can be extended to incorporate additional features or constraints, such as prior knowledge, multi-modal information, or semi-supervised learning objectives. The flexibility in model architecture enables researchers to explore novel variations and adaptations of NVDMs, pushing the boundaries of topic modeling and uncovering new insights from text data.

Implementation🔗

This implementation provides a solid foundation for training and utilizing an NVDM for topic modeling tasks. By leveraging PyTorch’s powerful features and the encoder-decoder architecture, the model can learn meaningful latent representations and generate interpretable topics from a collection of documents.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
import torch
import torch.nn as nn
import torch.nn.functional as F


class NVDM(nn.Module):
    def __init__(self, tokenizer, embed_dim, hidden_dim, latent_dim, dropout=0.5, num_layers=2, uniqueness_weight=0.0):
        super(NVDM, self).__init__()
        self.tokenizer = tokenizer
        self.vocab_size = len(tokenizer.vocabulary())
        self.embed_dim = embed_dim
        self.hidden_dim = hidden_dim
        self.latent_dim = latent_dim

        self.uniqueness_weight = uniqueness_weight
        # Encoder MLP
        encoder_layers = [
            nn.Linear(self.vocab_size, embed_dim),
            nn.BatchNorm1d(embed_dim),
            nn.LeakyReLU(),
            nn.Dropout(p=dropout),
            nn.Linear(embed_dim, hidden_dim),
            nn.BatchNorm1d(hidden_dim),
            nn.LeakyReLU(),
            nn.Dropout(p=dropout)
            ]
        
        for _ in range(num_layers):
            encoder_layers.append(nn.Linear(hidden_dim, hidden_dim))
            encoder_layers.append(nn.BatchNorm1d(hidden_dim))
            encoder_layers.append(nn.LeakyReLU())

        self.encoder = nn.Sequential(
            *encoder_layers
        )

        # intermediate representation
        self.mu = nn.Linear(hidden_dim, latent_dim)
        self.log_var = nn.Linear(hidden_dim, latent_dim)

        # Decoder MLP
        decoder_layers = [
            nn.Linear(latent_dim, hidden_dim),
            nn.BatchNorm1d(hidden_dim),
            nn.LeakyReLU()
        ]
        for _ in range(num_layers):
            decoder_layers.append(nn.Linear(hidden_dim, hidden_dim))
            decoder_layers.append(nn.BatchNorm1d(hidden_dim))
            decoder_layers.append(nn.LeakyReLU())
            decoder_layers.append(nn.Dropout(p=dropout))

        decoder_layers.append(nn.Linear(hidden_dim, latent_dim))
        self.decoder = nn.Sequential(*decoder_layers)

        self.dropout_layer = nn.Dropout(p=dropout)
        self.decoder_output = nn.Linear(latent_dim, embed_dim)
        
        self.output = nn.Linear(embed_dim, self.vocab_size)
        self.output.weight.data.normal_(0, 0.01)
        self.output.bias.data.fill_(0)

    def encode(self, x):
        h = self.encoder(x)
        mu = self.mu(h)
        log_var = self.log_var(h)
        return mu, log_var

    def reparameterize(self, mu, log_var):
        std = torch.exp(0.5 * log_var)
        eps = torch.randn_like(std)
        return mu + eps * std

    def decode(self, z):
        h = self.dropout_layer(self.decoder_output(z))
        logits = self.output(h)
        return logits
    
    def topic_word_distribution(self):
        word_embeddings = self.output.weight

        # Compute the topic-word scores
        topic_word_scores = torch.matmul(self.decoder_output.weight.t(), word_embeddings.t())
        topic_word_probabilities = torch.softmax(topic_word_scores, dim=1)

        return topic_word_probabilities
        
    def topic_words(self, top_k=10):
        topic_word_distributions = self.topic_word_distribution()

        # Get the top k words for each topic
        top_probs, top_indices = torch.topk(topic_word_distributions, k=top_k, dim=1)

        topic_words = [self.tokenizer.decode(indices.tolist()) for indices in top_indices]
        topic_probabilities = [probs.tolist() for probs in top_probs]
        return topic_words, topic_probabilities
    
    
    def topic_uniqueness_loss(self, batch_word_probabilities):
        # Normalize word probabilities to get document distributions
        document_distributions = F.normalize(batch_word_probabilities, p=1, dim=1)

        # Calculate the topic-word distribution matrix
        topic_word_dist = self.topic_word_distribution().t()

        # Calculate the document-topic distribution matrix
        document_topic_dist = torch.matmul(document_distributions, topic_word_dist)

        # Calculate the topic co-occurrence matrix
        topic_cooccurrence_matrix = torch.matmul(document_topic_dist.t(), document_topic_dist)

        # Normalize the topic co-occurrence matrix
        topic_cooccurrence_matrix = F.normalize(topic_cooccurrence_matrix, p=1, dim=1)

        # Calculate the average pairwise similarity between topics
        num_topics = topic_cooccurrence_matrix.shape[0]
        topic_similarity = torch.sum(topic_cooccurrence_matrix) - num_topics
        avg_topic_similarity = topic_similarity / (num_topics * (num_topics - 1))

        return avg_topic_similarity

    def kullback_liebler_loss(self, x, logits):
        # Normalize the document representations to get probability distributions
        original_dist = F.softmax(x, dim=1)
        reconstructed_dist = F.softmax(logits, dim=1)
        
        # Calculate KL divergence loss
        return F.kl_div(torch.log(reconstructed_dist), original_dist, reduction='batchmean')
    

    def reconstruction_loss(self, x, logits):
        # Calculate reconstruction/log-likehihood loss
        probs = F.log_softmax(logits, dim=1)
        batch_size = x.shape[0]
        return -torch.sum(x * probs) / batch_size

    def forward(self, documents):
        x = torch.stack([self.tokenizer.encode(document) for document in documents])
        mu, log_var = self.encode(x)
        z = self.reparameterize(mu, log_var)
        logits = self.decode(z)
        recon_loss = self.reconstruction_loss(x, logits)
        kl_loss = self.kullback_liebler_loss(x, logits)

        topic_uniqueness_loss = 0.0
        if self.uniqueness_weight:
            batch_word_probabilities = F.softmax(logits, dim=1)
            uniqueness_loss = self.topic_uniqueness_loss(batch_word_probabilities)
            #print('uniqueness_loss: ', self.uniqueness_weight, ' * ', uniqueness_loss, flush=True)
            topic_uniqueness_loss = self.uniqueness_weight * uniqueness_loss
        
        return recon_loss + kl_loss + topic_uniqueness_loss

The constructor takes various parameters such as the tokenizer, embedding dimension, hidden dimension, latent dimension, dropout rate, and number of layers. Both the encoder and decoder are implemented as multi-layer perceptrons (MLPs) using PyTorch’s nn.Sequential module. The encoder MLP maps the input document representation to a hidden representation and then to the parameters (mean and log-variance) of the variational posterior distribution. The decoder MLP takes a sample from the variational posterior and reconstructs the original document representation.

The loss implementation includes three loss functions: reconstruction loss, KL divergence loss, and topic uniqueness loss.

• reconstruction_loss method computes the negative log-likelihood of the reconstructed document representation given the original document representation.
• kullback_liebler_loss method calculates the KL divergence between the original document distribution and the reconstructed document distribution.
• topic_uniqueness_loss method computes the average pairwise similarity between topics based on the topic co-occurrence matrix, encouraging the model to learn distinct topics.

In natural language processing (NLP) tasks, such as topic modeling with NVDMs, it is essential to represent text documents in a format that can be easily processed and understood by machine learning models. One common approach is to use the Bag-of-Words (BoW) representation.

The Bag-of-Words representation treats each document as a collection of words, disregarding the order and grammatical structure of the words in the document. It represents a document as a vector, where each element corresponds to a unique word in the vocabulary, and the value of the element indicates the presence or frequency of that word in the document.

Here are a few reasons why the Bag-of-Words representation is necessary:

• Simplicity: BoW is a simple and intuitive way to represent text documents. It captures the essential information about the presence and frequency of words in a document, which is often sufficient for many NLP tasks, including topic modeling. Fixed-length representation: BoW converts variable-length text documents into fixed-length numerical vectors. This is crucial for machine learning models, which typically require input data to have a consistent shape and size.
• Computationally efficient: BoW representation is computationally efficient to generate and process. It allows for fast training and inference of models, especially when dealing with large datasets.
• Compatibility with machine learning algorithms: Many machine learning algorithms, including NVDMs, are designed to work with numerical data. The BoW representation converts text documents into numerical vectors, making them compatible with these algorithms.

However, it’s important to note that the BoW representation has some limitations. It ignores the order and context of words, which can lead to the loss of semantic information. More advanced techniques, such as word embeddings (e.g., Word2Vec, GloVe), can capture semantic relationships between words and provide more meaningful representations.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
import torch
from torch.utils.data import Dataset
from collections import defaultdict, Counter
import re
import csv


class CSVDataset(Dataset):
    def __init__(self, path, text_field, label_field, load=False, min_df=0, max_df=float('inf'), min_tf=0, max_tf=float('inf')):
        self.path = path
        self.text_field = text_field
        self.label_field = label_field
        self.data = []

        self.min_tf = min_tf
        self.max_tf = max_tf
        if load:
            with open(self.path, 'r') as input_file:
                    self.data = list(csv.DictReader(input_file))
        
        self.term_frequencies, self.total_tokens, self.document_frequencies, self.total_documents = self.process_corpus(path, text_field)

        self.min_df = min_df
        if isinstance(min_df, float) and 0.0 <= min_df < 1.0:
            self.min_df = min_df * self.total_documents

        self.max_df = max_df
        if isinstance(max_df, float) and 0.0 <= max_df < 1.0:
            self.max_df = max_df * self.total_documents

    def process_corpus(self, path, field):
        term_frequencies = Counter()
        document_frequencies = Counter()
        total_documents = 0
        total_tokens = 0
        with open(path, 'r') as input_file:
            total_tokens = 0
            for row in csv.DictReader(input_file):
                doc = row[field]
                tokens = doc.split()
                term_frequencies.update(tokens)
                document_frequencies.update(set(tokens))
                total_tokens += len(tokens)
                total_documents += 1
        return term_frequencies, total_tokens, document_frequencies, total_documents
    
    def is_valid(self, token):
        return self.document_frequencies[token] >= self.min_df and self.document_frequencies[token] <= self.max_df and self.term_frequencies[token] >= self.min_tf and self.term_frequencies[token] <= self.max_tf

    def __len__(self):
        if self.data:
            return len(self.data)

        with open(self.path, 'r') as input_file:
            for index, row in enumerate(csv.reader(input_file)):
                pass
        return index + 1

    def __getitem__(self, idx):
        if self.data:
            text = self.data[idx][self.text_field]
            label = self.data[idx][self.label_field]
        else:
            text = None
            with open(self.path, 'r') as input_file:
                for file_idx, row in enumerate(csv.DictReader(input_file)):
                    if file_idx == idx:
                        text = row[self.text_field]
                        label = row[self.label_field]
                        break
        return text, label

Advantages of the CSVDataset implementation:

• Flexibility in data loading: The CSVDataset class allows for flexible loading of data from a CSV file. It supports loading the entire dataset into memory at once (load=True) or loading data on-the-fly as needed. This flexibility enables efficient memory usage, especially when dealing with large datasets.
• Customizable text and label fields: The CSVDataset class takes text_field and label_field parameters, allowing you to specify the column names in the CSV file that correspond to the text data and labels, respectively. This makes it easy to adapt the dataset to different CSV formats and column names.
• Preprocessing and filtering: The CSVDataset class provides functionality for preprocessing and filtering the text data based on term frequencies and document frequencies. The process_corpus method computes the term frequencies, document frequencies, and total token counts for the entire corpus. The is_valid method allows for filtering tokens based on the specified thresholds.

Advantages of the min_df, max_df, min_tf, and max_tf parameters:

• Filtering rare and common words: The min_df and max_df parameters allow for filtering words based on their document frequency. By setting min_df, you can exclude words that appear in very few documents, which are often less informative. Similarly, by setting max_df, you can exclude words that appear in a large portion of the documents, as they may be too common and not discriminative.
• Filtering low and high frequency words: The min_tf and max_tf parameters enable filtering words based on their term frequency. By setting min_tf, you can exclude words that occur very rarely in the corpus, which may be noise or less relevant. On the other hand, by setting max_tf, you can exclude words that occur too frequently and may not provide much discriminative power.
• Vocabulary control: By applying these filtering parameters, you can control the size and quality of the vocabulary used for the BoW representation. This helps in reducing the dimensionality of the feature space and focusing on the most informative words for the given task.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
class TokenizedDataset(CSVDataset):

    def __getitem__(self, idx):
        text, label = super().__getitem__(idx)
        text = [token for token in text.split() if self.is_valid(token)]
        return ' '.join(text), label


class BowTokenizer:
    UNKNOWN_TOKEN = '<UNK>'

    def __init__(self, vocab_size=None, stopwords=None):
        self.vocab_size = vocab_size
        self.word2idx = None
        self.idx2word = None
        self.vocab = None
        self.stopwords = stopwords
        
    def fit(self, texts):
        # Tokenize the texts and build the vocabulary
        word_counts = Counter()
        for text in texts:
            words = self.tokenize(text)
            word_counts.update(words)

        # Sort the words by frequency and select the top vocab_size words
        vocab_size = self.vocab_size
        if not vocab_size:
            vocab_size = len(word_counts)
        
        self.vocab = [word for word, _ in word_counts.most_common(vocab_size)]
        
        # Create word-to-index and index-to-word mappings
        word2idx = {word: idx for idx, word in enumerate(self.vocab)}

        self.word2idx = word2idx
        self.idx2word = dict(zip(word2idx.values(), word2idx.keys()))
    
    def vocabulary(self):
        return list(self.vocab)
        
    def tokenize(self, text):
        # Tokenize the text into words
        words = re.findall(r'\b\w+\b', text.lower())
        return words
    
    def encode(self, text):
        # Tokenize the text and convert words to indices
        words = self.tokenize(text)
        word_counts = Counter(words)

        vocab_size = len(self.vocab)
        data = torch.zeros(vocab_size, dtype=torch.float)
        for word, count in word_counts.items():
            if word not in self.word2idx:
                continue
            word_index = self.word2idx[word]
            data[word_index] = count
        return data
    
    def decode(self, input_ids):
        # Convert token indices to
        words = [self.idx2word.get(idx, self.UNKNOWN_TOKEN) for idx in input_ids]
        return ' '.join(words)

Advantages of the BowTokenizer implementation:

• Vocabulary building: The BowTokenizer class provides functionality for building a vocabulary from the text data. It tokenizes the texts, counts the word frequencies, and selects the top vocab_size words to form the vocabulary. This allows for creating a compact and representative vocabulary for the BoW representation.
• Encoding and decoding: The BowTokenizer class offers methods for encoding text into BoW vectors and decoding BoW vectors back into text. The encode method tokenizes the text, maps the words to their corresponding indices in the vocabulary, and returns a vector representation. The decode method converts the BoW vector back into text by mapping the indices to their corresponding words.
• Handling unknown words: The BowTokenizer class handles unknown words (words not present in the vocabulary) by mapping them to a special <UNK> token. This ensures that the encoding process can handle out-of-vocabulary words gracefully.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
import torch
import torch.optim as optim
from torch.utils.data import DataLoader
import utils
from nvdm import NVDM

import csv


def train(model, dataloader, optimizer, device):
    model.train()
    total_loss = 0
    batch_index = -1
    for documents, labels in dataloader:
        batch_index += 1
        optimizer.zero_grad()
        loss = model(documents)
        loss.backward()

        # Clip gradients
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

        optimizer.step()
        if batch_index % 100 == 0:
            print(f'Loss #{batch_index}: ', loss.item(), flush=True)
        total_loss += loss.item()
    return total_loss / len(dataloader)


if __name__ == '__main__':
    import argparse
    parser = argparse.ArgumentParser(description='NVDM Training')

    # NVDM configuration parameters
    nvdm_config = parser.add_argument_group('NVDM Configuration')
    nvdm_config.add_argument('--vocab_size', type=int, default=10000, help='Number of topics')
    nvdm_config.add_argument('--topics', type=int, default=30, help='Number of topics')
    nvdm_config.add_argument('--alpha', type=float, default=1.0, help='Alpha parameter')
    nvdm_config.add_argument('--hidden_size', type=int, default=256, help='Hidden size')
    nvdm_config.add_argument('--embedding_size', type=int, default=100, help='Embedding size')
    nvdm_config.add_argument('--dropout', type=float, default=0.5, help='Dropout rate')
    nvdm_config.add_argument('--num_layers', type=int, default=4, help='Number of layers')
    nvdm_config.add_argument('--num_encoder_heads', type=int, default=12, help='Number of attention heads in the encoding step')
    nvdm_config.add_argument('--num_decoder_heads', type=int, default=3, help='Number of attention heads in the decoding step')
    nvdm_config.add_argument('--temperature', type=float, default=1.0, help='Temperature for topic mixture')

    # data parametersargs.path
    data_config = parser.add_argument_group('Preprocess Function Parameters')
    data_config.add_argument('--path', type=str, default='/samples/corpus.csv', help='CSV path to load data from')
    data_config.add_argument('--text', type=str, default='content', help='CSV field containing the text')
    data_config.add_argument('--label', type=str, default='content', help='CSV field containing text label')
    data_config.add_argument('--stream', action='store_true', help='Stream data directly from the file instead of loading it into memory')

    # preprocess parameters
    preprocess_config = parser.add_argument_group('Preprocess Function Parameters')
    preprocess_config.add_argument('--mindf', type=float, default=50, help='Minimum document frequency')
    preprocess_config.add_argument('--maxdf', type=float, default=float('inf'), help='Maximum document frequency')
    preprocess_config.add_argument('--mintf', type=float, default=0, help='Minimum term frequency')
    preprocess_config.add_argument('--maxtf', type=float, default=float('inf'), help='Maximum document frequency')

    # Loss function parameters
    loss_config = parser.add_argument_group('Loss Function Parameters')
    loss_config.add_argument('--reconstruction_weight', type=float, default=1, help='Reconstruction loss weight')
    loss_config.add_argument('--kl_weight', type=float, default=0.1, help='KL divergence loss weight')
    loss_config.add_argument('--sparsity_weight', type=float, default=1.0, help='Sparsity loss weight')
    loss_config.add_argument('--uniqueness_weight', type=float, default=1.0, help='Uniqueness loss weight')

    # Training parameters
    training_config = parser.add_argument_group('Training Parameters')
    training_config.add_argument('--epochs', type=int, default=3, help='Number of epochs')
    training_config.add_argument('--batch_size', type=int, default=256, help='Batch size')
    training_config.add_argument('--learning_rate', type=float, default=0.01, help='Learning rate')
    training_config.add_argument('--debug', action='store_true', help='Enable debug mode')

    args = parser.parse_args()
    
    # Device setup
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

    # Define the dataset and data loader
    load = True
    if args.stream:
        load=False

    dataset = utils.TokenizedDataset(path=args.path, text_field=args.text, label_field=args.label, load=load, min_df=args.mindf, max_df=args.maxdf, min_tf=args.mintf, max_tf=args.maxtf)
    dataloader = DataLoader(dataset, batch_size=args.batch_size, shuffle=True, drop_last=True)

    tokenizer = utils.BowTokenizer(vocab_size=args.vocab_size)
     # Load and preprocess the dataset
    with open(args.path, 'r') as input_file:
        documents = [row[args.text] for row in csv.DictReader(input_file)]
        tokenizer.fit(documents)

    # Assume we have a list 'documents' containing the input text documents
    print('preprocessing data', flush=True)

    # Create the NVDM model
    model = NVDM(tokenizer, args.embedding_size, args.hidden_size, args.topics, args.dropout, uniqueness_weight=args.uniqueness_weight).to(device)

    # Optimizer
    optimizer = optim.Adam(model.parameters(), lr=args.learning_rate)

    # Training loop
    print('training model', flush=True)
    for epoch in range(args.epochs):
        train_loss = train(model, dataloader, optimizer, device)
        print(f'Epoch [{epoch+1}/{args.epochs}], Loss: {train_loss:.4f}')

        topic_words, topic_probabilities = model.topic_words()
        for index, (words, probabilities) in enumerate(zip(topic_words, topic_probabilities)):
            print(f'\nTopic #{index + 1} Words: ', words, flush=True)
            print('Topic Probabilities:', probabilities, flush=True)

This implementation uses argparse to allow developers to configure the architecture, computational complexity and supported vocbaulary from the terminal, aong with the file/dataset the NVDM is trained on. The NVDM consistently produces high quality topics, even without topic coherence loss.

Topic #1 Words:  national_junior triple_jump wind_assist european_indoor_champion hurdles heptathlon organiser_calendar_doha_qat world_junior national_record_holder sprint_double
Topic Probabilities: [0.0005295022274367511, 0.000521067064255476, 0.0004984174738638103, 0.0004961825325153768, 0.0004958845092914999, 0.00047388693201355636, 0.0004721697769127786, 0.000470947299618274, 0.00046596929314546287, 0.00046446104533970356]

Topic #2 Words:  kilometre_reach tanui start_push_pace imane_merga kiprono pass_km_km athl_te splits kidane kamworor
Topic Probabilities: [0.0001887115795398131, 0.00016992191376630217, 0.00016913456784095615, 0.00016887785750441253, 0.00016846574726514518, 0.00016821929602883756, 0.00016521599900443107, 0.00016431287804152817, 0.0001635944499867037, 0.00016298888658639044]

Topic #3 Words:  fraser_pryce kaniskina maurice_greene bolt world_champion_ato farah greene gatlin korzeniowski tyson_gay
Topic Probabilities: [0.0002071216731565073, 0.00019888173846993595, 0.00019405571219976991, 0.00019289054034743458, 0.0001920316572068259, 0.00018928298959508538, 0.00018790591275319457, 0.00018012983491644263, 0.00017984140140470117, 0.000177944268216379]

Topic #4 Words:  fourth_overall nsw dobrynska ennis score combined_event effort decathlon kasyanov heptathlon
Topic Probabilities: [0.0001549244625493884, 0.00015470129437744617, 0.0001546467246953398, 0.00015251476725097746, 0.00015251018339768052, 0.00015153890126384795, 0.00014963878493290395, 0.00014948415628168732, 0.00014858909707982093, 0.00014747283421456814]

Topic #5 Words:  stage_km stage_km_stage km_stage pass_km_mark pass_kilometre se anti_dope_programme sato hold_steady tsegay
Topic Probabilities: [0.0002340153732802719, 0.00019602605607360601, 0.00019109177810605615, 0.00018873649241868407, 0.0001754782279022038, 0.00017319770995527506, 0.00017031136667355895, 0.00016833780682645738, 0.00016813860565889627, 0.00016755572869442403]

Topic #6 Words:  razorback_track_field tahoma_panose_font_font family_arial_narrow_panose arkansas_athletic_visit nixon color_windowtext_span_balloontextchar div_msonormal_margin_top font_font_family_itc stone_serif_std_medium tahoma_san_serif_msochpdefault
Topic Probabilities: [0.00020097965898457915, 0.00019553006859496236, 0.0001942943490575999, 0.00019251251069363207, 0.00019250520563218743, 0.0001921903167385608, 0.00019170019368175417, 0.0001908671110868454, 0.00019030227849725634, 0.0001891691208584234]

Topic #7 Words:  ncaas hellebaut yeah bergqvist pass_height xc harting anti_dope_programme low_stick swede
Topic Probabilities: [0.00020984900766052306, 0.00020819688506890088, 0.00020724012574646622, 0.00020646779739763588, 0.00020583244622685015, 0.00020467313879635185, 0.00020236226555425674, 0.0001983269612537697, 0.00019566438277252018, 0.00019545013492461294]

Topic #8 Words:  bend_knee plank set_rep abs knee posture push_harder main_protagonist dumbbell upper_body
Topic Probabilities: [0.00013974822650197893, 0.00013468462566379458, 0.00013362171011976898, 0.00013106761616654694, 0.00013051718997303396, 0.0001305115583818406, 0.00013051058340352029, 0.00013008633686695248, 0.00012852296640630811, 0.00012837933900300413]

Topic #9 Words:  datum_addition_shoe_weight stride_flexibility_forefoot_account usage_battery_mechanical_test lab_provide_objective_exclusive measure_sole_thickness_sit foot_road_foam_cushion review_shoe michigan_shoe_real_world pennsylvania_shoe_real_world tester_shoe
Topic Probabilities: [0.00029314172570593655, 0.00029293980333022773, 0.0002928385802078992, 0.000292605662252754, 0.00029258636641316116, 0.0002924941072706133, 0.00029232309316284955, 0.0002862800029106438, 0.00027762691024690866, 0.00026143176364712417]

Topic #10 Words:  sauce kasyanov round_foul wlodarczyk freimuth overnight_leader dressing round_throw pepper topping
Topic Probabilities: [0.00032631121575832367, 0.0003227636043448001, 0.0003218145575374365, 0.00032164782169274986, 0.00031468141241930425, 0.0003138505853712559, 0.0003078986192122102, 0.0003074088890571147, 0.0003049601800739765, 0.0002995755639858544]

Topic #11 Words:  bell_lap closing_lap jebet atapuerca foam_midsole stable_ride medial_post faith_kipyegon heath dibaba
Topic Probabilities: [0.00017135417147073895, 0.00016612840408924967, 0.000163451477419585, 0.00015913633978925645, 0.00015847643953748047, 0.00015797841479070485, 0.0001568600710015744, 0.00015681587683502585, 0.0001563854602864012, 0.00015422738215420395]

Topic #12 Words:  arron greene robles maurice_greene roble merritt wariner doucour powell kallur
Topic Probabilities: [0.0005106705939397216, 0.0005042121629230678, 0.0004950464935973287, 0.00046031526289880276, 0.0004537209460977465, 0.0004504402750171721, 0.00044670907664112747, 0.0004097057390026748, 0.0004081916413269937, 0.000406078586820513]

Topic #13 Words:  platform feature promote joint fitting argentina tech bra flexibility runners
Topic Probabilities: [0.00017971890338230878, 0.0001771343086147681, 0.0001728379720589146, 0.0001703346788417548, 0.000169746796018444, 0.00016817505820654333, 0.00016802331083454192, 0.00016549485735595226, 0.0001650949561735615, 0.00016503891674801707]

Topic #14 Words:  field_stay_tune complete_coverage_iu_track iu immediate_release_track_field federations clinic program nominee kerala indoor_track_field
Topic Probabilities: [0.00015527514915447682, 0.0001551182213006541, 0.00015354147762991488, 0.00014967596507631242, 0.0001477991754654795, 0.0001419319596607238, 0.00014136313984636217, 0.00014117403770796955, 0.00014114829536993057, 0.00014088221359997988]

Topic #15 Words:  km race_walk_challenge chihuahua km_loop ekiden km_km women_km stage_km km_stage world_marathon_champion
Topic Probabilities: [0.00045582986786030233, 0.00042629288509488106, 0.00041378289461135864, 0.00040664817788638175, 0.0003824761079158634, 0.000382121535949409, 0.000376749288989231, 0.00037533120485022664, 0.00037033201078884304, 0.00036467579775489867]

Topic #16 Words:  training_program tempo_pace hill_sprint key_workout training_schedule tempo_run build_mileage specific_workout energy_system speed_workout
Topic Probabilities: [0.00015735242050141096, 0.0001552267640363425, 0.00015487417113035917, 0.0001523850078228861, 0.00015200886991806328, 0.00015089314547367394, 0.00015057215932756662, 0.00015039686695672572, 0.00014915651991032064, 0.00014892316539771855]

Topic #17 Words:  road_foam_cushion_stride addition_shoe_weight_measure flexibility_forefoot_account_review sole_thickness_sit_foot shoe_real_world_usage provide_objective_exclusive_datum battery_mechanical_test_lab brightcove_id_video michigan_shoe_real_world review_shoe
Topic Probabilities: [0.0003328978782519698, 0.00033284176606684923, 0.00033272075233981013, 0.0003327052399981767, 0.0003280891396570951, 0.0003277743235230446, 0.0003274726332165301, 0.0003065402852371335, 0.00029554389766417444, 0.00029002776136621833]

Topic #18 Words:  true_freshman iu wartburg uw_oshkosh uw_la_crosse ncaa_meet ducks johns_hopkins lsu nyu
Topic Probabilities: [0.0005267398082651198, 0.0004757751594297588, 0.0004727425111923367, 0.00046986431698314846, 0.0004570367746055126, 0.0004565916024148464, 0.0004489163984544575, 0.0004444819933269173, 0.0004420706245582551, 0.0004363793705124408]

Topic #19 Words:  tzis score overnight_leader total_score chernova overnight_lead pts fajdek gotzis snug_fit
Topic Probabilities: [0.00024404078430961818, 0.0002303147193742916, 0.00022389950754586607, 0.00022108246048446745, 0.00021435876260511577, 0.0002082447026623413, 0.0002077820390695706, 0.00020295534341130406, 0.00020166372996754944, 0.00019990364671684802]

Topic #20 Words:  kipruto automatic_qualifying_spot kejelcha singletrack ayana koech kipyegon reach_kilometre kilometre_remain deba
Topic Probabilities: [0.0004376643046271056, 0.00041711798985488713, 0.0003987165109720081, 0.0003969542740378529, 0.0003965102368965745, 0.0003922139585483819, 0.0003744940913747996, 0.0003742391418199986, 0.00036667476524598897, 0.00035903151729144156]

Topic #21 Words:  div_msonormal_margin_margin msohyperlink_mso_style_priority style_priority_color_purple underline_visited_span_mso san_serif_link_span pt_font_family_calibri color_blue_text_decoration family_cambria_panose_font math_panose_font_font font_family_calibri_panose
Topic Probabilities: [0.0003982365597039461, 0.0003865780890919268, 0.00038599688559770584, 0.0003858261334244162, 0.00038576123188249767, 0.0003767758025787771, 0.0003719555970747024, 0.0003674782929010689, 0.0003656522312667221, 0.00036364945117384195]

Topic #22 Words:  training_program run_economy elite_runner program pennsylvania improve lactate consistent cross_training downhill
Topic Probabilities: [0.00014887095312587917, 0.00014796634786762297, 0.00014657336578238755, 0.00014462492254097015, 0.0001440627092961222, 0.00014293886488303542, 0.00014185230247676373, 0.0001412749697919935, 0.00014071576879359782, 0.0001404146896675229]

Topic #23 Words:  stage_km_stage fukuoka improved associated chiba km_stage live_japan increase muscle_fatigue function
Topic Probabilities: [0.0001788241497706622, 0.0001709579664748162, 0.00016188611334655434, 0.0001599854585947469, 0.00015961381723172963, 0.00015759470988996327, 0.00015517046267632395, 0.00015402464487124234, 0.00015372017514891922, 0.00015293341130018234]

Topic #24 Words:  grill pepper cook_minute garlic stir avocado sauce olive_oil syrup slice
Topic Probabilities: [0.00029838853515684605, 0.0002721975324675441, 0.00026485041598789394, 0.0002625574416015297, 0.0002616708807181567, 0.0002607014321256429, 0.00025924600777216256, 0.00025659558014012873, 0.00024157052394002676, 0.0002401871606707573]

Topic #25 Words:  eaton hardee follow_page trey_hardee brightcove_id claye sebrle ashton_eaton bayer attempt_clearance
Topic Probabilities: [0.00029643013840541244, 0.0002523841103538871, 0.00023353486903943121, 0.00023333514400292188, 0.00023249629884958267, 0.00023083601263351738, 0.00023046253772918135, 0.0002253647253382951, 0.00022452250414062291, 0.0002195353154093027]

Topic #26 Words:  bq vail pre_nats postseason cliff flagstaff deer west_regional hardrock dirt_road
Topic Probabilities: [0.0002408408181509003, 0.00023494136985391378, 0.00023437567870132625, 0.00022776515106670558, 0.00021635669691022485, 0.00021620663756038994, 0.00021123082842677832, 0.0002079781552311033, 0.0002075362717732787, 0.00020752135606016964]

Topic #27 Words:  ncaas span_amp_amp amp_amp_amp_lt gt_amp_amp_amp redactor_selection_marker_data amp_amp_lt_span verified_redactor_amp_amp marker_data_verified_redactor amp_amp_amp_gt marker_class_redactor_selection
Topic Probabilities: [0.0005251255352050066, 0.0005159128922969103, 0.0005129625787958503, 0.0005118255503475666, 0.0005064702709205449, 0.0004995728377252817, 0.0004968844004906714, 0.0004857392341364175, 0.00047573723713867366, 0.0004749214858748019]

Topic #28 Words:  allentown_shoe_real_world lab_test measure_sole_thickness_sit usage_battery_mechanical_test stride_flexibility_forefoot_account lab_provide_objective_exclusive datum_addition_shoe_weight foot_road_foam_cushion athl_te review_shoe
Topic Probabilities: [0.00015008448099251837, 0.00014843278040643781, 0.00014576577814295888, 0.00014571099018212408, 0.00014551033382304013, 0.00014550470223184675, 0.00014537919196300209, 0.00014508143067359924, 0.00014476204523816705, 0.0001446998940082267]

Topic #29 Words:  pace pace_slow hill_sprint mile_tempo_run run_faster lagat rupp training_cycle pacing hill
Topic Probabilities: [0.0004469976993277669, 0.000368804088793695, 0.00036803274997510016, 0.0003578069154173136, 0.0003511097456794232, 0.00034814863465726376, 0.0003470892843324691, 0.0003425931208766997, 0.0003401785215828568, 0.0003377698012627661]

Topic #30 Words:  closing_lap water_jump backstretch cherono houlihan berian ayana sowinski kipruto kilometre_reach
Topic Probabilities: [0.000206675409572199, 0.0001992114121094346, 0.00019332184456288815, 0.00019155119662173092, 0.00018596377049107105, 0.00018261026707477868, 0.00018221103528048843, 0.00018160931358579546, 0.00017999696137849241, 0.00017984103760682046]

References🔗

1. “Neural Variational Inference for Text Processing” by Yishu Miao, Lei Yu, and Phil Blunsom (2016)
2. “Discovering Discrete Latent Topics with Neural Variational Inference” by Yishu Miao, Edward Grefenstette, and Phil Blunsom (2017)
3. “Topic Modeling with Wasserstein Autoencoders” by Feng Nan, Ran Ding, Ramesh Nallapati, and Bing Xiang (2019)
4. “A Primer on Neural Network Models for Natural Language Processing” by Yoav Goldberg (2016)
5. “Autoencoding Variational Bayes” by Diederik P. Kingma and Max Welling (2014)

Github Projects🔗

1. PyTorch Examples: Text Generation with VAEs
2. Neural Variational Document Model (NVDM) Implementation
3. Variational Autoencoder for Topic Modeling
4. PyTorch-NLP: Natural Language Processing with PyTorch

In the world of image processing and computer vision, extracting dominant colors from an image has become an increasingly important task. Whether you’re a data-driven designer looking to create visually appealing color palettes or a machine learning enthusiast exploring the intricacies of unsupervised learning, understanding how to accurately identify and quantify the most prominent colors in an image is a valuable skill. In this blog post, we’ll dive into the fascinating realm of dominant color extraction using the powerful combination of X-Means clustering and the CIE2000 color distance metric.

The X-Means clustering algorithm, an extension of the well-known K-Means algorithm, has gained significant attention in recent years due to its ability to automatically determine the optimal number of clusters in a dataset. By leveraging the principles of unsupervised machine learning, X-Means allows us to group similar colors together without requiring prior knowledge of the number of dominant colors present in an image. This adaptability makes X-Means a valuable tool for color quantization and palette generation tasks.

However, the effectiveness of any color clustering algorithm heavily relies on the choice of color distance metric. Traditional color spaces, such as RGB or HSV, often fail to accurately represent the perceptual differences between colors as perceived by the human eye. Developed by the International Commission on Illumination (CIE), CIE2000 takes into account the intricacies of human color perception, ensuring that the clustering process yields results that are visually meaningful and intuitive. By combining the power of X-Means clustering with the perceptual accuracy of CIE2000, we can unlock new possibilities in dominant color extraction, and image quantization.

Understanding X-Means Clustering🔗

To understand X-Means clustering we need to start with K-Means clustering. K-Means is a widely used unsupervised learning algorithm for clustering data points. It partitions a given dataset into a specified number of clusters (K) based on the similarity between data points. The algorithm follows these steps:

1. Initialization: Randomly select K data points as initial cluster centroids.
2. Assignment: Assign each data point to the nearest centroid based on a distance metric (e.g., Euclidean distance).
3. Update: Recalculate the centroids of each cluster by taking the mean of all data points assigned to that cluster.
4. Iteration: Repeat steps 2 and 3 until the centroids no longer change significantly or a maximum number of iterations is reached.

The goal of K-Means is to minimize the sum of squared distances between each data point and its assigned centroid, resulting in compact and well-separated clusters.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
class KMeans {
    /**
     * A configurable implementation of the K-means clustering algorithm
     * 
     * 
     * @param {int} minK 
     * @param {Function} distanceMetric 
     * @param {object} options
     */

    constructor(minK, distanceMetric, options) {
        this.minK = minK;
        this.distanceMetric = distanceMetric;

        var options = options || {};
        this.distanceThreshold = options.distanceThreshold || 0.0; // minimum threshold of centroid changes to continue iterations
        this.maxIterations = options.maxIterations || Number.MAX_SAFE_INTEGER;
        this.meanFunc = options.meanFunc || this._arithmeticMean;
        this.kDTree = options.kDTree || false;
    }

    validateDataset(dataset) {
        if (!Array.isArray(dataset) || !dataset.length) {
            throw Error('dataset must be array');
        }

        if (dataset.length <= this.minK) {
            throw Error('dataset must have at least ' + this.minK + ' data points');
        }

        for (var i = 0; i < dataset.length; i++) {
            if (!Array.isArray(dataset[i])) {
                console.log(dataset[i]);
                throw Error('dataset points must be an array');
            }

            if (!dataset[i].length) {
                throw Error('dataset points must be a non-empty array');
            }
        }

        return true;
    }

    transform(dataset, options) {
        /**
         * Executes the k-means clustering algorithm against a dataset
         * with the given optional optimization parameters.  Parameters are
         * 
         * - validate (bool):  Indicates if the dataset should be validated
         * - kDTree (bool):  Indicates if a KD-tree should be used to find nearest centroids
         * 
         * @param {Array} dataset
         * @param {object} options
         */
        var options = options || {};
        options.kDTree = options.kDTree || this.kDTree;
        if (options.validate || true) {
            this.validateDataset(dataset);
        }

        let iterations = 0;
        let oldCentroids, labels, centroids;

        // Initialize centroids randomly
        if (options.useNaiveSharding || true) {
            centroids = this._getRandomCentroidsNaiveSharding(dataset);
        } else {
            centroids = this._getRandomCentroids(dataset);
        }

        // Run the main k-means algorithm
        while (!this._shouldStop(oldCentroids, centroids, iterations)) {
            // Save old centroids for convergence test.
            oldCentroids = [...centroids];
            iterations++;

            // Assign labels to each datapoint based on centroids
            labels = this._getLabels(dataset, centroids, options.kDTree);
            centroids = this._recalculateCentroids(dataset, labels);
        }

        const clusters = [];
        for (let i = 0; i < this.minK; i++) {
            clusters.push(labels[i]);
        }
        const results = {
            clusters: clusters,
            centroids: centroids,
            iterations: iterations,
            converged: iterations <= this.maxIterations,
        };
        return results;
    }

    _getLabels(dataSet, centroids, kDTree = true) {
        // prep data structure:
        const labels = {},
            centroidIndices = [];
        for (let c = 0; c < centroids.length; c++) {
            labels[c] = {
                points: [],
                centroid: centroids[c],
            };
            centroidIndices.push(c);
        }
        var tree;
        if (kDTree) {
            var centroidDimensions = centroids[0].map(function(v, i) {
                return i
            });
            tree = new KDTree(centroids, centroidIndices, centroidDimensions, this.distanceMetric);
        }
        // For each element in the dataset, choose the closest centroid. 
        // Make that centroid the element's label.
        for (let i = 0; i < dataSet.length; i++) {
            const a = dataSet[i];
            let closestCentroid, closestCentroidIndex
            if (kDTree) {
                var closestCentroids = tree.nearestNeighbor(a, 1);
                closestCentroid = closestCentroids[0],
                    closestCentroidIndex = closestCentroid.id;
            } else {
                let prevDistance;
                for (let j = 0; j < centroids.length; j++) {
                    let centroid = centroids[j];
                    if (j === 0) {
                        closestCentroid = centroid;
                        closestCentroidIndex = j;
                        prevDistance = this.distanceMetric(a, closestCentroid);
                    } else {
                        // get distance:
                        const distance = this.distanceMetric(a, centroid);
                        if (distance < prevDistance) {
                            prevDistance = distance;
                            closestCentroid = centroid;
                            closestCentroidIndex = j;
                        }
                    }
                }
            }
            // add point to centroid labels:
            labels[closestCentroidIndex].points.push(a);
        }
        return labels;
    }


    _compareDatasets(a, b) {
        for (let i = 0; i < a.length; i++) {
            if (a[i] !== b[i]) {
                return false;
            }
        }
        return true;
    }

    _getRandomCentroids(dataset) {
        // selects random points as centroids from the dataset
        const numSamples = dataset.length;
        const centroidsIndex = [];
        let index;
        while (centroidsIndex.length < this.minK) {
            index = this._randomBetween(0, numSamples);
            if (centroidsIndex.indexOf(index) === -1) {
                centroidsIndex.push(index);
            }
        }
        const centroids = [];
        for (let i = 0; i < centroidsIndex.length; i++) {
            const centroid = [...dataset[centroidsIndex[i]]];
            centroids.push(centroid);
        }
        return centroids;
    }

    _getRandomCentroidsNaiveSharding(dataset) {
        // implementation of a variation of naive sharding centroid initialization method
        // (not using sums or sorting, just dividing into k shards and calc mean)
        const numSamples = dataset.length;
        const step = Math.floor(numSamples / this.minK);
        const centroids = [];
        for (let i = 0; i < this.minK; i++) {
            const start = step * i;
            let end = step * (i + 1);
            if (i + 1 === this.minK) {
                end = numSamples;
            }
            centroids.push(this._calcMeanCentroid(dataset, start, end));
        }
        return centroids;
    }

    _calcMeanCentroid(dataSet, start, end) {
        const features = dataSet[0].length;
        const n = end - start;
        let mean = [];
        for (let i = 0; i < features; i++) {
            mean.push(0);
        }
        for (let i = start; i < end; i++) {
            for (let j = 0; j < features; j++) {
                mean[j] = mean[j] + dataSet[i][j] / n;
            }
        }
        return mean;
    }

    _shouldStop(oldCentroids, centroids, iterations) {
        if (iterations > this.maxIterations) {
            return true;
        }
        if (!oldCentroids || !oldCentroids.length) {
            return false;
        }

        if (oldCentroids.length === centroids.length) {
            let maxMovement = 0.0;
            for (let i = 0; i < centroids.length; i++) {
                const distance = this.distanceMetric(centroids[i], oldCentroids[i]);
                maxMovement = Math.max(maxMovement, distance);
            }

            if (maxMovement <= this.threshold) {
                return true;
            }

            let haveChanged = true;
            for (let i = 0; i < centroids.length; i++) {
                if (!this._compareDatasets(centroids[i], oldCentroids[i])) {
                    haveChanged = false;
                }
            }
            return haveChanged;
        }
        return false;
    }

    _recalculateCentroids(dataSet, labels) {
        // Each centroid is the arithmetic mean of the points that
        // have that centroid's label.  If no points have
        // a centroid's label, you should randomly re-initialize it.
        let newCentroid;
        const newCentroidList = [];
        for (const k in labels) {
            const centroidGroup = labels[k];
            if (centroidGroup.points.length > 0) {
                // find mean:
                newCentroid = this.meanFunc(centroidGroup.points);
            } else {
                // get new random centroid
                newCentroid = this._getRandomCentroids(dataSet, 1)[0];
            }
            newCentroidList.push(newCentroid);
        }
        return newCentroidList;
    }


    _arithmeticMean(dataset) {
        const totalPoints = dataset.length;
        const means = [];
        for (let j = 0; j < dataset[0].length; j++) {
            means.push(0);
        }
        for (let i = 0; i < dataset.length; i++) {
            const point = dataset[i];
            for (let j = 0; j < point.length; j++) {
                const val = point[j];
                means[j] = means[j] + val / totalPoints;
            }
        }
        return means;
    }

    _randomBetween(min, max) {
        return Math.floor(
            Math.random() * (max - min) + min
        );
    }
}

Despite its simplicity and effectiveness, K-Means has some limitations. One major drawback is that it requires the number of clusters (K) to be specified in advance. Determining the optimal value of K can be challenging, especially when dealing with complex datasets. If the chosen K does not align with the natural structure of the data, the resulting clusters may not be meaningful or representative. Moreover, K-Means is sensitive to the initial placement of centroids and may converge to suboptimal solutions. It can also struggle with clusters of different sizes, densities, or non-spherical shapes.

To address these limitations, an adaptive approach that can automatically determine the optimal number of clusters is desirable.

X-Means is an extension of the K-Means algorithm that aims to overcome the limitation of requiring a pre-specified number of clusters. It automatically determines the optimal number of clusters based on the structure of the data. The key idea behind X-Means is to start with a small number of clusters and iteratively split them into smaller clusters if doing so improves the clustering quality. The algorithm follows these steps:

1. Initialization: Start with a small number of clusters (e.g., K=1) and apply K-Means to the dataset.
2. Splitting: For each existing cluster, consider splitting it into two subclusters. Apply K-Means to each potential split and calculate a model selection criterion to assess the quality of the split.
3. Comparison: Compare the model selection criterion scores of the original cluster and the split subclusters. If the split results in a better score, accept the split and update the cluster assignments.
4. Iteration: Repeat steps 2 and 3 until no further splits improve the BIC score or a maximum number of iterations is reached.

By iteratively splitting clusters and evaluating the resulting models, X-Means can automatically determine the optimal number of clusters that best fit the data. This adaptive approach eliminates the need for manual tuning of the K parameter and allows the algorithm to discover the underlying structure of the dataset more effectively.

X-Means combines the simplicity and efficiency of K-Means with the flexibility of automatically determining the appropriate number of clusters. It provides a more robust and data-driven approach to clustering, making it a valuable tool in various domains such as customer segmentation, anomaly detection, and pattern recognition.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
class XMeans extends KMeans {
    /**
     * A configurable implementation of the X-means clustering algorithm.
     * The X-means clustering creates between minK and maxK clusters by
     * splitting a cluster into 2 clusters.  The algorithm splits the
     * cluster with contributes most towards the entropy of the model.
     * 
     * When minK === maxK, behaves exactly like KMeans clustering.
     * 
     * @param {int} minK minimum
     * @param {int} maxK max number of clusters to create
     * @param {Function} distanceMetric 
     * @param {object} options
     */

    constructor(minK, maxK, distanceMetric, options) {
        super(minK, distanceMetric, options);

        if (minK > maxK) {
            throw Error('maxK must be greater than, or equal to minK');
        }
        this.maxK = maxK;

        var options = options || {};
        this.splitThreshold = options.splitThreshold || 0.0; // threshold for splitting a cluster
        this.bicThreshold = options.bicThreshold || 0.0; // threshold for stopping the clustering
    }

    transform(dataset, options) {
        /**
         * Executes the k-means clustering algorithm against a dataset
         * with the given optional optimization parameters.  Parameters are
         * 
         * - validate (bool):  Indicates if the dataset should be validated
         * - kDTree (bool):  Indicates if a KD-tree should be used to find nearest centroids
         * 
         * @param {Array} dataset
         * @param {object} options
         */
        var options = options || {};
        options.kDTree = options.kDTree || false;
        if (options.validate || true) {
            this.validateDataset(dataset);
        }

        let iterations = 0;
        let oldCentroids, labels, centroids;

        // Initialize centroids randomly
        if (options.useNaiveSharding || true) {
            centroids = this._getRandomCentroidsNaiveSharding(dataset);
        } else {
            centroids = this._getRandomCentroids(dataset);
        }

        labels = this._getLabels(dataset, centroids, options.kDTree);
        var oldBIC = Number.MAX_SAFE_INTEGER,
            currentBIC = this._calculateBIC(dataset, labels, centroids.map(function(v, i) {
                return i;
            })),
            currentK = this.minK;
        // Run the main X-means algorithm
        while (!this._shouldStop(oldCentroids, centroids, iterations, oldBIC, currentBIC)) {
            console.debug('currentK: ' + currentK + ', maxK: ' + this.maxK + ', Iteration #' + iterations);
            // Save old centroids for convergence test.
            oldCentroids = [...centroids];
            iterations++;

            // Assign labels to each datapoint based on centroids
            labels = this._getLabels(dataset, centroids, options.kDTree);
            centroids = this._recalculateCentroids(dataset, labels);

            if (currentK === this.maxK) {
                continue;
            }
            // X-Means calculations

            // Calculate BIC for each cluster to determine the one with highest BIC
            const clusterBICs = [];
            for (var i = 0; i < centroids.length; i++) {
                const clusterBIC = this._calculateBIC(dataset, labels, [i]);
                clusterBICs.push({
                    clusterIndex: i,
                    BIC: clusterBIC
                });
            }
            clusterBICs.sort(function(a, b) {
                return b.BIC - a.BIC;
            });
            const highestBICClusterIndex = clusterBICs[0].clusterIndex;

            // Split the cluster with highest BIC
            const splitClusters = this._splitCluster(labels[highestBICClusterIndex], labels);

            // Calculate BIC for split model
            let splitLabels = Object.assign({}, labels);
            const splitCentroids = centroids.concat(splitClusters.centroids); // add clusters from split
            splitCentroids.splice(highestBICClusterIndex, 1); // remove split cluster
            splitLabels = this._getLabels(dataset, splitCentroids, options.kDTree); // assign labels based on new centroids
            const splitBIC = this._calculateBIC(dataset, splitLabels, splitCentroids.map(function(v, i) {
                return i;
            }));
            console.debug('current BIC:' + currentBIC + ', potential BIC: ' + splitBIC);

            // Replace clusters and centroids if BIC is improved
            if (this._isImproved(currentBIC, splitBIC, this.splitThreshold)) {
                centroids = splitCentroids;
                labels = splitLabels;
                currentK++;
                oldBIC = currentBIC;
                currentBIC = splitBIC;
            }
        }

        const clusters = [];
        for (const cluster in labels) {
            clusters.push(labels[cluster]);
        }
        const results = {
            clusters: clusters,
            centroids: centroids,
            iterations: iterations,
            converged: iterations <= this.maxIterations,
        };
        return results;
    }

    _isImproved(oldValue, newValue, threshold) {
        if (threshold > 0 && threshold < 1.0) {
            threshold = threshold * oldValue;
        }
        return oldValue - newValue > threshold;
    }

    _shouldStop(oldCentroids, centroids, iterations, oldBIC, bic) {
        if (!this._isImproved(oldBIC, bic, this.bicThreshold)) {
            return true;
        }
        return super._shouldStop(oldCentroids, centroids, iterations);
    }

    _splitCluster(cluster) {
        // Apply k-means with k=2 to the points in the cluster
        const kmeans = new KMeans(2, this.distanceMetric);
        return kmeans.transform(cluster.points);
    }

    _calculateBIC(dataset, labels, centroids) {
        const n = dataset.length;
        const k = centroids.length;
        let withinClusterVariance = 0.0;
        for (var i = 0; i < centroids.length; i++) {
            const centroidIndex = centroids[i];
            const clusterCentroid = labels[centroidIndex].centroid;
            const clusterPoints = labels[centroidIndex].points;

            for (var j = 0; j < clusterPoints.length; j++) {
                const squaredDistance = this.distanceMetric(clusterPoints[j], clusterCentroid);
                withinClusterVariance += squaredDistance;
            }
        }
        const BIC = n * Math.log(withinClusterVariance / n) + k * Math.log(n);
        return BIC;
    }

    _arithmeticMean(dataset) {
        const totalPoints = dataset.length;
        const means = [];
        for (let j = 0; j < dataset[0].length; j++) {
            means.push(0);
        }
        for (let i = 0; i < dataset.length; i++) {
            const point = dataset[i];
            for (let j = 0; j < point.length; j++) {
                const val = point[j];
                means[j] = means[j] + val / totalPoints;
            }
        }
        return means;
    }

}

This X-Means implementation uses the Bayesian Information Criterion as it’s model selection criterion for splitting clusters. The Bayesian Information Criterion (BIC) is a statistical measure used to assess the quality of a clustering model. Intuitively, the BIC balances the goodness of fit of the model with the complexity of the model. It takes into account how well the model fits the data (i.e., how close the data points are to their assigned cluster centroids) and penalizes the model for having a large number of clusters. The BIC encourages finding a clustering solution that captures the underlying structure of the data while avoiding overfitting.

A lower BIC value indicates a better clustering model. As the number of clusters increases, the model becomes more complex, and the BIC penalizes this complexity to prevent overfitting. The BIC helps strike a balance between the model’s ability to capture the data’s structure and its simplicity. The BIC is used in this X-Means implementation to determine whether splitting a cluster into two subclusters improves the overall clustering quality. If the BIC improves significantly after splitting a cluster, it suggests that the split captures meaningful substructures in the data. The algorithm continues splitting clusters until the improvement in BIC falls below a specified threshold, indicating that further splitting does not substantially enhance the clustering quality.

Deciding on an appropriate BIC threshold for the X-Means algorithm can be challenging, as it depends on the specific characteristics of your dataset and the desired granularity of the resulting clusters. One common approach is to use the elbow method to run the XMeans algorithm with different BIC thresholds and plot the resulting BIC values against the number of clusters. You can then look for an “elbow” point in the plot, where the rate of improvement in BIC starts to diminish. The BIC threshold corresponding to this elbow point can be a good choice, as it indicates a balance between model complexity and goodness of fit.

Here’s a high-level overview of how to use this XMeans implementation:

1. Create an instance of the XMeans class

   var minK = 2,
       maxK = 10,
       distanceMetric = euclideanDistanceFunc,
       options = {
         splitThreshold: 0.05,
         bicThreshold: 0.05
       }
   var clf = new XMeans(minK, maxK, distanceMetric, options);
   

   The splitThreshold and bicThreshold parameters play important roles in the XMeans algorithm:

   • splitThreshold: This parameter determines the threshold for splitting a cluster. If the difference between the current BIC and the potential BIC after splitting a cluster exceeds this threshold, the cluster is split into two. A higher value makes the algorithm more reluctant to split clusters, while a lower value encourages more splits.
   • bicThreshold: This parameter determines the threshold for stopping the clustering process. If the improvement in BIC between iterations falls below this threshold, the algorithm stops further splitting of clusters. A higher value leads to earlier termination of the clustering process, while a lower value allows for more iterations and potentially more clusters.

   If the splitThreshold or bicThreshold is greater than 1, the implementation will consider the value as an absolute value and only split a cluster, or end the execution if the BIC is below the specified value. If the specified threshold is between 0 and 1 in this implementation, the _isImproved method will interpret the given *threshold value as a proportion, and split the cluster or end the execution if the BIC has improved by more than the given proportion since the previous iteration.

2. Call the transform method on the XMeans instance, passing in the data to be clustered:

   var options = {
           kDTree: true,
           validate: true
       },
       results = clf.transform(data, options);
   

   The XMeans.transform method accepts the following parameters in the options object:

   • KDTree: Indicates the algorithm is to use a KDTree for identifying th4e nearest centroid to each point in the data. Default is false.
   • validate: Indicates the implementation is verify the data is a valid dataset: An array of arrays of numbers.

The CIE2000 Color Distance Metric🔗

RGB (Red, Green, Blue) and HSV (Hue, Saturation, Value) are commonly used color spaces in computer vision and image processing. However, they have inherent limitations when it comes to representing perceptual differences between colors.

In the RGB color space, the Euclidean distance between two colors does not accurately reflect the perceived difference by the human visual system. For example, the RGB distance between a dark blue and a light blue may be smaller than the distance between a red and an orange, even though the latter pair appears more similar to the human eye.

Similarly, the HSV color space, while better at separating color information from brightness, still has limitations in representing perceptual differences. The hue component in HSV is represented as an angle, which can lead to discontinuities and inconsistencies when measuring color distances.

To address the limitations of traditional color spaces, the International Commission on Illumination (CIE) developed the CIE2000 color distance metric. CIE2000 is designed to provide a more accurate representation of perceptual color differences, taking into account the complexities of human color perception.

The CIE2000 metric is based on the CIELAB color space, which is a perceptually uniform color space. In CIELAB, colors are represented using three components: L for lightness, a for green-red, and b for blue-yellow. The CIE2000 formula incorporates additional corrections and weighting factors to account for the non-uniformity of color perception across different regions of the color space.

The development of CIE2000 involved extensive research and empirical studies to understand and quantify human color perception. The CIE conducted experiments with human observers to gather data on perceived color differences, which was then used to refine the metric’s parameters and equations.

Using the CIE2000 color distance metric offers several advantages for color comparison and clustering tasks:

1. Perceptual Accuracy: CIE2000 provides a more perceptually uniform measure of color difference, aligning closely with human color perception. This means that colors that appear similar to the human eye will have smaller CIE2000 distances, while colors that appear more distinct will have larger distances.
2. Improved Clustering Results: When used in conjunction with clustering algorithms like K-Means or X-Means, CIE2000 can lead to more meaningful and visually coherent color clusters. By capturing perceptual differences accurately, CIE2000 helps group colors that are perceptually similar, resulting in clusters that align better with human perception.
3. Color Consistency: CIE2000 ensures consistent color comparison across different regions of the color space. It takes into account the non-uniformity of color perception, applying appropriate corrections and weighting factors to maintain perceptual uniformity.
4. Robustness to Illumination Changes: CIE2000 is designed to be more robust to changes in illumination compared to other color distance metrics. It incorporates corrections for the effect of illumination on color appearance, making it suitable for scenarios where lighting conditions may vary.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
function deltaE2000(labA, labB, options) {
    /*
    * CIE2000 equation for quantifying perceptual distance between colors.
    * For a very high-level explanation, see https://sensing.konicaminolta.us/us/blog/identifying-color-differences-using-l-a-b-or-l-c-h-coordinates/
    * or for a more mathematical approach: https://zschuessler.github.io/DeltaE/learn/
    * Implementation based on equations from http://www.brucelindbloom.com/index.html?Eqn_DeltaE_CIE2000.html 
    * 
    * Thresholds
    * 0: No perceivable difference between the colors.
    * 0-1: Very slight difference, likely imperceptible to most observers.
    * 1-2: Slight difference, noticeable to trained observers or under careful examination.
    * 2-3.5: Distinct difference, noticeable to most observers.
    * 3.5-5: Significant difference, readily apparent to all observers.
    * Above 5: Very large difference, colors are clearly distinct.

    * Limits
    * 0: Completely identical
    * 100:  Completely different
    * 
    * Weighting factors
    * kL (lightness weighting factor): Controls the influence of lightness differences on the overall Delta E 2000 value.
    * kC (chroma weighting factor): Controls the influence of chroma differences.
    * kH (hue weighting factor): Controls the influence of hue differences.
    * 
    * Higher values of a given factor emphasize that aspect of the color.  Each ranges from 0 to 2, and default to 1.
    */

    const l1 = labA[0],
        a1 = labA[1],
        b1 = labA[2],
        l2 = labB[0],
        a2 = labB[1],
        b2 = labB[2];

    // missing utility functions added to Math Object
    Math.rad2deg = function(rad) {
        return 360 * rad / (2 * Math.PI);
    };
    Math.deg2rad = function(deg) {
        return (2 * Math.PI * deg) / 360;
    };
    
    const avgL = (l1 + l2) / 2;
    const c1 = Math.sqrt(Math.pow(a1, 2) + Math.pow(b1, 2));
    const c2 = Math.sqrt(Math.pow(a2, 2) + Math.pow(b2, 2));
    const avgC = (c1 + c2) / 2;
    const g = (1 - Math.sqrt(Math.pow(avgC, 7) / (Math.pow(avgC, 7) + Math.pow(25, 7)))) / 2;

    const a1p = a1 * (1 + g);
    const a2p = a2 * (1 + g);

    const c1p = Math.sqrt(Math.pow(a1p, 2) + Math.pow(b1, 2));
    const c2p = Math.sqrt(Math.pow(a2p, 2) + Math.pow(b2, 2));

    const avgCp = (c1p + c2p) / 2;

    let h1p = Math.rad2deg(Math.atan2(b1, a1p));
    if (h1p < 0) {
        h1p = h1p + 360;
    }

    let h2p = Math.rad2deg(Math.atan2(b2, a2p));
    if (h2p < 0) {
        h2p = h2p + 360;
    }

    const avghp = Math.abs(h1p - h2p) > 180 ? (h1p + h2p + 360) / 2 : (h1p + h2p) / 2;

    const t = 1 - 0.17 * Math.cos(Math.deg2rad(avghp - 30)) + 0.24 * Math.cos(Math.deg2rad(2 * avghp)) + 0.32 * Math.cos(Math.deg2rad(3 * avghp + 6)) - 0.2 * Math.cos(Math.deg2rad(4 * avghp - 63));

    let deltahp = h2p - h1p;
    if (Math.abs(deltahp) > 180) {
        if (h2p <= h1p) {
            deltahp += 360;
        } else {
            deltahp -= 360;
        }
    }

    const deltalp = l2 - l1;
    const deltacp = c2p - c1p;

    deltahp = 2 * Math.sqrt(c1p * c2p) * Math.sin(Math.deg2rad(deltahp) / 2);

    const sl = 1 + ((0.015 * Math.pow(avgL - 50, 2)) / Math.sqrt(20 + Math.pow(avgL - 50, 2)));
    const sc = 1 + 0.045 * avgCp;
    const sh = 1 + 0.015 * avgCp * t;

    const deltaro = 30 * Math.exp(-(Math.pow((avghp - 275) / 25, 2)));
    const rc = 2 * Math.sqrt(Math.pow(avgCp, 7) / (Math.pow(avgCp, 7) + Math.pow(25, 7)));
    const rt = -rc * Math.sin(2 * Math.deg2rad(deltaro));

    options = options || {};

    /*
        * kl (lightness) weighting factor ( 0.0 <= kC <= 2.0)
        * Increasing kL amplifies the importance of lightness differences.
        * This can be useful for applications where subtle lightness variations are crucial, like in textile or paint industries.
        * Conversely, lowering kL downplays lightness changes, which might be beneficial for web design where screen brightness can affect perceived lightness. 
        * */
    const kl = Math.min(
            Math.max(
                options.kl || 1.0,
                0.0),
            2.0
        );
    /*
        * kC (chroma) weighting factor ( 0.0 <= kC <= 2.0)
        * Adjusting kC modifies the influence of chroma (color saturation) in the calculation.
        * Raising kC highlights chroma differences, making vibrant colors stand out more.
        * Lowering it reduces the impact of chroma variations, potentially minimizing the appearance of color shifts due to factors like lighting or viewing angle. 
        * */
    const kc = Math.min(
            Math.max(
                options.kC || 1.0,
                0.0),
            2.0
        );
    /*
        * kH (hue) weighting factor ( 0.0 <= kC <= 2.0)
        * Modifying kH alters the emphasis on hue (color tint) differences.Modifying kH alters the emphasis on hue (color tint) differences.
        * This is rarely used in practical applications due to the complexity of hue calculations and its potentially limited impact on perceived color changes in most contexts.
        * */
    const kh = Math.min(
            Math.max(
                options.kH || 1.0,
                0.0),
            2.0
        );

    const deltaE = Math.sqrt(Math.pow(deltalp / (kl * sl), 2) + Math.pow(deltacp / (kc * sc), 2) + Math.pow(deltahp / (kh * sh), 2) + rt * (deltacp / (kc * sc)) * (deltahp / (kh * sh)));
    return deltaE;
}
deltaE2000.MIN = 0.0;
deltaE2000.MAX = 100.0;

/**
    * Just Noticeable Difference threshold.
    * Represents the smallest color difference that a typical observer can perceive under specific viewing conditions.
    * For Delta E 2000, a JND value of around 1 is generally accepted. This means a Delta E 2000 difference of 1 or
    * less is often considered imperceptible to most people.
    */
deltaE2000.JND = 1.0;
/**
    * Barely Noticeable Difference threshold.
    * The color difference is barely perceptible. Only experienced observers under favorable viewing conditions may be able to
    * detect the difference.
    */
deltaE2000.BND = 2.0;
/**
    * Noticeable Difference threshold.
    * The color difference is noticeable but still considered acceptable in many applications. It represents a good color match.
    */
deltaE2000.ND = 3.5;
/**
    * Apparent Difference threshold.
    * The color difference is becoming more apparent and may be unacceptable in certain applications requiring precise color matching.
    */
deltaE2000.AD = 5.0;

In this code, the deltaE2000 function takes two LAB colors as input and computes the CIE2000 color distance between them. The implementation follows the steps outlined in the CIE2000 formula, including the calculation of various intermediate values and corrections.

CIE2000 Tuning🔗

The CIE2000 color distance metric provides a way to measure the perceptual difference between two colors, taking into account various aspects of color perception. One of the powerful features of CIE2000 is the ability to customize the perceptual distance calculation by adjusting the weight given to different color components. This is achieved through the use of three optional parameters: kL, kC, and kH, which are all 1 by default.

• kL (Lightness Weight): The kL parameter controls the weight given to the lightness component (L*) in the perceptual distance calculation.
  • Increasing the value of kL will make the perceptual distance more sensitive to lightness differences, while decreasing it will reduce the impact of lightness variations. Adjusting kL can be useful in scenarios where lightness differences are more or less important than other color aspects.
  • Increase Example: When comparing colors in a low-light environment or dealing with dark colors, increasing kL can help emphasize lightness differences and improve the perceptual distance measurement.
  • Decrease Example: In scenarios where the colors are primarily distinguished by their chroma and hue, such as comparing colors in a well-lit environment, decreasing kL can be appropriate.
• kC (Chroma Weight): The kC parameter determines the weight given to the chroma component (C*) in the perceptual distance calculation.
  • Increasing kC will make the perceptual distance more sensitive to chroma (saturation) differences, while decreasing it will reduce the impact of chroma variations. Modifying kC can be beneficial when dealing with colors where chroma differences are of particular importance or when you want to emphasize or de-emphasize the role of saturation in the distance calculation.
  • Increase Example: If you are comparing colors in a context where vividness and saturation are key factors, such as in graphic design or art, increasing kC can be beneficial.
  • Decrease Example: If you are working with pastel or desaturated colors, where chroma variations are subtle, decreasing kC can be appropriate.
• kH (Hue Weight): The kH parameter controls the weight given to the hue component (H*) in the perceptual distance calculation.
  • Increasing kH will make the perceptual distance more sensitive to hue differences, while decreasing it will reduce the impact of hue variations. Adjusting kH can be useful when working with colors where hue differences are more or less significant than other color aspects.
  • Increase Example: If you are working with a color dataset where colors are primarily distinguished by their hue, such as in color categorization or classification tasks, increasing kH can be beneficial.
  • Decrease Example: If you are comparing colors in a context where hue shifts are common or expected, such as in color correction or white balance adjustments, decreasing kH can be appropriate.

In a color search application for fashion, where users are looking for similar shades of a particular color, increasing kC and kH while slightly decreasing kL can help prioritize chroma and hue similarities over lightness variations.

When analyzing color differences in a printing process, where maintaining consistent lightness is crucial, increasing kL while keeping kC and kH at their default values can help ensure that lightness variations are given more weight in the distance calculation.

In a color palette generator tool, where creating visually appealing and harmonious color schemes is the goal, increasing kH while slightly decreasing kL and kC can help prioritize hue relationships and create palettes with balanced color combinations.

The interpretation of the deltaE2000 distance values can vary depending on the context and application, but some thresholds for general guidelines are included for gauging the perceptual difference from the distance metric which may prove useful in post-processing and analysis of dominant colors.

Dominant Color Extraction with X-Means and CIE2000🔗

Extracting the dominant colors from an image takes the following steps:

1. Image Loading: Begin by loading the image into your program or website.
2. Color Space Conversion: Convert the image from its original color space (e.g., RGB) to the CIELAB color space. The CIELAB color space is perceptually uniform and is the basis for the CIE2000 color distance metric. This conversion ensures that the color distances calculated in later steps are perceptually meaningful.
3. Pixel Extraction: Extract the pixel values from the image. Each pixel should be represented as a three-dimensional vector containing the color of the pixel in the CIELAB color space.
4. Preprocessing the Image Data for Clustering:
   • Pixel Sampling: Depending on the size of the image and computational resources available, we may sample a subset of pixels for clustering. This can be done by randomly selecting a fixed number of pixels or by using a systematic sampling approach.
5. Execute the X-Means Algorithm. Initialize the X-Means algorithm with the preprocessed pixel data. Specify the initial number of clusters (e.g., K=1) and set the CIE2000 (deltaE2000) color distance metric as the distance function.
6. Dominant Color Representation: Once the X-Means clustering is complete, the cluster centroids represent the dominant colors in the image. These centroids are typically stored as CIELAB color values and can be converted back to the original color space (e.g., RGB) for visualization or further processing.
7. Post-processing and Analysis: This can include merging similar colors, filtering out less significant colors based on their cluster sizes, or sorting the colors based on their prominence in the image.

Color Space Conversion🔗

First we need to convert the pixels in our image from RGB to the LAB color space, as the deltaE2000 function expects the colors to be in the CIELAB color space:

function rgb2lab(rgba) {
    var r = rgba[0] / 255,
        g = rgba[1] / 255,
        b = rgba[2] / 255,
        alpha = rgba[3], // alpha, untouched
        x,
        y,
        z;

    r = r > 0.04045 ? Math.pow((r + 0.055) / 1.055, 2.4) : r / 12.92;
    g = g > 0.04045 ? Math.pow((g + 0.055) / 1.055, 2.4) : g / 12.92;
    b = b > 0.04045 ? Math.pow((b + 0.055) / 1.055, 2.4) : b / 12.92;

    x = (r * 0.4124 + g * 0.3576 + b * 0.1805) / 0.95047;
    y = (r * 0.2126 + g * 0.7152 + b * 0.0722) / 1.0;
    z = (r * 0.0193 + g * 0.1192 + b * 0.9505) / 1.08883;

    x = x > 0.008856 ? Math.pow(x, 1 / 3) : 7.787 * x + 16 / 116;
    y = y > 0.008856 ? Math.pow(y, 1 / 3) : 7.787 * y + 16 / 116;
    z = z > 0.008856 ? Math.pow(z, 1 / 3) : 7.787 * z + 16 / 116;

    return [116 * y - 16, 500 * (x - y), 200 * (y - z), alpha];
}

Clustering with X-Means🔗

function clusterPixels(imgElement, minK, maxK, maxIterations) {
    var imgData = loadData(imgElement),
        data = [];
    for (var i = 0; i < imgData.data.length; i += 4) {
        var rgbColor = [imgData.data[i], imgData.data[i + 1], imgData.data[i + 2], imgData.data[i + 3]],
            labColor = rgb2lab(rgbColor);
        data.push(labColor);
    }
    var clusterer = new XMeans(minK, maxK, euclidean, {maxIterations: maxIterations});
    var results = clusterer.transform(data, {kDTree: false});
    return results;
}

function loadData(imgElement) {
    const canvas = document.createElement("canvas"),
        ctx = canvas.getContext("2d");
    canvas.width = imgElement.width;
    canvas.height = imgElement.height;
    ctx.drawImage(imgElement, 0, 0);
    const imageData = ctx.getImageData(0, 0, imgElement.width, imgElement.height);
    return imageData;
}

Post-processing and Analysis🔗

Post-processing allows you to refine the extracted dominant colors and create visually cohesive color palettes. By analyzing the identified colors, we can prioritize dominant colors, merge similar shades, remove outlying colors with a low proportion of pixels, or adjust the color representation to suit specific design or application requirements. This refinement process ensures that the resulting color palettes are aesthetically pleasing and representative of the image’s overall color scheme.

Evaluate Quality of Clusters🔗

Clustering evaluation helps assess the quality of the color clusters in terms of compactness and separation. Compactness refers to how tightly the colors within a cluster are grouped together, while separation refers to how well-distinguished the clusters are from each other. Evaluation metrics such as silhouette score or Davies-Bouldin index can quantify these aspects. A high silhouette score indicates that colors within a cluster are more similar to each other than to colors in other clusters, suggesting well-defined and distinct color groups. By evaluating cluster quality, you can determine if the XMeans algorithm is effectively partitioning the color space and creating cohesive color clusters.

Clustering evaluation can also guide the selection of appropriate parameters for the XMeans algorithm, such as the BIC threshold or the range of the number of clusters (minK and maxK). By evaluating the clustering results for different parameter settings, you can identify the combination that yields the best color clusters. For example, you can compare the average silhouette score or other evaluation metrics for different BIC thresholds and choose the threshold that produces the most perceptually meaningful and well-separated color clusters.

Centroid distance and Intracluster distance🔗

meanDistanceBetweenCentroids(clusters) {
    const output = new Array(clusters.length).fill(0),
        centroids = clusters.map(function(cluster) { return cluster.centroid; });
    for (var i = 0; i < clusters.length; i++) {
        output[i] = this._avgDistanceToPoints(clusters[i].centroid, centroids);
    }
    return output;
}

meanDistanceWithinClusters(clusters) {
    const output = new Array(clusters.length).fill(0);
    for (var i = 0; i < clusters.length; i++) {
        output[i] = this._avgDistanceToPoints(clusters[i].centroid, clusters[i].points);
    }
    return output;
}

Silhouette Coefficient🔗

The Silhouette Coefficient is an intuitive measure of how well each data point fits into its assigned cluster compared to other clusters. It quantifies the notion of “belonging” by considering the similarity of a data point to its own cluster and its dissimilarity to the nearest neighboring cluster. A high Silhouette Coefficient indicates that a data point is well-matched to its assigned cluster and poorly matched to neighboring clusters, suggesting a good clustering result. Conversely, a low or negative Silhouette Coefficient suggests that a data point may be on the border between clusters or even assigned to the wrong cluster. By computing the average Silhouette Coefficient across all data points, you can assess the overall quality and consistency of the clustering solution. A higher average Silhouette Coefficient indicates better-defined and more separated clusters, while a lower value suggests overlapping or less distinct clusters. Interpreting the Silhouette Coefficient provides valuable insights into the effectiveness of the clustering algorithm and helps in determining the optimal number of clusters for a given dataset.

A value close to 1 indicates that the color data point is well-matched to its assigned cluster and poorly matched to neighboring clusters, suggesting a good clustering result. A value close to 0 suggests that the color data point is on the border between two clusters. A negative value indicates that the color data point may have been assigned to the wrong cluster, as it is closer to points in a neighboring cluster.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
/**
* Calculates the Silhouette coefficient for all points in a given clustering result.
*
* The Silhouette coefficient is a widely used metric for evaluating the quality of clustering.
* It measures how well data points are assigned to their clusters by considering both the intra-cluster
* distance (distance between points within the same cluster) and the inter-cluster distance
* (distance between points in different clusters).
*
* @param {Array<Array<number>>} dataset - The dataset used for clustering.
* @param {Array<object>} clusters - Array of cluster labels for each data point.
*
* @returns {Array} The Silhouette coefficient  for each cluster.
*/
silhouetteCoefficient(dataset, clusters) {
    const numSamples = dataset.length;
    const numClusters = clusters.length;
    const silhouetteScores = new Array(numSamples).fill(0);

    for (let i = 0; i < numSamples; i++) {
        const point = dataset[i];
        let clusterIndex;
        for (let k = 0; k < numClusters; k++) {
            if (clusters[k].points.includes(point)) {
                clusterIndex = k;
                break;
            }
        }

        const clusterPoints = clusters[clusterIndex].points;
        const withinClusterDist = this._avgDistanceToPoints(point, clusterPoints);

        let nearestClusterDist = Infinity;
        for (let k = 0; k < numClusters; k++) {
            if (k !== clusterIndex) {
                const otherClusterPoints = clusters[k].points;
                const distToCluster = this._avgDistanceToPoints(point, otherClusterPoints);
                nearestClusterDist = Math.min(nearestClusterDist, distToCluster);
            }
        }

        const silhouetteScore = (nearestClusterDist - withinClusterDist) / Math.max(withinClusterDist, nearestClusterDist);
        silhouetteScores[i] = silhouetteScore;
    }
    return silhouetteScores;
}

_avgDistanceToPoints(point, points) {
    const options = options || {},
        numPoints = points.length;
    let totalDist = 0;

    for (let i = 0; i < numPoints; i++) {
        const otherPoint = points[i];
        const dist = this.distanceMetric(point, otherPoint);
        totalDist += dist;
    }

    return totalDist / numPoints;
}