Dependencies and Their Role in the Project
The page covers the third-party stack used in the winner notebook, with each library tied to its actual purpose.
2.8.0+cu126Core deep learning framework: embeddings, BiLSTM layers, custom attention modules, Siamese classifier, BCE loss, AdamW optimizer, schedulers, gradient clipping, checkpoint tensors, and CUDA execution.1.8.2Training and validation metrics: BinaryAccuracy, BinaryPrecision, BinaryRecall, BinaryF1Score, AUROC, AveragePrecision, and PrecisionRecallCurve for threshold calibration.3.9.0Experiment tracking for run names, parameters, per-epoch metrics, best-model logging, reproducibility summaries, and comparison across LSTM-attention variants.2.3.3 & NumPy 2.4.1CSV ingestion, dataframe cleaning, null handling, class distribution checks, train/validation preparation, embedding matrix initialization, probability arrays, and diagnostic summaries.1.8.0 & SciPy 1.17.0Stratified train/validation split, precision-recall support, calibration analysis, PCA-style feature projection in error analysis, and scientific-computing support for sklearn workflows.3.9.4English stopword list used for tokenizer stop masks and token-frequency inspection.3.10.8Diagnostic visualizations: class/probability distributions, calibration curves, attention/error plots, and feature-space views.4.67.1 & IPython 9.5.0Progress bars for training/validation loops and notebook extension support for the custom requirements refresh magic.Architecture Search Before the Winner
The winner was selected after a wide set of LSTM-attention experiments, not from a single baseline attempt.
Raw Fields and Modeling Target
The notebook keeps the problem clean: two questions in, binary duplicate probability out.
| Column | Type | Role |
|---|---|---|
id | Identifier | Row-level identifier from the Quora question-pair training data. |
qid1, qid2 | Question IDs | Original Quora question identifiers for both sides of the pair. |
question1, question2 | Raw text | The two natural-language questions cleaned, tokenized, padded/truncated, embedded, and encoded by the Siamese model. |
is_duplicate | Target | Binary label: 1 means duplicate semantic intent, 0 means different intent. |
Class balance
The label distribution is moderately imbalanced: 36.92% duplicate and 63.08% non-duplicate. The training code therefore optimizes threshold-sensitive metrics such as F1, precision, recall, AUROC, and Average Precision instead of relying only on accuracy.
Text normalization
QuoraPreproccesor fills missing questions, lowercases text, removes punctuation noise, normalizes spacing, and prepares both question columns before vocabulary building and dataset construction.
Model and Training Hyperparameters
These values are pulled from the winner run artifacts and config JSON.
hidden_dim=384, num_heads=4, dropout 0.35, attention dropout 0.0.<PAD> index 0, <UNK> index 1, lowercase text, maximum length 50, vocabulary cap 20000.0.05, learning rate 0.0005, weight decay 0.001, gradient clipping 1.5.F1Score in max mode, watches validation loss for early stopping, and recalibrates the threshold from the precision-recall curve.Custom Attention and Siamese Forward Pass
The main architecture is not a black-box layer; attention is implemented directly in PyTorch.
class AttentionHead(nn.Module): def forward(self, lstm_output, mask): proj = self.W(lstm_output) energy = torch.tanh(proj) scores = self.V(energy).squeeze(-1) scores = scores.masked_fill(mask == 0, self.mask_fill_num) weights = F.softmax(scores, dim=-1) context = torch.bmm(weights.unsqueeze(1), proj * mask.unsqueeze(-1)) return context.squeeze(1) class MultiHeadAttention(nn.Module): def forward(self, hidden_state, mask): x = torch.cat([h(hidden_state, mask) for h in self.heads], dim=-1) return self.out_linear(x)
def forward(self, q1, q2): h1 = self._encode(q1) h2 = self._encode(q2) cosine_sim = F.cosine_similarity(h1, h2).unsqueeze(-1) feat = torch.cat([ h1, h2, abs(h1 - h2), h1 * h2, cosine_sim ], dim=1) logits = self.fc_dims(feat) return logits.squeeze(-1)
ModelConfig and Encoder Internals
The architecture is driven by explicit configuration, then assembled into a reusable Siamese encoder.
class ModelConfig: MODEL_TYPE = "LSTM_attention" ATTENTION_TYPE = "MultiHead-Bahdanau" TOKEN_EMBEDDING = "gloVe-6B-100d" EMB_DIM = 100 FREEZE_TOKEN_EMBEDDING = True LOSS = "BCE with Logits" NUM_HEADS = 4 BIDIRECTIONAL = True HIDDEN_DIM = 384 LSTM_OUT = HIDDEN_DIM * 2 NUM_LAYERS = 2 DROPOUT = 0.35 LABEL_SMOOTHING = 0.05 FC_DIMS = [1024, 256] FC_DP = 0.4 INPUT_FC_DIM = 3073
def _encode(self, question): emb = self.embedding(question) emb = self.emb_dropout(emb) mask = self._create_mask(question) if self.stop_mask is not None: token_stop_mask = self.stop_mask[question] mask = mask * token_stop_mask.float() out, _ = self.LSTM(emb) ctx = self.attention(out, mask) return ctx
Attention concept in plain terms
Winner encoderEach question becomes a sequence of LSTM hidden states. For each head, the model projects every token state, scores it, masks padding and stop tokens, normalizes scores with softmax, and produces a weighted context vector. Multi-head attention repeats that in parallel subspaces, concatenates the heads, and projects them back into one 768-dimensional question representation.
How QuoraSiameseClassifier Is Built
The constructor wires together pretrained embeddings, recurrent encoding, attention pooling, and a pair-classification head.
class QuoraSiameseClassifier(nn.Module): def __init__(self, vocab_size, config, embedding=None, stop_mask=None): self.embedding = nn.Embedding(vocab_size, config.EMB_DIM) self.emb_dropout = nn.Dropout(config.EMB_DP) if embedding is not None: self.embedding.weight.data.copy_(embedding) self.embedding.weight.requires_grad = not config.FREEZE_TOKEN_EMBEDDING self.LSTM = nn.LSTM( input_size=config.EMB_DIM, hidden_size=config.HIDDEN_DIM, bidirectional=config.BIDIRECTIONAL, num_layers=config.NUM_LAYERS, dropout=config.DROPOUT, batch_first=True ) self.attention = MultiHeadAttention(config.LSTM_OUT) self.proj = nn.Identity() self.fc_dims = self._build_fc_layers(config.INPUT_FC_DIM, config.FC_DIMS, config.FC_DP)
| Component | Purpose |
|---|---|
nn.Embedding | Maps token IDs to dense vectors initialized from GloVe when available. |
nn.LSTM | Encodes token order and contextual dependencies in each question independently. |
MultiHeadAttention | Compresses sequence output into a context vector while respecting masks. |
nn.Identity | Keeps projection optional; winner run uses full 768-dimensional context vectors. |
_build_fc_layers | Builds the GELU + Dropout MLP classifier from pair-comparison features. |
Validation and Calibration Summary
The final checkpoint was selected by validation F1, then the threshold was calibrated from the precision-recall curve.
What the Custom MLflowTracker Automates
The project builds a small tracking abstraction that keeps each experiment consistent and comparable.
Versioned experiment identity
The tracker builds run names from model_type and attention_type, then generates a versioned suffix. This keeps repeated experiments understandable during comparison.
Full hyperparameter visibility
Each config section is logged to MLflow parameters. That includes tokenization, model dimensions, optimizer choices, scheduler settings, and training controls.
Metric curves, not just final numbers
The tracker logs train_loss, val_loss, train/validation metrics, learning rate, and best_threshold at each epoch.
Reproducible run package
The fit loop captures vocabulary state, label mapping, configs, exported architecture, dependency files, checkpoint state, the notebook, and the final summary.
Champion Experiment Snapshot
A concise public-facing view of the winner result and training controls.
| Field | Winner Value |
|---|---|
run_type | exploring-best-architecture |
model_type | LSTM_attention |
attention_type | MultiHead-Bahdanau |
checkpoint_metric | F1Score in max mode |
early_stop_metric | loss in min mode, patience 2 |
best_threshold | 0.4843311608 |
best_calibrated_f1 | 0.8131223321 |
Custom Tokenizer Inspired by Production NLP Pipelines
The tokenizer mirrors familiar NLP ecosystem ideas: build, encode, decode, save, load, and hydrate embeddings.
class QuoraTokenizer: build_vocabs(df) # count q1 + q2 tokens _build_stop_mask() # create token-level stopword mask encode(text) # ids, truncate, pad to max_length decode(ids) # inspect encoded samples save(path) # persist vocabs and masks load(path) # restore tokenizer state load_embedding(dim) # create GloVe matrix
Vocabulary is part of the model package
The tokenizer state is treated as a first-class output, so inference can reuse the same token IDs, padding behavior, stop mask, vocabulary size, and index-to-word mapping.
The loaded GloVe embedding matrix is copied directly into the PyTorch embedding layer, with a staged training strategy: frozen at first, then unfrozen from epoch 4.
QuoraDataset Converts Clean Text Into Model Batches
The dataset class keeps question-pair encoding and label packaging in one predictable interface.
class QuoraDataset(Dataset): def __init__(self, df, tokenizer): self.q1 = df["question1"].values self.q2 = df["question2"].values self.label = df["is_duplicate"].values self.tokenizer = tokenizer def __getitem__(self, idx): encoded_q1 = self.tokenizer.encode(self.q1[idx]) encoded_q2 = self.tokenizer.encode(self.q2[idx]) label = torch.tensor(self.label[idx], dtype=torch.float32) return {"q1": encoded_q1, "q2": encoded_q2, "label": label}
Every batch has a stable shape
Both questions are encoded to the same fixed length, so the trainer receives q1, q2, and label tensors without needing extra notebook-side glue code.
The dataset also includes a pos_class_weight helper, useful for experiments that need explicit positive-class weighting.
Training Loop With Calibration Built In
The trainer wraps optimization, metric computation, threshold search, scheduler behavior, checkpointing, and artifact logging.
class Trainer: def fit(self, num_epochs): for epoch in range(num_epochs): if epoch == self.config.UNFREEZE_EPOCH: self._unfreeze_embedding() train_loss, train_results = self.train_one_epoch() val_loss, val_results = self.evaluate(use_optimal_threshold=True) self._check_scheduler(val_loss) self.history.update(train_loss, val_loss, train_results, val_results, self.optimizer) self.tracker.log_epoch(epoch, train_loss, val_loss, train_results, val_results, lr) self.tracker.log_metric("best_threshold", self.best_threshold, epoch) if self._is_checkpoint_better(val_results["F1Score"]): self.best_checkpoint_score = val_results["F1Score"] self.tracker.save_state_dict(self.model.state_dict(), checkpoint_path) if self.early_stopper.step(val_loss): break final_threshold, calibrated_results = self.find_optimal_threshold() self.tracker.log_summary(calibrated_results)
Requirements and Model Script Creators
Two utilities improve portability: dependency extraction and architecture export from the notebook.
requirements.py
A custom IPython magic scans notebook imports with ast, filters standard-library modules, maps imports to installed distributions, and updates requirements.txt. The winner artifact records 11 packages, including mlflow==3.9.0, torch==2.8.0+cu126, and torchmetrics==1.8.2.
export_model_from_notebook
The notebook extracts model-defining cells into model_architecture.py. That creates a deployable architecture script containing config and model classes, separate from the exploratory notebook itself.
BCE With Label Smoothing
The winner model uses binary cross entropy with smoothed labels to reduce overconfident duplicate/non-duplicate decisions.
class BCEWithLabelSmoothing(nn.Module): def __init__(self, epsilon, reduction="mean"): super().__init__() self.epsilon = epsilon self.criterion = nn.BCEWithLogitsLoss(reduction=reduction) def forward(self, logits, labels): labels = labels.float() labels = labels * (1 - self.epsilon) + (1 - labels) * self.epsilon return self.criterion(logits, labels)
Why smoothing matters here
Duplicate-question detection contains many ambiguous examples: two questions can be lexically different but semantically close, or lexically similar but logically different. Label smoothing softens hard 0/1 targets, which helps the classifier avoid brittle confidence spikes.
The trainer then searches for the best F1 threshold on the validation precision-recall curve, so calibration is handled both in the loss and in post-processing.
How Attention Was Used for Debugging
Attention weights are not treated as absolute explanation, but they are useful for sanity-checking what the model emphasizes.
Top attended tokens
The analysis decodes token IDs back to words and aggregates high-attention tokens across validation examples. This checks whether the model attends to meaningful semantic words or collapses into punctuation and stopwords.
Hard examples by error type
For false positives and true positives, the notebook prints question pairs, predicted probability or distance, and token-level attention values to compare what changed between correct and incorrect duplicate decisions.
Variance and feature checks
The diagnostic code inspects LSTM outputs before attention and final feature vectors after similarity construction, looking for collapse, low variance, or poorly separated clusters.
From raw scores to deployable decisions
The winner training loop internalizes this lesson by searching the precision-recall curve and using the best threshold rather than assuming default sigmoid thresholding.
What This Project Demonstrates
The research value is both modeling depth and engineering maturity.
Modeling: the project implements attention manually, evaluates several attention/loss/matching families, and settles on a calibrated Siamese BiLSTM attention architecture.
Experimentation: MLflow gives the work a real lab structure: every meaningful run can carry parameters, metrics, artifacts, checkpoint files, and summaries.
Engineering: the code is no longer just notebook logic. It is organized into classes with clear responsibilities, closer to the style of reusable ML libraries.
Research direction: this foundation is designed to accept additional architecture families later, because the dataset layer, metric layer, tracker, and artifact packaging are already separated from the model implementation.