Knowledge-Augmented Visual Question Answering

Overview

My MSc final project investigated whether task-specific ConceptNet knowledge graph slices, combined with a frozen visual question answering model through late fusion, could improve performance on OK-VQA. I implemented the end-to-end experimental system, established a frozen baseline, built deterministic knowledge retrieval and caching, implemented multiple fusion strategies, and evaluated the approach with full-validation runs, ablations, a random-slice control, and qualitative error analysis.

Initial Ask

Can bounded, task-specific external knowledge improve visual question answering without retraining the underlying vision-language model?

Problem

Visual question answering often requires knowledge that is not directly visible in an image. The project tested whether retrieving a small, question-specific slice of ConceptNet and combining that evidence with a frozen ViLT baseline could improve answer accuracy under matched experimental conditions.

Desired Outcome

Produce a reproducible system that could fairly test whether task-specific knowledge augmentation improves a fixed VQA baseline, and expose enough diagnostics to explain positive, negative, or null results.

Constraints

• The ViLT baseline was frozen for the matched comparison, so improvements had to come from the external-knowledge branch and fusion mechanism rather than end-to-end retraining.
• ConceptNet contains noisy and generic relations, making retrieval quality and answer-space alignment major risks.
• The fixed 10,000-answer vocabulary covered 96.5% of validation answers but still constrained what the knowledge branch could influence.
• Evaluation had to distinguish real knowledge contribution from gains caused by extra parameters, reranking, or other fusion side effects.

Approach

I built two separable branches: a ViLT answer-classification baseline and a ConceptNet knowledge branch. Question-derived entities were used to construct bounded graph slices with configurable hop depth, relation filtering, edge-weight thresholds, neighbour limits, and top-K controls. Retrieved facts were encoded into a knowledge-derived answer signal and combined with baseline logits using weighted, gated, and top-N late-fusion strategies. Configuration-linked run logging, deterministic slice caching, ablations, controls, and error-analysis tooling were used to keep the evaluation reproducible and diagnosable.

Key Decisions

Solution

A reproducible Python-based experimental pipeline for OK-VQA that loads image-question pairs, runs a frozen ViLT answer classifier, extracts question entities, retrieves and ranks bounded ConceptNet facts, produces a knowledge answer signal, performs late fusion, scores predictions with VQA-soft accuracy, and records experiment artifacts for comparison and error analysis.

Personal Contribution

I designed and implemented the research system, experiment configurations, dataset and evaluation pipeline, ConceptNet slicing and caching logic, knowledge encoding and fusion mechanisms, control experiments, result logging, quality checks, and qualitative error-analysis outputs.

Technical Implementation

The final frozen baseline used dandelin/vilt-b32-finetuned-vqa with a 10,000-answer vocabulary and was evaluated on all 5,046 OK-VQA validation examples. The knowledge branch used question-derived entity extraction and a locally queryable ConceptNet store to build deterministic bounded slices. Fusion experiments included naive weighted addition, gated fusion, and constrained top-N reranking. Results and configurations were stored by run so that the reported comparisons could be reproduced and audited.

Tech Stack

• Python
• PyTorch
• Hugging Face Transformers
• ViLT
• ConceptNet
• OK-VQA
• COCO

Result & Impact

• 5,046
  Validation examples
• 0.163892
  Frozen baseline VQA-soft accuracy
• 0.110517
  Naive weighted fusion
• 0.163892
  Top-20 gated fusion

The implemented knowledge branch did not improve the frozen baseline. Naive weighted fusion substantially degraded accuracy, while constrained weighted fusion reduced the damage to a near-zero negative delta and gated fusion preserved baseline performance by suppressing unreliable knowledge. A random-slice control also failed to improve the baseline, strengthening the conclusion that the tested ConceptNet signal was not providing useful answer-aligned evidence. The result narrowed the likely bottleneck to retrieval relevance and answer-space alignment rather than the existence of a fusion mechanism alone.

Learnings

• External knowledge is only useful when retrieval relevance and the model's answer representation align; adding more context can actively make a system worse.
• Gating can be valuable as a safety mechanism even when an auxiliary signal is not strong enough to improve performance.
• Controls and reproducible experiment logging matter because they let a null result answer the research question rather than merely report that tuning failed.
• A technically successful project does not require a positive benchmark delta when the system is implemented correctly and the evaluation exposes why the hypothesis was not supported.