Arachnet Market News Monitor
Whats new: Highest movement today observed within Nifty system indexing tracking frames.
Arachnet Intelligence Terminal
Automated financial news aggregation, NLP clustering, and sentiment-driven market analytics pipeline
Scraping Engine
Three-tier anti-bot evasion pipeline. Tier 1 cycles rotating real browser headers via requests. Tier 2 escalates to CloudScraper for TLS fingerprint spoofing. Tier 3 deploys headless undetected-chromedriver when static layers fail — ensuring maximum data ingestion resilience.
Parsing & ETL
Dual-strategy parsing engine: JSON-LD script extraction for structured page metadata and HTML DOM traversal for rendered layouts. Config-driven per-source, stored in arachnet.db. Pandas deduplication normalizes headlines cross-site before SQL insertion.
Clustering Engine
TF-IDF vectorization converts raw headlines and summaries into numerical feature matrices. Cosine similarity finds semantically related articles across sources. Levenshtein + Jaccard metrics provide precision verification. A 72-hour freshness gate prevents stale topic merging.
Scoring & Sentiment
VADER lexicon analysis produces compound sentiment scores bounded at ±1.0. A non-linear volume saturation curve prevents high-volume clusters from dominating. Historical scores undergo 33.3% daily exponential decay to surface emerging trends over stale narratives.
Pipeline Execution Sequence
Orchestrated by run_sentiment_analysis.py with OS-level singleton locking to prevent concurrent pipeline collisions.