SEO audit work has a particular kind of tax on marketing teams. The tasks themselves are not conceptually hard — crawl the site, check the meta titles, review the backlink profile, flag the technical errors. But they are repetitive, time-consuming, and require maintaining a working knowledge of a dozen different tool outputs simultaneously. The result is that most teams do a thorough SEO audit once a quarter, do a partial one when something goes wrong, and let the rest slide.
The problem with quarterly audits is that SEO degrades faster than that. A canonical tag breaks on a page template and affects 200 URLs before the next audit cycle. A competitor earns a significant backlink cluster that shifts a keyword ranking. A Core Web Vitals issue appears on mobile after a theme update. None of these announce themselves — they accumulate quietly until the next audit, by which point the compounding effect has already landed in the rankings data.
The engine described here solves that by making the audit continuous, not periodic. Screaming Frog handles the technical crawl. SEMrush — connected via its MCP integration — provides on-demand keyword, competitive, and backlink data. Custom Python scripts parse, prioritise, and format the output. Claude receives the structured output and produces the analysis, the prioritised fix list, and the drafted implementations. The human role in the loop is review and approval — not execution.
The Stack — What Each Tool Does
The Stack
Four tools, one connected workflow
Paid · Desktop
Screaming Frog
Technical Crawler
Crawls the site on a schedule and exports structured data on every URL — status codes, meta titles, descriptions, H1s, canonical tags, response times, internal links, images, and structured data. The raw technical data source. Configured to run weekly and export directly to a watch folder that triggers the script pipeline.
Paid · MCP Connector
SEMrush
Keywords + Backlinks + Competitive
Connected via MCP integration — Claude can query SEMrush directly within the workflow. Provides on-demand keyword position data, backlink profile changes, toxic link flagging, and competitor movement. The MCP connection removes the manual export step — data flows directly into the Claude session when called.
Free · Custom Built
Python Scripts
Parse · Prioritise · Format
Custom scripts that sit between the raw tool exports and Claude. They ingest the Screaming Frog CSV, apply prioritisation rules (issue type × affected URL count × estimated impact), and produce a structured markdown summary Claude can reason against. The translation layer between raw crawl data and actionable AI input.
Free · Cowork
Claude (Cowork)
Analysis · Fixes · Drafts
Receives the structured script output and SEMrush data, analyses patterns across all three audit pillars simultaneously, produces a prioritised issue list, drafts fix recommendations with specific implementation details, and writes updated meta copy where needed. The decision and drafting layer — the human reviews output, not inputs.
The SEMrush MCP Connection — Why This Changes the Workflow
MCP Integration
How the SEMrush MCP connector removes the biggest manual bottleneck
The traditional SEO workflow involves logging into SEMrush, running the reports you need, exporting CSVs, cleaning the data, and pasting it somewhere Claude can read it. The MCP connector collapses this into a single step — Claude queries SEMrush directly.
Keyword position pull during the session. Instead of a manual export, Claude calls the SEMrush connector mid-session and retrieves current keyword rankings for specified terms. The data is live, not last week’s export — which matters when you are diagnosing a ranking movement that happened in the last 48 hours.
Backlink profile delta on demand. Claude can query new backlinks acquired since the previous session, lost backlinks, and toxic link flags — without a manual export. The off-page audit section of the workflow calls this automatically, and Claude reasons against the delta rather than the full static snapshot.
Competitive movement tracking. Claude can query competitor ranking movements for target keywords mid-session and incorporate that context into the prioritisation logic — so a technical fix recommendation is weighted differently if a competitor just gained three positions on the same keyword.
How to connect it. SEMrush’s MCP server is available through Claude’s connector directory. Once connected, Claude accesses it via tool calls within the Cowork session. You authorise the connection once and it persists — subsequent sessions can query SEMrush without re-authentication as long as your API access is active.
Pillar One: On-Page Optimisation
Pillar 01
On-Page SEO — Content, Titles, Structure
Screaming Frog
SEMrush MCP
Claude
SEMrush MCP
Claude
On-page optimisation is the most time-consuming part of a manual SEO audit because it involves reading and evaluating copy, not just flagging technical errors. A script can tell you a meta title is 72 characters. Only AI can tell you whether those 72 characters match the search intent of the target keyword, are differentiated from the competitor titles ranking above, and include the right semantic signals for the page’s content cluster. That reasoning layer is what Claude adds — and it is the layer that most automated SEO tools skip entirely.
Screaming Frog exports meta titles, descriptions, H1s, and word count for all pages. The Python script filters to pages below traffic threshold or with recent position drops — focusing Claude’s attention on the pages that actually need attention rather than processing the entire site.
SEMrush MCP pulls current keyword ranking for each flagged URL and the top three competitors ranking for the same term. Claude receives both the current page metadata and the competitive context simultaneously.
Claude evaluates each flagged URL against four on-page criteria: keyword-intent alignment, title differentiation from competitors, meta description click-through quality, and H1/H2 structural coherence. It produces a priority score and a specific rewrite for each flagged element.
For each URL in this list [structured table], review the current meta title and description against the target keyword [X] and the competitor titles currently ranking in positions 1–3 [from SEMrush]. Identify where the current title fails on intent alignment, differentiation, or click-through quality. Produce a rewritten title under 60 characters and a rewritten description under 155 characters for each. Flag where the H1 needs updating to match the target keyword cluster.
Per-page rewrites with specific reasoning for each change — not just “optimise for keywords” but “current title misses the transactional intent signal; competitor in P1 uses ‘free trial’ language that this page’s title omits; recommended rewrite adds the intent signal without keyword stuffing.” Ready to implement without further editing in most cases.
Pillar Two: Off-Page and Backlink Analysis
Pillar 02
Off-Page SEO — Backlinks, Authority, Toxicity
SEMrush MCP
Claude
Claude
Off-page analysis in a manual workflow means logging into SEMrush, running the backlink audit, exporting the new and lost link lists, cross-referencing toxic flags, and writing up what it means for ranking trajectory. With the MCP connection, Claude does this directly — querying the delta since the previous session and reasoning against it in context with the keyword movement data from the same session.
SEMrush MCP query pulls the backlink delta — new links acquired, links lost, and any newly flagged toxic domains since the previous audit date. The delta approach keeps the analysis focused on what changed, not a review of the entire backlink profile every week.
Claude assesses each new link for relevance, authority, and anchor text quality. It flags any anchor text patterns that are over-optimised or potentially manipulative. For lost links, it assesses whether the loss is likely to affect ranking for any tracked keyword and whether re-acquisition is worth pursuing.
Toxic link recommendations are produced with disavow-ready formatting. Claude outputs any disavow candidates as a formatted list ready to upload directly to Google Search Console — no reformatting required.
Competitor link gap analysis on demand. Claude queries SEMrush for domains linking to competitors but not to the audited site — producing a prioritised outreach list of link-building opportunities sorted by domain authority and topical relevance to the site’s content clusters.
Pull the backlink delta for [domain] since [last audit date] via SEMrush. For new links: assess authority, relevance to content clusters, and anchor text pattern. For lost links: flag any that were pointing to pages ranking in top 5 positions. For toxic flags: produce a disavow file in Google’s required format. Then query competitor link gap for top 3 competitors and produce a prioritised outreach list of the 10 highest-authority domains linking to competitors but not to us.
Structured backlink delta report, disavow file formatted for direct upload, and a 10-site outreach priority list with domain authority scores and the specific content angle that would justify a link request based on what each site has linked to for competitors.
Pillar Three: Technical Audit
Pillar 03
Technical SEO — Crawl, Core Web Vitals, Structure
Screaming Frog
Python Scripts
Claude
Python Scripts
Claude
The technical audit is where the Python scripts do the heaviest lifting. Screaming Frog exports can contain thousands of rows across dozens of dimensions — filtering, prioritising, and structuring that data for human review is exactly the kind of task that should be automated. The scripts apply a weighted scoring model to the crawl output, surface the issues most likely to affect ranking and crawl budget, and produce a structured input Claude can reason against in under 30 seconds.
Screaming Frog runs on schedule, exports to watch folder. A lightweight script monitors the folder, detects the new export, and immediately begins processing — so the pipeline runs without any manual trigger once the crawl completes.
Python script applies the prioritisation model. Issues are scored by type (4xx errors highest, thin content second, duplicate meta third, slow response time fourth), multiplied by affected URL count, and further weighted by whether the affected URLs are in the top-traffic or top-ranking tier. The output is a ranked issue list, not a raw export.
Canonical and redirect chain analysis runs separately. A dedicated script function identifies pages in redirect chains of three or more hops, canonical mismatches between rendered and declared canonicals, and self-referencing canonicals on paginated content — common issues that Screaming Frog flags but does not prioritise.
Claude receives the prioritised issue list and drafts remediation. For each technical issue, Claude produces a specific fix recommendation with implementation detail — not “fix your meta titles” but “17 product category pages have duplicate meta titles defaulting to the site template; here is the dynamic title pattern to implement across the template and the five highest-priority pages to update manually first.”
Here is this week’s prioritised technical issue list from the site crawl [structured markdown]. For each issue category, provide: the specific cause of the issue pattern, the exact implementation fix with any code snippet or configuration change required, the order in which to prioritise fixes given the URL impact scores, and any issue where the fix requires developer involvement vs content team action. Flag any issues that represent a ranking risk if unaddressed within the next two weeks.
Structured fix list with ownership tags (developer / content / SEO), implementation-ready code snippets for common fixes (canonical tag additions, redirect configurations, schema additions), and a two-tier priority split: urgent (ranking risk within two weeks) vs scheduled (address in next sprint). Formatted for direct input into a Jira or Linear ticket system.
What the Engine Surfaces — A Typical Weekly Output
Sample Output
Issues from a typical weekly run — prioritised by impact
#
Issue Identified
What Claude Produced
Fix Complexity
01
14 product pages returning 404 after URL restructure
Critical
Full redirect map with source and destination URLs, formatted as .htaccess rules ready to implement. Identified that 3 of the 14 had inbound backlinks — flagged for priority fix.
Same day
02
Blog pagination creating duplicate meta descriptions across 40 pages
High
Dynamic meta description pattern using the page number variable, plus rel=”canonical” pointing all paginated pages to the root URL. Code snippet for WordPress and custom CMS variants both included.
1 sprint
03
3 high-ranking pages with missing structured data schema
High
Complete JSON-LD schema blocks for each page type (Article, FAQ, HowTo) with all required fields populated from the existing page content. Ready to paste into the page head.
1–2 days
04
Competitor gained 18 backlinks from industry publication cluster
Medium
Identified the publication cluster via SEMrush MCP, assessed their editorial focus, produced a pitch angle and suggested content asset that matches their linking pattern. Formatted as an outreach brief.
Outreach
05
Top 10 pages with meta titles below 40 or above 60 characters
Medium
Rewritten titles for all 10 pages, incorporating target keyword, intent signal, and differentiation from P1–3 competitors. Character count verified. Ready for review and upload.
Same day
06
Internal link depth: 6 cornerstone pages buried 4+ clicks from homepage
Medium
Identified 8 existing pages from which a contextual internal link to each cornerstone page would be natural. Produced the anchor text and placement suggestion for each. No new content required.
1 sprint
Audit Cadence
Weekly
vs quarterly before automation — issues caught 12× earlier on average
Human Time Per Cycle
20 min
Review and approve — not execute. Down from 3–4 hours of manual work
Fix Drafts Produced
100%
Every flagged issue comes with a specific drafted fix — no blank-page implementation
Tools in Stack
4
Screaming Frog + SEMrush MCP + Python scripts + Claude Cowork
Before and After — What the Workflow Replaced
Before Automation
Manual weekly audit — what it actually cost
Open Screaming Frog, run crawl, wait 20–40 minutes, export to CSV, open in Excel, manually filter by issue type
Log into SEMrush separately, run keyword report, run backlink audit, export both, bring into the same spreadsheet
Cross-reference the three data sources manually — which technical issues affect ranking pages, which backlinks are new, what changed this week
Write up findings from scratch, format for whoever needs to see it, write fix recommendations in plain English with no implementation detail
Quarterly cadence in practice — weekly was the intention, quarterly was the reality because the process was too expensive to run every week
After Automation
Automated engine — what the 20 minutes actually covers
Screaming Frog crawl ran overnight, script processed the export, structured issue list is ready when you open your laptop
SEMrush data queried live within the Claude session — no separate login, no export, no reformatting
Cross-source analysis done by Claude — ranking pages cross-referenced against technical issues, backlink delta cross-referenced against keyword movement, automatically
Fix recommendations arrive with implementation detail — code snippets, redirect rules, schema blocks, rewritten copy — ready to hand to a developer or implement directly
Weekly cadence is now real, not aspirational — because the execution cost is 20 minutes of review, not 4 hours of manual work
“The engine doesn’t make better SEO decisions than a skilled SEO professional. It makes sure SEO decisions happen every week instead of every quarter — and that every flagged issue arrives with a drafted fix, not just a flag.”
On building the Python scripts: The prioritisation scripts were built with Claude’s help — described the Screaming Frog export format and the weighting logic I wanted, and Claude produced the initial script. The whole process took about 90 minutes for a working first version, including two revision loops to adjust the scoring weights. No prior Python experience required — the script logic is straightforward and Claude can explain and modify any part of it.
This use case is also available as
Same content, different formats
📝 This Article
💼 LinkedIn Post
🎙️ Podcast Episode
💼 LinkedIn Post
🎙️ Podcast Episode
Build Yours
Want to build this engine
for your own site?
for your own site?
Book a 15-minute call. Tell me your current SEO audit workflow — tools you have, what takes longest, where things fall through. I will map out which parts of this engine apply to your setup and what to build first.
Follow for more on marketing, business & strategy
I write about what it actually takes to build a real point of view — no recycled tips, no engagement-bait contrarianism.
Follow on LinkedIn →
Want AI actually working in your marketing?
Book a free 15-minute call and we'll map one AI workflow you can put to work in your marketing this week — no fluff, real tools.
Book an AI Workflow Call →
Read next: How I Built a Detailed Market Research Plan With AI — Qual + Quant
Designing a research plan from scratch usually means starting with a blank page, a literature review, and two weeks of scoping conversations. With AI…
Read the Essay →