<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[IndexerPal Blog]]></title><description><![CDATA[IndexerPal Blog]]></description><link>https://indexerpal.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Fri, 30 Oct 2026 01:58:41 GMT</lastBuildDate><atom:link href="https://indexerpal.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Getting a site indexed in Bing: Webmaster Tools, URL Submission and the API]]></title><description><![CDATA[Import your site from Search Console in a few minutes, then use Bing's own submission tools for the pages that matter. With curl examples and the quotas…]]></description><link>https://indexerpal.hashnode.dev/getting-a-site-indexed-in-bing-webmaster-tools-url-submission-and-the-api</link><guid isPermaLink="true">https://indexerpal.hashnode.dev/getting-a-site-indexed-in-bing-webmaster-tools-url-submission-and-the-api</guid><category><![CDATA[SEO]]></category><category><![CDATA[Bing]]></category><category><![CDATA[Web Development]]></category><category><![CDATA[Tutorial]]></category><dc:creator><![CDATA[IndexerPal]]></dc:creator><pubDate>Fri, 23 Oct 2026 13:20:50 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6ad3c4d2ea2a6916603a2ba7/9e1509d7-13a9-4ec8-b910-478edb676634.jpg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Most indexing advice is written for Google, and Bing gets a line at the end. That's a mistake if any of your traffic comes from Bing. Bing also gives site owners more direct control than Google does: a published daily submission quota, a URL Submission API and native IndexNow support.</p>
<p>This guide goes from zero to a site Bing knows about, then covers how to push individual URLs.</p>
<h2>Step 1: add the site, the fast way</h2>
<p>If your site is already verified in Google Search Console, import it. Bing's <a href="https://blogs.bing.com/webmaster/september-2019/Import-sites-from-Search-Console-to-Bing-Webmaster-Tools">import announcement</a> describes the flow:</p>
<ol>
<li>Sign in at <a href="https://www.bing.com/webmasters">bing.com/webmasters</a>.</li>
<li>Choose <strong>Import</strong> under "Import your sites from GSC".</li>
<li>Authorize access to your Google account.</li>
<li>Select the sites (and their sitemaps) and click <strong>Import</strong>.</li>
</ol>
<p>Imported sites are verified automatically. You can import up to 100 sites at a time, within a limit of 1,000 sites per Bing Webmaster Tools account. Bing re-validates ownership by syncing with Search Console, so if you later revoke that access, verify again with another method.</p>
<p>If you don't use Search Console, verify manually with an XML file at the root, a <code>&lt;meta&gt;</code> tag in the home page <code>&lt;head&gt;</code>, or a DNS CNAME record. Traffic data can take up to 48 hours to appear for a newly verified site.</p>
<h2>Step 2: give Bing a sitemap</h2>
<p>If you imported from Search Console, your sitemaps may already be there. Check <strong>Sitemaps</strong> in the left menu. Otherwise submit the full URL, for example <code>https://www.example.com/sitemap.xml</code>.</p>
<p>Add it to robots.txt as well, so every crawler finds it:</p>
<pre><code class="language-text">Sitemap: https://www.example.com/sitemap.xml
</code></pre>
<p>The Sitemaps page shows when Bing last read each file, how many URLs it found and any errors. A sitemap stuck on "Pending" for days usually means Bing couldn't fetch it: check that it returns <code>200</code> with an XML content type and isn't behind a bot challenge on your CDN.</p>
<h2>Step 3: check how Bing sees a page</h2>
<p>Bing's <a href="https://www.bing.com/webmasters/help/url-inspection-55a30305">URL Inspection</a> tool works like Google's. Paste a URL to see whether it's indexed, when it was crawled and any SEO or markup issues. "Live URL" fetches the page now, which is the quickest way to confirm Bingbot isn't blocked.</p>
<p>From the outside, <code>site:example.com/page</code> on bing.com gives a rough answer for any URL, including pages on sites you don't own.</p>
<h2>Step 4: submit URLs directly</h2>
<p>Bing gives you two built-in options. The limits below are from Bing's URL Submission help page.</p>
<table>
<thead>
<tr>
<th></th>
<th>URL Submission (dashboard)</th>
<th>URL Submission API</th>
</tr>
</thead>
<tbody><tr>
<td>Quota</td>
<td>10,000 URLs per domain per day</td>
<td>10,000 URLs per domain per day</td>
</tr>
<tr>
<td>Resets</td>
<td>Midnight UTC</td>
<td>Midnight UTC</td>
</tr>
<tr>
<td>Scope</td>
<td>Domain level, not per subdomain</td>
<td>Domain level</td>
</tr>
<tr>
<td>History</td>
<td>Last 1,000 submissions visible</td>
<td>Your own logs</td>
</tr>
<tr>
<td>Good for</td>
<td>A handful of pages after a launch</td>
<td>CMS hooks and publishing pipelines</td>
</tr>
</tbody></table>
<p>Bing notes that additional quota can be requested through support.</p>
<h3>Using the API</h3>
<p>Create an API key in Bing Webmaster Tools under <strong>Settings &gt; API access</strong>. Bing's URL Submission help page, checked on 23 Oct 2026, also describes OAuth 2.0 for setting up the API, so use whichever method your account offers. The examples below use the API key. Then check your quota:</p>
<pre><code class="language-bash">curl -s "https://ssl.bing.com/webmaster/api.svc/json/GetUrlSubmissionQuota?siteUrl=https://www.example.com&amp;apikey=$BING_API_KEY"
# {"d":{"__type":"UrlSubmissionQuota:#Microsoft.Bing.Webmaster.Api","DailyQuota":...,"MonthlyQuota":...}}
</code></pre>
<p>And submit a batch:</p>
<pre><code class="language-bash">curl -s -X POST \
  "https://ssl.bing.com/webmaster/api.svc/json/SubmitUrlbatch?apikey=$BING_API_KEY" \
  -H "Content-Type: application/json; charset=utf-8" \
  -d '{
        "siteUrl": "https://www.example.com",
        "urlList": [
          "https://www.example.com/blog/new-post/",
          "https://www.example.com/pricing/"
        ]
      }'
</code></pre>
<p>A small Python wrapper that respects the quota:</p>
<pre><code class="language-python">import os
import requests

API = "https://ssl.bing.com/webmaster/api.svc/json"
KEY = os.environ["BING_API_KEY"]
SITE = "https://www.example.com"


def remaining_quota() -&gt; int:
    r = requests.get(f"{API}/GetUrlSubmissionQuota",
                     params={"siteUrl": SITE, "apikey": KEY}, timeout=30)
    r.raise_for_status()
    return r.json()["d"]["DailyQuota"]


def submit(urls: list[str]) -&gt; None:
    allowed = min(len(urls), remaining_quota())
    if allowed == 0:
        print("No quota left today")
        return
    for i in range(0, allowed, 500):
        batch = urls[i:min(i + 500, allowed)]
        r = requests.post(f"{API}/SubmitUrlbatch", params={"apikey": KEY},
                          json={"siteUrl": SITE, "urlList": batch}, timeout=60)
        print(r.status_code, len(batch))
        r.raise_for_status()
</code></pre>
<p>The 500-per-call batch size is a cautious choice. Check Bing's API reference for the current maximum per call.</p>
<h2>Should you use the API or IndexNow?</h2>
<p>Bing's own help page answers this: Microsoft still supports the URL Submission API, but "adoption of IndexNow is strongly recommended where possible", and the API "may be deprecated in the future". The API only notifies Bing. IndexNow notifies every participating engine, including Yandex, Naver and Seznam, with one request. If you want to set it up, <a href="https://dev.to/indexerpal/indexnow-in-ci-notify-bing-and-yandex-on-every-deploy-f20">IndexNow in CI</a> shows how to send a notification on every deploy.</p>
<p>Practical advice:</p>
<ul>
<li><strong>New integration:</strong> use IndexNow.</li>
<li><strong>Existing API integration that works:</strong> keep it, and plan a move to IndexNow.</li>
<li><strong>Occasional manual submissions:</strong> the dashboard URL Submission tool is fine.</li>
</ul>
<p>Don't run both for the same URLs. You spend quota twice for the same signal.</p>
<h2>Why a submitted URL still isn't in Bing</h2>
<p>Submission only puts the URL in front of Bingbot. Bing then applies its <a href="https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a">Webmaster Guidelines</a>. The usual reasons a page stays out:</p>
<ul>
<li>Thin or duplicate content, or a page that mostly repeats other pages on the site.</li>
<li>Few or no internal links pointing to it.</li>
<li>A <code>noindex</code>, a robots.txt block for <code>bingbot</code>, or a canonical to another URL.</li>
<li>The CDN serving Bingbot a challenge page instead of the content.</li>
<li>A new domain with no links from elsewhere, which Bing may crawl slowly at first.</li>
</ul>
<p>Bing Webmaster Tools' <strong>Site Scan</strong> runs an on-demand SEO audit of your site and flags many of these issues. Run it once on a new site.</p>
<h2>A 30-minute setup checklist</h2>
<ul>
<li><input disabled type="checkbox" /> Site imported from Search Console or verified manually</li>
<li><input disabled type="checkbox" /> Sitemap submitted and showing a recent "Last read" date</li>
<li><input disabled type="checkbox" /> Sitemap listed in robots.txt</li>
<li><input disabled type="checkbox" /> One important URL checked in URL Inspection, live test passing</li>
<li><input disabled type="checkbox" /> IndexNow (or the API) wired into publishing, not both</li>
<li><input disabled type="checkbox" /> Site Scan run once, critical issues fixed</li>
</ul>
<hr />
<p><strong>Written by the IndexerPal team.</strong> We build tools for URL and backlink indexing. IndexerPal includes Bing and Yandex alongside Google in every credit; see <a href="https://indexerpal.com/how-it-works">how it works</a>.</p>
]]></content:encoded></item><item><title><![CDATA[Why your backlinks aren't indexed, and which ones are worth fixing]]></title><description><![CDATA[A link on a page Google never indexed is unlikely to do much. Here's how to find out why a linking page is missing, with a script that checks the page and…]]></description><link>https://indexerpal.hashnode.dev/why-your-backlinks-aren-t-indexed-and-which-ones-are-worth-fixing</link><guid isPermaLink="true">https://indexerpal.hashnode.dev/why-your-backlinks-aren-t-indexed-and-which-ones-are-worth-fixing</guid><category><![CDATA[SEO]]></category><category><![CDATA[Link Building]]></category><category><![CDATA[Google]]></category><category><![CDATA[Python]]></category><dc:creator><![CDATA[IndexerPal]]></dc:creator><pubDate>Thu, 22 Oct 2026 13:17:04 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6ad3c4d2ea2a6916603a2ba7/62f2806e-7ccd-4f8e-8028-e62a45c616ee.jpg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>You build or earn a link, it shows up in your backlink tool, and weeks later nothing has moved. One common reason is simple: Google hasn't indexed the page the link sits on. Backlink tools list links their own crawlers found. Whether Google ever kept that page is a separate question.</p>
<p>This post looks at why linking pages stay out of the index, how to diagnose a specific link in a few seconds, and how to decide which links are worth any effort at all.</p>
<h2>The page has to be in Google first</h2>
<p>Google discovers links by crawling pages. If a linking page was never crawled, Google hasn't seen your link. If it was crawled but not indexed, Google decided the page wasn't worth keeping, which says something about how much weight anything on it is likely to carry.</p>
<p>So "is my backlink indexed?" really means "is the page that links to me indexed?" That's the page to check, not your own.</p>
<h2>Why linking pages don't get indexed</h2>
<p>Some link types index well and some rarely do. The reasons are the same ones that apply to any page.</p>
<table>
<thead>
<tr>
<th>Link type</th>
<th>Typical problem</th>
</tr>
</thead>
<tbody><tr>
<td>Profile pages (forums, SaaS directories, social sites)</td>
<td>Thin pages, often <code>noindex</code> by the platform, few internal links pointing to them</td>
</tr>
<tr>
<td>Directory listings</td>
<td>Buried deep in pagination, near-identical to thousands of other listings</td>
</tr>
<tr>
<td>Guest posts on small blogs</td>
<td>The host blog itself is crawled rarely; new posts wait weeks</td>
</tr>
<tr>
<td>Press release syndication</td>
<td>The same text on dozens of sites; Google keeps one and drops the rest</td>
</tr>
<tr>
<td>Web 2.0 and free hosted blogs</td>
<td>New subdomains with no links in, and content Google sees as low value</td>
</tr>
<tr>
<td>Comment and forum links</td>
<td>Usually <code>nofollow</code> or <code>ugc</code>, on long threads that get crawled infrequently</td>
</tr>
<tr>
<td>Niche edits on old articles</td>
<td>Often fine: the page is already indexed, Google just needs to recrawl it</td>
</tr>
</tbody></table>
<p>The pattern: pages with real content, linked from other pages on the same site, index well. Pages that exist only to hold a link, or that look like many other pages, tend not to.</p>
<h2>Diagnose one link in under a minute</h2>
<p>Before spending money or time, check whether the linking page can be indexed at all, and whether your link is actually on it in a form Google can use. This script does both:</p>
<pre><code class="language-python"># check_backlink.py
import re
import sys
from urllib.parse import urlparse

import requests

# An honest user agent. Don't pretend to be Googlebot: many sites block
# fake Googlebot requests, and you'd be testing the block, not the page.
UA = "Mozilla/5.0 (compatible; backlink-check/1.0)"


def check(page_url: str, my_domain: str) -&gt; list[str]:
    notes = []
    r = requests.get(page_url, headers={"User-Agent": UA}, timeout=30,
                     allow_redirects=True)
    if r.history:
        notes.append(f"redirects to {r.url}")
    if r.status_code != 200:
        notes.append(f"status {r.status_code}")
        return notes

    html = r.text
    if "noindex" in r.headers.get("X-Robots-Tag", "").lower():
        notes.append("X-Robots-Tag: noindex")
    if re.search(r'&lt;meta[^&gt;]+name=["\'](robots|googlebot)["\'][^&gt;]+noindex', html, re.I):
        notes.append("meta robots noindex")

    canon = re.search(r'&lt;link[^&gt;]+rel=["\']canonical["\'][^&gt;]+href=["\']([^"\']+)', html, re.I)
    if canon and canon.group(1).rstrip("/") != r.url.rstrip("/"):
        notes.append(f"canonical points to {canon.group(1)}")

    links = re.findall(
        r'&lt;a\b[^&gt;]*href=["\']([^"\']*' + re.escape(my_domain) + r'[^"\']*)["\'][^&gt;]*&gt;',
        html, re.I)
    if not links:
        notes.append("link to your domain NOT in the raw HTML (JS-inserted or removed)")
    for tag in re.findall(r'&lt;a\b[^&gt;]*' + re.escape(my_domain) + r'[^&gt;]*&gt;', html, re.I):
        rel = re.search(r'rel=["\']([^"\']+)', tag, re.I)
        if rel:
            notes.append(f"link rel={rel.group(1)}")

    words = len(re.sub(r"&lt;[^&gt;]+&gt;", " ", html).split())
    if words &lt; 300:
        notes.append(f"thin page (~{words} words)")
    return notes or ["no blocking issue found"]


if __name__ == "__main__":
    domain = sys.argv[1]               # yoursite.com
    for line in sys.stdin:
        url = line.strip()
        if url:
            print(url, "|", "; ".join(check(url, domain)))
</code></pre>
<pre><code class="language-bash">cat linking-pages.txt | python check_backlink.py yoursite.com
</code></pre>
<p>Also check robots.txt by hand for the host (<code>https://host/robots.txt</code>), since the script doesn't parse it. The word count is crude: it includes navigation text, so treat it as a warning, not a verdict.</p>
<p>What the results mean:</p>
<ul>
<li><strong><code>noindex</code>, a canonical elsewhere, or a robots.txt block:</strong> this page won't be indexed by any method. Ask for the link on a different page, or let it go.</li>
<li><strong>Link not in the raw HTML:</strong> it may be inserted by JavaScript. Google can render JavaScript, but a link that only appears after rendering is a weaker bet. If it's missing entirely, the link was removed.</li>
<li><strong><code>rel="nofollow"</code>, <code>ugc</code> or <code>sponsored</code>:</strong> Google treats these as hints rather than strict rules, but don't expect them to pass the same value as a plain link.</li>
<li><strong>No blocking issue, but not indexed:</strong> a discovery problem. This is the case worth working on.</li>
</ul>
<h2>Getting a clean page indexed</h2>
<p>For pages that pass the checks above but still aren't in Google:</p>
<ol>
<li><strong>Ask the site owner for an internal link.</strong> A link to the guest post from the blog's home page, a category page or a related article is the strongest signal you can get on someone else's site.</li>
<li><strong>Ask whether the page is in their sitemap.</strong> Many small blogs have sitemaps that exclude certain post types or haven't regenerated in months.</li>
<li><strong>Give it a real reason to be crawled.</strong> Linking to the page from your own site, newsletter or social profiles gives crawlers another path to it.</li>
<li><strong>Wait a fair amount of time.</strong> On a small blog, a few weeks is normal. Check again before doing anything else.</li>
<li><strong>Use an indexing service</strong> for the ones that matter and still haven't moved. You can't request indexing in Search Console for a site you don't own, so this is the remaining option. IndexerPal is one such service: it works on any public URL, checks each one in Google afterwards, and its instant mode indexes pages in 2–5 minutes on average (IndexerPal's own measurement). It can't override a <code>noindex</code> or a thin page, which is why the checks come first.</li>
</ol>
<h2>Decide which links are worth the effort</h2>
<p>Not every unindexed link deserves attention. A quick triage:</p>
<table>
<thead>
<tr>
<th>Link</th>
<th>Worth fixing?</th>
</tr>
</thead>
<tbody><tr>
<td>Editorial link on a relevant site, page has real content</td>
<td>Yes, always</td>
</tr>
<tr>
<td>Paid guest post, page clean but not indexed after a month</td>
<td>Yes, ask the host for an internal link first</td>
</tr>
<tr>
<td>Directory or citation for local SEO</td>
<td>Yes for the main ones; the listing also matters to users</td>
</tr>
<tr>
<td>Profile page with <code>noindex</code></td>
<td>No, nothing will change it</td>
</tr>
<tr>
<td>Syndicated press release copy number 30</td>
<td>No, Google keeps one copy</td>
</tr>
<tr>
<td>Comment link on a thread from 2014</td>
<td>No</td>
</tr>
</tbody></table>
<p>Also keep Google's <a href="https://developers.google.com/search/docs/essentials/spam-policies#link-spam">link spam policies</a> in mind. Buying links that pass ranking credit is against them whatever happens with indexing. Getting a page indexed doesn't change what kind of link it is. Google's most recent spam update on the <a href="https://status.search.google.com/products/rGHU1u87FJnkP6W2GwMi/history">Search Status Dashboard</a> is the September 2026 spam update, which started on 24 Sep 2026 and took 13 days to roll out, so this is not a theoretical risk.</p>
<h2>A monthly routine</h2>
<ul>
<li>Export new linking pages from your backlink tool.</li>
<li>Check which are in Google's index (a bulk checker saves hours here).</li>
<li>Run the script above on the ones that aren't.</li>
<li>Fix, request or drop each one using the tables above.</li>
<li>Recheck the fixed ones next month.</li>
</ul>
<p>Most of the value is in the second and third steps: finding out which links are actually missing and why, before spending anything on them.</p>
<hr />
<p><strong>Written by the IndexerPal team.</strong> We build tools for URL and backlink indexing. To get clean linking pages crawled and checked in Google, see the <a href="https://indexerpal.com/backlink-indexer">IndexerPal backlink indexer</a>.</p>
]]></content:encoded></item><item><title><![CDATA[XML sitemaps that actually help indexing: split, date honestly, and audit]]></title><description><![CDATA[A sitemap is a hint, not a command. Here's how to make it a hint Google trusts, and how to use it as a diagnostic tool for pages that aren't indexed.]]></description><link>https://indexerpal.hashnode.dev/xml-sitemaps-that-actually-help-indexing-split-date-honestly-and-audit</link><guid isPermaLink="true">https://indexerpal.hashnode.dev/xml-sitemaps-that-actually-help-indexing-split-date-honestly-and-audit</guid><category><![CDATA[SEO]]></category><category><![CDATA[Google]]></category><category><![CDATA[Python]]></category><category><![CDATA[Web Development]]></category><dc:creator><![CDATA[IndexerPal]]></dc:creator><pubDate>Wed, 21 Oct 2026 13:18:38 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6ad3c4d2ea2a6916603a2ba7/d6a7f0dd-7aa8-4da4-a687-42fa88691a1f.jpg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Most sitemaps are generated once by a plugin and never looked at again. They list redirects, <code>noindex</code> pages and every tag archive, with today's date as <code>lastmod</code> on all of them. Google still reads them, but it has little reason to trust them.</p>
<p>A sitemap built with a bit of care does two jobs: it helps Google find and recrawl the right pages, and it lets you see in Search Console exactly which groups of pages aren't getting indexed. This post covers both.</p>
<h2>What Google says a sitemap is</h2>
<p>From Google's <a href="https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap">sitemap guide</a>:</p>
<ul>
<li>Submitting a sitemap "is merely a hint: it doesn't guarantee that Google will download the sitemap or use the sitemap for crawling URLs on the site."</li>
<li>Each sitemap file is limited to <strong>50 MB uncompressed or 50,000 URLs</strong>. Above that, split it and optionally use a sitemap index file.</li>
<li>Use <strong>fully qualified, absolute URLs</strong>. Google crawls them exactly as listed.</li>
<li>Include the URLs you <strong>want to see in search results</strong>, meaning the canonical versions.</li>
<li>Google <strong>ignores <code>&lt;priority&gt;</code> and <code>&lt;changefreq&gt;</code></strong>.</li>
<li>Google uses <code>&lt;lastmod&gt;</code> "if it's consistently and verifiably accurate", and it should reflect the last <strong>significant</strong> change: main content, structured data or links. A new copyright year doesn't count.</li>
</ul>
<p>That last point is the one most sites get wrong.</p>
<h2>Rule 1: only list URLs you want indexed</h2>
<p>Every URL in the sitemap should pass all of these:</p>
<table>
<thead>
<tr>
<th>Check</th>
<th>Why</th>
</tr>
</thead>
<tbody><tr>
<td>Returns <code>200</code></td>
<td>Redirects and errors waste the crawl and blur your reports</td>
</tr>
<tr>
<td>No <code>noindex</code> (meta tag or <code>X-Robots-Tag</code>)</td>
<td>You're sending two contradicting signals</td>
</tr>
<tr>
<td>Self-referencing canonical</td>
<td>A URL that canonicalizes elsewhere isn't the one you want indexed</td>
</tr>
<tr>
<td>Not blocked in robots.txt</td>
<td>Google can't crawl it to confirm anything</td>
</tr>
<tr>
<td>Has content worth indexing</td>
<td>Empty filters, thin tag pages and internal search results dilute the file</td>
</tr>
</tbody></table>
<p>Parameter variations (<code>?sort=price</code>, <code>?utm_source=...</code>) don't belong in a sitemap at all.</p>
<h2>Rule 2: make <code>lastmod</code> true</h2>
<p>Set <code>lastmod</code> from the date the content actually changed, not from the build time. If your CMS only stores one <code>updated_at</code> that changes on every save, add a separate <code>content_updated_at</code> that you only set when the body, title, structured data or links change.</p>
<p>A date-only value (<code>2026-10-14</code>) is fine. If you include a time, include the timezone (<code>2026-10-14T09:12:00+00:00</code>).</p>
<p>When <code>lastmod</code> is honest, it becomes useful: Google can see which pages changed since its last crawl and prioritize those.</p>
<h2>Rule 3: split by page type</h2>
<p>One giant file works, but splitting by type gives you a diagnostic tool for free. In Search Console's Page indexing report, you can filter by sitemap and see indexed vs not indexed for each one.</p>
<pre><code class="language-text">sitemap_index.xml
â”œâ”€â”€ sitemap-pages.xml        (home, pricing, about, features)
â”œâ”€â”€ sitemap-blog.xml         (articles)
â”œâ”€â”€ sitemap-products-1.xml   (first 50,000 products)
â”œâ”€â”€ sitemap-products-2.xml
â””â”€â”€ sitemap-categories.xml
</code></pre>
<p>If 95% of <code>sitemap-blog.xml</code> is indexed and 30% of <code>sitemap-categories.xml</code> is, you know where to look without reading thousands of rows.</p>
<h2>A generator that follows the rules</h2>
<p>This Python script builds a sitemap index and type-based sitemaps from a database. Adapt the query to your schema.</p>
<pre><code class="language-python"># build_sitemaps.py
from datetime import timezone
from xml.sax.saxutils import escape
import gzip
import pathlib

BASE = "https://www.example.com"
OUT = pathlib.Path("public")
LIMIT = 50_000

QUERIES = {
    "pages": "select path, content_updated_at from pages where indexable",
    "blog": "select path, content_updated_at from posts where published and indexable",
    "products": "select path, content_updated_at from products where in_catalog and indexable",
}


def url_entry(path, updated):
    loc = escape(f"{BASE}{path}")
    lastmod = updated.astimezone(timezone.utc).strftime("%Y-%m-%d")
    return f"&lt;url&gt;&lt;loc&gt;{loc}&lt;/loc&gt;&lt;lastmod&gt;{lastmod}&lt;/lastmod&gt;&lt;/url&gt;"


def write(name, rows):
    body = "".join(url_entry(p, u) for p, u in rows)
    xml = ('&lt;?xml version="1.0" encoding="UTF-8"?&gt;'
           '&lt;urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"&gt;'
           f"{body}&lt;/urlset&gt;")
    (OUT / name).write_bytes(gzip.compress(xml.encode("utf-8")))
    return max(u for _, u in rows)


def build(db):
    index = []
    for kind, sql in QUERIES.items():
        rows = db.execute(sql).fetchall()
        for n, i in enumerate(range(0, len(rows), LIMIT), start=1):
            chunk = rows[i:i + LIMIT]
            name = f"sitemap-{kind}-{n}.xml.gz"
            newest = write(name, chunk)
            index.append((name, newest))

    entries = "".join(
        f"&lt;sitemap&gt;&lt;loc&gt;{BASE}/{name}&lt;/loc&gt;"
        f"&lt;lastmod&gt;{newest.astimezone(timezone.utc):%Y-%m-%d}&lt;/lastmod&gt;&lt;/sitemap&gt;"
        for name, newest in index
    )
    (OUT / "sitemap_index.xml").write_text(
        '&lt;?xml version="1.0" encoding="UTF-8"?&gt;'
        '&lt;sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"&gt;'
        f"{entries}&lt;/sitemapindex&gt;",
        encoding="utf-8",
    )
</code></pre>
<p>Notes on the choices:</p>
<ul>
<li><code>escape()</code> handles <code>&amp;</code> in URLs, which otherwise breaks the XML.</li>
<li>Gzip is allowed; the 50 MB limit applies to the uncompressed size.</li>
<li>The index's <code>lastmod</code> per file is the newest <code>lastmod</code> inside it, so Google can skip files that haven't changed.</li>
</ul>
<h2>Submit it in the places Google reads</h2>
<p>Google lists three ways in its guide:</p>
<ol>
<li><strong>The Sitemaps report</strong> in Search Console. Submit <code>sitemap_index.xml</code> once; the child files appear under it.</li>
<li><strong>robots.txt</strong>: <code>Sitemap: https://www.example.com/sitemap_index.xml</code>. Every crawler that reads robots.txt finds it.</li>
<li><strong>The Search Console API</strong>, if you manage many properties.</li>
</ol>
<p>The old "ping" URL is gone. Google <a href="https://developers.google.com/search/blog/2023/06/sitemaps-lastmod-ping">deprecated the sitemap ping endpoint in 2023</a> and pointed to accurate <code>lastmod</code> values instead. Scripts that still call it get nothing back.</p>
<h2>Audit the sitemap before Google does</h2>
<p>Run this weekly, or in CI on a sample. It flags URLs that break Rule 1:</p>
<pre><code class="language-python"># audit_sitemap.py
import gzip
import random
import re
import sys
import xml.etree.ElementTree as ET

import requests

NS = {"s": "http://www.sitemaps.org/schemas/sitemap/0.9"}


def locs(url):
    raw = requests.get(url, timeout=30).content
    if url.endswith(".gz"):
        raw = gzip.decompress(raw)
    root = ET.fromstring(raw)
    if root.tag.endswith("sitemapindex"):
        for loc in root.findall("s:sitemap/s:loc", NS):
            yield from locs(loc.text.strip())
    else:
        for loc in root.findall("s:url/s:loc", NS):
            yield loc.text.strip()


def check(url):
    r = requests.get(url, allow_redirects=False, timeout=30,
                     headers={"User-Agent": "sitemap-audit"})
    problems = []
    if r.status_code != 200:
        problems.append(f"status {r.status_code}")
    if "noindex" in r.headers.get("X-Robots-Tag", "").lower():
        problems.append("X-Robots-Tag noindex")
    if re.search(r'&lt;meta[^&gt;]+name=["\']robots["\'][^&gt;]+noindex', r.text, re.I):
        problems.append("meta noindex")
    m = re.search(r'&lt;link[^&gt;]+rel=["\']canonical["\'][^&gt;]+href=["\']([^"\']+)', r.text, re.I)
    if m and m.group(1).rstrip("/") != url.rstrip("/"):
        problems.append(f"canonical -&gt; {m.group(1)}")
    return problems


if __name__ == "__main__":
    urls = list(locs(sys.argv[1]))
    sample = random.sample(urls, min(200, len(urls)))
    bad = {u: p for u in sample if (p := check(u))}
    for u, p in bad.items():
        print(u, "; ".join(p))
    print(f"{len(bad)} of {len(sample)} sampled URLs have problems")
</code></pre>
<p>The regexes are deliberately simple. They catch the common cases, not every HTML variant.</p>
<h2>Reading the Sitemaps report</h2>
<table>
<thead>
<tr>
<th>What you see</th>
<th>Likely meaning</th>
</tr>
</thead>
<tbody><tr>
<td>Status "Success", discovered URLs match your count</td>
<td>Google read the whole file</td>
</tr>
<tr>
<td>"Couldn't fetch"</td>
<td>Wrong URL, a non-200 response, or a bot challenge on the CDN</td>
</tr>
<tr>
<td>"Has errors"</td>
<td>Invalid XML, relative URLs or a bad date format</td>
</tr>
<tr>
<td>Discovered count far below your count</td>
<td>Truncated file, or URLs outside the sitemap's directory</td>
</tr>
<tr>
<td>"Last read" weeks old on an active site</td>
<td>Google sees little change; check that <code>lastmod</code> reflects real updates</td>
</tr>
</tbody></table>
<p>A sitemap gets URLs found. Whether they're indexed still depends on the pages themselves, which is where the per-sitemap view in the Page indexing report earns its keep.</p>
<hr />
<p><strong>Written by the IndexerPal team.</strong> We build tools for URL and backlink indexing. IndexerPal can import a sitemap directly and verifies each URL it sends in Google; see <a href="https://indexerpal.com/website-indexing">website indexing</a>.</p>
]]></content:encoded></item><item><title><![CDATA[A new website's first 14 days in Google: a day-by-day indexing plan]]></title><description><![CDATA[What to check on launch day, what to submit on day 1, and what to look at on days 7 and 14 so a new domain gets crawled and indexed.]]></description><link>https://indexerpal.hashnode.dev/a-new-website-s-first-14-days-in-google-a-day-by-day-indexing-plan</link><guid isPermaLink="true">https://indexerpal.hashnode.dev/a-new-website-s-first-14-days-in-google-a-day-by-day-indexing-plan</guid><category><![CDATA[SEO]]></category><category><![CDATA[Google]]></category><category><![CDATA[search console]]></category><category><![CDATA[Web Development]]></category><dc:creator><![CDATA[IndexerPal]]></dc:creator><pubDate>Tue, 20 Oct 2026 13:21:21 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6ad3c4d2ea2a6916603a2ba7/bfdafbad-b65b-4cc1-9bee-6619a1ee4024.jpg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>A brand-new domain starts with nothing. Google doesn't know it exists, no other site links to it, and Search Console has no data. The first two weeks decide how quickly that changes, and most of what matters is done in the first two days.</p>
<p>This is the plan we'd follow for any new site, from the day before launch to day 14. It assumes you own the site and can edit its code or settings. Google's own <a href="https://developers.google.com/search/docs/essentials">Search Essentials</a> and <a href="https://developers.google.com/search/docs/fundamentals/seo-starter-guide">SEO starter guide</a> are the background reading.</p>
<h2>Day 0: the launch-day check that saves weeks</h2>
<p>The most common reason a new site isn't indexed after a month is a setting left over from staging. Before you announce anything, open these four things on the live domain:</p>
<ol>
<li><p><code>https://yourdomain.com/robots.txt</code>. It should not contain <code>Disallow: /</code> for all user agents. Staging sites often do.</p>
</li>
<li><p>The page source of the home page. Search for <code>noindex</code>. Many CMSs have a "discourage search engines" checkbox that adds it to every page.</p>
</li>
<li><p>The response headers: <code>curl -sI https://yourdomain.com/</code>. Look for <code>X-Robots-Tag: noindex</code>, which some hosts add to preview domains.</p>
</li>
<li><p>The canonical tag on a few pages. It should point to the live domain, not to <code>staging.</code> or a preview URL.</p>
</li>
</ol>
<p>Also make sure <code>http://</code>, <code>https://</code>, <code>www</code> and non-<code>www</code> all redirect with a single 301 to the one version you want. Decide this now; changing it later means waiting for Google to process the move.</p>
<h2>Day 1: tell Google and Bing the site exists</h2>
<p><strong>Search Console.</strong> Add a <strong>Domain property</strong> (verified with a DNS TXT record). It covers every subdomain and protocol in one place, so you won't miss data because Google picked a variant you didn't add.</p>
<p><strong>Sitemap.</strong> Submit it in the Sitemaps report and add a <code>Sitemap:</code> line to robots.txt. On a new site, keep it to the pages you actually want found: home, main sections, key product or service pages, and published articles. Leave out tag archives, empty categories and search results.</p>
<p><strong>URL Inspection.</strong> Inspect the home page, run "Test live URL", and if it passes, request indexing. Do the same for three to five of your most important pages. That's enough. Google's <a href="https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl">recrawl guide</a> notes there's a quota and that repeated requests for the same URL don't speed things up. For the full picture of what the button does and why it sometimes fails, see our guide to <a href="https://indexerpal.hashnode.dev/request-indexing-in-search-console-what-the-button-does-its-quota-and-why-it-fails">Request Indexing in Search Console</a>.</p>
<p><strong>Bing Webmaster Tools.</strong> Use "Import from Google Search Console". It takes two minutes and verifies the site automatically.</p>
<h2>Days 2 to 3: give crawlers a path in</h2>
<p>Google mainly finds new URLs by following links from pages it already crawls. A sitemap helps, but a new domain with zero links from anywhere is easy to deprioritize.</p>
<p>Places that are crawled often and where a link to a new site is legitimate:</p>
<ul>
<li><p>Your own profiles: company page on LinkedIn, GitHub organization, social accounts.</p>
</li>
<li><p>Partners, suppliers or clients who can mention you on a page that already exists.</p>
</li>
<li><p>A launch post on a community where your audience actually is.</p>
</li>
<li><p>Relevant, curated directories for your industry or location. Skip the mass-submission kind.</p>
</li>
</ul>
<p>You're not building links for ranking here. You're making sure a crawler following ordinary links can reach your home page.</p>
<h2>Days 3 to 5: fix internal linking before you add more pages</h2>
<p>Check that every page you want indexed is linked from at least one other page with a normal <code>&lt;a href&gt;</code> link. Pages that only appear in the sitemap, or that are reachable only through a search box or a JavaScript menu that renders late, are the ones that stay "Discovered, currently not indexed".</p>
<p>A simple structure for a small site:</p>
<table>
<thead>
<tr>
<th>Page type</th>
<th>Linked from</th>
</tr>
</thead>
<tbody><tr>
<td>Main sections</td>
<td>Header navigation on every page</td>
</tr>
<tr>
<td>Key pages (pricing, contact, main services)</td>
<td>Navigation and the home page</td>
</tr>
<tr>
<td>Articles</td>
<td>A blog index, plus links from related articles</td>
</tr>
<tr>
<td>Product or service pages</td>
<td>Their section page, and the home page for the important ones</td>
</tr>
</tbody></table>
<h2>Day 7: first look at the data</h2>
<p>By now Search Console usually has some data. Look at three places:</p>
<p><strong>Page indexing report.</strong> Filter by your sitemap. Expect a mix: some pages indexed, some "Discovered, currently not indexed", maybe some "Crawled, currently not indexed". On a week-old domain, that's normal.</p>
<p><strong>Crawl stats</strong> (Settings &gt; Crawl stats). Confirm Googlebot is visiting and getting mostly <code>200</code> responses. A spike in <code>5xx</code> or timeouts here means your hosting is the bottleneck.</p>
<p><strong>URL Inspection for your key pages.</strong> If the home page still isn't indexed after a week, re-check the Day 0 list. A forgotten <code>noindex</code> is far more likely than a slow Google.</p>
<p>What not to do on day 7: request indexing for every page again, rewrite the whole site, or buy a batch of low-quality links. None of those make a new site more trustworthy.</p>
<h2>Days 8 to 13: publish, link, wait</h2>
<p>Use the second week to add the content that gives Google a reason to keep the site: pages that answer the questions your customers ask. Each new page should be linked from an existing one the day it goes live, and appear in the sitemap with an accurate <code>lastmod</code>.</p>
<p>If you publish regularly, set up IndexNow (it covers Bing, Yandex and others, though not Google) and keep the sitemap regenerating automatically.</p>
<h2>Day 14: review</h2>
<p>Make a short list:</p>
<table>
<thead>
<tr>
<th>Question</th>
<th>Where to look</th>
</tr>
</thead>
<tbody><tr>
<td>Is the home page indexed?</td>
<td>URL Inspection</td>
</tr>
<tr>
<td>What share of sitemap URLs are indexed?</td>
<td>Page indexing report, filtered by sitemap</td>
</tr>
<tr>
<td>Which important pages aren't indexed, and with which status?</td>
<td>Page indexing report</td>
</tr>
<tr>
<td>Is Googlebot crawling without errors?</td>
<td>Crawl stats</td>
</tr>
<tr>
<td>Are any impressions showing yet?</td>
<td>Performance report</td>
</tr>
</tbody></table>
<p>Google's guide says crawling can take "anywhere from a few days to a few weeks", so a partly indexed site at day 14 isn't a failure. A site where nothing at all is indexed after two weeks almost always has a technical block. Go back to Day 0.</p>
<h2>Where outside help fits</h2>
<p>Most new sites get indexed with the steps above and some patience. An indexing service can help when a handful of clean, important pages are still waiting after the first week and the launch can't wait longer. It won't fix a <code>noindex</code>, a staging robots.txt or thin pages, so do the checks first.</p>
<hr />
<p><strong>Written by the IndexerPal team.</strong> We build tools for URL and backlink indexing. For new pages that need to be in Google sooner, see <a href="https://indexerpal.com/website-indexing">IndexerPal website indexing</a>.</p>
]]></content:encoded></item><item><title><![CDATA[Request Indexing in Search Console: what the button does, its quota, and why it fails]]></title><description><![CDATA[A field guide to the one indexing control Google gives every site owner, including the error messages and what to do after you press it.]]></description><link>https://indexerpal.hashnode.dev/request-indexing-in-search-console-what-the-button-does-its-quota-and-why-it-fails</link><guid isPermaLink="true">https://indexerpal.hashnode.dev/request-indexing-in-search-console-what-the-button-does-its-quota-and-why-it-fails</guid><category><![CDATA[SEO]]></category><category><![CDATA[Google]]></category><category><![CDATA[search console]]></category><category><![CDATA[Web Development]]></category><dc:creator><![CDATA[IndexerPal]]></dc:creator><pubDate>Mon, 19 Oct 2026 13:25:42 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6ad3c4d2ea2a6916603a2ba7/80150980-939b-4e12-a2df-bc4c840f3c70.jpg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>"Request indexing" is the most-pressed button in Google Search Console and probably the least understood. People press it ten times for the same URL, hit a quota message, and conclude the tool is broken. It isn't. It does one specific thing, with limits Google documents only loosely.</p>
<p>This guide covers what happens when you press it, the order of steps that avoids wasted requests, what each message means, and when another method is the better choice.</p>
<h2>What the button actually does</h2>
<p>From Google's <a href="https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl">recrawl guide</a>:</p>
<ul>
<li>It asks Google to crawl one URL. It doesn't add the page to the index directly.</li>
<li>"Requesting a crawl does not guarantee that inclusion in search results will happen instantly or even at all."</li>
<li>"There's a quota for submitting individual URLs and requesting a recrawl multiple times for the same URL won't get it crawled any faster."</li>
<li>You must be an owner or full user of the property, and you can't request indexing for URLs you don't manage.</li>
</ul>
<p>Google doesn't publish the size of the quota. Treat it as a small daily allowance per property and spend it on pages that matter.</p>
<h2>The four-step routine</h2>
<p>Pressing the button first and reading later is how quota gets wasted. Use this order instead.</p>
<p><strong>1. Inspect the URL.</strong> Paste the full URL, exactly as it appears in your sitemap and canonical tag, into the search bar at the top of Search Console. Trailing slashes, <code>http</code> vs <code>https</code> and <code>www</code> matter.</p>
<p><strong>2. Read the indexed result.</strong> It shows what Google stored the last time it crawled: whether the URL is on Google, the last crawl date, and the canonical Google chose. If "Google-selected canonical" is a different URL, requesting indexing for this one won't help. Fix the canonical signals first.</p>
<p><strong>3. Run "Test live URL".</strong> This fetches the page now. It tells you whether Google can crawl it today, whether <code>noindex</code> is present, and how the rendered HTML looks. If the live test shows a problem, the request will be rejected anyway.</p>
<p><strong>4. Request indexing.</strong> Only once the live test passes. Then leave it alone for several days.</p>
<h2>What each message means</h2>
<table>
<thead>
<tr>
<th>What you see</th>
<th>What it means</th>
<th>What to do</th>
</tr>
</thead>
<tbody><tr>
<td>"URL is on Google"</td>
<td>The page is indexed</td>
<td>Request only if you changed it significantly and want a fresh crawl</td>
</tr>
<tr>
<td>"URL is on Google, but has issues"</td>
<td>Indexed, with enhancement or structured data warnings</td>
<td>Fix the listed items, then request</td>
</tr>
<tr>
<td>"URL is not on Google"</td>
<td>Not indexed; the reason is listed under Page indexing</td>
<td>Read the reason before pressing anything</td>
</tr>
<tr>
<td>"Indexing requested"</td>
<td>The URL is in a priority crawl queue</td>
<td>Wait. Don't request again for the same URL</td>
</tr>
<tr>
<td>"Indexing request rejected"</td>
<td>The live test found an indexability issue</td>
<td>Run "Test live URL", fix what it reports</td>
</tr>
<tr>
<td>"Quota exceeded"</td>
<td>Your daily allowance for the property is used</td>
<td>Try again the next day, or use a sitemap</td>
</tr>
<tr>
<td>"URL is not available to Google" in the live test</td>
<td>Blocked, error or login wall</td>
<td>Check robots.txt, status code and authentication</td>
</tr>
</tbody></table>
<h2>Why requests fail or seem to do nothing</h2>
<p><strong>The page isn't indexable.</strong> <code>noindex</code> in the HTML or an <code>X-Robots-Tag</code> header, a robots.txt block, a 4xx or 5xx response, or a canonical pointing to another URL. The live test catches most of these. Check the response headers too, since an <code>X-Robots-Tag: noindex</code> added by a CDN or framework doesn't show in the page source.</p>
<p><strong>Google crawled it and chose not to index it.</strong> The request worked: Google fetched the page. The status then often becomes "Crawled, currently not indexed", which is a judgement about the page, not a technical failure. Another request won't change that judgement.</p>
<p><strong>You requested the wrong variant.</strong> You inspected <code>https://example.com/page</code> but your canonical says <code>https://www.example.com/page/</code>. Google treats them as different URLs.</p>
<p><strong>It's too soon.</strong> Google's guide says crawling can take "anywhere from a few days to a few weeks". The indexed result in URL Inspection updates only after a new crawl is processed.</p>
<p><strong>The page is on a site you don't own.</strong> You can't request indexing for it. This is the usual case for backlinks.</p>
<h2>When to use it, and when not to</h2>
<p>Good uses:</p>
<ul>
<li>A new page that matters (a launch, a pricing page, a key article) on an established site.</li>
<li>A page you fixed after a <code>noindex</code> or error, to get the corrected version crawled.</li>
<li>A significant update to an important page, such as new prices or a corrected fact.</li>
</ul>
<p>Poor uses:</p>
<ul>
<li>Every new blog post on a site that publishes daily. Your sitemap and internal links already cover these.</li>
<li>Hundreds of URLs after a migration. Submit the sitemap in the Sitemaps report and let Google work through it.</li>
<li>Pages Google has crawled and decided not to index. Improve the page first.</li>
<li>Pressing it again because nothing has happened after a day.</li>
</ul>
<h2>Request indexing vs Validate fix</h2>
<p>The Page indexing report has its own <strong>Validate fix</strong> button. It's a different tool:</p>
<table>
<thead>
<tr>
<th></th>
<th>Request indexing</th>
<th>Validate fix</th>
</tr>
</thead>
<tbody><tr>
<td>Scope</td>
<td>One URL</td>
<td>All URLs with one issue type</td>
</tr>
<tr>
<td>Where</td>
<td>URL Inspection</td>
<td>Page indexing report, inside an issue</td>
</tr>
<tr>
<td>What it does</td>
<td>Queues a crawl of that URL</td>
<td>Starts a validation run that recrawls affected URLs over time</td>
</tr>
<tr>
<td>Feedback</td>
<td>Indexed result after a recrawl</td>
<td>Validation status: started, passed or failed, with an email</td>
</tr>
<tr>
<td>Use when</td>
<td>A single important page</td>
<td>You fixed a template bug affecting many pages</td>
</tr>
</tbody></table>
<p>After fixing a sitewide issue, Validate fix is the right button, and you don't have to request each URL one by one.</p>
<h2>A simple log for important requests</h2>
<p>If your team requests indexing for important pages, keep a short log. It stops duplicate requests and tells you whether the button is helping.</p>
<pre><code class="language-text">date        url                                   reason             live test   result on day 3 / 7
2026-10-14  https://www.example.com/pricing/      new prices         pass        indexed / new version shown
2026-10-14  https://www.example.com/blog/launch/  new page           pass        not indexed / indexed
2026-10-15  https://www.example.com/docs/api/     removed noindex    pass        indexed / indexed
</code></pre>
<p>After a few weeks you'll have your own data on how long requests take on your site.</p>
<h2>When you need more than the button</h2>
<p>The button covers a few URLs on sites you've verified. Two situations fall outside it:</p>
<ul>
<li><strong>More pages than the quota allows,</strong> for example a product launch with fifty pages that all need to be crawled the same day. A sitemap is the official route, at Google's pace.</li>
<li><strong>URLs on sites you don't own,</strong> such as backlinks, guest posts or listings. Search Console can't see or request them.</li>
</ul>
<p>In both cases, outside indexing services exist. IndexerPal is one: its instant mode indexes pages in 2–5 minutes on average (IndexerPal's own measurement), and it checks each URL in Google afterwards. No service can make Google index a page it has decided not to keep, so the indexability checks above apply either way.</p>
<hr />
<p><strong>Written by the IndexerPal team.</strong> We build tools for URL and backlink indexing. For pages that need to be in Google quickly, see <a href="https://indexerpal.com/instant-indexing">IndexerPal instant indexing</a>.</p>
]]></content:encoded></item><item><title><![CDATA[Is this URL indexed? Five ways to find out]]></title><description><![CDATA[Pick the method by who owns the page and how many URLs you have, then read the result without fooling yourself.]]></description><link>https://indexerpal.hashnode.dev/is-this-url-indexed-five-ways-to-find-out</link><guid isPermaLink="true">https://indexerpal.hashnode.dev/is-this-url-indexed-five-ways-to-find-out</guid><category><![CDATA[SEO]]></category><category><![CDATA[Web Development]]></category><category><![CDATA[Google]]></category><category><![CDATA[Python]]></category><dc:creator><![CDATA[IndexerPal]]></dc:creator><pubDate>Sat, 17 Oct 2026 18:58:59 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6ad3c4d2ea2a6916603a2ba7/7df24b0f-a6ed-4319-9748-c944a9f09d99.jpg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Checking whether a page is in Google looks like a yes or no task. The answer you get depends on the tool you use and on whether you own the site. This post covers the five methods that exist and includes a small Python script for the API route. It ends with the mistakes that make a page look missing when it isn't, or indexed when it's not.</p>
<h2>Start with two questions</h2>
<p>Before you pick a method, answer these:</p>
<ol>
<li><p><strong>Do you control the site?</strong> If it's verified in Search Console, you get Google's own answer. If it's someone else's site, such as a page carrying your backlink, you don't.</p>
</li>
<li><p><strong>How many URLs?</strong> One URL by hand is fine. Fifty is tedious. Five thousand needs automation.</p>
</li>
</ol>
<table>
<thead>
<tr>
<th></th>
<th>One URL</th>
<th>Hundreds or more</th>
</tr>
</thead>
<tbody><tr>
<td><strong>Your site</strong></td>
<td>URL Inspection tool</td>
<td>URL Inspection API, Page indexing report</td>
</tr>
<tr>
<td><strong>Other sites</strong></td>
<td>site: search</td>
<td>Bulk index checker</td>
</tr>
</tbody></table>
<h2>Method 1: the site: operator</h2>
<p>Type <code>site:</code> plus the full URL into Google:</p>
<pre><code class="language-text">site:https://example.com/guides/setup
</code></pre>
<p>It works for any public URL and costs nothing. The catch is in Google's own wording: an indexed URL "can show up in search results for site: queries", but "it's not guaranteed" (<a href="https://developers.google.com/search/docs/monitor-debug/search-operators/all-search-site">source</a>). Treat a hit as strong evidence and a miss as "unknown". It also doesn't scale. Google starts showing CAPTCHAs if you script it, and automated queries are against its terms.</p>
<h2>Method 2: URL Inspection in Search Console</h2>
<p>For a page on your own verified property, this is the answer to trust. Paste the URL into the top bar of <a href="https://search.google.com/search-console/about">Search Console</a> and you get:</p>
<ul>
<li><p>whether the URL is on Google</p>
</li>
<li><p>the last crawl date</p>
</li>
<li><p>the canonical you declared and the one Google picked</p>
</li>
<li><p>the reason, if it isn't indexed</p>
</li>
</ul>
<p>Help page: <a href="https://support.google.com/webmasters/answer/9012289">URL Inspection tool</a>. It's one URL at a time, and "Request indexing" in the same panel has a daily quota.</p>
<h2>Method 3: the Page indexing report</h2>
<p>Indexing &gt; Pages in Search Console gives counts of indexed and non-indexed URLs, grouped by reason. Use it to spot trends, such as a sudden jump in "Duplicate without user-selected canonical" after a deploy. Two limits: example lists stop at 1,000 URLs per reason, and you can't search for a single URL. Google itself points you back to URL Inspection for individual pages (<a href="https://support.google.com/webmasters/answer/7440203">source</a>).</p>
<h2>Method 4: the URL Inspection API</h2>
<p>The same data as Method 2, from code. Limits are 2,000 inspections per day and 600 per minute per property (<a href="https://developers.google.com/webmaster-tools/limits">usage limits</a>). That's enough to monitor the pages that matter most on a mid-sized site every day.</p>
<p>A minimal script with a service account that you've added as a user on the property:</p>
<pre><code class="language-python"># pip install google-api-python-client google-auth
from google.oauth2 import service_account
from googleapiclient.discovery import build

SITE = "sc-domain:example.com"   # or "https://example.com/"
URLS = [
    "https://example.com/",
    "https://example.com/guides/setup",
]

creds = service_account.Credentials.from_service_account_file(
    "service-account.json",
    scopes=["https://www.googleapis.com/auth/webmasters.readonly"],
)
gsc = build("searchconsole", "v1", credentials=creds)

for url in URLS:
    res = gsc.urlInspection().index().inspect(
        body={"inspectionUrl": url, "siteUrl": SITE}
    ).execute()
    r = res["inspectionResult"]["indexStatusResult"]
    print(
        url,
        r.get("verdict"),          # PASS, NEUTRAL, FAIL
        r.get("coverageState"),    # e.g. "Submitted and indexed"
        r.get("lastCrawlTime"),
        r.get("googleCanonical"),
        sep=" | ",
    )
</code></pre>
<p>Field reference: <a href="https://developers.google.com/webmaster-tools/v1/urlInspection.index/inspect">urlInspection.index.inspect</a>. Write the rows to a CSV each day and you have an indexing history you can diff after every release.</p>
<p>Two things trip people up. The <code>siteUrl</code> must match the property exactly (a Domain property uses the <code>sc-domain:</code> prefix). And the API only reads status. It doesn't request indexing.</p>
<h2>Method 5: a bulk index checker</h2>
<p>For URLs on sites you don't own, Search Console and its API can't help. A bulk checker takes a list, looks each URL up in Google and returns a status per row, usually as a CSV. It answers the same question as a site: search, at volume.</p>
<p>When you compare checkers, look at:</p>
<ul>
<li><p><strong>Unit of billing:</strong> per URL or per batch of URLs. This changes the cost of a 5,000-link audit a lot.</p>
</li>
<li><p><strong>Batch size:</strong> how many URLs one run accepts.</p>
</li>
<li><p><strong>Export:</strong> a CSV with the exact URL you submitted, so you can join it back to your link list.</p>
</li>
<li><p><strong>Read-only:</strong> checking should not submit anything anywhere. Make sure the tool keeps those steps separate.</p>
</li>
</ul>
<p>For your own pages, a checker is an outside view. URL Inspection still has the final word.</p>
<h2>Reading the result without fooling yourself</h2>
<p>Most "my page isn't indexed" reports we see come down to one of these:</p>
<p><strong>You checked a different URL.</strong> <code>http://</code> and <code>https://</code>, <code>www</code> and bare domain, trailing slash or not, uppercase in the path, a tracking parameter. Each is a distinct URL to Google. Normalize your list before you check it:</p>
<pre><code class="language-python">from urllib.parse import urlsplit, urlunsplit

def normalize(u: str) -&gt; str:
    s = urlsplit(u.strip())
    return urlunsplit((s.scheme.lower(), s.netloc.lower(), s.path or "/", s.query, ""))
</code></pre>
<p>Then check the version that actually returns 200, not the one that redirects.</p>
<p><strong>Google indexed the canonical instead.</strong> If the "Google-selected canonical" differs from your URL, the content is in the index under the other address. For a backlink, that can mean the indexed copy is a different page from the one carrying your link.</p>
<p><strong>Indexed is not ranking.</strong> An indexed page is eligible to appear. It won't show for every query. To confirm presence, search for a distinctive sentence from the page in quotes.</p>
<p><strong>The status changed.</strong> Pages drop out of the index and come back. A single check is a snapshot.</p>
<p><strong>Not indexed was correct.</strong> Redirects, deliberate duplicates and pages with noindex are supposed to stay out. Filter them from your list before you count failures.</p>
<h2>A backlink audit, step by step</h2>
<ol>
<li><p>Export the URLs of the pages that host your links, not your own URLs.</p>
</li>
<li><p>Normalize and dedupe them with a function like the one above.</p>
</li>
<li><p>Drop anything that redirects or returns 4xx or 5xx. A quick <code>curl -o /dev/null -w "%{http_code}"</code> loop does it.</p>
</li>
<li><p>Run the rest through a bulk index checker and save the CSV.</p>
</li>
<li><p>For each "not indexed" row, open the page. Confirm your link is still there and look at its <code>rel</code> value. Google treats <code>nofollow</code>, <code>sponsored</code> and <code>ugc</code> differently (<a href="https://developers.google.com/search/docs/crawling-indexing/qualify-outbound-links">qualify outbound links</a>).</p>
</li>
<li><p>Decide per row: submit it to an indexer, ask the site owner to fix the page, or write it off.</p>
</li>
<li><p>Re-check new links about two weeks after they go live, then every one to three months for links that matter.</p>
</li>
</ol>
<h2>Quick answers</h2>
<p><strong>Does checking a URL get it indexed?</strong> No. Every method above only reads status. Submitting is a separate action.</p>
<p><strong>Can I check Bing too?</strong> For your own site, yes, with the <a href="https://www.bing.com/webmasters/help/url-inspection-55a30305">URL Inspection tool in Bing Webmaster Tools</a>. Most bulk checkers focus on Google.</p>
<hr />
<p><strong>Written by the IndexerPal team.</strong> We build tools for URL and backlink indexing. To check up to 10,000 URLs in Google in one run, see the <a href="https://indexerpal.com/index-checker">IndexerPal index checker</a>.</p>
]]></content:encoded></item></channel></rss>