Sphinx generates headerlink anchors (the paragraph symbol ¶) next to
each heading. These appear as ¶ in the output due to a UTF-8/Latin-1
decode mismatch in BeautifulSoup. Fix by removing .headerlink elements
in NOISE_SELECTORS and stripping any residual ¶/¶ in clean_text().
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The Odoo 18 sitemap.xml returns 404. The fallback URL list also failed
because urljoin(BASE_URL, /applications/...) strips the /documentation/18.0
path (absolute path arg replaces the whole path component in urljoin).
Changes:
- Add discover_urls_by_crawl(): fetches each module index page and
collects all internal links — replaces sitemap as primary source
- crawl() now chains: sitemap → crawl discovery → hardcoded fallback
- Fix fallback_urls() to use BASE_URL + path (not urljoin) and trim
the list to known-good pages
- Keep crawl discovery rate-limited (0.5s between module seeds)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Scraper was using a bot User-Agent that triggered Cloudflare bot
detection, returning challenge pages with < 100 chars of content.
Switched to a standard Chrome UA with Accept headers.
Qdrant healthcheck used /healthz which does not exist in v1.9.0.
Changed to GET / which is always available. Added start_period: 30s
so the check does not fire before Qdrant has time to initialise.
Added --debug flag to scraper for future extraction diagnostics.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
