tocmo0nlord/duplicate-finder
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Pipeline discovery and indexing — workers start immediately

Browse Source 
Instead of walk-everything-first then index, workers now receive files
the instant os.walk yields them. The thread pool is open before the
walk starts; each discovered file is submitted immediately. Completed
futures are drained after each directory to keep memory flat.

Progress message shows:
  "Discovering & indexing (8w): 1,234 — 5,678 found so far"
  then once walk finishes:
  "Indexing (8w): 8,000 / 9,100"

UI: merged Discovery + Indexing into a single "Discover + Index" phase pill.
Indeterminate progress bar stays on until total file count is known.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
tocmo
2026-04-05 01:54:43 -04:00
parent fef364162c
commit a6748de6e0
2 changed files with 155 additions and 135 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
templates/index.html
View File

@@ -759,9 +759,8 @@
<div class="progress-bar-fill" id="progress-fill" style="width:0%"></div>
</div>
<div class="phase-pills">
<span class="phase-pill" data-phase="discovery">Discovery</span>
<span class="phase-pill" data-phase="takeout">Takeout</span>
<span class="phase-pill" data-phase="indexing">Indexing</span>
<span class="phase-pill" data-phase="indexing">Discover + Index</span>
<span class="phase-pill" data-phase="phash">Phash</span>
<span class="phase-pill" data-phase="grouping">Grouping</span>
</div>
@@ -1025,7 +1024,7 @@ async function refreshStats() {
// ── Scan polling ──────────────────────────────────────────────────────────────
let scanPoller = null;
const PHASES = ['discovery','takeout','indexing','phash','grouping'];
const PHASES = ['takeout','indexing','phash','grouping'];
function startPoller() {
if (scanPoller) return;
@@ -1065,7 +1064,7 @@ function updateScanUI(s) {
if (isRunning) {
el('progress-msg').textContent = s.message || '';
const indeterminate = s.phase === 'discovery' || s.phase === 'takeout' || s.total === 0;
const indeterminate = s.phase === 'takeout' || s.total === 0;
const fill = el('progress-fill');
fill.classList.toggle('indeterminate', indeterminate);
if (!indeterminate) {