Best practise for automization of data for large number of document stores

[HenryUSU]

Hi everyone!
We've been using flowise with great success for almost a year now in a project, that is getting pretty large now. Without the flexibility and power of Agentflows v2 we wouldn't be able to do that and scale so easy, so thanks for that awesome improvement!

Our challenge is that we have an increasing number of document stores (currently 50, one document loader per document store), 3 mio to 21 mio characters per document store and about 1500 total chunks per doc store. We use postgres as a vector db currently.

With each new document loader, the time to process and upsert documents increases. I think to process and upsert every document loader takes about 5-6 hours, one by one. Currently because of the issue with pgvector and the record manager not clearing up everything properly, we need to upsert everything all the time. And we need to do that on a regular basis, as our data changes regularly as well.

We are using Cypress to automize this and also be able to track in detail if there is an issue for a specific document loader and what might be the reason for it. Sometimes we have the issue that the Flowise UI is not responding or the processing took too long etc. In general it works fine, but we expect the project to further grow to maybe few hundred document stores. Our current approach is not ideal and we want to improve it.

I am curious if others have best practises for automizing lots of document stores/document loaders with regular data updates.
Or maybe features are planned to make this easier and/or faster. I saw that there is already a feature on the roadmap: "Cron Job For Upsert/Refresh Doc Store, AgentflowV2". This could make things a bit easier I assume.

So I am looking forward for any input, ideas and would like to hear how you do it in your projects.

[Ganesh-403]

@HenryUSU
First of all, it's great to hear you've been running Flowise successfully at that scale. 50 document stores with millions of characters each is already beyond what many deployments handle, so you've probably reached the point where UI-driven ingestion becomes the bottleneck rather than the vector database itself.

A few suggestions that I'd consider for a deployment of this size:

1. Move away from UI automation where possible

Using Cypress works, but you're effectively automating a browser to trigger backend operations. Since Flowise exposes Document Store Upsert and Refresh APIs, I'd recommend driving ingestion directly via the API instead. This tends to be more reliable and easier to monitor than UI automation. :contentReference[oaicite:0]{index=0}

For example, instead of:

Cypress
    ↓
Browser
    ↓
Flowise UI

consider:

Scheduler
    ↓
Your ingestion service
    ↓
POST /document-store/refresh/{id}

or

POST /document-store/upsert/{id}

This also makes retries and logging much easier.

---

2. Refresh only the document stores that changed

If your source systems can tell you which datasets changed, only call the Refresh API for those document stores instead of processing all 50 every run.

That can significantly reduce processing time if only a subset of your data changes regularly. Flowise's Refresh API is designed to reprocess existing document loaders within a document store. :contentReference[oaicite:1]{index=1}

---

3. Make sure the Record Manager cleanup mode matches your workflow

You mentioned having to rebuild everything because of the current pgvector/Record Manager issue.

Normally, the Record Manager is intended to avoid unnecessary re-embedding by hashing chunks and only indexing changes. Depending on your use case, Incremental Cleanup is generally preferable for continuously evolving datasets, while Full Cleanup is intended for complete replacements. :contentReference[oaicite:2]{index=2}

If you're forced to perform full re-upserts due to the pgvector issue, I'd keep an eye on fixes for that rather than optimizing around the workaround.

---

4. Parallelize cautiously

If your infrastructure allows it, consider refreshing multiple independent document stores concurrently (with a sensible concurrency limit).

For example:

Refresh Store A
Refresh Store B
Refresh Store C

instead of strictly serial execution.

The optimal concurrency depends on:

• embedding provider rate limits,
• PostgreSQL/pgvector capacity,
• CPU available for parsing/chunking.

Running everything in parallel usually isn't ideal, but a small worker pool often improves throughput substantially.

---

5. Separate ingestion from serving

At your scale, I'd treat ingestion as a dedicated background process:

Source systems
        ↓
Change detection
        ↓
Queue
        ↓
Workers
        ↓
Flowise Document Store API

This gives you retries, monitoring, and better fault isolation than a long-running browser automation job.

---

Regarding the roadmap

I also noticed the roadmap item you mentioned for Cron Job for Upsert/Refresh Doc Store. From what I can tell, that would simplify scheduling inside Flowise, but it likely wouldn't change the underlying indexing performance—it would mainly reduce the operational overhead of orchestrating refreshes yourself.

---

One question: are your 50 document stores completely independent knowledge bases, or are they partitioned versions of the same dataset (e.g., one per customer/tenant)? If it's the latter, there may be opportunities to consolidate ingestion while filtering at retrieval time, which could reduce the number of separate upsert jobs you need to manage.

If this solves your problem, feel free to mark it as the accepted answer so others can find it easily.