Skip to content

feat: add kubernetes support#284

Draft
kyteinsky wants to merge 51 commits intomasterfrom
feat/noid/rev-content-flow
Draft

feat: add kubernetes support#284
kyteinsky wants to merge 51 commits intomasterfrom
feat/noid/rev-content-flow

Conversation

@kyteinsky
Copy link
Copy Markdown
Contributor

@kyteinsky kyteinsky commented Mar 5, 2026

No description provided.

marcelklehr
marcelklehr reviewed Mar 5, 2026
View reviewed changes
@kyteinsky kyteinsky force-pushed the feat/noid/rev-content-flow branch from 4ca116f to 8e53243 Compare March 5, 2026 10:53
marcelklehr
marcelklehr reviewed Mar 6, 2026
View reviewed changes
marcelklehr
marcelklehr reviewed Mar 10, 2026
View reviewed changes
@marcelklehr
Copy link
Copy Markdown
Member

marcelklehr commented Mar 10, 2026 •
edited
Loading

Shall we set the context_chat php app to the branch of the corresponding PR here, so we can use CI properly? (temporarily)

@kyteinsky
Copy link
Copy Markdown
Contributor Author

kyteinsky commented Mar 10, 2026

Shall we set the context_chat php app to the branch of the corresponding PR here, so we can use CI properly? (temporarily)

good idea!

@kyteinsky kyteinsky force-pushed the feat/noid/rev-content-flow branch 2 times, most recently from 83b7587 to 84f6896 Compare March 12, 2026 12:20
kyteinsky and others added 22 commits March 26, 2026 23:02
@kyteinsky
feat: add kubernetes app role selection
5dad714 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
feat: add thread start and stop logic
089d27a 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
wip: migrate the indexing process
64ffdaf 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
wip: parallelize file parsing and processing based on cpu count
03a3f43 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
ci: use the kubernetes branch of context_chat
0dc404b 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix typo
c733982 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
migrate the update process to be thread based
dda312f 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix pydantic types
b09a93c 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix: use a dedicated event to allow app halt without app being disabled
11b436c 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix fetch url and pydantic types
c88e153 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix: use the correct file id
cd5241e 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix: wip: improve embeddings exception handling
4958d1d 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix(ci): update to the latest changes
a049121 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix(ci): use file to store stderr
795380c 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix(ci): add cron jobs
7bc0ed7 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix(ci): do a occ files scan before cron jobs
d94c687 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
feat: record indexing errors in content decode function
dadc8fa 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
chore: move file fetch inside injest
f9d86dc 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix: truly parallel file parsing and indexing
1ade191 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@marcelklehr @kyteinsky
initial pass at request processing
12fd1ca
@marcelklehr @kyteinsky
implement request processing
8aa2471
@kyteinsky
request processing fixes
2093936 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky kyteinsky force-pushed the feat/noid/rev-content-flow branch from 3459ba5 to 85d29f1 Compare March 26, 2026 20:26
kyteinsky and others added 18 commits March 27, 2026 02:57
@kyteinsky
fix: seek to 0 to read the full buffer
4c6d01b 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix(ci): 3% tolerance
51774ff 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix(ci): wait longer for EM server
c81b675 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix: don't process files or requests until the EM server is healthy
6817f89 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@marcelklehr
tests: Increase testing time to allow backend to injest more sources
104a37a
@marcelklehr
fix: More log statements
b3b461a
@marcelklehr
tests: Set wait time back to 90
a4a88da
@marcelklehr
fix: Reduce worker count on github actions
0c52747 
to prevent oom
@marcelklehr
fix(exec_in_proc): Raise RuntimeError if exitcode is non-zero
e676c32
@marcelklehr
fix(indexing): Reduce memory pressure on gh actions
b027ff3
@marcelklehr
fix(indexing): Fallback to batch_size=1 if embed_sources is killed
19b773f 
and do not retry afterward if one these single item batches get killed
@kyteinsky
fix: log stdout and stderr from subprocesses
bde0bc5 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix: don't raise before std* is captured
4de591f 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
feat: log cpu count and memory info of the system
4deda84 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix: catch BaseException in subprocess
ad0eac7 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@marcelklehr
fix(utils): Improve exec_in_proc to handle more failure modes
36bcfb7
@kyteinsky
one more stab at a fix
47eaf72 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
do not throw away the valid result even with exitcode 1
309ab2b 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky kyteinsky force-pushed the feat/noid/rev-content-flow branch from a43b72b to e874b37 Compare April 3, 2026 07:01
@kyteinsky
fix: use forkserver as process start method
e1763ac 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky kyteinsky force-pushed the feat/noid/rev-content-flow branch from e874b37 to e1763ac Compare April 3, 2026 07:19
kyteinsky added 8 commits April 3, 2026 13:16
@kyteinsky
fix(ci): consider eligible files as the total files count
3301652 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix: use logging config in forkserver and other fixes
32aa374 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix: remove extra diagnostics
33ee38a 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix: use zip on the subset of filtered sources
d9ebdac 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix(em): use tcp socket connection check
ea77480 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix(ci): remove github CI restrictions
1ce237a 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix: remove unused code and some de-duplication
d82e01b 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix(mp): run repairs and config file check only in MainProcess
286db22 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@marcelklehr marcelklehr marcelklehr left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kyteinsky @marcelklehr