Repository navigation
feat: 실거래가 데이터 추출 스케줄링 및 배치 적재 구현 - #3
Merged
Merged
Conversation
scripts/results의 월별 RTMS JSON(8개 소스: 매매/전월세 x 아파트/오피스텔/ 연립다세대/단독다가구)을 읽어 SaleTransaction/RentTransaction 스키마에 맞게 변환한 뒤 배치 upsert하는 작업을 APScheduler로 매일 04:00에 실행하도록 main.py에 추가. 적재 검증 중 UniqueConstraint에 포함된 nullable 컬럼(jibun, building_name, floor, exclusive_use_area)이 단독·다가구 타입에서 항상 NULL이라 PostgreSQL의 NULL != NULL 규칙으로 인해 dedup이 동작하지 않던 문제를 발견하여, 복합 자연키 대신 행 전체 내용을 해싱한 dedup_hash 컬럼 기반 UNIQUE로 교체. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDnVFzm6DW7CjG9hXN3QeV
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
scripts/results의 월별 RTMS JSON(매매/전월세 x 아파트/오피스텔/연립다세대/단독다가구, 8개 소스)을 읽어SaleTransaction/RentTransaction스키마에 맞게 변환한 뒤 배치 upsert하는 로직을api/app/main.py에 추가AsyncIOScheduler)로 매일 04:00에 추출 작업이 실행되도록 스케줄링postgres:17) 중 기존 복합UniqueConstraint가 nullable 컬럼(단독·다가구는 jibun/building_name/floor/exclusive_use_area가 항상 NULL) 때문에 dedup이 동작하지 않던 문제를 발견 → 행 전체 내용을 해싱한dedup_hash컬럼 기반 UNIQUE로 교체(api/app/model/prop_transaction.py)Test plan
ruff format ./ruff check .통과postgres:17컨테이너에 3개월(202606~202608) x 8소스 샘플을 두 번 연속 적재하여 idempotency(재실행 시 중복 없음) 확인 — SINGLE_MULTI 포함 모든 property_type에서count(*) == count(DISTINCT dedup_hash)일치TestClient로 FastAPI 앱을 기동해 lifespan(스케줄러 시작/종료) 및/health정상 동작 확인🤖 Generated with Claude Code
https://claude.ai/code/session_01YDnVFzm6DW7CjG9hXN3QeV