DSM-176: CSV Exports API by wtomaszewska · Pull Request #77 · cul/atc

wtomaszewska · 2026-08-14T14:28:53Z

Ticket DSM-176

Overview

This PR adds a CSV Exports feature to the S3 Browser API. A user selects files and/or folders across one or more buckets and the app generates a downloadable CSV describing every matching object (S3 URI, name, modification date, storage tier, size and restore status). Generation runs in the background using Resque so large selections don't block the request. Finished exports can be listed, viewed and downloaded through the API. Exports are retained for 6 months and then cleaned up automatically via a daily cron job.

API Endpoints

All endpoints live under /api/csv_exports, return JSON and require an authenticated user.

`POST /api/csv_exports` - create an export

Accepts a selections payload (one entry per bucket), each entry listing files (individual object keys) and directories (folder prefixes). The action:

Normalizes each selection (strips leading slashes, drops blank entries and removes exact-duplicate files/folders)
Validates the request and rejects it with an error when:
- there are no selections
- a selection overlaps (eg. a file that already lives inside a selected folder or a folder nested inside another selected folder)
- a selection tries to export an entire bucket (a / directory)
Creates a CsvExport record (status pending) owned by the current user, enqueues PrepareCsvExportJob and responds 202 Accepted with the new record's id.

The S3 lookups and file writing happen asynchronously in the job (see below), not during the request.

Permissions: any authenticated user may create an export.

`GET /api/csv_exports` - list exports

Returns the current user's exports (summary + pagination metadata), newest first.

Permissions: non-admins see only their own exports; admins see everyone's.

`GET /api/csv_exports/:id` - export detail

Returns a single export's status, selections and any per-object errors.

Permissions: non-admins may view only their own exports, admins may view any.

`GET /api/csv_exports/:id/download` - download the CSV

Returns the generated CSV file (text/csv). Returns 404 if the file isn't available yet (still processing or the export failed).

Permissions: same as detail/show - non-admins can download only their own exports, admins can download any.

CSV Generation (`PrepareCsvExportJob`)

A Resque job (in the CSV_EXPORT queue) builds the file:

For folders it lists all objects under the prefix. For files it fetches object metadata directly. An extra head_object call is made only for INTELLIGENT_TIERING objects, which are the only ones that can report archive/restore status.
It writes a CSV with: S3 URI, file name, modification date, storage tier, human-readable size, size in bytes and whether a restore is in progress.
The record moves from "pending" to "processing", then finishes as "success", "completed_with_errors" or "failure".
Errors are handled per object: if a single file or folder can't be read, that failure is recorded on the export and the rest of the CSV is still produced, ending as completed_with_errors. Only an error that prevents producing any file at all results in failure.

Data Model & Permissions

New CsvExport model / csv_exports table: path_to_csv_file, user reference, status (enum: pending, processing, success, failure, completed_with_errors), export_errors, export_paths, timestamps. Destroying a record also removes its CSV file from disk (after_destroy).
Added an is_admin boolean to users (default false). Admins can manage all exports, while regular users can create exports and read only their own.

Automatic Cleanup

Atc::CsvExports::ExpiredPurger deletes CsvExport records (and their CSV files) older than 6 months
ExpiredPurger is called in the atc:csv_exports:delete_expired rake task and scheduled via whenever to run daily.

…t information for the csv file; use HeadObject to retrieve archive_status and restore_status

…b for preparing Csv and port existing logic; modify the received data format so files and folders are split by bucket, allowing to export object data across different s3 buckets

…figured in aws.yml; add endpoints for GET all csv exports and GET single export

…e S3 URI; install kaminari and paginate csv exports list; create export directory if missing

…ath for failed records, update csv path after writing to file, include timestamp and export id; add csv download endpoint

…export detail info

…intelligent tiering; make sure that selections are not empty

…ject URI / folder prefix if available

…r controller to buckets controller; move duplicate methods into a new BucketValidation concern

…), add tests for PrepareCsvExportJob; use Rails logger

…6 months, run it via cron

goldsmithb

This looks great! I left a few questions and suggestions, but overall it looks ready to go.

goldsmithb · 2026-08-14T21:54:46Z

    end

-    render json: { folders: folders, objects: objects }
+    render_camelized_json({ folders: folders, objects: objects })


I keep reading this as 'render_caramelized_json' in my head and thinking "sounds delicious" 😆

goldsmithb · 2026-08-14T21:55:21Z

+  csv_exports_directory = AWS_CONFIG[:s3_browser][:csv_exports_directory]
+
+  if csv_exports_directory.blank?
+    logger.error('CSV export directory is not configured in aws.yml')


Slight typo:

Suggested change

logger.error('CSV export directory is not configured in aws.yml')

Rails.logger.error('CSV export directory is not configured in aws.yml')

goldsmithb · 2026-08-14T22:20:30Z

+
+  # A request must include at least one selection and every selection must target
+  # at least one file OR directory
+  def validate_selections!(selections)


Should we add a rule to reject selections that include an entire bucket? Those requests would have the form:

{ "selections": [ { "bucket": "some-bucket", "directories": ["/"], "files": [] } ] }

We already do it in the normalize_prefixes method

goldsmithb · 2026-08-17T15:47:17Z

+  def csv_export_detail_json(csv_export)
+    {
+      id: csv_export.id,
+      export_paths: JSON.parse(csv_export.export_paths), # do we want to limit the number of paths returned here?


I don't think it's necessary; a copy of the original export request body would be nice to have on an export's details page.

My comment here is a bit confusing because it was meant to be added to csv_export_summary_json used in the index action. We definitely want to display all export paths on the detail page.

The list view doesn't need to show all exported paths, and since the selection can include thousands of files, it makes sense to avoid sending unnecessary data. Similarly, we don't need to send export_errors when fetching the export list, since those can also contain many entries. I think it makes sense to keep summary (used in index) and detail (used in show) objects separate in this case and avoid overfetching.

goldsmithb · 2026-08-17T16:00:53Z

+    }
+  end
+
+  def csv_export_summary_json(csv_export)


perhaps we can combine this method and csv_export_detail_json to render responses for both index and show actions, and show/hide fields on the front end as needed.

See #77 (comment)

goldsmithb · 2026-08-17T16:00:54Z

+      status: csv_export.status,
+      export_paths: JSON.parse(csv_export.export_paths),
+      export_errors: csv_export.export_errors,
+      path_to_csv_file: csv_export.path_to_csv_file,


I know we put it in the API spec, but I'm not sure why a user would need this information. Is it worth rethinking this? In either case I think we can consolidate this with the other csv export json method.

I can remove this field if we just want to render a download button. For the consolidation part, see #77 (comment)

goldsmithb · 2026-08-17T17:41:40Z

+  end
+
+  def set_csv_export
+    @csv_export = CsvExport.find(params[:id])


We tend to use strong params for all controller actions so that interfaces are super explicit. It doesn't add any security benefits in this case, but worth considering

goldsmithb · 2026-08-17T17:54:41Z

    AWS_CONFIG[:aws_access_key_id],
-    AWS_CONFIG[:aws_secret_access_key]
+    AWS_CONFIG[:aws_secret_access_key],
+    AWS_CONFIG[:aws_session_token]


What is the session token for?

I sometimes use it locally when testing but I removed it to avoid confusion

…g params when passing export id

…e list view, add tests

wtomaszewska added 15 commits July 23, 2026 13:46

DSM-176 WIP csv export endpoint - accept folders and files and extrac…

0de0b7c

…t information for the csv file; use HeadObject to retrieve archive_status and restore_status

DSM-176 add is_admin field to Users, create csv exports table

3355669

modify CsvExport to capture errors, paths and status; create a new jo…

8d827ed

…b for preparing Csv and port existing logic; modify the received data format so files and folders are split by bucket, allowing to export object data across different s3 buckets

create CSV file based on the data from export_paths; use a folder con…

4b78e5d

…figured in aws.yml; add endpoints for GET all csv exports and GET single export

add permissions to index, show actions; fix missing bucket name in th…

fb08146

…e S3 URI; install kaminari and paginate csv exports list; create export directory if missing

merge main

22eb832

move csv endpoints into new CsvExportsController; allow for nil csv_p…

1d19a3a

…ath for failed records, update csv path after writing to file, include timestamp and export id; add csv download endpoint

add render_camelized_json method, send export errors when retrieving …

e8e1203

…export detail info

avoid extra call to retrieve head object for objects that are not in …

bc5e0b4

…intelligent tiering; make sure that selections are not empty

add completed_with_errors status to CsvExport, capture errors with ob…

e787e80

…ject URI / folder prefix if available

throw error when overlapping selections are detected; rename s3browse…

1eff2b9

…r controller to buckets controller; move duplicate methods into a new BucketValidation concern

add tests for CSV export API endpoints (index, show, create, download…

7f48de3

…), add tests for PrepareCsvExportJob; use Rails logger

add whenever gem, add ExpiredPurger to remove csv exports older than …

060ec5e

…6 months, run it via cron

refactor storage_tier method, remove unnecessary rubocop exclude lines

554ac85

throw error when a whole bucket is selected

9b385f1

wtomaszewska requested review from elohanlon and goldsmithb August 14, 2026 14:28

remove skip verify_authenticity_token used for postman calls

b774cb6

goldsmithb approved these changes Aug 17, 2026

View reviewed changes

wtomaszewska added 2 commits August 18, 2026 14:42

move build_selections to private methods, use Rails logger, use stron…

4c00e32

…g params when passing export id

send a sample of selected paths and total number of selections for th…

5941eaf

…e list view, add tests

	logger.error('CSV export directory is not configured in aws.yml')
	Rails.logger.error('CSV export directory is not configured in aws.yml')

Uh oh!

Conversation

wtomaszewska commented Aug 14, 2026

Ticket DSM-176

Overview

API Endpoints

POST /api/csv_exports - create an export

GET /api/csv_exports - list exports

GET /api/csv_exports/:id - export detail

GET /api/csv_exports/:id/download - download the CSV

CSV Generation (PrepareCsvExportJob)

Data Model & Permissions

Automatic Cleanup

Uh oh!

goldsmithb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

`POST /api/csv_exports` - create an export

`GET /api/csv_exports` - list exports

`GET /api/csv_exports/:id` - export detail

`GET /api/csv_exports/:id/download` - download the CSV

CSV Generation (`PrepareCsvExportJob`)