Data pipelines

Build a reusable sequence, inspect each step, and prepare a complete output file.

Import and conversion workflow

Saved recipes

Find a collection
Showing up to 100 of 6 matches. The current source remains selectable.
StepDEDUPLICATE
Cleanup operation examples

Choose Clean, cast, compute or flatten, then edit its JSON configuration. Supported kinds: trim, lower, upper, replace, regex, substring, split, merge, default, cast_string, cast_number, cast_boolean, add, subtract, multiply, date, flatten. Column names are literal top-level names. Flatten uses dotted output names and rejects collisions.

{"op":"transform","transform":{"kind":"multiply","field":"price","value":1.2,"as":"price_with_tax"}}
{"op":"transform","transform":{"kind":"date","field":"date","layout":"02/01/2006","output_layout":"2006-01-02"}}

Substring offsets count Unicode characters. Arithmetic preserves decimal precision and rejects unsupported precision instead of rounding.

Run and export pipeline
How steps work

Drag cards to reorder them, or use the arrow buttons. Configurations use JSON: filter values keep their types, map supports constant values and copied fields, and select/drop/rename reuse response transformation rules. A pipeline supports up to 20 steps. Global operations use disk-backed SQLite; preview returns up to 20 records.

Validate output before publication

Rules use the same syntax as Data quality. An invalid output record rejects the export; no partial download is published. Final previews check only their first 20 rows. Intermediate step previews do not check final-output rules.

Import a pipeline definition

Choose a source above to reuse a recipe with local data. Join steps must reference collections on this instance.

Recipe settings and API

POST /api/pipelines/5cc34cda80d7a7ef08d839216b8d7a6a/run

Pipeline activity and downloads
Automatic runs

Run the latest saved recipe at a fixed interval while FluxPort is serving. Runs never overlap for this schedule. Outputs appear in execution history and expire after 24 hours.

Next run (Unix seconds): 0. Checks run every 15 seconds; missed intervals are coalesced into one run after restart.

Recipe revisions

Each save creates a revision. The latest 50 are retained. Restore creates a new revision.

RevisionSavedAction
12026-10-12T08:22:17Z
Execution history
RunStatusRecordsStartedUpdatedResult
Cleaned demo-orders.yaml · pipeline → xlsxready1002026-10-12T08:22:17Z2026-10-12T08:22:17ZDownload