The Pipeline Generation Lifecycle
The workflow moves linearly from Discovery to Output Generation, but you can revisit earlier steps at any point before generating your final artifacts. The five step are:Discovery → Selection → Transformations → Enrichments → Output Generation
Step 1: Discovery
After launching a new session and selecting a sample file, the first step is the Discovery. it is where PrettyWhale.ai reads your sample file and automatically builds a structural understanding of your data. You don’t write any schema definitions or configuration files, you simply upload a file or connect your source and PrettyWhale.ai does the analysis. During Discovery, PrettyWhale.ai detects field names, infers data types, computes value distributions, identifies nullable fields, and flags anomalies such as mixed types, unexpected nulls, or values that don’t match their apparent format. The result is a field-level profile of your dataset that forms the foundation for every subsequent step. What would normally take an engineer hours of exploratory work, profiling a new dataset, documenting its shape, and catching edge cases, happens automatically in seconds.Step 2: Selection
After Discovery, PrettyWhale.ai presents you with a detailed field-level breakdown of everything it found in your sample. Your job in this step is to decide what belongs in your pipeline. Check the fields you want to include and uncheck the ones you don’t. Fields you exclude are completely removed from scope, they won’t appear in generated code, tests, or schemas. This keeps your pipeline clean and focused, and ensures generated artifacts don’t carry unnecessary complexity from fields your downstream systems don’t need.Step 3: Transformations
With your fields selected, PrettyWhale.ai’s analyzes each one and suggests a set of transformations designed to make your data consistent, valid, and ready for downstream consumption. Suggestions are field-specific and contextually appropriate. Typical suggestions include normalizing date and timestamp formats to a consistent standard, standardizing text casing, trimming leading and trailing whitespace, validating string patterns such as email addresses or phone numbers, and coercing numeric types. For each suggestion you can accept it as-is, modify the parameters or remove it entirely. The final transformation configuration you approve is what drives the logic in your generated ingestion script. The transformation library is constantly growing. We’ll keep you updated on future additions on a regular basis.Step 4: Enrichments
Incoming step - in development Enrichment is an optional step that lets you extend your data with context from internal or external sources before it reaches its destination. PrettyWhale.ai analyzes your selected fields and suggests relevant enrichments based on what it finds (for example, resolving acountry_code field to a full country name and region, or deriving company information from an email domain).
You connect external data sources through the Enrichment panel and configure which fields trigger which lookups. Enrichments are integrated directly into the generated code, the output script handles the lookup logic, fallback behavior, and any necessary caching or batching, so you don’t have to write that boilerplate yourself.
Step 5: Output Generation
Once your configuration is complete, click Generate. PrettyWhale.ai’s compiles your Discovery profile, field selections, transformation rules, and enrichment configuration into a complete, validated artifact set. The time it takes to generate the deliverables depends on the size of the source file and the number of transformations. During this time, you can continue working on other tasks. PrettyWhale.ai runs in the background and will save the outputs to your VS Code folder once generation is complete. The output is production-ready from the moment you download it. Tests cover the validation and transformation logic you configured, and documentation reflects the exact pipeline you built, not a generic template. Every artifact is consistent with the others because they are generated together from the same configuration and not assembled from independent templates.What’s Generated
Every pipeline generation produces the following artifacts:- Executable code
- Schemas and metadata parameters
- Unit tests
- Configuration files
- Documentation
Where to start
Quickstart
Get your first pipeline generated in minutes. Follow a step-by-step walkthrough from sample file to downloadable code.