Introducing Petl

Many moons ago I ran into a problem building an eCommerce site. Sourcing all the relevant data for any product proved problematic due to the multiple systems and their overall lack of performance. Getting up-to-date inventory, for instance, required querying SAP with abysmal multi-second performance. I flipped the script and wrote a system that allowed me to combine multiple sources of data and push the result to a number of targets including a search index and an optimied view database.

This pattern is repeated often -- at the end of the day, developers largely are just moving data from one system to another and possibly changing it along the way. Petl is designed to make that a more simple and repeatable process.

Petl is built on the concept of Pipelines and Steps, which give you control over how data is changed and status is tracked. A Pipeline can be as simple as copying data from one object into another of the same type, or be as complex as involving multiple steps, each querying a different external system or database. Petl provides the foundation on which you can build.

Let's start with the most basic example. In some situations, you may need to query multiple systems and combine their data into a single object. Let's say you are combining Contact information from multiple sources -- a CRM, custom database, and so on. We provide a name, so that we can use ASP.NET's keyed services.

builder.Services.AddPetl()
    .WithAutoMapping<Contact, Contact>("combine-contacts");

To use this, we simply inject the keyed service.

public class ContactsController {
    public ContactsController(
        [FromKeyedServices("combine-contacts")]IPipeline<Contact, Contact> pipeline
    ) {
        // .. do something
    }
}

If we want to filter out null properties during the copy, we can provide a callback to the configuration.

services.AddPetl()
    .WithAutoMapping<Contact, Contact>((source, target, value) =>
    {
        // Only copy non-empty strings
        if (value is string str)
            return !string.IsNullOrEmpty(str);

        return value is not null;
    });

Perhaps Dependency Injection is not what you need -- I've got you covered there as well. You can create a PipelineBuilder instance anywhere, build a pipeline and execute it.

var builder = new PipelineBuilder<InputModel, OutputModel>();

builder
    .WithStep("Transform User Data")
        .Property(x => x.FirstName, y => y.FullName)
        .Property(x => x.BirthYear, y => y.Age)
        .Transform((source, target) => {
            target.DisplayName = $"{source.FirstName} ({source.Age})";
        });

var pipeline = builder.Build();

pipeline.Exec(input);

There are a number of additional ways to configure Petl, and more on the way. Start using it today, and provide questions and feedback on the GitHub! It's available on NuGet