Total Page Preview:   000000000815

What is Pipeline in Azure Data Factory

In this article we will discussed about Pipeline In Azure Data Factory.

Pipeline :

A pipeline is a group of activities. Together, the activities in a pipeline perform a task. Data factory may have one or more pipelines. Example, a pipeline could contain a group of activities that take data from an Azure blob, and then run a Hive query on an HDInsight cluster to partition the data. Benefit of using pipeline allows you to manage the activities as a set instead of each one individually. Example, you can deploy and schedule the pipeline, rather than the activities independently.

Activity :

Activities define the actions to perform on your data. A pipeline may have one or more activities. Example, you may use a Copy activity to copy data from one data store to another data store. Similarly, you may use a Hive activity, which runs a Hive query on an Azure HDInsight cluster to transform or analyse your data. 

Data Factory supports two types of activities: 

  • Data movement activities
  • Data transformation activities.

Create Pipeline in Azure Data Factory using portal:

 
 
{
    "name": "PipelineTemplate",
    "properties": {
        "description": "pipe line create for the make activity",
        "activities": [
            {
                "type": "Copy",
                "typeProperties": {
                    "source": {
                        "type": "BlobSource",
                        "recursive": false
                    },
                    "sink": {
                        "type": "SqlSink",
                        "writeBatchSize": 0,
                        "writeBatchTimeout": "00:00:00"
                    },
                    "translator": {
                        "type": "TabularTranslator",
                        "columnMappings": "ProductId:ProductId,ProductName:ProductName,Quantity:Quantity,City:City"
                    }
                },
                "inputs": [
                    {
                        "name": "SourceAzureBlobDataset"
                    }
                ],
                "outputs": [
                    {
                        "name": "DestinationAzureSQLDataset"
                    }
                ],
                "policy": {
                    "timeout": "1.00:00:00",
                    "concurrency": 1,
                    "executionPriorityOrder": "NewestFirst",
                    "style": "StartOfInterval",
                    "retry": 1,
                    "longRetry": 0,
                    "longRetryInterval": "00:00:00"
                },
                "scheduler": {
                    "frequency": "Minute",
                    "interval": 15
                },
                "name": "Activity-0-Products_txt->[dbo]_[tblProducts]"
            }
        ],
        "start": "2017-07-20T10:10:43.355Z",
        "end": "2099-12-30T18:30:00Z",
        "isPaused": false,
        "pipelineMode": "Scheduled"
    }
}
 

 

Thank You

About Author

Brijesh Kumar

Database Developer

I have more then 6 years Experience in Microsoft Technologies - SQL Server Database, ETL Azure Cloud - Azure SQL Database, CosmosDB, Azure Data Factory, PowerBI, Web Job, Azure Function, Azure Storage, Web Apps, Powershall and Database Migration On-Premise to Azure Cloud.
LinkedIn : https://www.linkedin.com



Comments


                           
                           

                           

                           

Facebook User: