Using Azure Data Factory to get data from a REST API
Solution 1
Data factory offers a generic HTTP connector and a specific REST connector, allowing you to do retrieve data from HTTP endpoints by using GET or POST methods.
Example: HTTP Linked Service
{
"name": "HttpLinkedService",
"properties":
{
"type": "Http",
"typeProperties":
{
"authenticationType": "Anonymous",
"url" : "https://en.wikipedia.org/wiki/"
}
}
}
Solution 2
I have done this using Custom .Net Activities. I had a need to pull data from Salesforce API. I have a write up on how to do this here: http://eatcodelive.com/2016/02/06/accessing-azure-data-lake-store-from-an-azure-data-factory-custom-net-activity/ it is utilizing the data lake store but you can store the data anywhere.
Also have a look here for an example from microsoft: https://azure.microsoft.com/en-us/documentation/articles/data-factory-use-custom-activities/
Hope that helps.
Solution 3
This can be achieved with Data Factory. Data Factory is useful if you want to run batches on a schedule and have a single place for monitoring and management. There is sample code in our GitHub repo for an http loader to blob here https://github.com/Azure/Azure-DataFactory. Then the blob can be consumed for other processing, like putting it into your table. To do this, you would create a pipeline that contains your custom loader, then an activity with blob as the input dataset and table as the output dataset.
Solution 4
https://github.com/Azure/Azure-DataFactory/tree/master/Samples/HttpDataDownloaderSample
Hi this is a custom data activity that dowloads a file by http request. You can achieve what you desire making small changes. Hope it helps.
dagrun
Updated on July 09, 2022Comments
-
dagrun almost 2 years
Is it possible to use Azure Data Factory to get data from a REST API and insert it to a Azure database table?
-
dagrun about 8 yearsThis tutorial uses a azure storage blob as the input data. I want to retrieve data from another source using http calls.
-
Pedro G. Dias about 8 yearsIndeed, but the principle is exactly the same, just replace blob with REST client and you're green :)
-
J Marlow about 4 yearsThe link is dead, use this instead: github.com/Azure/Azure-DataFactory/tree/master/SamplesV1/…