Debugging remote HTTP clients - just a pipedream?

If you use Azure Data Factory, Power Automate or Logic Apps you’ll be familar with the REST and HTTP connectors. These permit you to connect to remote APIs and process the result in your pipeline.

The challenge comes when it’s time to debug. Unlike local development, we can’t run Fiddler or Charles to debug headers. Instead we’re running blind.

We spent a bunch of time last year trying to figure out why the ADF REST datasource wouldn’t work with an API when Postman did. Finally we got our vendor to look in their logs: they wanted application/x-javascript. ADF sends application/json. We were trying to override, but ADF was silently ignoring the override - this restriction is now helpfully documented.

So how to debug? I’ve found Pipedream to be really helpful: sign up for a free account, create an event source and you get an endpoint:

Point your HTTP client at the endpoint, make a request and you can see all the request headers. Here’s some from the ADF HTTP client calling the Wikidata Linked Data Fragments service:

It’s a great little service and all you need is a Github or Google login. I hope you find it useful.