Data Streams - API Quickstart

In this tutorial we’ll cover the basics of streaming real-time content through the API.

Note that you need access to the “Data Streams” feature in your API package to use this feature.

Before you start

Take a look at the Platform Overview guide to understand the key concepts of the Meltwater platform.

To run through this tutorial, you will need:

A location that is able to accept HTTP POST requests
- For this tutorial we use webhook.site which is free to use
Your Meltwater API token
A Saved Search in your Meltwater App account

Authentication

You need to provide your API token when you call any Meltwater API endpoint.

You provide the API token as a header called apikey, for example:

curl -X GET \
  --url "https://api.meltwater.com/v3/searches" \
  --header "apikey: **********"

For full details take a look at the Authentication page.

Obtaining a Meltwater search

Hooks are created with an existing Meltwater search.

In this tutorial we’ll cover using an existing search, but you can create / edit searches using the API. See the Managing Saved Searches guide for more details.

To create a hook you need to provide the id of the required search. You can use the GET /v3/searches endpoint to list your current searches:

curl -X GET \
  --url "https://api.meltwater.com/v3/searches" \
  --header "apikey: **********"

Example response:

{
  "searches": [
    {
      "updated": "2018-01-10T14:42:10.000Z",
      "name": "Elon Musk",
      "id": 2382415
    }
  ]
}

Setup a destination service

You are going to need a service that accepts HTTP POSTs with Content-type: application/json and a JSON body. We expect your service to return HTTP status code 200 on success.

For this tutorial we are going to use webhook.site as a replacement for your own server.

If you already have a destination service, you can skip ahead to step 4.

Go to webhook.site you should be presented with a dashboard where you will see events being delivered. Copy “your unique url” this is your destination, we will refer to it as the <target_url>.

To make sure your webhook.site endpoint is working, make a curl call to it:

curl -X POST -i -d "fizz=buzz" "<target_url>"

The response should be similar to following:

HTTP/1.1 200 OK
Server: nginx/1.14.2
Content-Type: text/plain; charset=UTF-8
Transfer-Encoding: chunked
Vary: Accept-Encoding
X-Request-Id: 2c061fd0-f465-11eb-9a03-0242ac130003
X-Token-Id: 28c9f8c8-f465-11eb-9a03-0242ac130003
Cache-Control: no-cache, private
Date: Tue, 03 Aug 2021 14:14:06 GMT

If you check back on the webhook.site you should see your request. Your destination is set up! Let’s proceed and create a hook.

Choosing an output template

The API allows you to choose from a selection of templates for your data stream, which you specify using the template field in your request. The available templates are documented on the Export & Streaming Output Templates page.

We recommend using the general purpose “API JSON” template for most integrations, as this includes all the data fields most customers need.

To use the “API JSON” template you would use the following as part of your request:

"template": {
  "name": "api.json"
}

If you do not specify a template in your request the API will use the legacy output template as documented here.

Creating a data stream

With the destinations target URL primed and ready for data, we are now going to setup a data stream using the search id we looked up previously.

Note that for historical reasons, data streams are called ‘hooks’ in the API endpoints.

Select one <search_id> from any of your searches and run following request to setup a data stream (see POST /v3/hooks):

curl --location 'https://api.meltwater.com/v3/hooks' \
--header 'apikey: **********' \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--data '{
    "search_id": <search_id>,
    "target_url": "<target_url>", 
    "template": {"name": "api.json"}
}'

As response you will receive the created data stream with its id (note the <hook_id>), e.g.:

{
  "search_id": <search_id>,
  "target_url": "<target_url>",
  "updated": "2024-01-01T05:57:47.00Z",
  "hook_id": "<stream_id>",
  "template": {"name": "api.json"}
}

Understanding streamed results

Each time a batch of documents (1 or more) match your search, the Meltwater API will deliver a JSON payload to your target URL. The structure of the payload is as follows:

{
    "request": {
        "company_id": <the id of the account that owns the inputs used>,
        "hook_id": <the id of the stream/hook in the Meltwater system that pushed the data>,
        "inputs": [<the inputs used for the stream, ie. the search]>
    },
    "docs": [
      <an object for each document, according to the chosen template>
    ]
}

Note that prior to the templates feature being introduced, streams used a legacy output format as described here.

Verifying incoming search results

Depending on the volume of results your search produces, you should be receiving incoming results to your <target_url>.

For your production setup you may want to verify the payload signature, see details in Document Verification.

Troubleshooting

Start by taking a look at the streaming section of our Frequently Asked Questions.

No results? If you do not see any incoming requests for some time and you suspect something is not quite working please contact our support team.

You can simulate a webhook request from the Melwater API with following cURL call. This is a typical request that your target_url should expect to receive.

Try making this call from outside your own network to verify there’s no network restrictions preventing external requests reaching your target_url.

curl <target_url> \
  -X POST \
  -H "Content-Type: application/json" \
  -d @- << EOF
[
  {
    "source_reach": 712,
    "source_id": "C054DBBA56E1722EFEC5BF468DF229E8",
    "source_name": "Myinforms",
    "source_country_code": "us",
    "source_subregion": "Pennsylvania",
    "source_ave": 6.59,
    "metadata": {
      "search_name": "hello",
      "search_id": 1234,
      "hook_id": "0a815fa3ee87489de1b932f178541f80c1f1038c",
	  "company_id": "company_id"
    },
    "source_information_type": "news",
    "document_url": "http://myinforms.com/en-us/a/43897824-musk-throws-pie-at-naysayers-as-tesla-posts-rare-profit/",
    "document_title": "Musk Throws 'Pie' at Naysayers as Tesla Posts Rare Profit",
    "document_sentiment": "negative",
    "document_publish_date": "2016-02-01T05:10:14.371Z",
    "document_language_code": "en",
    "document_key_phrases": ["fourth quarter"],
    "document_opening_text": "Tesla Motors Inc. posted an unexpected profit.",
    "document_image_link": "http://i1.myinforms.com/300x200/default.jpg",
    "document_id": "MweugEjvEhiuFbQQmid40Gs1Y-I",
    "document_hit_sentence": "Musk Throws 'Pie'",
    "document_matched_keywords": ["Musk", "Tesla"],
    "document_authors": [{"name": "John Doe", "author_information_type": "news"}]
  }
]
EOF

Listing active hooks

You can query for all active hooks you have created so far (see GET /v3/hooks):

curl -v -XGET \
  --url "https://api.meltwater.com/v3/hooks" \
  --header "apikey: **********"

The response will provide a list of hooks with <search_id>, your defined <target_url> and the <hook_id>:

{
  "hooks": [
    {
      "search_id": <search_id>,
      "hook_id": "<hook_id>",
      "target_url": "<target_url>",
      "updated": "2024-02-02T13:55:13Z",
      "template": {"name": "api.json"},
      "version": "v2"
    }
  ]
}

Deleting your hook

You have verified your hook is working and you are receiving results. You now plan to stop receiving results by deleting the hook again. All you need to do is to provide the <hook_id> in following curl request (see DELETE /v3/hooks/{hook_id}):

curl -v -XDELETE \
  --url "https://api.meltwater.com/v3/hooks/<hook_id>" \
  --header "apikey: **********"

The response should give back HTTP/1.1 204 No Content.

MELTWATER API

Your account

Analyst guides

Developer guides

API reference

Help