Tweet Rehydration

Due to Twitter terms of service, when you export tweets from the Meltwater API you will receive only the ID of the tweet and the tweet’s author. In this guide we look at how you can use these IDs to fetch full details from the Twitter API, in a process called ‘rehydration’.

This process does add a step to your process when using the Meltwater API, but it means that both Meltwater and our clients are compliant with the required terms.

Exported tweet documents

When you run an export, any tweets that are returned will not have content or the details of the tweet author. This is intentional because of the terms Twitter enforces. To get the content of a tweet (and many other useful fields) you will need to rehydrate each tweet using the Twitter API.

Each tweet returned from the Meltwater API will have the following ID fields:

{
  "document_external_id": "1228393702244134912",
  "document_authors": [
    {
      "author_external_id": "1000001234567891234",
      ...
    }
  ],
  ...
}

Use the value of the document_external_id field to fetch tweet details from the Twitter API.

Use the value of the author_external_id field to fetch author details from the Twitter API.

Requesting access to the Twitter API

To be able to call the Twitter API you must first request access.

Details of this process are documented by Twitter and are not covered here.

Twitter have recently updated their API packaging and pricing. Please see the official Twitter API documentation for up-to-date pricing details.

Rehydrating tweets

You can fetch up to 100 tweets in a single API call using the GET /2/tweets endpoint on the Twitter API.

The process is extensively documented on the Twitter developer - Tweet lookup.

The limits applied to this endpoint vary by the authentication method you use, but at time of writing the limits would allow you to rehydrate millions of tweets per month.

When you call the endpoint you express which fields you’d like included in the response. By default the id and text fields are returned, but you can also request ‘expansion’ on fields such as the author.

For example, this request returns when each tweet was created, the author id, and additional details about the author:

curl -X GET \
  --url https://api.twitter.com/2/tweets?ids=1228393702244134912,1227640996038684673,1199786642791452673&tweet.fields=created_at&expansions=author_id&user.fields=created_at \
  --header "Authorization: Bearer $BEARER_TOKEN"

This would return:

{
  "data": [
    {
      "author_id": "2244994945",
      "created_at": "2020-02-14T19:00:55.000Z",
      "id": "1228393702244134912",
      "text": "What did the developer write in their Valentine’s card?\n  \nwhile(true) {\n    I = Love(You);  \n}"
    },
    {
      "author_id": "2244994945",
      "created_at": "2020-02-12T17:09:56.000Z",
      "id": "1227640996038684673",
      "text": "Doctors: Googling stuff online does not make you a doctor\n\nDevelopers: https://t.co/mrju5ypPkb"
    },
    {
      "author_id": "2244994945",
      "created_at": "2019-11-27T20:26:41.000Z",
      "id": "1199786642791452673",
      "text": "C#"
    }
  ],
  "includes": {
    "users": [
      {
        "created_at": "2013-12-14T04:35:55.000Z",
        "id": "2244994945",
        "name": "Twitter Dev",
        "username": "TwitterDev"
      }
    ]
  }
}

We won’t cover all the options here as they are covered in the Twitter documentation, but in summary the Twitter API allows you to be very specific about the fields you’d like returned, and allows you to fetch author details alongside tweet fields in one request.

Rehydrating authors

For most use cases rehydrating tweets will give all the details needed. However, it is also possible to lookup authors (users) using the Twitter API.

You can fetch up to 100 users in a single API call using the GET /2/users endpoint on the Twitter API.

The process is extensively documented on the Twitter developer - User lookup.

Similar to the tweet lookup you can specify which fields you’d like fetched, but as a basic example you can fetch users with the following call:

curl -X GET \
  --url "https://api.twitter.com/2/users?ids=2244994945,6253282" \
  --header "Authorization: Bearer $BEARER_TOKEN"

This would return:

{
  "data": [
    {
      "id": "2244994945",
      "username": "TwitterDev",
      "name": "Twitter Dev"
    },
    {
      "id": "6253282",
      "username": "Twitter",
      "name": "Twitter"
    }
  ]
}