Rehydrating X Posts

Due to X terms of service, when you export X posts from the Meltwater API you will receive only the ID of the post and the post’s author. In this guide we look at how you can use these IDs to fetch full details from the X API, in a process called ‘rehydration’.

This process does add a step to your process when using the Meltwater API, but it means that both Meltwater and our clients are compliant with the required terms.

Exported X posts

When you run an export, any X posts that are returned will not have content or the details of the X author. This is intentional because of the terms XS enforces. To get the content of a post (and many other useful fields) you will need to rehydrate each post using the X API.

Each post returned from the Meltwater API will have the following ID fields:

{
  "document_external_id": "1228393702244134912",
  "document_authors": [
    {
      "author_external_id": "1000001234567891234",
      ...
    }
  ],
  ...
}

Use the value of the document_external_id field to fetch post details from the X API.

Use the value of the author_external_id field to fetch author details from the X API.

Requesting access to the X API

To be able to call the X API you must first request access.

Details of this process are documented by X and are not covered here.

X have recently updated their API packaging and pricing. Please see the official X API documentation for up-to-date pricing details.

Rehydrating posts

You can fetch up to 100 posts in a single API call using the GET /2/tweets endpoint on the X API.

The process is extensively documented on the X developer site - Tweet lookup.

The limits applied to this endpoint vary by the authentication method you use, but at time of writing the limits would allow you to rehydrate millions of posts per month.

When you call the endpoint you express which fields you’d like included in the response. By default the id and text fields are returned, but you can also request ‘expansion’ on fields such as the author.

For example, this request returns when each post was created, the author id, and additional details about the author:

curl -X GET \
  --url https://api.twitter.com/2/tweets?ids=<POST_IDs>&tweet.fields=created_at&expansions=author_id&user.fields=created_at \
  --header "Authorization: Bearer $BEARER_TOKEN"

This would return:

{
  "data": [
    {
      "author_id": "2244994945",
      "created_at": "2020-02-14T19:00:55.000Z",
      "id": "1228393702244134912",
      "text": "What did the developer write in their Valentine’s card?\n  \nwhile(true) {\n    I = Love(You);  \n}"
    },
    {
      "author_id": "2244994945",
      "created_at": "2020-02-12T17:09:56.000Z",
      "id": "1227640996038684673",
      "text": "Doctors: Googling stuff online does not make you a doctor\n\nDevelopers: https://t.co/mrju5ypPkb"
    },
    {
      "author_id": "2244994945",
      "created_at": "2019-11-27T20:26:41.000Z",
      "id": "1199786642791452673",
      "text": "C#"
    }
  ],
  "includes": {
    "users": [
      {
        "created_at": "2013-12-14T04:35:55.000Z",
        "id": "2244994945",
        "name": "Twitter Dev",
        "username": "TwitterDev"
      }
    ]
  }
}

We won’t cover all the options here as they are covered in the X documentation, but in summary the X API allows you to be very specific about the fields you’d like returned, and allows you to fetch author details alongside post fields in one request.

Rehydrating authors

For most use cases rehydrating posts will give all the details needed. However, it is also possible to lookup authors (users) using the X API.

You can fetch up to 100 users in a single API call using the GET /2/users endpoint on the X API.

The process is extensively documented on the X developer site - User lookup.

Similar to the post lookup you can specify which fields you’d like fetched, but as a basic example you can fetch users with the following call:

curl -X GET \
  --url "https://api.twitter.com/2/users?ids=2244994945,6253282" \
  --header "Authorization: Bearer $BEARER_TOKEN"

This would return:

{
  "data": [
    {
      "id": "2244994945",
      "username": "TwitterDev",
      "name": "Twitter Dev"
    },
    {
      "id": "6253282",
      "username": "Twitter",
      "name": "Twitter"
    }
  ]
}