I'm using npm-twit to get followers of a specific account.
The Twitter API returns up to 5000 results from a single GET request.
If the user I'm querying has over 5000 followers a "next_cursor" value is returned with the data.
To get the next 5000 results, I need to re-run the GET function, passing it the "next_cursor" value as an argument. I just can't seem to work out how to do it.
I was thinking a while loop, but I can't reset the global variable, I think because of scope:
var cursor = -1
while ( cursor != 0 ) {
T.get('followers/ids', { screen_name: 'twitter' }, function (err, data, response) {
// Do stuff here to write data to a file
cursor = data["next_cursor"];
})
}
Obviously I'm not a JS genius, so any help would be much appreciated.
I'm using npm-twit to get followers of a specific account.
The Twitter API returns up to 5000 results from a single GET request.
If the user I'm querying has over 5000 followers a "next_cursor" value is returned with the data.
To get the next 5000 results, I need to re-run the GET function, passing it the "next_cursor" value as an argument. I just can't seem to work out how to do it.
I was thinking a while loop, but I can't reset the global variable, I think because of scope:
var cursor = -1
while ( cursor != 0 ) {
T.get('followers/ids', { screen_name: 'twitter' }, function (err, data, response) {
// Do stuff here to write data to a file
cursor = data["next_cursor"];
})
}
Obviously I'm not a JS genius, so any help would be much appreciated.
Share Improve this question asked Jan 18, 2015 at 10:56 user1385827user1385827 3351 gold badge7 silver badges23 bronze badges3 Answers
Reset to default 8The issue you are having is due to Node.js being asynchronous.
T.get('followers/ids', { screen_name: 'twitter' }, function getData(err, data, response) {
// Do stuff here to write data to a file
if(data['next_cursor'] > 0) T.get('followers/ids', { screen_name: 'twitter', next_cursor: data['next_cursor'] }, getData);
})
}
Please note:
- I gave a name to the internal callback function. That is so that we can recursively call it from the inside.
- The loop is replaced with a recursive callback.
- If there is a next_cursor data, then we call
T.get
using the same functiongetData
.
Be aware that Do stuff here code will be executed many times (as many as there are next cursors). Since it is recursive callback - the order is guaranteed.
If you do not like the idea of recursive callbacks, you can avoid it by:
- Finding out beforehand all the next_cursor's if possible, and generate requests using for loop.
- Alternatively, use asynchronous-helper modules like Async (though for learning purposes, I would avoid modules unless you are fluent in the concept already).
Consider testing with some 5K+ account.
const T = new Twit(tokens)
function getFollowers (screenName, followers = [], cur = -1) {
return new Promise((resolve, reject) => {
T.get('followers/ids', { screen_name: screenName, cursor: cur, count: 5000 }, (err, data, response) => {
if (err) {
cur = -1
reject(err)
} else {
cur = data.next_cursor
followers.push(data.ids)
if (cur > 0) {
return resolve(getFollowers(screenName, followers, cur))
} else {
return resolve([].concat(...followers))
}
}
})
})
}
async function getXaqron () {
let result = await getFollowers('xaqron')
return result
}
console.log(getXaqron().catch((err) => {
console.log(err) // Rate limit exceeded
}))
Struggled with this one.. Everything seemed to work, but data['next_cursor'] didn't change, EVER!
Code should be like this:
T.get('followers/ids', { screen_name: 'twitter' }, function getData(err, data, response) {
// Do stuff here to write data to a file
if(data['next_cursor'] > 0) T.get('followers/ids', { screen_name: 'twitter', cursor: data['next_cursor'] }, getData);
})
}
Parameter for Twit isn't "next_cursor", it's just "cursor" ;)