Always Test Everything – Twitter App Update

I’m a firm believer that you should always test things in advance. It helps avoid unexpected last-minute panics when things don’t work as you expected them to. On Friday I’m going to pick a winner from my twitter followers using the twitter app script I wrote back in October 2013.

It Still Works, But…

Since I had no idea where the SD card with that old Raspbian image was, I decided this would be an ideal opportunity to check that the script is still current and working as it should. You’ll be glad to hear that the instructions still work fine and the original script still works. But – yes there’s always a but – I have a lot more twitter followers than I did two years ago and I stumbled across an issue that only kicks in once you have >5000 followers.

I ran the original script and it told me I had exactly 5000 followers. I knew this to be wrong because yesterday I hit 9000 (you tend to notice the even milestones). Why could this be happening? Is it a limitation of tweepy? Some googling revealed that the twitter API is rate-limited. In terms of querying followers it will only allow you to query 5000 followers once per minute, so you have to split your query into sections <=5000.

How To Search More Than 5000 Followers In Tweepy?

It’s easy and obvious when you know, but actually finding out how to do this was quite tricky. It took several rounds of looking at the tweepy documention, the twitter API documentation and stack exchange to work out the code that would do the job for me. In truth, I would never have worked it out from the documentation because it is too hard to understand and there are too few useful examples. This code snippet on stack exchange saved the day for me, but it took a long time to find it. Lines 9 onwards are the interesting bits…

import time
import tweepy

auth = tweepy.OAuthHandler(..., ...)
auth.set_access_token(..., ...)

api = tweepy.API(auth)

# this is the part I actually used, changing McDonalds to raspitv...
ids = []
for page in tweepy.Cursor(api.followers_ids, screen_name="McDonalds").pages():
    ids.extend(page)
    time.sleep(60)

print len(ids)

Documentation Without Examples Is Almost Useless (to me)

This code allows you to Cursor() through multiple pages() of results. I’d seen reference to Cursor() and pages() in both tweepy and Twitter API documention, but it was incomprehensible to me. All I needed was 4 lines of code in an example. A well explained example is worth far more than generalised documentation.

So how does it work? It loops through the pages() of follower_ids, waiting 60 seconds in between each page. Each page is added to the ids list. Once finished, it tells you how many entries there are in the ids list.

1 Minute Delay Per 5000 Followers

This was exactly what I needed! So I tweaked my script to incorporate this code, and it now works as it should. If used on an account with a very large number of followers it will take a long time to run, as it waits 65 seconds between each page of 5000 followers. To run this script for the official @Raspberry_Pi twitter account (with 257k followers) would take >55 minutes (and it might well break something else – WJDK).

Here’s The Code

This script now works and is fit for purpose…

#!/usr/bin/env python2.7
# twitdraw.py based on twitterwin.py by Alex Eames https://raspi.tv/?p=5281
import tweepy
import random
from time import sleep

# Consumer keys and access tokens, used for OAuth
consumer_key = 'insert_your_consumer_key'
consumer_secret = 'insert_your_consumer_secret'
access_token = 'insert_your_access_token'
access_token_secret = 'insert_your_access_token_secret'

# OAuth process, using the keys and tokens
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)

# Creation of the actual interface, using authentication
api = tweepy.API(auth)

user = api.get_user('raspitv')
print user.screen_name
print user.followers_count

ids = []
for page in tweepy.Cursor(api.followers_ids, screen_name="raspitv").pages():
    ids.extend(page)
    if len(page) == 5000:
        print "waiting 65 seconds before requesting next page of results..."
        sleep(65)

print "you have %d followers" % len(ids)

show_list = raw_input("Do you want to list the followers array?")
if show_list in ('y', 'yes', 'Y', 'Yes', 'YES'):
    print ids

def pick_winner():
    random_number = random.randint(0, len(ids)-1)
    winner = api.get_user(ids[random_number])
    print winner.screen_name, random_number

while True:
    pick = raw_input("Press Enter to pick a winner, Q to quit.")
    if pick in ('q', 'Q', 'quit', 'QUIT', 'Quit'):
        break
    pick_winner()

(Edit to add: Thanks to Andrew Scheller for comment 1 below. Most of those tips have been incorporated to improve the code.)

Odd SSL Error

The only other thing that isn’t quite as it ought to be is that there is an odd SSL error message appearing in the output for each tweepy query. There doesn’t appear to be much I can do about this – apart from switching to twython – which I probably will do at some point anyway. But for now, it works, so I don’t much care about that.

And Tomorrow We Use It For Real!

Tomorrow I will use this script to pick a winner at random from my twitter followers. The winner will receive a Joey board from Gooligum Electronics

If you fancy having a go at this yourself, start with the original page here which has all the installation instructions on.