Howto tweet from BASH scripts using OAuth

 

You may or may not be aware that Twitter have stopped third party applications from using Basic Authentication. For those who are not quite clear on the mechanics of authentication, when using Basic Authentication you (or your program) send a Username and Password with each request.

 


The use of Basic Authentication raises a number of concerns;

  • Application has to securely store the users credentials. Could be exposed through a bug or other means


  • If the user changes their password, they will need to update the settings in every application they use


  • Other applications can masquerade as your application - usually spammers/malware


  • Once a user has provided his/her credentials, the only way to guarantee that authorisation is revoked is to change his/her password


  • Gives the Operations Team of the service provider (i.e. Twitter) better visibility into their network

 

 

Although you can encrypt the stored credentials, you cannot use a one-way hash as you need to be able to reverse it for authentication. If you can reverse the encryption, so can a potential attacker. Developers are therefore given the extra responsibility of storing users credentials securely. OAuth removes this problem. Rather than storing a username and password, you simply have to store a unique hash that identifies your application and the user account to use. Indeed, all the concerns above are resolved by OAuth!

 

Without going into too much depth, the OAuth process is as follows;

  1. Developer creates OAuth tokens for his/her application (Known as Consumer Key and Consumer Secret)


  2. User uses Third Party Application


  3. Application requests a Request Token from Twitter (by sending the Consumer Key and Consumer Secret to identify itself)


  4. User is directed to Twitter's User Authorisation URL with the Request Token in the Request URI


  5. The User is asked to log in to Twitter (unless they have 'remember me' set)


  6. The User is asked whether they want to allow the application to access their account.


  7. If the user allows the application access, Twitter marks the Request Token as authorised


  8. The User is directed back to the application, the authorised Request Token is provided to the application (To inform the application that it is now authorised).


  9. The application contacts Twitter and exchanges the authorised Request Token for an Access Token


  10. The application accesses the users account using the Access Token


  11. All further requests use the Access Token


Note: Once the application is authorised, it will skip steps 1 to 8 unless its authorisation is revoked (either by Twitter or the user).

 

This process, quite understandably, can be very intimidating to developers who are not familiar with it. Before we look further into using OAuth to post to Twitter from a BASH script, let's break down what each token actually does;

Token

Information
Consumer Key
Identifies the application
Consumer Secret
The key used to generate a hash of the message contents (i.e. the Tweet)
Note: Must be kept Secret!

Request Token
Used to identify the applications request for authorisation. If authorisation is granted, the Request Token is marked as such and later exchanged for an Access Token

Access Token
The token used by the application to prove that it has been authorised. The authorised Request Token is exchanged for this Access Token



Now that we know what each token does, how do we go about using OAuth in a BASH script? You'll notice that Step 4 requires the user to be directed to Twitter's authentication page, so how do we do that from a shell? We could use a text based browser such as lynx but that would inconvenience our users. So we won't redirect them to Twitter's page, we'll simply ask them to go there when the need arises.

However, as the Tokens used are long and complex, it would be unrealistic to expect the user to manually enter these into the system. The OAuth specification does, however, cater for this eventuality with a slightly modified process for these 'Out of Band (oob)' requests. Steps 1 - 3 and 10 - 11 in the above process remain the same. Steps 4 - 9 are as follows;

  1. User is asked to visit a Twitter URL with the Request Token forming part of the Request URI


  2. The User is asked to log in to Twitter (unless they have 'remember me' set)


  3. The User is asked whether they want to allow the application to access their account.


  4. If the user allows the application access, Twitter Provides the user with a PIN


  5. The User enters the PIN into the application


  6. The application contacts Twitter and exchanges the PIN for an Access Token


Although a small change to the process, it does make our lives far easier, and therefore gives the end-user an easier time! So we need a script that will allow us to use OAuth's Out Of Band process in order to authenticate with the users Twitter account.

 

You'll be pleased to know that we don't need to invent the wheel. An OAuth module has already been written in BASH, the author has even created a Twitter specific library and a CLI made them all available for free under a BSD License.

The only slight drawback is that they are written to work as an installed application, in this article we are more interested in using the scripts as modules. This has the advantage that we can distribute the Twitter code with our application (rather than requiring the user to install the code as a dependancy). The disadvantages of using the scripts in this way are;

  • We have to amend the scripts to work to our needs


  • The user could potentially end up with multiple copies of the same script if many applications use it


The first disadvantage is obviously of little concern to the end user, although they may be a little more concerned about the duplication of libraries (especially if disk space is tight). However, it is possible for us to work around the second disadvantage, and inconvenience our users as little as possible. We will assume that the user won't mind the duplication, as searching for other instances and setting the appropriate configuration is outside the scope of this guide.

So let's write a simple BASH script to post information to Twitter.

 

  1. Register your application with Twitter here.


  2. Make a note of the keys that Twitter gives you (Note: They are further down the page than you might expect), we'll need them very soon!


  3. Create a directory called tmpdir_maintenance


  4. Create a directory inside tmpdir_maintenance called mod_Twitter


  5. Download OAuth, TwitterOAuth and tcli.sh  from Google Code or grab my pre-modified version (read the rest of the article though, it's important to know how it works!).


  6. Move OAuth.sh, TwitterOAuth.sh and tcli.sh to mod_Twitter (Or extract the tarball in tmpdir_maintenance)


  7. Create a text file called tmpdir_maintenance.sh in the tmpdir_maintenance directory


Now put the following code into tmpdir_maintenance.sh;


    #!/bin/bash
    #
    #
    # Filename: tmpdir_maintenance.sh
    #
    # Copyright (c) 2010, B Tasker
    # All rights reserved.
    #
    # Redistribution and use in source and binary forms, with or without
    # modification, are permitted provided that the following conditions are met:
    #
    # * Redistributions of source code must retain the above copyright notice,
    # this list of conditions and the following disclaimer.
    #
    # * Redistributions in binary form must reproduce the above copyright notice,
    # this list of conditions and the following disclaimer in the documentation
    # and/or other materials provided with the distribution.
    #
    # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
    # AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
    # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
    # ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
    # LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
    # CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
    # SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
    # INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
    # CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
    # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
    # POSSIBILITY OF SUCH DAMAGE.
    #
    #
    # A simple script to clear temporary files and
    # report it's status to the adminsitrator
    #
    #
    # Utilises the BASH OAuth libraries from Yu-Jie Lin
    # http://blog.yjl.im/2010/05/bash-oauth.html
    #
    # We want to report the following information to the SysAdmin
    #
    # Date/Time
    # Number of Files in Temporary Directory
    # Number of Files successfully removed
    #
    # As this is a simple example script, we're not going to
    # recurse into directories.


    ########### Variables ###################


    # Where is the log file to write to?
    LOGFILE="/var/log/tmpdir_maintenance.log"

    # Where are your temporary files stored?
    TEMP_DIRECTORY="/tmp"

    # Where is this script?
    PROG_ROOT="/home/ben/sandbox/src/BASH/tmpdir_maintenance/"

    ########### Variables End ###############


    # We want to be in the Temporary Directory
    cd "$TEMP_DIRECTORY"


    # Use ls to grab the file count
    # Double space after -F\
    FILE_COUNT=$( ls -l | grep total | awk -F\ '{print $2}')


    # Take note of the number of directories
    # For some reason ls doesn't display total when limited to directories
    # So we need to count them
    #
    # Again, note the double space after -F\
    DIR_COUNT=$( ls -ld | wc -l | awk -F\ '{print $1}')

    # Subtract the number of directories from the File Count
    FILE_COUNT=$(( $FILECOUNT - $DIR_COUNT))

    # Set our Date Stamp
    DATESTAMP=$( date +'%d%m%Y %H%M')

    # We've not deleted anything yet, so lets zero our counters
    DEL_COUNT="0"
    FAILED_COUNT="0"

    # We now have a file count. Begin deleting files one by one

    for a in *;
    do

    # Check whether it's a directory

    if [ -d "$a" ]
    then

    # It is so don't touch it, or the count
    echo "Skipping Directory $a"

    else

    # It's not a Dir, try and delete it
    rm -f "$a"

    # Check whether it worked

    if [ "$?" == "0" ]
    then
    # Was successful
    # Add one to the Del_Count
    DEL_COUNT=$(( $DEL_COUNT + 1 ))

    else
    # Was Unsuccessful
    # Add one to the failed count
    FAILED_COUNT=$(( FAILED_COUNT + 1 ))

    fi

    fi

    done


    # We've now attempted to delete all files in TEMPDIR. So lets report our success;
    #
    # We need to ensure it doesn't exceed 140 characters or posting to Twitter will fail
    # Note: HOSTNAME will display the DNS hostname of the system
    STATUS_MESSAGE="$DATESTAMP $HOSTNAME $FILE_COUNT files in $TEMP_DIRECTORY. \
    $DEL_COUNT Removed. $FAILED_COUNT Failed"

    # Record the activity in a local log
    /bin/cat << EOM >> "$LOGFILE"
    $STATUS_MESSAGE
    EOM

    # Use the Twitter modules to post the information
    cd "$PROG_ROOT"/mod_Twitter
    ./tcli.sh -c statuses -s "$STATUS_MESSAGE" > "$TEMP_DIRECTORY"/tmpdir_maintenance.$$.tmp

    # Status should have posted to Twitter. Parse the response to check that it has;
    cat "$TEMP_DIRECTORY"/tmpdir_maintenance.$$.tmp | grep \ > /dev/null

    if [ "$?" == "1" ]
    then

    # An error occurred
    echo "An error occurred posting to Twitter" >> "$LOGFILE"

    fi

    # Tidy up
    rm -f "$TEMP_DIRECTORY"/tmpdir_maintenance.$$.tmp

    # Exit the script
    echo "Saying Goodbye!"
    exit

 

We now have a very simple script that will count all files in the Temp Directory, delete any that aren't directories, and report to the SysAdmin any files that couldn't be deleted. However, we're not quite ready to add it to our crontab just yet! We need to configure and modify mod_Twitter to work in the way we want.

Note: If you downloaded my tarball of ready modded files, you only need to follow step J.

  1. Enter the mod_Twitter directory


  2. Create a text file called .tcli.rc


  3. Enter the following information into .tcli.conf;

      oauth_consumer_key="YOURCONSUMERKEY"
      oauth_consumer_secret="YOURCONSUMERSECRET"


  4. Close and save .tcli.rc


  5. Open tcli.sh


  6. Locate the two lines beginning with;

      OAuth_sh=$(which TwitterOAuth.sh)(( $? != 0 )) && echo 'Unable to locate TwitterOAuth.sh! 
      source "$OAuth_sh"
  7. And replace with;
      source TwitterOAuth.sh

  8. Open TwitterOAuth.sh


  9. Locate the following lines;

      OAuth_sh=$(which OAuth.sh)
      (( $? != 0 )) && echo 'Unable to locate OAuth.sh! Make sure it is in searching PATH.' && exit 1
      source "$OAuth_sh"


    And replace with;

      source OAuth.sh

 

We've now edited the modules to behave in the way we need.

There is one last thing that we need to do before the script will work. You'll recall that in Stages 4 - 7 of the OAuth process, the user is asked to authenticate and allow access to the application. The script we've written isn't able to do this for us (as that would require storing their username and password - negating the point of OAuth). We only need to authenticate in this way once per account that we wish to post to (in this example - one).


So complete the following steps.

  1. Enter the directory mod_Twitter (You should already be here!)


  2. Run the following commands;

      chmod +x *.sh
      ./tcli.sh -c statuses -s "$HOSTNAME Ready to start logging"

  3. When the script asks you to visit a Twitter page, copy and paste the link to your Web browser

  4. Log in to Twitter

  5. You should be asked whether you wish to authorise the application to access your account - Select 'Allow'

  6. Twitter will provide you with a PIN - Enter this into the PIN prompt of tcli.sh

  7. The program will exchange the PIN for an Access Token and then publish your Tweet

 



The application should now be able to tweet without requiring the user to enter their credentials, a PIN or anything else. The user can manage which applications are authorised from their Twitter settings, and our application will continue to work even if the users password changes.

If you want to be doubly sure that the system is able to tweet, run the following command from within the mod_Twitter directory;


    ./tcli.sh -c statuses -s "Tempdir Maintenance Script is Good to Go"

You simply need to make the script tmpdir_maintenance.sh executable;


    # From the mod_Tweet Directory
    cd .. && chmod +x tmpdir_maintenance.sh

Now you simply have to schedule tmpdir_maintentance.sh to run, whether by adding it to your crontab or by calling it from another script that is run regularly.

 

 

As intimidating as OAuth may have seemed when we started, we've managed to write a simple script that uses OAuth to authenticate without user intervention. Our script will only fail to authenticate if either the user, or Twitter themselves, revoke its authorisation.

Although using Twitter as a remote log is an unusual use of the service, the creating this script should hopefully have equipped you with the knowledge necessary to write more complex scripts. Whether it's a script for your website, a custom Twitter interface, or even a logging system, you should now have the resources you need to allow you to properly utilise OAuth.


.