TikTok Tweet 1.0 GA Release

Hey guys, I’m back again and very pleased to announce that TikTok Tweet 1.0 GA has just been released. This has been an exciting and rewarding journey of learning (and a little frustration!) for me, and I’m pleased to be able to give you a tour of the finished product.

Getting Started

Once you clone the repo from Github, you can check out the project’s structure:
Screen Shot 2015-04-23 at 4.41.58 PM

Go into the conf folder and right off the bat, you need to edit config.txt to your liking:
Screen Shot 2015-04-24 at 12.06.38 PM

[connection]
host = 127.0.0.1
keyspace = tiktok

Here you need to specify the address of your cluster/node, as well as the keyspace where the tweets and related data are going to be held. This is so that cqlengine can connect to your database.

[replies]
in_reply_to = @CassPopQuiz
account_id = 2815304775

Add “@YourUserName” and your twitter count id such that the program can collect replies to yor tweets

[conf]
CONSUMER_KEY = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
CONSUMER_SECRET = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
ACCESS_KEY = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
ACCESS_SECRET = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Add your twitter auth information so that TikTok Tweet can have access to you Twitter account

[time]
my_time = 10

Set how frequently you would like TikTok Tweet to check the queues for new tweets to send (in seconds)

[db_settings]
keyspace = tiktok
host = 127.0.0.1
# for replication
strategy_class = SimpleStrategy
replication_factor = 1

database settings for the initial set up of the keyspace and tables. I’m am assuming the user has some knowledge of Cassandra replication strategies here.

Once you’ve set your config file, you can start up

$ bash job.sh

and the app should be available in your browser:

http://127.0.0.1:8000/tiktokadmin/

Create a Tweet

First type the text of your tweet into the Create a Tweet page (limit of 140 characters of course!)
Screen Shot 2015-04-24 at 3.08.36 PM

For now we are just going to save it in the database, not going to add it to a queue just yet. Just type in your tweet and hit the Create button.
Screen Shot 2015-04-24 at 3.09.18 PM
Now let’s create a queue though the Manage a Queue section. Give your new queue a name and hit Submit:
Screen Shot 2015-04-24 at 4.48.07 PM

Head into the Schedule a Tweet our tweet part of our app and find our tweet waiting to be added to a queue:
Screen Shot 2015-04-24 at 3.12.27 PM
Let’s add this to a queue. Set a date:
Screen Shot 2015-04-24 at 3.15.38 PM

…and time:
Screen Shot 2015-04-24 at 3.15.58 PM

Add viola! Our tweet appears on our timeline at approximately the time we chose:
Screen Shot 2015-04-24 at 3.31.49 PM

When someone responded to one of our tweets, it gets stored in our database:
Screen Shot 2015-04-24 at 3.37.10 PM

To see responses to tweets, to the Manage Responses section of the site and select the queue from which you would like to see associated responses:
Screen Shot 2015-04-24 at 3.37.41 PM

We can see the replies to our tweets in the order they came in
Screen Shot 2015-04-24 at 3.37.53 PM

I think this application will help me a lot in managing my daily Cassandra Pop Quiz on Twitter, now all I need is to come up with some great questions! Any body who wants to download the source good and give me some feedback, it would be much appreciated. This is the first full stack application I’ve completed all the way through, so I’m definitely looking for some helpful tips and hints. I’d still like to add some other features, like a UI for administration. I’ll be continuously looking for ways to improve this app and more cool features to add.

Advertisements
Posted in Uncategorized | Leave a comment

Using a Datepicker and jQuery Validation with TikTokTweet

You may remember I was using a not very fun looking text field for date/time input. This is pretty much a users worst nightmare, but at the time I was focused on getting the bulk of the app developed and I swore I would return to this later with a better solution. This is pretty sad/funny:

Screen Shot 2015-03-30 at 7.02.30 PM

I went in search of better UI component for my users, because they deserve better, and typing a proper date is almost impossible. I envisioned a jQuery dropdown calendar after doing some research on better methods, Bootstrap 3 Date/Time Picker turned out to be the solution I chose. I think it works great and looks really nice. Here are some screen shots:

Screen Shot 2015-03-30 at 7.02.46 PM

Screen Shot 2015-03-30 at 7.02.58 PM

I installed the datepicker using bower. Bower will find all the necessary packages and dependencies needed and put them in bower_components folder with the bootstrap stuff.

bower install eonasdan-bootstrap-datetimepicker#latest --save

I made sure I had necessary head entries for script and styles needed to use datepicker in the template. Some of these were already present from using bootstrap.

<head>
  <!-- ... -->
  <script type="text/javascript" src="/bower_components/jquery/jquery.min.js"></script>
  <script type="text/javascript" src="/bower_components/moment/min/moment.min.js"></script>
  <script type="text/javascript" src="/bower_components/bootstrap/dist/js/bootstrap.min.js"></script>
  <script type="text/javascript" src="/bower_components/eonasdan-bootstrap-datetimepicker/build/js/bootstrap-datetimepicker.min.js"></script>
  <link rel="stylesheet" href="/bower_components/bootstrap/dist/css/bootstrap.min.css" />
  <link rel="stylesheet" href="/bower_components/eonasdan-bootstrap-datetimepicker/build/css/bootstrap-datetimepicker.min.css" />
</head>

Now in place of my old date input, get to use this. A little chunkier, but the javascript makes it a whole lot swankier.

<div class="container">
   <label>Time to tweet </label>
   <div class="row">
       <div class='col-sm-6'>
           <div class="form-group">
               <div class='input-group date' id='datetimepicker1'>

                   <input type='text' class="form-control" name="when_to_tweet" />
               <span class="input-group-addon"><span class="glyphicon glyphicon-calendar"></span>
               </span>
               </div>
           </div>
       </div>
       <script type="text/javascript">
           $(function () {
               $('#datetimepicker1').datetimepicker();
           });
       </script>
   </div>
</div>

We can even do some cool tricks here. On my tweet_edit template, I made it such that upon editing a tweet, the time to tweet is shown as well, and the user can observe it and decide on a different time if they so choose. This involves adding a default date to the function, mine being the original time to tweet held by the time_to_send variable:

<script type="text/javascript">
   $(function () {
       $('#datetimepicker1').datetimepicker({
           defaultDate: new Date('{{time_to_send}}')
       });
   });
</script>

Another important feature that I thought should be implemented was form validation. We need to make sure that even if the user makes a mistake on input, we don’t have anything unusual going into our database that could cause trouble down the line. Like a blank tweet for instance. I used the jQuery Validation plugin for this. I downloaded and installed the plugin, just like bootstrap and datepicker, with Bower:


bower install jquery-validation

And then added an additional script to the top of my template in the head:

<script type="text/javascript" src="/static/bower_components/jquery-validation/dist/jquery.validate.js"></script>

Now that that the plugin was ready to use, I went looking through my templates for possible scenarios that could trip the user up. For instance, if they tried to create “empty” tweet with no text. I wanted to prevent this blank tweet from ever entering the database.

Screen Shot 2015-03-30 at 7.03.22 PM

jQuery validation made it super easy to put safeguards in place. Adding “required” to the form field will cause an alert to come up and stop things from going any further. Also notice as well that I put in maxlength=”140″, you may notice that this is the max number of characters for a tweet on Twitter. The user will not be permitted to enter any more characters once they have reached 140. You can also write your own rules and messages to use with jQuery plugin, but I didn’t see the need to go there just yet.

<h2>Create a Tweet</h2>

<textarea class="form-control"  id="tweettext" name="tweettext"  maxlength="140"  required></textarea>

jQuery form validation is seriously easy to use, and they have some great docs and a nice instructional video on their site. You should go check it out, as well as their other methods.

Posted in Uncategorized | Leave a comment

Using Twitter Bootstrap to build a front-end for TikTokTweet

Hey folks! I’m back again with another update on my Cassandra/Django project, TikTokTweet! Has you may recall from my last couple blog posts, my UI was pretty sad/non-existent, and so once I felt the backend of the this app was working pretty good, the frontend was definitely in need of some love and attention.

Screen Shot 2015-03-20 at 2.58.14 PM

This looks similar to a Tech.ed project i did when I was twelve. My CSS/Javascript skills are questionable at best. Enter Twitter Bootstrap, like a knight in shining “UI” armor. Bootstrap is a very popular HTML, CSS and JavaScript framework. With its free and open-source tools, it makes frontend development quick and easy for everyone. It’s great for people who want to sleek looking websites or apps, but don’t have the skill or aptitude to design them. It has predefined CSS classes and HTML designed templates for a bunch of components, like forms, buttons, navigation, and lots more, as well as Javascript extensions to make any website look pro and stylish. Since the arrival of Bootstrap 3.0, the focus has been on responsive design by default, to give mobile users the best experience possible.

To get going with Bootstrap on my Django project, I first need to set up a file to serve my static files from within the project, and make configuration changes to my settings.py file. This is necessary due to the dynamic nature of Django. I used these blog post as a guideline, and it seemed to work out well. First, I went into my settings.py file and make these changes:

import os
PROJECT_DIR = os.path.dirname(os.path.abspath(__file__))
STATIC_ROOT = os.path.join(PROJECT_DIR, '../static')

# Set this:
STATIC_URL = ‘/static/’

# This is what my TEMPLATE_CONTEXT_PROCESSORS looked like:
TEMPLATE_CONTEXT_PROCESSORS = (
   'django.core.context_processors.static',
   'django.contrib.auth.context_processors.auth'
)
 
# In the urls.py folder (the one in the same folder as settings.py) I added this line to the end of import statments:
from tiktoktweet.settings import STATIC_ROOT

# And added this to the end of the file:

urlpatterns += patterns('',
                       url(r'^static/(.*)$', 'django.views.static.serve', {'document_root': STATIC_ROOT, 'show_indexes' : True}),

Now I would be able to serve my bootstrap files from the static folder.

I then went into this file and install bootstrap using Bower. Bower is a package management system for client-side programming. It requires Node, npm, and Git. It should bring in any other required dependencies as well with your download.

bower install bootstrap

In comes everything you’ll be needing in the bower_components folder,

bower_components|⇒ ls
bootstrap jquery

There is a whole lot of stuff in here. The bootstrap/dist folder contains most of the necessary the CSS and Javascript I need.

bootstrap/dist
├── css
│ ├── bootstrap-theme.css
│ ├── bootstrap-theme.css.map
│ ├── bootstrap-theme.min.css
│ ├── bootstrap.css
│ ├── bootstrap.css.map
│ └── bootstrap.min.css
├── fonts
│ ├── glyphicons-halflings-regular.eot
│ ├── glyphicons-halflings-regular.svg
│ ├── glyphicons-halflings-regular.ttf
│ ├── glyphicons-halflings-regular.woff
│ └── glyphicons-halflings-regular.woff2
└── js
├── bootstrap.js
├── bootstrap.min.js
└── npm.js

So now that I had my bootstrap files, how would make my HTML “Bootstrapped”? Let’s take a look at one of my original HTML files that allows a user to input a tweet.

<!DOCTYPE html>
<html>
<body>

<form action="{% url 'tiktokadmin:create_post'  %}" method="post">
    {% csrf_token %}
    <h2>Create a Tweet</h2>
    <textarea class="medium" name="tweettext" cols="20" rows="5" ></textarea>

<br>

    <label>Add to queue:</label>
    <select name = "queue_id">
        <option value = 0>Do not add to queue</option>
        {% for queue in queue_list %}
        <option value = "{{queue.id}}">{{queue.name}}</option>
        {% endfor %}
    </select>
    <br><br>
    <label>Time to tweet in the form yyyy-mm-dd hh:mm:ss</label>
    <div class="element-input"><label class="title"></label><div class="item-cont"><input class="large" type="datetime" name="when_to_tweet" placeholder="yyyy-mm-dd hh:mm:ss"/><span class="icon-place"></span></div></div>

    <br>

    <input type="submit" value="Submit"/>

</form>
<br>
<form action="/tiktokadmin/">
    <input type="submit" value="Home" id="jsSubmit1" >
</form>

</body>
</html>

Now in the , lets include all the files we need to make this thing Bootstrap saavy if we want to. Remember these files are being served from our static folder. Bootstrap makes use of HTML5 doctype, so be sure to include it at the beginning.

<!DOCTYPE html>
<html>
<head>
    <!-- ... -->
    <script type="text/javascript" src="/static/bower_components/jquery/dist/jquery.min.js"></script>
    <script type="text/javascript" src="/static/bower_components/bootstrap/dist/js/bootstrap.min.js"></script>
    <link rel="stylesheet" href="/static/bower_components/bootstrap/dist/css/bootstrap.min.css" />
</head>

<body>
....

We can now start adding the magic of bootstrap to our template. One of the really powerful things about Bootstrap is the responsive grid system, allowing your app to look just as nice on a desktop’s browser as a mobile device. This grid system will scale up to 12 columns, and so your content will be compartmentalized into rows and columns on the page. Having control over the number and arrangement of these “blocks” allows us to organize the layout of our page quite easily. For example, top row here, made up of .col-md-* grid classes, will start off stacked on a mobile phone, but become horizontal on a desktop browser.

Screen Shot 2015-03-19 at 12.30.57 PM

 <div class="row">
  <div class="col-md-1"><p>.col-md-1</p></div>
  <div class="col-md-1"><p>.col-md-1</p></div>
  <div class="col-md-1"><p>.col-md-1</p></div>
  <div class="col-md-1"><p>.col-md-1</p></div>
  <div class="col-md-1"><p>.col-md-1</p></div>
  <div class="col-md-1"><p>.col-md-1</p></div>
  <div class="col-md-1"><p>.col-md-1</p></div>
  <div class="col-md-1"><p>.col-md-1</p></div>
  <div class="col-md-1"><p>.col-md-1</p></div>
  <div class="col-md-1"><p>.col-md-1</p></div>
  <div class="col-md-1"><p>.col-md-1</p></div>
 <div class="col-md-1"><p>.col-md-1</p></div>
</div>
  <div class="row">
    <div class="col-md-6"><p>.col-md-6</p></div>
    <div class="col-md-6"><p>.col-md-6</p></div>
</div>
  <div class="row">
    <div class="col-md-4"><p>.col-md-4</p></div>
  <div class="col-md-4"><p>.col-md-4</p></div>
    <div class="col-md-4"><p>.col-md-4</p></div>
</div>
  <div class="row">
    <div class="col-md-8"><p>.col-md-8</p></div>
    <div class="col-md-4"><p>.col-md-4</p></div>
</div>

One can also make offset columns if you would like to move your columns to the right., using the .col-md-offset-* classes. This increases the left margin of a column by *

Screen Shot 2015-03-19 at 12.48.58 PM

 <div class="row">
      <div class="col-md-4"><p>.col-md-4</p></div>
      <div class="col-md-4 col-md-offset-4"><p>.col-md-4 .col-md-offset-4</p></div>
</div>
<div class="row">
  <div class="col-md-3 col-md-offset-3"><p>.col-md-3 .col-md-offset-3</p></div>
  <div class="col-md-3 col-md-offset-3"><p>.col-md-3 .col-md-offset-3</p></div>
</div>
<div class="row">
  <div class="col-md-6 col-md-offset-3"><p>.col-md-6 .col-md-offset-3</p></div>
</div>

Using the various predefined CSS classes in bootstrap, I was able to bring TikTokTweet from something like this:

Screen Shot 2015-03-20 at 2.58.14 PM

To something a little more like this, in a matter of a few hours. Maybe not the most striking example you’ll find, but pretty good for someone with some with nearly no experience in frontend development I think.

Screen Shot 2015-03-20 at 3.04.41 PM

Simple adding .form-control to my queue selection drop down menu brings its from this:

Screen Shot 2015-03-19 at 2.39.11 PM
to this:

Screen Shot 2015-03-19 at 2.40.07 PM

<select class = "form-control" name = "queue_id">
        <option value = 0>Do not add to queue</option>
        {% for queue in queue_list %}
        <option value = "{{queue.id}}">{{queue.name}}</option>
        {% endfor %}
    </select>

I’m really glad someone recommended Bootstrap. It took me a bit to get the hang of, but once I caught, it became clear as day, and I know without it, doing the frontend of my app would have been a real uphill battle. I’d really like to know a bit more about implementing templates, like some of the really sleek looking ones on Start Bootstrap. In my next blog post, I’m gonna talk a bit about how I changed my date/time input, as well as form validation. See you then!!

Posted in Uncategorized | Leave a comment

Tying TikTokTweet with Twitter using Tweepy and CQLEngine

Hello World! I’m back to discuss the backend of the app I’m creating. The backend and frontend of the application both make calls to the database, but they operate independently from each other. The front deals with the user input and putting it into database, but has no direct contact with Twitter. The back end apps relay information between Twitter and the database (stored tweets to send out and replies) but remains untouched by the user. I think of Twitter as the hinge between the two different parts of the app.

This part of the application, send_tweets.py, is responsible for checking the “queues” and sending the tweets in the databases on a schedule according to their associated timestamp. You must first connect CQLEngine to your database using the ip address and keyspace, and sync appropriate models that this:

logging.basicConfig(level=logging.INFO)
connection.setup(['127.0.0.1'], "tiktok")
logging.info("Connected to tiktok database")

sync_table(Tweets)
sync_table(Queue)
sync_table(Tweets_queue)

I also used logging to track events that happen as the program runs. There are a few logging levels to choose from, and the best choice will differ depending on your use case. I chose ‘INFO’ to ensure that I can watch to make sure things are working as I expected them to. Check here to see more info on logging.

You enter your twitter application keys and tokens accordingly. This part of the app will be grabbing tweet from the database and sending them to the twitter account it has the credentials for, and will therefore need access. I used the Configuration File Parser module to keep my authentication hidden. This is important because I’m pushing all my code up to Github to for others to experiment with and use as a learning tool. The idea would be for them to connect to their own Twitter account. The parser reads all my auth data from a YAML file of sorts. Obviously I don’t include that in the code I push to Github.

parser = SafeConfigParser()
parser.read('config.txt')

CONSUMER_KEY = parser.get('conf', 'CONSUMER_KEY')
CONSUMER_SECRET = parser.get('conf', 'CONSUMER_SECRET')
ACCESS_KEY = parser.get('conf', 'ACCESS_KEY')
ACCESS_SECRET = parser.get('conf', 'ACCESS_SECRET')
auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
auth.set_access_token(ACCESS_KEY, ACCESS_SECRET)
api = tweepy.API(auth)

g= Queue.objects.all()

For each one of our Queue objects, we take the ids, and we filter out objects from the Tweet_queue table with a datetime less than the current time. These represent the tweets that are due to be sent out on Twitter.The app will look for tweets in the queues where a time_to_send is less than the current time, and send these tweets out to the twitter feed

for x in g:

    q= Tweets_queue.objects.filter(Tweets_queue.queue_id == x.id, Tweets_queue.time_to_send < datetime.utcnow())

Now we get the tweet text for each of the tweet_queue objects we’ve filtered out, and update our twitter status with it. Meanwhile, we update the tweet_sent table with our newly tweeted tweet, and delete it from tweets_queue table. This is to help stop things from getting messy. A twitter id is also created by the twitter API once the status has been update. We collect this as status.id and make an entry in the Tweets_sent_by_twitter_id. This will be useful in collecting to replies to this tweet later.

for entry in q:
        tweet = Tweets.get(id = entry.tweet_id)
        logging.info("Checking for tweets in for queue_id = %s" %  entry.queue_id)
        status = api.update_status(tweet.tweet)
        Tweets_sent.create(queue_id = entry.queue_id, time_sent = status.created_at, tweet_id = tweet.id)
        Tweets_queue.objects(queue_id = entry.queue_id, time_to_send = entry.time_to_send).delete()
        Tweets_sent_by_twitter_id.create(twitter_id = status.id, queue_id = entry.queue_id, tweet_id = tweet.id)
        logging.info("Sent tweet")

What about check_for_replies.py?This part of the application will look at the Twitter timeline and capture replies to tweets in specific queues. These will be stored in your database in the queue_tweet_responses table.

First we set up the connection for CQLEngine with the database, including the ip address and keyspace:

logging.basicConfig(level=logging.INFO)
connection.setup(['127.0.0.1'], "tiktok")
logging.info("Connected to tiktok database")

Next we enter the twitter application keys and tokens such that the webapp may have access to our twitter account.

consumer_key = parser.get('conf', 'CONSUMER_KEY')
consumer_secret = parser.get('conf', 'CONSUMER_SECRET')
access_token = parser.get('conf', 'ACCESS_KEY')
access_token_secret = parser.get('conf', 'ACCESS_SECRET')

The StdOutListener class will be responsible for receiving data receive data from the stream. Twitter offers a set of streaming APIs that enables us to have low latency access to the tweet data on Twitter. A streaming client will be pushing messages from twitter, indicating that tweets and events have occurred. We don’t experience the same overhead as we would with REST polling to pull data.The fill_in_timeline method will determine if the tweet on your timeline is a reply. The metadata for these tweets will be collected into the queue_tweet_responses table.

class StdOutListener(tweepy.StreamListener):
    ''' Handles data received from the stream. '''

    def on_status(self, status):
        list = Tweets_sent_by_twitter_id.objects.filter(twitter_id = status.in_reply_to_status_id)
        tweet = list.get()
        Queue_tweet_responses.create(queue_id = tweet.queue_id , time_received = columns.TimeUUID.from_datetime(status.created_at), tweet_id = tweet.tweet_id, response = status.text, user = status.author.screen_name )

        return True

How do we restrict this to just replies and not grab everything on the timeline? I’ve put in a filtration device, the fill_in_timeline function. This uses the Twitter API search function, that can look for tweets on the timeline that contain a specific string of text. I choose ‘CassPopQuiz’ as it is alway in a reply.

Screen Shot 2015-02-21 at 11.31.04 PM

def fill_in_timeline(auth):
    api = tweepy.API(auth)
    public_tweets = api.search('@CassPopQuiz', count =100)
    for status in public_tweets:

At this point, I also made sure sure we were only dealing with tweets on the timeline that were replies by ensuring that they had a in_reply_to_status_id property. If not, we can assume that the tweet wasn’t a reply, and is not of use to us here.

        if status.in_reply_to_status_id != None:
#Ensures the tweet is a reply by limited them to those with a in_reply_to_status_id
            list = Tweets_sent_by_twitter_id.objects.filter(twitter_id = status.in_reply_to_status_id)
            tweet = list.get()
            Queue_tweet_responses.create(queue_id = tweet.queue_id , time_received = status.created_at, tweet_id = tweet.tweet_id, response = status.text, user = status.author.screen_name )

I now use Twitter’s streaming API, using my authentication info and the listener I created as arguments. The listener is responsible for receiving the Twitter will determine what is done with it. In this case if there is a in_reply_to_status_id, and entry is created in the queue_tweet_responses table. I make sure to filter the stream using the user id of my account, as I only want tweet info from my timeline, not the whole wider Twitter universe.

stream = tweepy.Stream(auth, listener)
    stream.filter(follow=['2815304775'])

I’m pretty happy with what I have so far for my scheduler, as bare bones as it is at this point. I’m looking forward to moving on to Phase 2 with this project and smoothing out the rough edges as well as adding some additional helpful features.For instance I would like to make the scheduler do cron jobs. Once again, any suggestion would be greatly appreciated.

Posted in Uncategorized | Leave a comment

Cassandra and Django: Building the front end for TikTokTweet

Hey there, I’m back again to talk about my latest project, TikTokTweet, a tweet scheduler for python. This is the third blog post the series dealing with the project, if you need to catch up, go back and have a look at the first blog post in the series. Today I’m going to get into a lot more details on what I’ve been up to. I went over the models in my last blog post, they have changed a bit since. Just as an overview, these models were derived from my original CQL schema:

/* Table to store tweets.
 * There is timestamp for the initial creation of the tweet and one for date/time
 * modified incase any edits are made.
 */
class Tweets(Model):
    id = columns.UUID(primary_key=True)
    tweet = columns.Text()
    description = columns.Text()
    created = columns.DateTime()
    modified = columns.DateTime()

/* Table to store the different queues created.
 * A queue is a list of tweets of a
 * particular category. Each queue type gets its own unique id.
 */
class Queue(Model):
    id = columns.UUID(primary_key=True)
    name = columns.Text()
    created = columns.DateTime()
    modified = columns.DateTime()

/* Table showing what tweets are associated
 * with which queues and time for the tweet to be sent out.
 * Each tweet is in a particular queue with a certain set
 * time to be set out, hence the chosen primary key.
 * Clustering order for time_to_send is descending
 * such that next tweet to go out is at the top of the list.
 */
class Tweets_queue(Model):
    queue_id = columns.UUID(primary_key=True)
    time_to_send = columns.DateTime(primary_key=True, clustering_order="DESC")
    tweet_id = columns.UUID(primary_key=True, clustering_order="ASC")

/* Tables to hold responses to tweets.
 * TIMEUUID is used as a distinct identifier here
 * to deal with the unlikely possibility that two
 * people could reply to a tweet at the exact same time.
 * The queue_id is including in the primary key such that
 * we can select the responses for a particular queue.
 * Ordering by time descending is chosen such that the l
 * latest reply will be at the top of the list.
 */
class Queue_tweet_responses(Model):
    queue_id = columns.UUID(primary_key=True)
    time_received = columns.DateTime(primary_key=True, clustering_order="DESC")
    user = columns.Text(primary_key=True)
    tweet_id = columns.UUID()
    response = columns.Text()

/* Table to hold tweets which have already been sent.
 * This is so we can keep track of tweets that have already
 * been sent out. Eventually this could be work to help us
 * if the program crashes and we don't end up sending the
 * same tweets out again by accident.
 */
class Tweets_sent(Model):
    queue_id = columns.UUID(primary_key=True)
    time_sent = columns.DateTime(primary_key=True, clustering_order="DESC")
    tweet_id = columns.UUID()

/* Table to hold Twitter ids which have already been sent through 
 * Twitter. This id is can be obtained once the tweet is sent out.
 */
class Tweets_sent_by_twitter_id(Model):
    twitter_id = columns.BigInt(primary_key=True)
    queue_id = columns.UUID()
    tweet_id = columns.UUID()

Queue_tweet_responses has a new column ‘users’, so I can keep track of who the replies are from. Also, a new table, tweets_sent_by_twitter_id has been added.
Now, I wanted to go through how to set up Cassandra with Django (and enable the use of cqlengine along with it) in one shot. Django comes with SQLite by default, and some tinkering needs to be done to connect Cassandra up as the database. Using Django Cassandra Engine, a database wrapper for Django that uses cqlengine, this becomes pretty quick and painless. I first installed Django Cassandra Engine using PyPi:

pip install django-cassandra-engine

After that, I went into my settings.py file. Here I see all the default stuff Django had set up for SQLite. I made these changes to the settings.py file so that my app would be ready to use Cassandra. I added Django Cassandra Engine to the top of my INSTALLED_APPS.(It should be on top.)


INSTALLED_APPS = (
'django_cassandra_engine',
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'tiktokadmin',

)

Next I changed my DATABASES setting. I keep the default as SQLite though and add Cassandra as an option. ‘NAME’ will be the name of your keyspace. I did not add Cassandra as the default database, as Django will be using SQLite for administrative functions in the backend (i.e. user sessions). It is possible that Cassandra could be used for such a purpose, but I will not be addressing that at this time.


DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
},
'cassandra': {
'ENGINE': 'django_cassandra_engine',
'NAME': 'tiktok',
'TEST_NAME': 'test_db',
'HOST': '127.0.0.1',
'OPTIONS': {
'replication': {
'strategy_class': 'SimpleStrategy',
'replication_factor': 1
}
}
}
}

I have my models based on my schema in my models.py file. Make sure your Cassandra instance is running. From the command line, run the command:

./manage.py sync_cassandra

You could be able to see the tables derived from the models being created in your Cassandra database. You will need to run this every time you make changes to the models in your models.py file, so that the changes can synced with the database.

views.py

Now I was ready to put some stuff in views.py. Views are a fundamental part of the Django framework. A view is a Python function that will take a Web request and return a Web response. It contains whatever arbitrary logic is needed to return that response. For example, supposing we wanted to create a tweet:

def create(request):
    template = loader.get_template('tiktokadmin/create.html')
    queue_list = Queue.objects.all()
    context = RequestContext(request, {'queue_list': queue_list})
    return HttpResponse(template.render(context))

Here is the ‘template’:

Screen Shot 2015-02-12 at 7.10.17 PM

Screen Shot 2015-02-12 at 7.25.56 PM

The create view provides the necessary data from the database needed to make this part of the web app happen. The list of queue names is pulled from the queue table in the database and displayed as a list on the webpage, where the user gets to make a selection when they create their tweet as to which queue to add the tweet to. We get the ‘template’ from actually grabbing the associated HTML file we’ve constructed to give the layout the user see at this particular part of our webapp. These can be found inside the ‘templates/tiktokadmin’ folder.

This code loads the template called 'tiktokadmin/create.html'

template = loader.get_template('tiktokadmin/create.html')

When you use a template in Django, it is compiled only once. For optimization, it is stored for future use. A template can have variable names, such as queue_list in the code snippet below from ‘tiktokadmin/create.html’.

...
<label>Add to queue:</label>
<select name="queue_id">
<option value="0">Do not add to queue</option>{% for queue in queue_list %}
<option value="{{queue.id}}">{{queue.name}}</option>{% endfor %}
</select>
...

For this template to work, its going to need the list of queues available from the queue table. We make a CQLEngine query to retrieve all the objects from the queue table. This can be accomplished with the .all() method, which will return a QuerySet of all objects in a table. We obtain all Queue objects:

queue_list = Queue.objects.all()

Now the template we loaded earlier gets context passed to it. A context is a is a dictionary mapping template variable names to Python objects. This is a dictionary with variable names as the “key” and their values as the “value”In this case, it is the queue_list we obtained. ‘queue_list’ is the key, and the queue_list obtain from the code above is the value.

context = RequestContext(request, {'queue_list': queue_list})

We now return a HTTPResponse object with the result of the rendered template.

return HttpResponse(template.render(context))

The render() function takes the request object as its first argument, a template name as its second argument and a dictionary as its optional third argument. When you pass the context to the render method, the variables in the template get replaced by their dictionary values in the context.‘queue_list’ in the template gets replaced with the value of queue_list = Queue.objects.all(). queue_list isn’t actually a list but a ModelQuerySet which gives a future. The actually list is not obtained until get() is run. I can it also be invoked in the iterator, like the for loop in our HTML.The view then returns an HttpResponse object of the given template rendered with the given context.

Screen Shot 2015-02-12 at 7.13.51 PM

The information the user provides on this page is posted to the create_post view to handle.

def create_post(request):
    template = loader.get_template('tiktokadmin/create_post.html')
    text = request.POST['tweettext']
    queueid = request.POST['queue_id']
    tweetid = uuid.uuid4()
    if queueid != '0':
        tweet_time = request.POST['when_to_tweet']
        if tweet_time == '':
            template = loader.get_template('tiktokadmin/create_post_no_time.html')
            context = RequestContext(request)
            return HttpResponse(template.render(context))
        tweet_time = datetime.strptime(tweet_time, '%Y-%m-%d %H:%M:%S')
        Tweets_queue.create(queue_id = queueid, time_to_send = tweet_time, tweet_id = tweetid)
    Tweets.create(id = tweetid, created = datetime.utcnow(), modified = datetime.utcnow(),  tweet = text)
    context = RequestContext(request, {'text': text})
    return HttpResponse(template.render(context))

To come…

This is just the completion of Phase 1 of my TikTokTweet project, still have quite a bit of work to do. I haven’t done much of anything with the front end as you may be able to tell. I was considering trying out Bootstrap to make the app look sleek and shiny without too much hassle, given my limited experience with design in the front end. The other possibility would be JQuery. I also need to figure out a better way to do a datetime input as opposed to the yyyy-mm-dd hh:mm:ss string format I’m requesting from the user currently. I’m welcome any suggestions and, I’m pretty new to web application development and definitely eager to learn more. Up next, I’ll be looking into the backend parts of the app where the scheduler and lister reside. I think that’s where the real magic of this app lies.

Posted in Uncategorized

Using Cassandra with Django and CQLEngine: A brief example from my project

CQLEngine is a Cassandra CQL3 ORM for python. As I began my TikTokTweet project, I figured that cqlengine would be a excellent way to integrate Cassandra database technology with the Django framework I was using. I’ll be digging a lot more into Django in blog posts to come, but for now I’ll just focus on setting up the database.

CQLEngine’s syntax has a very Django like feel, making nice and easy to work with in Django’s logic. While there is no official support by Django for NoSQL although there is a good community plugin, Django Cassandra Engine. This is a database wrapper for Django framework, which, as the name might suggest, uses CQLEngine. Check out this post on Planet Cassandra talking a little more about that.

Django Cassandra Engine was pretty quick and painless to set up with my app in Django. The next part was translating my schema into models. Your models will be the definitive source of info about your data. You may recall my schema from the last post (with a few minor changes):

/* Table to store tweets.
 * There is timestamp for the initial creation of the tweet and one for date/time
 * modified incase any edits are made.
 */
CREATE TABLE tweets (
    id UUID,
    tweet text,
    description text,
    created timestamp,
    modified timestamp,
    PRIMARY KEY(id)
);

/* Table to store the different queues created. 
 * A queue is a list of tweets of a
 * particular category. Each queue type gets its own unique id.
 */
CREATE TABLE queue (
    id UUID,
    name text,
    created timestamp,
    modified timestamp,
    PRIMARY KEY (id)
);

/* Table showing what tweets are associated 
 * with which queues and time for the tweet to be sent out.
 * Each tweet is in a particular queue with a certain set 
 * time to be set out, hence the chosen primary key. 
 * Clustering order for time_to_send is descending 
 * such that next tweet to go out is at the top of the list.
 */
CREATE TABLE tweets_queue (
    queue_id UUID,
    time_to_send timestamp,
    tweet_id UUID,
    PRIMARY KEY(queue_id, time_to_send, tweet_id)
) WITH CLUSTERING ORDER BY (time_to_send DESC, tweet_id ASC);

/* Tables to hold responses to tweets. 
 * TIMEUUID is used as a distinct identifier here 
 * to deal with the unlikely possibility that two 
 * people could reply to a tweet at the exact same time. 
 * The queue_id is including in the primary key such that 
 * we can select the responses for a particular queue. 
 * Ordering by time descending is chosen such that the l
 * latest reply will be at the top of the list.
 */
CREATE TABLE queue_tweet_responses (
    time_received TIMEUUID,
    tweet_id UUID,
    queue_id UUID,
    response text,
    PRIMARY KEY(queue_id, time_received)
) WITH CLUSTERING BY (time_received desc);

/* Table to hold tweets which have already been sent. 
 * This is so we can keep track of tweets that have already 
 * been sent out. Eventually this could be work to help us 
 * if the program crashes and we don't end up sending the 
 * same tweets out again by accident.
 */
CREATE TABLE tweets_sent (
    queue_id UUID,
    time_sent timestamp,
    tweet_id UUID,
    PRIMARY KEY(queue_id, time_sent)
) WITH CLUSTERING ORDER BY (time_sent desc);

Now I needed to translate my CQL schema into models. In the case of cqlengine, a model is Python class representing a CQL table. Even though I’m basing my models of the theoretical tables designs above, the actual tables in the database will be derived from the models I create in the model.py file in my django application. I wrote these models based on the above schema:

class Tweets(Model):
    id = columns.UUID(primary_key=True)
    tweet = columns.Text()
    description = columns.Text()
    created = columns.DateTime()
    modified = columns.DateTime()

class Queue(Model):
    id = columns.UUID(primary_key=True)
    name = columns.Text()
    created = columns.DateTime()
    modified = columns.DateTime()

class Tweets_queue(Model):
    queue_id = columns.UUID(primary_key=True)
    time_to_send = columns.DateTime(primary_key=True, clustering_order="DESC")
    tweet_id = columns.UUID(primary_key=True, clustering_order="ASC")

class Queue_tweet_responses(Model):
    queue_id = columns.UUID(primary_key=True)
    time_received = columns.TimeUUID(primary_key=True, clustering_order="DESC")
    tweet_id = columns.UUID()
    response = columns.Text()

class Tweets_sent(Model):
    queue_id = columns.UUID(primary_key=True)
    time_sent = columns.DateTime(primary_key=True, clustering_order="DESC")
    tweet_id = columns.UUID()

The columns in my model will map to columns in my CQL tables in the database. The column attributes are defined within the model classes. The order in which you define your columns will be the same order as they will be in the corresponding table, so be mindful of this. Have a look at the cqlengine documentation on defining a model. From inside my app I run ./manage.py sync_cassandra to sync these models with my database. If running for the first time, it will create the keyspace and tables for you, which is really nice. Now from cqlsh, I can see that my formerly empty keyspace has been populated with tables.

cqlsh:tiktok> desc tables;

queue  tweets  tweets_sent  tweets_queue  queue_tweet_responses

As we can see, there tables reflect the schema.cql I came up with in the beginning to base my models on:

cqlsh:tiktok> desc table tweets_queue;

CREATE TABLE tiktok.tweets_queue (
    queue_id uuid,
    time_to_send timestamp,
    tweet_id uuid,
    PRIMARY KEY (queue_id, time_to_send, tweet_id)
) WITH CLUSTERING ORDER BY (time_to_send DESC, tweet_id ASC)
. . . . . .

I can access my data by referencing the models defined in models.py:

tweetqueue_list = Tweets_queue.objects.filter(queue_id = 'ceb55051-01b6-4b41-b965-c61adb5ad817')
#do a foreach loop on tweetqueue_list to get all tweet_ids for retrieving tweets from Tweets table.
for entry in tweetqueue_list:
    tweets = Tweets.get(id = entry.tweet_id)
    print tweets.tweet

And everywhere that Mary went
The lamb was sure to go
Mary had a little lamb
His fleece was white as snow

Next up, I’ll be discussing how I constructed my views to combine these models with dealing with web requests and responses.
.

Posted in Uncategorized | Leave a comment

Operation TikTokTweet

Hey everyone, I wanted to give you a brief run down on a little  project I’ve been planning. For a while I was doing this daily “Cassandra Pop Quiz” thing on my. Everyday at the same time, I would send out a pop quiz question of a Cassandra nature and people would respond. I was a lot of fun, but I times I would be busy or in meetings and I would forget, leading the question being sent out late. I got this idea that if I could automate my pop quiz question to be sent out on a schedule, as in the same time everyday.  it would be easy and I could just forget about, knowing that it would be taken care of. This lead to the concept of “TikTokTweet”, a Twitter client for scheduling tweets. Tik Tok was the round mechanical man in the Land of Oz books. He was a true and faithful companion to dorothy, he would never let her down. He was always there when she was in need, like “clockwork” we’ll say. He would need to be wound up on a schedule to continue to his normal functioning. Hopefully this little “bot” will be equally as reliable.

Some the features I hope to include in TikTokTweet:

  • Ability to create a Tweet
    • you can save it in the database for it for later
    • optionally add it directly to a queue to be tweeted out at a set time.
  • Add your saved tweets to a queue
    • assign a date and time to saved tweets to be sent out from the Schedule a Tweet page
    • Choose a to particular queue to add the tweet to
  • Create queues for tweets
    • Create and name queues for a specific categories of tweet,
    • add tweets to these queues to be sent out on schedule .
    • add tweets directly from Create Tweet or add a saved tweet from the Schedule a Tweet page.
    • Add new queues from the Manage queues page, and edit or delete queued tweets as well.


A brief overview of the application flow:

Screen Shot 2014-12-21 at 10.55.08 PM

 

From the home page of the application, you have four options. If you are just starting out, you may want to go to the “Create new Tweet” part of the site, or potentially start out creating a new queue in “Manage Queues”. Lets have a look at creating a new tweet with the app as a starting point.

Screen Shot 2014-12-21 at 10.55.17 PM

On the “Create new Tweet” page, we see are usual 140-char limit box that most of us are familiar with from Twitter. We can fill this out and send this into our database for later usage. Optionally, we can make use of that tweet right away and send it to a queue from our list(I’ll be explaining where we make those shortly) and attach a date and time for it to be tweeted out. Because I have yet to explain queuing, lets say we just submit this tweet to the database without any other options, and we’ll deal with it later.

Screen Shot 2014-12-21 at 10.55.34 PM

Now say we want to add the tweet we sent to the database to a queue. We would click on the “Manage Tweets” page from the front page. Here we have the list of tweets sitting in our database. We will select a tweet and choose a queue from the list and date and time for it to be tweeted out.

Screen Shot 2014-12-21 at 10.55.43 PM

Now lets looking at how queuing works. A logical step early in the game would be to create a queue to add tweets to. A queue is a list of tweets on a schedule to be tweeted out at different set time. We can select a queue to look at from the drop down list, or create a new one. Initially, there won’t be any queues until you create one.

Screen Shot 2014-12-21 at 10.55.53 PM

Once we choose a particular queue to look at, we can make changes if we like. We can edit a tweet in the queue or remove it from the queue entirely (it will still exist in the database). We may even want to delete the entire queue.

Screen Shot 2014-12-21 at 10.56.03 PM

If we choose to edit a tweet, we will be brought to the “Edit Tweet” page for that tweet. Here we have the option to change the text of that tweet, or even change it’s place in the queue, assigning it a new date and time to be tweeted out. Hit the “Submit” button for the changes to take effect.

Screen Shot 2014-12-21 at 10.56.14 PM

Of course, we are going to want to have a look at the response to these tweets that we get. Responds are organized on a queue basis, and from the “Manage Response” page, we will select a queue for which we want to see the associated responses, as in responses for tweets in this queue.

Screen Shot 2014-12-21 at 10.56.23 PM

From the next section of the “Manage Responses” pages, we will be able to see responses associated with the particular queue we have show, with the most recent response at the top. Because the timing of the responses is important to me, it’s necessary that I be able to get these responses in order. TikTokTweet should list responses in order from newest to oldest.

A little look at the potential data model:

Because this is Cassandra we are using, we need think about the kind of queries we be performing against the database before we can really start developing our application. Firstly I knew I would need a table as to store tweets in as I created them from the “Create Tweet” page. The plan was to be able to add these tweets to various queues which i had created, at and they would be tweeted out at a particular time. This would be set from the “Schedule a Tweet” page. Since a tweet can be in multiple queues, we’ll store those in their own table, tweets. To keep track of and reference the list of queues I have created, I would need a queues table, holding the list of queues. Once a queue is selected for a particular tweet, we set a date time for that tweet within the queue. I will need a tweets_queue table to hold tweets and their associated dates/times for tweeting. I’ll use ‘CLUSTERING ORDER BY’ to store the dates in descending order, which will make the queries much easier. To select the next tweet to be sent, I’ll just have to do a ‘SELECT time_to_send, tweet_id FROM tweets_queue WHERE queue_id = <id> LIMIT 1’. Simple query now.  A tweets_sent table might also be necessary eventually to keep track of tweets that have already been sent to avoid redundancy. Finally, responses to these tweets would be held in queue_tweet_responses table, which could be access on a per queue basis in descending time order, as can be seen from the manage responses pages.

/* Table to store tweets. There is timestamp for the initial
 * creation of the tweet and one for date/time modified incase any edits are made.
 */
CREATE TABLE tweets (
    id UUID,
    tweet text,
    description text,
    created timestamp,
    modified timestamp,
    PRIMARY KEY(id)
);
/* Table to store the different queues created. A queue is a list
 * of tweets of a particular category. Each queue type gets its
 * own unique id.
 */
CREATE TABLE queue (
    id UUID,
    name text,
    created timestamp,
    modified timestamp,
    PRIMARY KEY (id)
);
/* Table showing what tweets are associated with which queues and time for
 * the tweet to be sent out. Each tweet is in a particular queue with a
 * certain set time to be set out, hence the chosen primary key.
 * Clustering order for time_to_send is descending such that next tweet
 * to go out is at the top of the list.
 */
CREATE TABLE tweets_queue (
    queue_id UUID,
    time_to_send timestamp,
    tweet_id UUID,
    PRIMARY KEY(queue_id, time_to_send)
) WITH CLUSTERING ORDER BY (time_to_send desc);
/* Tables to hold responses to tweets. TIMEUUID is used as a
 * distinct identifier here to deal with the unlikely possibility
 * that two people could reply to a tweet at the exact same time.
 * The queue_id is including in the primary key such that we can
 * select the responses for a particular queue. Ordering by time
 * descending is chosen such that the latest reply will be at the top of the list.
 */
CREATE TABLE queue_tweet_responses (
    time_received TIMEUUID,
    tweet_id UUID,
    queue_id UUID,
    response text,
    PRIMARY KEY(queue_id, time_received)
) WITH CLUSTERING BY (time_received desc);
/* Table to hold tweets which have already been sent. This is so
 * we can keep track of tweets that have already been sent out.
 * Eventually this could be work to help us if the program crashes
 * and we don't end up sending the same tweets out again by accident.
 */
CREATE TABLE tweets_sent (
    queue_id UUID,
    time_sent timestamp,
    tweet_id UUID,
    PRIMARY KEY(queue_id, time_sent)
) WITH CLUSTERING ORDER BY (time_sent desc);

I know I want TikTokTweet to be lightweight and easy to use, and enable me to manage questions and responses to my daily quiz questions so I could focus on other things. My plan is to write my app  in Python, using Tweepy API for the backend features, and Django framework along with Django Cassandra Engine (a database wrapper for Django framework, using cqlengine).

 

 

 

 

 

 

Posted in Uncategorized | Leave a comment