Analytics and Twitter: intro

I'm a developer working for a project at university trying to figure out what the future of transportation is. To do this, we have looked into trends in modern technology and culture around transportation, and after hours of deliberation, we realized that our "findings" had been essentially brought out of thin air through our own opinions. What we really wanted was data on who was thinking what about transportation. Unfortunately, we didn't really have a large spread for running surveys. However, having worked with the big data department before, I asked around and found out how I might mine social media data around certain search terms and see what we could find out.

For the record, this blog is just a rough document of stuff I'm doing. I don't really care if it's readable at this point. Maybe I'll update it later for real use, but we'll just have to see how it goes. At the moment I'm flying by the seat of my pants and would like to document this for future use by anyone else running in my path.

At any rate, I began my journey searching for twitter analysis, and frankly, what I found was unusable for my needs. I wanted to see what people were saying around certain keywords over time, but there was simply no way to do this. I also discovered, to my dismay that Twitter does not keep its data public for more that a few weeks explicitly. They remove tweets from search results after a while even though they might stay in the database. Companies profit off of this by selling that data or analyzing it in real time. the best service I found for my needs was tweetarchivist.com, but even this was not quite what I wanted, and it required payment after a certain amount of time.

And so, as I often do in these situations, I turned to my old friend python, the swiss army chainsaw of programming languages. I wanted to gather my own data, analyze it, and see what I could find. That's what I'm going to do with these posts now.

Are you a wizard?

One time, when I was about 4 or so, I got to go visit my cousins who lived out by the lake. I remember that they had a boat, and I remember that it was fun to be on the boat, but I don't really remember being on the boat.

What I do remember is going upstairs to the spare bed in my cousins' over sized penthouse of a bedroom and watching attentively as they took turns systematically running and jumping through the technicolor dream that was Super Mario Brothers. I sat in front of the NES, my vision overwhelmed by those figures on the television, as a wave of terrible curiosity overtook me. I knew that when you watched a movie, there wasn't really a troupe of little actors behind the screen; that would just be silly. Even so, I couldn't help but to wonder: how on Earth was it possible for this plumber to be following my cousins' every direction, every run, every jump? Was there really a small man with turtles and mushrooms up there? Was it alive? At the time, I thought it was magic, and somewhere deep down, I still kind of do.

See, I've been programming computers for about a decade now. I've used may fair share of languages and operating systems and I've even made an ethernet cable or two (nifty!). In fact, I once wrote a rudimentary compiler for the English language that translates into python code. In high school, if you wanted some help with computer stuff, I was usually the guy to talk to. Most people probably thought what I did was lame or nerdy, but I liked to think of myself as a wizard, like Harry Potter or Gandalf or something. And in more ways than one I kind of was.

Now, I suppose I should explain. The thing about computers is that at a very fundamental level, you can just think of them as really complex adding machines. They aren't actually all that intimidating when you get up and close. Most of what you really get in a computer box is wires, transistors, and memory slots with a few bells and whistles here and there. It's not even a particularly good adding machine either. All it really does is look at a bunch of 1's and 0's and then decide based on other bunches of 1's and 0's whether or not it wants to change them around or move to yet another bunch of 1's and 0's. This might not seem particularly magical, but please bear with me.

For a number of years, the way we used computers was pretty much limited to this mechanism. The computer could do a few things, and we would tell it to do those things in a certain order by feeding it certain sequences of 1's and 0's. Somewhere along the way, we learned that we could teach the computer to get whole bunches of those binary 1's and 0's out of a larger digit like '7' for 0111 or 'B' for 1011 (and yes you read that correctly). Soon after, we convinced the computer to translate a few magic words, like 'mov' and 'add,' into those larger digits making the whole process "abstracted," which is just programming jargon for "a whole lot less complicated looking." Instead of typing a bunch of chaotic digits everywhere, we could just use a few words that looked like English instead. This was called the assembly language, and it was our first foray into real wizardry. Later on, we learned that we could use assembly to interpret and compile other languages like Fortran and C. The rest is history.

So, on to my point. First, ask yourself, what does a wizard do? Well, if he's anything like Dumbledore or Gandalf, he waves a wand or a staff around as he mumbles some magic words and presto! You've got magic. Maybe he mixes up a few potions or flies a broomstick too, but for now, what really matters is that wizards (and witches) can speak in some esoteric language that causes certain things to happen that one might classify as "magical." I am of the opinion that programmers fit this definition quite nicely. Where Lord Voldemort might belt out an "Avada Kedavra" towards Harry by the switch of his phoenix core wand, Richard Stallman might type "sudo rm -r closed-source-software" in the terminal of his universal Lemote machine. Where Gandalf might mutter some incantations over his fireworks, John Carmack might write out a nice OpenGL sequence with some groovy shaders. Where a wizard might conjure up a portkey, any discerning web developer might write out an "href" attribute. You get the idea. Clearly the analogy breaks when it comes to controlling the real world, but in the realm of the computed, it's like Tron - the users are gods.

When I started programming, my drive to learn came from the desperate single minded urge to solve the mystery of the pixelated plumber. Little did I know just how deep that rabbit hole went. I started out with game maker (great peice of software by the way) and a few simple things on my TI-84. I looked up tutorials on the internet and coded my way through all the "Hello World!"s and "Draw a triangle to the screen"s I could. When calculators weren't enough, I moved to QBasic. When that became limiting, I tried my hand at, and became proficient with, Java, Python, C, and C++ and their respective SDL and OpenGL libraries. When I became curious about the internet, I had the chops to pick it right up with HTML, CSS, PHP, and a little Django for good measure (not that you'll probably ever see that in action, this is posted on blogger). Somewhere along the way, I lost track of what I was looking to discover, and instead, I stumbled upon something much more profound. It took me a while to realize what I loved so much about programming, but eventually, I figured it out. It wasn't the prestige or the novelty behind it -- that had worn off years before. No, the real draw was in the magic of it. Now that I had learned the spells, I could do almost anything I wanted to. I was suddenly more than just some conjurer of cheap tricks; I had become a wizard.

Still, just because I now understand the magic doesn't mean I've stopped enjoying it. To me, the sorcery that is Super Mario Bros. remains just as fun and enchanting as the day I was first introduced. I've made my own video games, I know what's going on down there, the mystery is over, but the magic never stopped...

If I were you, and I didn't know how to program, I would get on it that right away. Seriously, go do that. Right now! It's just about the most rewarding of all my hobbies. This is partly because of the fact that I enjoy it so much, and partly because it's literally rewarding -- it is, after all, a marketable skill. So go on! Get to coding, and programming, and scripting, and "hacking." Before you know it, you just might suddenly realize that yer' a wizard. A few resources I would recommend are as follows:

• Codeacademy.com: A great easy place to start if you've never worked with computers before. I'd probably start with the javascript or python courses if I were you.
• TheNewBoston.org: Has all kinds of video tutorials for just about anything you could think of.
• /r/LearnProgramming: A sub-reddit for programmers who need to be shown the ropes.
• StackOverflow.com: Got a question? If it's not answered here already, ask it yourself.
• Google.com: Because some people just don't think to look stuff up for themselves.

Happy coding everyone!