Google Docs + cURL = Fun

Well for a little back story I'll just explain why this all came about. I play WoW and my guild uses an online site to display DKP (This doesn't really matter if you don't know what WoW and DKP are. Long story short I didn't like the way the site was displaying the information so I wrote a script that pulls the data off of the original site and stores the data in a CSV file on my local computer.

Ok, Cool. Now I have a CSV file that I could easily upload to Google docs from the web interface, but I wanted to automate this since I'm lazy. :).

Problems:

1. I have a CSV I want to automatically upload to google docs with one command.
2. Which programing interface to the gdocs api should I use?

Well since I am a linux system admin by trade I love to use pure command-line tools that are available on most linux machines. Initially I was going to use Python because I like it, but when I was viewing the different interfaces I noticed an interface called "Protocol". Well basically this interface allows me to use curl inside a bash script which warms my heart :). Ok now that story time is over lets get into the nitty gritty.

Sometimes the Google's documentation of their api is kind of confusing so I will try to explain how I did things the best I can and maybe someone will find it useful. Also keep in mind that there are separate api's for spreadsheet's etc, but I am using a broad approach and just using the docs api which includes spreadsheets as well.

First things first, authentication. We will use curl to get an authentication token so that we can use this token to perform other needed tasks.

curl https://www.google.com/accounts/ClientLogin \
-d Email=USERNAME@gmail.com -d Passwd=CHANGEME \
-d accountType=GOOGLE \
-d source=cURL \
-d service=writely

Ok, now this will return something like the following:

SID=XXXXXX
LSID=XXXXX
Auth=XXXXX

Now all we really care about here is the "Auth" portion so we can use some bash magic if you pipe the output to the following it would do what you are needing.

grep Auth | cut -d\= -f2

In my scripts I stored this value as ${token}. Next thing that we will do now that we have our authentication token is get a list of our documents from google.

curl -L --silent --header "Authorization: GoogleLogin auth=${token}" "https://docs.google.com/feeds/documents/private/full"

Now this returns unformatted XML so if you need to view the output in a more human readable fashion so that you can get an idea of what you need to parse out you can pipe the output through the following, but generally your scripts shouldn't need a human readable version ;).

tidy -xml -indent -quiet

Now that we have pretty output you need to find the document that you are wanting to update. Since I am dynamically creating the spreadsheet on my end I am just wanting to upload a completely new version of the file. To do this you will need to parse out the URL for the "edit-media" link as this will be the URL that you will need to send your post command to. You should end up with a link similar to the following.

https://docs.google.com/feeds/media/private/full/spreadsheet%3A0ApXXXxxkCdHXXaWZOWjZyU2JVUlVHYjZwZ25WNHc/g3qaf1ch

Again I will store this value in a variable called ${document} as I will need it later. It is also handy to parse out the documents name which I store as ${slug}. Ok now we have our auth token, the correct URL for the document that we are wanting to edit, and document name. All that's left is issuing the POST via cURL to our ${document} to upload the local temp.csv.

curl -L --silent --request PUT --header "Authorization: GoogleLogin auth=${token}" --header "Slug: ${slug}" --header "Content-Type: text/csv" --data-binary "@temp.csv" "${document}"

The curl command should spit some XML back at you once the upload is complete. Woot! Our document has now been uploaded.

Note if you are wanting to do this for something other than a CSV. You will need to use a different "Content-Type" which can be found here:

http://code.google.com/apis/documents/faq.html#WhatKindOfFilesCanIUpload

References:

http://code.google.com/apis/gdata/articles/using_cURL.html
http://code.google.com/apis/documents/docs/2.0/developers_guide_protocol.html

NetCat! RAWR!

Big time in tiny town.

So this is a story about a small program that has a myriad of different uses. This tool is called netcat. To start out I am going to go over some of the more practical uses of netcat. The most common use that is probably the most well known is using netcat as a simple port scanner.

nc -v -w 1 localhost -z 1-3000

This example is passively scanning localhost for any open ports in the range of 1 to 3000. Pretty simple eh? Try it on you local machine you might be surprised what is open. Unless you are a paranoid freak who uses hardened linux with every port closed so the aliens can't invade your hard drive to steal your password to your TiVO to read your mind through television waves.

Ok the aliens aside this next example I haven't really found a practical use for as of yet but I could see it being helpful in a pinch. We are going to use netcat to make a copy of a hard drive over the network with the aid of dd.

Server:
nc -l -p 9000 | dd of=/dev/sda

Client:
dd if=/dev/sda | nc 192.168.0.1 9000

To explain what is going on here first we create a netcat server that listens on port 9000 and pipe this output to dd with the output set to /dev/sda (the hard drive we want to write the image to). Now we setup the client connection by using dd to read the disk we want copied and piping it's output to netcat which connects to our server on port 9000. Make sure to be careful if you try this we don't want anyone to accidentally nuke an important drive.

Another simple little thing you can do with netcat is create a makeshift webserver to server up some content.

One shot single connection:
nc -l -p 80 -q 1 < darkpages.html

Accept multiple connections:
while true; do nc -l -p 80 -q 1 < stuff.html; done

This is a pretty hack way to do this sort of thing but it would work. Now you can do this this sort of thing alot prettier with socat but that is subject for a post of it's own.

Now lets see what else is there to cover here. Ah yes I always forget about this one but it's not the most usefull as there are MANY other tools to aid in something like this on linux. If you haven't guessed I am talking about using netcat to transfer large files.

Server:
nc -lp 1337 > file.gz

Client:
nc -w 1 192.168.0.10 1337 < file.gz

Ok as you can tell this is very similar to the previous example using dd. First we create our server that is writing to our destination file. Next we create the client which if fed our source file and connect to the server. The "-w 1" will close the connection once the file is done transfering. Ok so since this is pretty useless I am going to kick it up a notch just for fun =).

Server:
nc -lp 1337 | pv | gunzip > file

Client:
cat file | gzip | nc -w 1 192.168.0.10 1337

Now in this example we are compressing the data one the fly on the client side. On the server side we pipe the output to pv so that we get a nice little display of the current data transfered, current transfer speed, as well as uncompressing on the fly.

Another silly thing that can be done with netcat would be using it as a simple chat server. Granted this isn't pretty but it would get the job done if you needed to chat it up without installing other crap.

Server:
nc -lp 9000

Client:
nc 192.168.0.10 9000

Either you or your friend start the server and the other one starts the client, then you type. Yes it's that simple!

I guess I should cover the more advanced uses now. You can use netcat as a light telnet server doing something similar to the following.

Server:
nc -lp 1337 -e /bin/bash

Client:
nc 192.168.0.10 1337

Once you created the server you can now connect and use any commands you normally would in bash it's pretty cool I think. Now a spin off of this would be what is called the revrse telnet server. What is a reservse telnet server? I am glad you asked =). Lets say your work firewall is locked down and you can't poke any hole to be able to access your work server outside of work, but you can access the internet via port 80. Well you are in LUCK!

Server:
nc -lp 80

Client:
nc 192.168.0.10 80 -e /bin/bash

Now how this works is you create the server on your home machine it's listening for connections on port 80 since we know our work firewall will allow port 80. Ok now the cool part from the server behind the firewall we connect out to our home pc but tell netcat that /bin/bash will be interpreting any commands that it gets in return. Ok we are connected and from our home machine we can now run commands directly on the work server!

Well I think this sums up this blog post I hope you all learned alot and find netcat as fun as I do =)! Please feel free to leave any comments about this blog.

Sharing is caring.

I'm going to try and post interesting things I learn here and there. My goal is to learn something new every day. So here goes first post. I recently had a job interview and there were a few questions that stuck in my mind.

First on the list was this, "If you just added a new virtual host host to your apache config file, how would you make your new vhost live without downing the server?"... granted not the exact words but same basic problem. Now all be it simple I did not know the answer because I have never used this function before. Well the answer is quite simple once you verify that your settings are correct and that you did not modify any of the other already running vhosts you can simply use the "reload" option with apache.

Next up, this question I knew was possible I just did not know the exact method to do it. The question, "Is it possible to have multiple IP's on a single NIC, if so how would you do this?" Well I knew it was possible, but how to do it is another question. So since I hate not knowing I took the time to find the answer. The most common method of doing this would be called "aliasing". To do this from the command line you would use ifconfig in a fashion similar to this...

ifconfig eth0:0 192.168.1.99 netmask 255.255.255.0 up

Now to kind of explain what is happening there "eth0" is our hardware device we use ":" to create and alias of that device call "0". You could have named the device "eth0:1" etc.

Now I'll explain the better way of doing this so that your system will manage the connection instead of manually creating it every time. What we need to do is create "/etc/sysconfig/network-scripts/ifcfg-eth0:0" so that it will start up automatically when the system starts. Your config will look something similar to this...

DEVICE=eth0:0
ONBOOT=yes
BOOTPROTO=static
IPADDR=192.168.1.99
NETMASK=255.255.255.0

Well that is all I have for you currently. Thank you for your time.

Have a nice day!