Nowadays, people are really into this whole “Skitch” thing, and being able to send images/screenshots to each other quickly. I’d been doing the same thing with TinyGrab for a long time, but I like to host things myself. Yes, TinyGrab has the ability to upload to your own server… but it uses FTP. This was causing me no end of issues, so I sought out something else.

I found Scrup, and I’ve been using it for the last year or so very happily. It’s open source, and has been hanging around on Github for 2 years now. There are some pretty sweet forks of it, including one that has support for a sound on upload completion, and Growl notifications too.

What does Scrup do?

So, you need to share an image, or a screenshot really quickly? Using the standard OS X screenshot features (Command + Shift + 4, and so on), you can hit one button and upload the image to your webserver and put the link to it into your clipboard ready for pasting anywhere.

It does also have the ability to edit the screenshot pre-upload (such as adding arrows to point to important, or awesome things)

What you need

• A server somewhere with some disk space
• PHP 5
• A webserver

How?

1. Install the Scrup.app onto your Mac. I have a pre-compiled a version with the sound and Growl patch included. You don’t have to use mine, you can compile it yourself using Xcode if you wish. (The source is on github here: https://github.com/rsms/scrup)
2. Create a folder on your webserver that you want to store your images in. I call mine “grb” (short for “grab”) because I like short URLs. (/var/www/grb/ -> http://laur.ie/grb/)
3. In that folder, put a script that will receive the files, and then return the URL to where it stored the file. You can view the one I use here (modified from the one that ships with Scrup) which names files something like “1s-euobfpq1xcwos.png” and has no authentication (so make sure you go the security-by-obscurity route of naming the script something random or add auth yourself)
4. Open Scrup, and point it at your upload script. For example, http://yourhost/grb/receiver.php?name={filename}

   

5. Take screenshots! They should get uploaded and you should see a green tick in the menu bar. The URL of the uploaded image should also be in your clipboard, ready for pasting wherever.

The best thing about Scrup is that it has a simple, fast UI for just uploading things quickly, and because it uses a regular HTTP POST, it works on whatever weird internet connection you may be on.

I experienced this today, very frustrating; sendmail all locked up and outputting this bizarre “timeout waiting for input from local during Draining Input” error into the logs.

tl;dr: figure out what sendmail is waiting for.

In my case, it was stuck on procmail. But why? Turns out the local mailbox (a user that runs a lot of crons) had hit 3GB, at which point it didn’t seem to be accepting any more email into that inbox. Moving that file out of the way and allowing a new one to be created caused the queue to be flushed instantly.

As much as most of us would love to not have to do it, most people reading this now will have to be on call at some point. It sucks, but Pagerduty makes it a little easier to manage when your team starts to grow.

Whilst we still have Nagios sending to all contacts directly (a personal preference) we still rely on Pagerduty for emergency pages from the rest of the company, and to arrange who is on call when (their calendar is pretty good for us, allows for exceptions etc).

We’re also a user of the IRC bot “irccat” which, briefly explained, allows input/output to scripts from an IRC chat.

I wanted to combine the two for a long time, and when Pagerduty released their API to access schedule data it wasn’t long before we had a command that allows anyone in the company to ask irccat who is on call and until when.

I’ve finally got around to releasing this today, a “library” of useful Pagerduty API functions (pagerduty.php) (note currently it has just two, to see who is on call for a given schedule. Pull requests for additional useful functions please!) and more importantly, pagerdutycron.php – A script to run on an interval that will then either broadcast in IRC a new person on call, and/or send an email.

As usual, I’ve stuck the code on Github: https://github.com/lozzd/PagerdutyPHP

A quick note to anyone setting up a new Hadoop cluster and hoping to quickly use the built in Ganglia metrics collection (which you should! If it moves, graph it!): This works out of the box with Ganglia 3.0, but the protocol changed with Ganglia 3.1.

The official GangliaMetrics pages talks about this, and talks about patching (which is already available if you use the Cloudera releases) but doesn’t go into more detail than that. I recently set up a new cluster, and remembered there was something I had to change in the default config to make it work out of the box… After inquiring (and finding the comment I left in my old config file!) I remembered, you must change the default class to have “31″ (e.g. Ganglia 3.1) on the end.

For example, the default config file: (Replacing @GANGLIA@ with your multicast address)

dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
dfs.period=10
dfs.servers=@GANGLIA@:8649

mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
mapred.period=10
mapred.servers=@GANGLIA@:8649

jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
jvm.period=10
jvm.servers=@GANGLIA@:8649

rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext
rpc.period=10
rpc.servers=@GANGLIA@:8649

Is changed to this:

dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
dfs.period=10
dfs.servers=@GANGLIA@:8649

mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
mapred.period=10
mapred.servers=@GANGLIA@:8649

jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
jvm.period=10
jvm.servers=@GANGLIA@:8649

rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
rpc.period=10
rpc.servers=@GANGLIA@:8649

Restart the cluster, and the graphs will appear under each host in the Ganglia interface.

There is a LOT of detail in these graphs, with metrics ranging from DFS (things like bytes written, and how many operations were transferred from other nodes) to the JVM (monitor those heap memory sizes!)

This is probably old news to most people I’m sure, but I have a rule that if I didn’t find it within 30 minutes, maybe this will help someone in the same boat as me

I recently took delivery of the rather splendid Thecus N5500 which I love; it’s the perfect mix between “it just works” and “oh, let’s stick SSH on there and poke around”. With 5 hot swap disk shelves, and 2TB hard drives you’ve got a serious amount of storage.

For your money you get a very nice little piece of hardware in a pretty nice shell (it strikes me as a touch tacky in places but then again it’s hardly going on show) with software that gets the job done. NFS, AFP, Samba, iSCSI, iTunes DAAP support, and plenty of modules to tickle your fancy (Logitech Squeezecenter, for instance).

But who am I kidding, I’m a sysadmin. 10 minutes after powering the thing on I was dying to log in using SSH so I could watch /proc/mdstat to see the RAID build. Luckily, the modules from the Thecus N5200 work fine; which means you’re a couple of clicks away from a root terminal.

1. Grab the SSH and SYSUSER N5200 modules, and unzip them (a mistake I made.. How embarrassing.)
2. Upload them using the webinterface, and enable them.
3. SSH to the NAS box using the user “sys” and the password “sys”
4. Enjoy your shell, and remember to run `passwd sys` to change the password to something else.

Now, you’ve got yourself a pretty handy, albeit it BusyBox-ridden install of Linux. The whole point of this post, is so I can pimp a few statically compiled binaries that might come in useful to you; they did to me anyway.

(You may wish to install the UTILITIES module, which gives you a proper version of top and ps, amongst other things, available here)

You can simply untar and drop the binaries into /raid/data/modules/bin folder so that they’re in your path, and stored on your disks rather than the flash units which are rather limited in space. By the way, these modules should also work fine on the Thecus N5200 NAS boxes too.

The binaries are available here: http://denness.net/thecus/binaries/

The list includes (all the latest versions as of the date of this blog post):

• ethtool, handy for network interface prodding
• iftop, a very useful “GUI” app that shows incoming/outgoing network bandwidth (let’s face it, this is fun on a NAS. NOTE: you may need to execute this one using `TERM=vt100; iftop`)
• iostat, for hard core disk stats porn. Run it with `iostat -mx 1` and watch the megabytes fly
• rsync, particularly handy if you want to synchronise/backup data from one place to another, so particularly handy on a NAS.
• vim, just in case you were planning on writing a lot of code on the Thecus
• GNU screen, a nice place to store your terminals and detach and come back later. (NOTE: you may need to execute this one using `TERM=vt100; screen`)
• The command line version of PHP, in case you were planning on writing any scripts in PHP to run on the Thecus.

Any suggestions/comments, let me know.

When I ended up using Windows to power the overhead information screens at Last.fm, I lost the ability to have a one line crontab entry that shut the monitors into DPMS standby (and wake them up) when we’re in and out of office hours. Makes no sense wasting power, but more importantly shortening the length of screens having them on when the office is empty.

I didn’t think I would have any issue finding a utility to place the screens in to standby mode. I didn’t; but unfortunately they were either not free, massively complicated or simply didn’t work.

So I found a code snippet online, fired up a copy of Visual Studio and compiled two exe files; MonitorOn.exe and MonitorOff.exe. MonitorOff sends a signal to all attached monitors on the system to go in to sleep mode, and if you move the mouse you can wake them up as normal. Or you can run MonitorOn which will send the signal manually. Simply place these into the Windows Task Scheduler, and you have a simple, effective way to manage your information screens.

You can download MonitorOn and MonitorOff here.

I’ve spent 3.43 years at Last.fm, which seems almost like a lifetime. For a long time, I couldn’t ever imagine leaving; every morning I would wake up excited to go and face new challenges and do fascinating new things. In the last 6-12 months so much has changed, as Last.fm gradually slips out of being a startup to being a company that, for better or for worse, has to make some money. I will certainly think twice before working for a company that has anything to do with the music industry… it’s a pain of a situation.

I’ve babysat the wonderful creation that is Last.fm through launches (both expected and unexpected), crashes (always unexpected), overheatings (and break-ins, and power failures… All the kind of thing that should never happen to a datacentre) and plenty of blood, sweat and tears.

It’s been an amazing experience, working with some of the most amazing people I have ever met (some of which have come and gone), but it’s time for me to help another startup through getting up at 4am to fix databases and exciting scaling questions.

And that will be Etsy; another website that has an awesome product that I love, plenty of traffic and graphs that point upwards and a bunch of guys who are passionate and have an awesome method of working. I’m really excited about getting involved and learning things again, as well as enabling a different group of passionate users go about their day to day business. I’ll still be in London, but popping to NY on occasion.

Let’s hope the next 3.43 years will be just as exciting.

Last night I upgraded my Exchange 2010 server to 2010 SP1. Unlike many other people’s experiences, this actually went rather smoothly, until I realised 2 hours later my T-Mobile G1 (aka HTC Dream) is no longer syncing my Email. I currently use Cyanogen’s Mod version 6 which brings the features of Android 2.2 aka Froyo to my poor ancient device (which I’m hanging on to because of the excellent keyboard!) and use the built-in Exchange ActiveSync (EAS) support to sync my email, contacts and calendar to my phone.

I read online about a few other people running Android 2.2 who had the same issue and some suggested it was a policy problem, so I deleted the default profile and recreated it, and deleted the device association on my phone and my account. Unfortunately this didn’t work either, leaving me with no email but more importantly, no contacts!

After enabling some extra logging on the server I could see it was unhappy about Android trying to get a folder sync without provisioning, but for some reason the device wasn’t attempting to apply the policy.

AccessState :
Blocked
AccessStateReason :
Policy
ResponseHeader :
HTTP/1.1 449 Retry after sending a PROVISION command
MS-Server-ActiveSync: 14.1

Curiously, enabling extra logging on the device just showed it successfully understanding the first OPTIONS request, then timing out.


[12:36:00] EAS SyncManager | !!! EAS SyncManager, onCreate
[12:36:00] AbstractSyncService | Testing EAS: example.contoso.com, domain\user, ssl=1
[12:36:14] AbstractSyncService | Validation (OPTIONS) response: 200
[12:36:14] AbstractSyncService | Server supports versions: 2.0,2.1,2.5,12.0,12.1,14.0,14.1
[12:36:14] AbstractSyncService | Try folder sync
[12:36:35] AbstractSyncService | IOException caught: Read timed out
[12:36:35] EAS SyncManager | !!! Eas SyncManager, onDestroy


Thinking I was going to have to apply my highly dubious Java programming skills to this solution, I downloaded the Android SDK and started up the emulator for 2.2. I attempted to set up my account on the emulator, and to my surprise it worked first time.

I can only imagine, Google have already patched up this issue and not yet released it, as the Android Open Source Project page for Mail.apk shows the last update 5 months ago. So I applied the best logic that I have as a Sysadmin… Scp the working Email.apk from the emulator to my server, and install it on my device. And it works! I lose the a few additions that have been made to the CyanogensMod version of Email.apk but I’d rather have it working.

If you’re experiencing this issue too and you want to apply my brute force method of fixing it and get your email syncing again, simply follow these instructions (you need root for this, but I suspect this is only a problem on modified builds):

1. Download the 2.2 stock Email.apk from here using the Android browser. Don’t attempt to install it, let it go in to your download folder on your SD card.
2. Open a terminal on your device. I use ConnectBot. Gain root using “su”
3. Run: mount -o remount,rw /system
4. Run: mv /system/app/Email.apk /sdcard/backup_Email.apk
5. Run: cp /sdcard/download/Email.apk /system/app
6. Reboot for good measure
7. Set up your sync, and enjoy a refreshing beverage
   for your hard work

Disclaimer: I have no idea if this has any bad affects on your phone, your sex life, or anything and I make no guarantees about anything. If anything bad happens, it’s not my fault, you do this at your own risk! The process above makes a backup, so hopefully if you regret this you can put back the old version.

It’s been a long time coming (even longer than CactiView!) but finally I’ve cleaned up (as much as possible) and released Naglite2, a full screen easy to read status screen backed on to Nagios.

Perfect for a NOC or operations room, you get a at-a-glance view of your hosts and services status, which not only helps in sudden emergencies but also incentivise  your staff to get a “clean board” and fix the remaining niggly problems in your network!

The screen also compresses down quite nicely into a mobile browser, perfect for checking on the status of your systems whilst on the move.

The code is up over at Github, feel free to use/distribute/fork and modify or send me comments.

Get Naglite2 now

I recently purchased a Draytek Vigor 2710n router to replace my aging 2600plus. I’ve always had a very high opinion of Draytek routers. They’re fairly expensive compared to most home routers, but they do have a bucketload of fairly advanced features, that I was disappointed were lacking in others.

I changed ISP at the same time as getting the new router, taking out service with the (so far) brilliant Zen Internet after recommendations from colleagues especially Caitlin who used to work there before joining us at Last.fm. They’re a proper Sysadmin’s ISP, giving you a netblock of 8 IPs on a pretty inexpensive home connection.

Unfortunately for me I (foolishly) didn’t check to see whether the 2710n still supported this, since my 2600 did and I just assumed a higher model number meant it was actually an upgrade rather than a downgrade.

After a quick chat to Draytek support they told me this indeed was not available and never would be. I contemplated giving up with the 2710n and returning it, or just using it as a WAP, but I was somewhat peeved when I looked in the HTML and noticed the only reason the button was missing (the interface has hardly changed in the 5 years between the models) was because it was hidden using CSS:

<input type="button" name="btnWIpAlias" id="wIpAlias" style="display: none;" value="WAN IP Alias" onclick="wanIpAlias()">

Using Firebug, you can easily show this button and it’s functions work fine. I decided to do some more digging.

The Draytek have an excellent Telnet (and now SSH) interface too, and the functions are entirely available there:
admin> ip aux ?
%% ip aux add [IP] [Join to NAT Pool]
%% ip aux remove [Index]
%% Where IP = Auxiliary WAN IP Address.
%% Join to NAT Pool = 0 or 1.
%% Index = The Index number of table.
Now auxiliary WAN1 IP Address table:
Index no. Status IP address NAT IP pool
----------------------------------------------------
1 Enable X.X.X.110 Yes
2 Enable X.X.X.105 No
3 Enable X.X.X.106 No
4 Enable X.X.X.107 No
5 Enable X.X.X.108 No
6 Enable X.X.X.109 No


Sorted! The IPs then appear in the relevant places in the web interface too, for example in the “open ports” or “DMZ” sections. If IP routing is what you’re after (used to be called “For IP Routing Usage”) simply use the “ip 2ndaddr” command.

If you like configuration via the web interface, that’s pretty simple to enable too. Grab Firebug for Firefox, and use the “DOM” tab to display all the variables used on the current page.

Firebug DOM console

There are tonnes of variables in here, many are set to “0″. Incredibly, all these functions do is tell the web interface to display extra features. They’re all there and available in the background, from static routing to Multi-Nat, to IP routing and extra VPN settings. You can even enable the WAN, Wizard and VoIP options if you really want to.

For Multi-NAT, double click on SHOW_INT_MULTINAT and set it to 1. For 2nd subnet IP routing, find SHOW_LAN_GEN_2NDSUBNET and set it to 1. Have a search through and enable the functions are need. By the way, this only lasts for as long as you’re logged in, so you’ll have to repeat it if you ever need to change the variables.

Before:

After:

I will note that you should probably buy the 2820 as suggested by Draytek if you really want these features.. It’s only £40 more expensive, but saves you a bit of hassle. This guide is only for people who have been mislead and don’t want to send their routers back!