TheRealTimeWeb.com - Amazon AWS

Installing git and gitolite on Ubuntu 11.10

Mon, 13 Feb 2012 11:19:00 +0100

Don't ask me how I did it (ok, you can ask, in fact I'm going to tell you...), but somehow I now have a remote git server and gitolite for repo and user administration installed and running on Ubuntu 11.10 using an Amazon EC2 Micro instance.

I've spent all morning on this, tested it on 3 different instances (I love it how you can just throw away an EC2 instance and start again with a new one in a matter of minutes) and have kept a log of the steps which got me there. Please note that there may be errors or illogical jumps in this summary, but maybe it will be helpful nonetheless - or it may even work outright, who knows.

BEfore we get started, here are some links that helped me (but note I had to pick some steps from some links, and other steps from others...). In particular I did not clone gitolite to my local machine (as this post suggests but ended up using sudo apt-get install git

Links I used:
git + gitolite + git-daemon + gitweb setup on Ubuntu 11.10 server
How to setup git server using gitolite in Ubuntu 11.10 Oneiric
Install Gitolite To Manage Your Git Repositories

First of all, get yourself an instance of Ubuntu (you may swap this for your preferred Linux distro). If you are in love with EC2 then head over to alestic.com and click the selection at the top of the page (I'm using the us-east region), and then pick your preferred image.

The neat thing is that if you already have an Amazon Web Services account you can launch your instance with virtually one click, very neat. I picked the Ubuntu 11.10 Oneiric EBS boot image, 64 bit.

Once booted, SSH into your instance (consult other blogs for details on this if you need help).
I then ran the following commands first:
sudo apt-get update sudo apt-get upgrade
Then create a tmp folder
mkdir tmp
which is where you will copy your public key into. Now open another local Terminal window and copy your public key - consult the steps for generating new keys from other sources if needed.
# local machine scp ~/.ssh/id_rsa.pub yourEC2server.com:/tmp/stefan.pub
Note that SCP is not always straight forward - I struggle with the syntax at times - and EC2 complicates it a bit. So the format may in fact look more like this: scp -i ~/.ec2/yourEC2pair.pem ubuntu@ec2-999-239-176-93.compute-1.amazonaws.com:/tmp
I then installed git, gitolite and some stuff which I am not sure is strictly needed:
sudo apt-get install git gitolite git-daemon-run git-doc
Then added my git user details and added the git user
git config --global user.name "Stefan Richter" git config --global user.email "my@domain.com" sudo adduser --system --shell /bin/bash --gecos 'git version control' --group --disabled-password --home /home/git git
The system should reply with
Adding system user `git' (UID 105) ... Adding new group `git' (GID 111) ... Adding new user `git' (UID 105) with group `git' ... Creating home directory `/home/git' ...
I then switched to the git user account, added a PATH variable and then actually ran the gitolite setup.
sudo su git echo "PATH=$HOME/bin:$PATH" > ~/.bashrc # run gitolite setup, passing a reference to the .pub file we uploaded earlier. In my case this file was in the ubuntu/tmp folder. gl-setup /home/ubuntu/tmp/stefan.pub
Hit enter when asked, and quit the file that opens in VIM with :wq.
I was now able to exit the remote SSH session and could now clone a the remote gitolite-admin repository:
git clone git@ec2-999-239-176-93.compute-1.amazonaws.com:gitolite-admin.git

Amazingly this worked. Please note that I may well have left out a step, or got the order slightly wrong. If needed I could create a screencast sometime (please shout), I think this would make things clearer. Make sure you read the gitolite instructions for how you add users and repos. One mistake I made was to not push my changes to the remote and then was left wondering why my new repos were not showing... Git newbie, I know.

Lastly I downloaded a trial of Tower, a GUI based git client that comes highly recommended - they even have a 25% off offer running right now.

Good luck.

Troubleshooting SWF Loading Issues in Chrome - MIME Type Issue With S3

Mon, 30 Jan 2012 08:48:00 +0100

As some of you may already know, I run and maintain a few of my own products, the most popular of which being Scribblar which pushes hundreds of sessions every day.

Recently I started getting reports from users that the main page which hosts the applications' main SWF file was not loading properly, or it would work in one browser but not another. Within the handful of reports I had, Google Chrome appeared to be the browser that posed most of the issues - this seemed odd as Chrome effectively has Flash Player built-in and always auto-updates to the latest release version which is why I recommend it as the preferred browser to anyone who asks.

My first look was towards SWFObject - I figured that maybe something in Chrome had changed and broken the Flash Player detection. A common trap that some developers fall into is to check for specific Flash Player versions, for example only allowing access to Player 11 or below, which then locks users out once Player 11.5 (or similar) is released. But this wasn't the issue here.

After much more digging and more back-and-forth emails with some users I noticed a very odd behaviour when trying to access my SWF directly (without an HTML wrapper) in Chrome. This image shows the request in the Chrome Debugger.

Notice how the shows as 'canceled', and that the content type is coming up as the generic binary/octet-stream? Clearly this pointed towards Chrome not being able to deal with a wrongly set MIME type correctly, whereas other browser may have handled this is a more flexible way.

My files are served via Amazon S3, and the MIME type is usually set during upload and forms part of the file's metadata. My FTP client of choice is Transmit, and after some digging I spotted the 'Cloud' panel in Transmit's preferences.

Here you can specify a particular MIME type to go along with a file extension. For SWF this would be application/x-shockwave-flash (but while you are there you may as well set it up for other file types such as CSS).

After setting these Transmit preferences, all subsequent SWF uploads had the correct MIME type set and hitting the SWF directly in Chrome now gave the desired results. If you need to update the MIME type post-upload then have a look at tools such as BucketExplorer or S3 Browser (Windows only).

It now makes sense why my file showed as MIME type binary/octet-stream: this is the default MIME type used by S3 when no other MIME type is specified during the PUT operation into S3.
This issue took me a while to track down and I hope the information above helps someone.

Enable Ping ICMP Replies For Amazon EC2 Windows Instances

Fri, 28 Oct 2011 05:23:00 +0100

Here's a quick tip on how to configure a Windows Amazon instance to successfully respond to ping requests.

By default an EC2 security group does not allow ICMP ping requests, and in some cases the internal Windows firewall will also block it. You therefore should check both settings if you want to be able to ping your EC2 Windows instance.

Step 1: Check Windows Firewall Settings
To enable the Windows firewall to allow ping request check that under 'Inbound Rules' the setting 'File and Printer Sharing (Echo Request - ICMPv4-In)' is enabled. The icon should turn green if the rule is enabled.
Alternatively you can use the commandline option:
netsh advfirewall firewall add rule name="ICMP Allow incoming V4 echo request" protocol=icmpv4:8,any dir=in action=allow

This should take care of the Windows firewall.Step 2: Configure EC2 Security Group
EC2 security groups use the so-called Classless Inter-Domain Routing method to configure IPs and routes. You can read more about it on Wikipedia and I will not pretend that I know all the details, but what I do know is that the syntax to specify a single IP as part of whichever rule you are trying to set up is [the_IP]/32. The /32 part means that the first 32 bits (which in case of IPv4 addresses means all of them) have to match for the rule to match. So just remember: to create a particular rule for one IP address only just specify the IP followed by a forward slash.

On the other hand to allow all IPs (I tend to grant access to services normally to one IP or all) you would specify 0.0.0.0/0.

So first you should create a custom ICMP rule of type Echo Request as shown in the first image.

Next you specify the IP address (or optional CIDR blocks) and then apply your rule as shown in the second image.

Your pings to the instance should now work.

'The format of the specified network name is invalid' - IIS Error 0x800704BE

Mon, 24 Oct 2011 21:45:00 +0100

Oh don't you just love cryptic error messages that could mean one hundred and one things? Yeah, me too.

So in the interest of some poor soul (maybe it's you ;-) searching on this topic in the year 2142 I decided to point out what resolved this issue for me.

Background: I am running Windows Server 2008 R2 with IIS inside a Hyper-V instance. The VM was configured with a static IP and each IIS site was configured to bind to that IP, and that IP alone.
I transferred the VM to Amazon EC2 (using the ec2-import-instance API) since I wanted to move away from having to maintain my own physical hardware. Long story short, once transferred I was unable to start any of the IIS sites, they all failed with the error 'The format of the specified network name is invalid - Error 0x800704BE'.

I knew that this error was likely related to IP bindings of some kind (EC2 usually expects you to use DHCP for IP address assignment since even an elastic IP can change) so I tried binding a few IIS sites to 'all' IPs. Still the sites would not start, but throw the above error.

I dug deeper and used the netsh utility (Windows commandline) to show which the network configuration for the machine, and in particular which IPs the HTTP service listens to:

netsh http show iplisten

This listed just one (the previous static) IP of my VM - this was now wrong. So I removed the binding with

netsh http delete iplisten ipaddress=11.22.33.44 (where 11.22.33.44 is the actual IP that needs removing)

Next set the service up to listen to all IPs:

netsh http add iplisten ipaddress=0.0.0.0

I then restarted IIS using

iisreset

and bingo, the sites started to work.

Hope this helps someone.

Introduction To Amazon Web Services

Thu, 15 Sep 2011 16:01:00 +0100

Yesterday afternoon I presented at Flash on the Beach 2011. The title of the sessions was 'AWS(omeness) - An Introduction to Amazon Web Services.
Overall the session went pretty well and I had a good number of bums on seats despite going up against the awesome Jared Ficklin, the inspiring James White and legendary Keith Peters who was presenting in the Influxis lounge.

The conference itself was immensely interesting, and even though at one day it felt as if we were laying Flash to bed, the next day was the resurrection with elevator pitches that showed so much creativity and skill that I felt that Flash has little to worry about - there's just too much talent and diversity in this community. But only time will tell. Personally I enjoy being part of it all, whether it lasts or not.

Ok, enough of that, here are my slides (also available as PDF - 4.8MB).

Introduction To Amazon Web Services

Speaking At Flash on the Beach 2011

Mon, 06 Jun 2011 13:55:00 +0100

I'm happy to announce that I will be speaking at Flash on the Beach this year, and for a change my session will not be directly about Flash but about my experience with Amazon Web Services (AWS) over the last few months. During my work on Scribblar.com I have learned to love AWS and think that many of my fellow developers would find it very interesting and useful.

The plan is to give attendees an overview of AWS and then take a closer look at its main products such as EC2 and S3. I'd like to cover things such as setting up a static site directly in S3, and go into some more detail about EC2's capabilities when it comes to building highly available and scalable websites. If time allows I will also cover options for running your database on AWS. No previous knowledge about AWS is needed.

I'm very excited to be back at FOTB again this year having previously been there once as a pseaker and twice as an attendee. I hope to see you in Brighton in September.

Dynamic Streaming Using F4M And Flash Media Playback Via CloudFront

Sat, 07 May 2011 15:31:00 +0100

Here is a quick heads up on an issue you may encounter when streaming video using Flash Media Playback and f4m files to provide dynamic streaming whereby the player will automatically pick the correct bitrate version depending on the user's connection speed.

In my case I wanted to stream my videos using Amazon's Cloudfront service. A typical RTMP URL will look something like this: rtmp://saaabbbccc.cloudfront.net/cfx/st/mp4:myvideo.mp4 If you go ahead and create an f4m file using this you may end up with something like the following (presuming 3 bitrates at 500, 1000 and 1500 kbps): Dynamic Streaming video/mp4 rtmp://saaabbbccc.cloudfront.net/cfx/st Unfortunately this file will not work when fed into Flash Media Playback. The reson (and fix) is quite simple and one some of us may remember from using the FLVPlayback component in Flash. It is a missing application instance name. In our case this is the default instance _definst_ that needs to be added to the baseURL. The correct f4m listing would therefore be as follows: Dynamic Streaming video/mp4 rtmp://saaabbbccc.cloudfront.net/cfx/st/_definst_ I hope this helps someone. I was slightly confused by this as any one of my files would play fine using the FLVPlayback component without specifying the _definst_ in the video RTMP URL.

And one final gotcha: if you host your f4m files in an Amazon S3 bucket (but not your streaming bucket, you need to use a separate non-streaming bucket for non-video files) you may require your own crossdomain file inside it or the Adobe hosted Flash Media Playback SWF won't be able to load it.

A free tool to help you manage your S3 buckets (if you are on Windows - I run this tool in a VM) is CloudBerry Explorer. It's one of the better S3 related tools out there. Do you know an equally good one for OSX?

Online User Group: FMS 4 on Amazon Web Services

Tue, 11 Jan 2011 11:46:00 +0100

Join FMS Product Manager Kevin Towes for an online session titled 'FMS 4 on Amazon Web Services'. This presentation is part of the FMS Online User Group and takes place on Tuesday, January 18th at 12:00 Eastern / 9:00am Pacific (GMT-8).

Join Kevin as he reviews and demos Flash Media Enterprise Server 4 running on Amazon Web Services, Adobe's newest deployment option that makes it easy to access new Peer to Peer functionality and much more. You'll learn how to deploy your FMS applications quickly and affordably on Amazon Web Services and access the power of new features including major enhancements to playback experience, and quality of service, and support for F4F packaging for HTTP streaming.

If you're looking to include the FMS platform in your business, and are curious about the opportunities with Amazon Web Services, or just want to get the most out of your investment in Adobe Flash Media Server technology, this session is one you don't want to miss.

RSVP: http://fms.groups.adobe.com/index.cfm?event=post.display&postid=32832

Adobe Flash Media Server on Amazon Web Services

Wed, 15 Dec 2010 12:04:00 +0100

Today Adobe announced the availability of a new hosting service for FMS: Flash Media Server on Amazon Web Services (AWS). Kevins Towes, FMS Product Manager, has full details on his blog.

In a nutshell, FMS on AWS allows you to quickly deploy a fully licensed instance of FMS 4 in the cloud. The charges are a combination of FMS license fees and Amazon AWS charges, plus a mark-up on the hourly AWS fees. For example a lareg FMS instance which is limited to 100 RTMFP P2P connections will cost $0.44 per hour to run. A normal AWS large instance (without FMS) is charged at around $0.38 per hour. To run such an FMs instance for a month will cost just over $300, plus any bandwidth on top plus a $5 recurring monthly fee. But do not forget the benefits this type of deployment brings. Imagine having to run a massive live event - previously you would need to rent your own hardware, buy FMS licenses and set everything up by hand. Combining FMS with AWS essentially gives you FMS on tap - you may use it for a whole month, or just a few hours for your actual event. In such cases the AWS deployment can work out extremely cost effective.

I was able to test the AWS instances with FMS and I must say it was very easy to get up and running, especially when you are already an AWS customer as I was. I simply logged into the AWS portal, chose the FMS AMI bundle, went through the subscription sign-up and within minutes I had a fully working FMS 4 instance up and running, ready to stream.

I suggest you give FMS on AWS a try next time you need to run a large scale event or need to scale up your existing FMs infrastructure.

Streaming Video with Amazon Cloudfront

Thu, 25 Feb 2010 10:36:00 +0100

Streamingmedia.com have published a great tutorial and overview on how to get started with Amazon's new Cloudfront Streaming service. While the service currently only supports on-demand streaming of Flash video content the company also has plans to support live streaming in the near future. This service looks like a very cost effective and scalable video streaming service, and is relatively easy to set up.

If you are interested in this service and require any help or advice then please feel free to get in touch. I've set up several video streaming environments on Amazon Cloudfront as well as EC2 in the past.

Amazon Offers FMS Powered CloudFront Streaming

Wed, 16 Dec 2009 11:01:00 +0100

I'm a little surprised that I had to pick this info out of an Amazon email newsletter as this seems to be pretty major news to me. What is Adobe's PR department doing to not send me some up-front info on this?

Anyways, Amazon Web Services is now offering a CloudFront based streaming service (supporting audio and video) that is powered by Flash Media Server. What this means is that everyone can now have an on-demand, highly available and highly scalable audio and video streaming CDN at their finger tips. How does this differ from traditional CDNs? The main difference is that you are only charged for actual usage, there are no tie-ins or long term contracts as is the case with most (if not all) of Amazon's Web Service offerings. Depending on Edge Server location the fees start at around $0.17 per GB of transfer up to the first 10TB used, then decrease as data usage rises.

What is interesting from a Flash developer's point of view is that Amazon's streaming server of choice which has been rolled out on a massive scale here is Flash Media Server. It should be noted that this service can only be used for on demand and live streaming, so allows no server side code or RTC based apps which include any server side logic.
You can find more info here.