I’ll be interested to see how long it takes for someone to build a Time Machine-style backup service using this new core functionality; retaining all revisions of all files stored within that volume. People are going to have to be a little careful about how much they store with this new model, because I imagine it will be very easy to start adding up a lot of space when storing duplicates of objects over time. It seems as if AWS stores full copies of objects (not partial differences between versions), so storage space should be easier to calculate/account for at least.

Versioning is enabled on a per-bucket basis, and you may also optionally require multi-factor authentication with a hardware device to delete versioned objects. The new functionality introduces a versionid concept for each object within a versioning-enabled bucket which keeps track of specific versions of an object, while a normal GET request will get the most recent version available.

It’s great to see Amazon continue innovating and listening to their customers. They are definitely not the only players in the cloud storage/computing game, but they seem to be doing a pretty good job at staying ahead of the curve.

Check out the KRead project page for some more details or jump over to http://kread.dentedreality.com.au/ and try it out.

According to iTunes, I have just under 6,000 “items” in my music library. That’s 16.5 days of music, playing 24/7. I like to find new music, and have pretty eclectic musical tastes (literally everything from The Corrs to korn appears in my library). I’d like a tool that helps me do a few things:

1. Find more music by the artists that I like
2. Find new artists that are similar to artists I like
3. Find new genres/categories of music to try out
4. Download/purchase all this music and add it to my iTunes library quickly and easily

Putting all this together, here’s what I’d like (Lazyweb?):

• An Adobe AIR application (that runs on my Mac)
• Full-screen ability, accessed ideally via the Apple Remote, a la Front Row
• Load my iTunes library (or Last.fm account perhaps) and use the play counts as a starting point for exploring music
• UI something like LivePlasma to start with, allowing me to browse related artists, as determined via the Amazon API.
• Navigate the hierarchy/relation-graph of genres to find new/related genres
• Use a CoverFlow-style UI to flick through the album art of albums within genres, showing artist and title on each album.
• Flip an album to view track listing.
• Preview all tracks available via MP3 right there (add them to a queue and let me keep browsing)
• Enable 1-click purchase of MP3 tracks, automatically adding them to my iTunes account once they are downloaded.
• It’d also be cool if this UI/app could run in a “local” mode, where it just provided a navigation experience on my own music collection, while still pulling in related information from Amazon.

That’s all for now – what do you think? What features would you want in a music exploration service? Do you even collect a local library anymore, or do you rely on services like Pandora to provide your beats? I’m also still waiting for a decent system to manage my media on my laptop + home theatre (Mac Mini).

So far I would estimate around 90 – 95% males, and perhaps even a higher than SF-usual ratio of about 90% Mac laptops in use.

NOTE: You know you’re at a “corporate” rather than a “community” event when the only wifi available requires payment…

Opening Statements – Adam Selipsky, VP AWS

It all started with Amazon building our own application (amazon.com), and needing to scale it, provide performance, etc. We spent over a couple of billion dollars to provide the infrastructure. We learned lessons:

• Moved away from expensive hardware
• Moved to standard hardware and lots of it
• Service-based architecture

Perhaps other people have the same or similar problems? So we started talking internally and externally about what we could provide. Asked people what’s important for all of this stuff

1. Reliability: It has to Just Work, not necessarily 100% of the time, but close,
2. Scalability: Upward and downward scalability,
3. Performance: It has to operate as quickly as if it were all in one data center
4. Simplicity: don’t make people have to learn too much (make it as simple as possible)

Enabling business to build and get paid for scalable, high-performance applications. Good results so far, including over 400,000 registered developers and S3 contains over 22 billion objects. Why do people love this stuff so much?

1. Get rid of the muck: don’t do things you don’t want to do or can’t do,
2. Focus: allows you to focus on product development, marketing, sales strategy etc,
3. Time to Market: allows start-ups to shave 2-6 months off delivery cycles (provisioning, configuration, etc)

Q: Is this a serious business or an experiment?

A: This is not an experiment, this is Amazon’s third business line. Retail Business, Seller Business (1.4 million active sellers) and now Developer Business.

Mention of Amazon Start-Up Challenge and invite people to check it out.

AWS Presentation – Jeff Barr, Sr. Evangelist AWS

As a consultant, I was invited to a small little conference hosted by Amazon. I was one of about 5 or 6 other freelancers there. I realized that they were talking about taking the covers off amazon.com and making the core of some of their system available to “everyone”. I was hired into a slightly different position and around 15 – 20% of his time was to “help out” when he could with AWS.

Summary of the 3 main business streams of Amazon.

Bandwidth consumed by AWS has surpassed the rest of all Amazon properties combined.

Why do people like this so much?

The “muck” or “undifferentiated heavy-lifting” can take up 70% of the time involved in building and scaling an online application.

Talks about the differential between projected usage and real usage, and how if you (traditionally) purchased assets to handle your projections, you will almost always have waste (or a deficit!).

Mentions “being famous” as showing up on: Techmeme, Reddit, Digg, Slashdot, TechCrunch

Amazon doesn’t usually see too many spikes across all users because of the averaging out effect of all of their clients’ usage cycles.

AWS principles: Easy to use, Fast, Elastic, Highly available, Secure, Pay as you go

Summaries of EC2 (+ EBS), S3 SimpleDB, DevPay, SQS, FPS, MTurk.

81 million Amazon.com users who you can enable to pay through FPS/DevPay.

Look at Podango, who have an impressive system that transcodes podcasts and automatically re-assembles them all with new advertising, announcements and raw content (daily). Everything is powered by AWS and automatically scales up and down to handle load (transcoding/assembly).

What has happened in the last year? FPS, DevPay, Elastic IPs, Availability Zones, Large Instances, CPU-Intensive Instances, EBS, S3 in EU, SimpleDB additions, Premium Developer Support, Service Health Dash, OpenSolaris/MySQL, plus more.

Trends We’re Seeing

• Systems using more than one service
• Massive datasets and large-scale parallel processing
• Enterprise adoption
• Increased need for support and transparency
• Running more sophisticated software in the cloud

Future Roadmap

• Security features and certifications
• Focus on operational excellence
• US and Int’l expansion
• Localization of technical resources (translations etc)
• EC2 GA (general availability) and SLA (we’re close)
• Windows Server support
• Additional services (we’re not anywhere near done!)

Start-Up Challenge

Over 900 entries last year in the competition to win $100,000 (cash + AWS credits). Ooyala won with their video content management system. Entrants are judged on:

1. Creativity and Originality
2. Likelihood of long-term success
3. How well it addresses a market need
4. Usage of AWS

Customer Presentations

Tap in Systems – Peter Loh

IT monitoring and systems management, powered by AWS. Their specialty is heterogeneous environments. They are recently funded and are now available under a limited beta. Monitoring usually comes last (requirements –> application –> monitoring), even though it’s a critical piece of the puzzle.

Deploy a monitoring solution in the cloud, capable of monitoring cloud devices or internal IT systems. Works like enterprise monitoring – gathers stats from all your systems, compiles them and provides reports. Their custom-developed systems are open source and will be released back to the community. Everything is priced per usage.

Provide all sorts of custom adapters/components to cater to specific services and their relevant metrics. Runs with Nagois, Ganglia, Cacti, Linux, Windows etc! They also ensure the security of your information because the EC2 instance runs under your AWS account, not theirs (they never see your data, just provide the engine to “do stuff with it”).

Vertica – Jerry Held

Grid-based, columnar DBMS for data warehousing and analytics. Founded in 2005 and now has 50+ customers worldwide, with great leadership/advisors. Moving some of their business to the cloud has provided a whole new playing field to expand their business etc.

Vertica provides:

• Scale out grid architecture
• Aggressive data compression
• Automatic high availability (failover, replication and recovery all in the cloud)

Some examples of what people are using it for:

• Analytic Software as a Service (companies providing specialized/assisted analysis of data)
• Short-term Analytics (competitve analysis, marketing campaign metrics, event post-mortem analysis etc)
• Frictionless Vertica Proof of Concepts (using the cloud lowers costs for Vertica AND for their customers)
• Vertica R&D (stress testing, benchmarking, etc)

“Don’t look at what you used to do, and figure out how to do it cheaper and faster in the cloud. Look at things that couldn’t be done before, but how this technology allows us to do something fundamentally different.”

Animoto – Brad Jefferson

Completely automated videos based on photographs and music. The Animoto engine analyzes the audio and syncs the video with the nuances of the music, rendering a completely custom video every time. It takes about 5 minutes to render a 30 second Animoto clip.

Halted development for 3 months and converted everything over to Amazon’s infrastructure. Probably the best decision they’ve made as a company. They use EC2, S3 and SQS. They don’t own a single server, and all work from laptops.

They have Editor, Producer and Renderer Queues to handle each component of the system, based on how it works in the real world.

Facebook application absolutely spiked them out – around 5 instances up to 3,600 instance in a matter of days. 25K users to 700K users on Facebook. It would have taken them 7,000 servers to handle this traffic spike.

Elastra – Stuart Charlton

A suite of software applications and components, provisioned and managed in the cloud using technologies created by Elastra. A company that’s around a year old, with 20-ish people. Software can manipulate hardware – game-changing.

Giving the system architects the direct ability to provision and manage the pieces of the system they design. EDML and ECML provide abstracted configuration of how this will all work together.

Q&A

Q: What about intellectual property rights etc?

A: It’s your data, it’s your code. We don’t look at it and we claim no ownership over it.

Q: Do you use Amazon for storage as well, or just for processing?

A: Everything is stored on S3.

Q: What is missing from AWS?

A: CDN performance, latency/initial data across the wire, EBS solves a lot of problems, Elastic IPs are a little limited, DevPay could use some more granularity on its reporting, a better way to handle software licensing.

Q: How are your experiences using EBS vs S3

A: Similar to Enterprise SAN or Network Storage. [Elastic Fox]. You can stripe, RAID, and performance is close to the local ephemeral storage. It will be the primary way that we configure our systems.

Q: How can you speed up the “spin-up” time for these services dealing with large volumes of data.

A: We’re looking at parallel loading of content using multiple instances to speed things up.

Salil Despandi – Bay Partners

We always ask if people are using EC2/S3. If not, why not? What about your architecture prevents you from working on AWS infrastructure? As a startup, you’re in a good position to re-architect so that you can.

“Java has become the COBOL of the 21st century.”

Issues (a year ago)

• ephemeral storage (now EBS!)
• cookie-cutter-ness of machines (more options now)
• scale out, not up (can get bigger sizes though)
• IP stability etc
• still very machine focused (not minutes/hours/etc)
• commercial software licensing is an unknown
• configuring and managing thousands of nodes is still difficult

Opportunities

• tools & higher layers of infrastructure (fault tolerance, load balancing, dashboard, etc etc)
• higher levels of service (like engineyard.com for RoR, ways to get data in, CDNs, security, etc)
• every layer of the stack, offered as a distributed scalable service in the cloud
• AWS-compatible private clouds
• open source and cloud intersection
• big ecosystem

Cocktails/Networking

Cheese/bread/fruit platter provided. Hors’ dourves came around care of the SFD staff and there were a few different beers and a red/white wine available. Not too bad at all. I had a few interesting conversations with a few interesting people, and it seemed like others were doing the same. Cards I collected were from people at:

• Right Scale: really interesting services providers who help you manage all your AWS stuff
• FathomDB: upcoming provider of database services, backed by AWS technologies (MySQL on EC2/S3)
• rPath: software appliances powered by AWS
• syncplicity: interesting synchronizing service that syncs not just web <—> desktop, but also desktop <—> web applications (think Flickr, Facebook, etc).
• Shoppertron: provides white-labeled storefronts which aggregate big providers (eBay, Shopping.com, etc) and integrate smaller providers as required
• Carticipate: cool iPhone App which helps you find car-pooling/car-sharing buddies who are near you, going in your direction.

Feedback

I find these specific events to be a great “intro to AWS”, but personally I’d love to see them do some more advanced events where they are quite technical and explore, in depth, some of the solutions customers have built, what their architecture looks like etc. I talked with Tracy Laxdal (AWS Marketing Manager) at length about this, and about how a lot of the people using their services are developers, and could benefit from more education on the systems architecture side of things.

In retrospect, I think there is also some great value to be had (for AWS and for customers) to do some perhaps on-the-spot case studies of looking at how someone has an application built, and how they could take what they have and implement more/other Amazon services to enhance their architecture and improve on what they’ve got. This could even be done using existing web apps as examples, and breaking down how you could build it on AWS. For example, you could take Flickr.com and talk about spinning up instances for handling image uploads, resizing etc, then look at a possible architecture of databases and web servers, fault tolerance at all levels, etc.

Disclaimer: This is a theory, I haven’t tried this out (anyone at EdgeCast want to confirm/give me a free account to try it out? )

I was looking at ways to store large volumes of user-uploaded resources (images) in a web environment tonight and had a bit of an epiphany. I had defaulted to Amazon’s S3 web service as the storage platform, since I’ve worked with it before and love the idea of an “unlimited”, fully-redundant storage device. The main problem with S3 however, is that it’s got less than stellar performance as far as latency and geographically-optimized delivery. That, and the bandwidth is relatively expensive.

So — what happens if you combine S3 with a Content Delivery Network of some sort?

I have recently done some work with the guys over at EdgeCast, who provide a nice, proxying CDN that’s really easy to use. Basically you CNAME a sub-domain of yours to point to their network, and then all requests to that sub-domain are mapped via their CDN nodes to your static resources. So normally, that’d look something like this (if you had the CDN configured to load files from www.example.com):

User Request (cdn.example.com/images/test.gif) --> edgecastcdn.net --> www.example.com/images/gif

The CDN node closest to the User will handle the request, loading the resource from the source domain (www.example.com in this example) if required, then passing it along to the User. That got me to thinking — could you combine the performance of the CDN with the redundancy and reliability of S3? Sure, why not?

Initially I was going to say that you’d want to do the CNAME configuration of S3 to point a sub-domain on your main site to S3, but I’m going to take a stab in the dark and suggest that you can skip that step if you like. Instead, jump straight to EdgeCast and configure an “HTTP Customer Origin” to point to your bucket, let’s say it’s “cdn_s3″. You would enter “http://cdn_s3.s3.amazonaws.com/” in your admin screen. From my experience you’ll also need to contact EdgeCast support and ensure that they configure the “Customer Origin” in their system correctly. You’re going to want to have them point it to your S3 bucket’s URL, so that it’ll proxy content directly from there. You will also need to configure a DNS entry (CNAME) for “cdn.example.com” pointing to (in my case) “edgecastcdn.net” to handle requests from your web pages for images in your S3 bucket.

Let’s say you had an <img> tag in a page that referenced “http://cdn.example.com/images/test.gif”. Theoretically, you should be able to upload “images/test.gif” to your S3 bucket, then when that URL was used in a page, the request would be passed through to edgecastcdn.net (via the cdn.example.com CNAME), which would distribute it across it’s CDN to the node nearest to the requesting user. If that node had a “fresh” copy of the content, it would be served immediately. If it didn’t have a fresh copy, then it would request it back via the “Customer Origin” configured at EdgeCast, which is to say it would load it directly from http://cdn_s3.s3.amazonaws.com/.

Voilà – redundant, high-performance, geo-distributed and optimized content delivery. Right?