Why did Koding move away from containers and Docker to virtual machines?

great writeup on scaling containers

From their newsletter: Celebrating 500k dev milestone with a global hackathon and a $35,000 prize!

PS: In the past Koding suffered from a few stability issues and it pained us. Now, we're powered by AWS and every VM you get is an AWS VM. No more LXC's, no hacks and no dockerization which means no more downtime. Just this month we have turned on more than a 100,000 VMs – yes you read that right… a 100,000 VMs with 99.9% uptime. If you have not done it yet, come give us another try… we guarantee that you will like it very much.

Why did Koding move away from containers and Docker to virtual machines?

Launching spot instances on AWS from the Command Line

It is no secret that I love Amazon Web Services, and that we run all of our business on AWS. One of the things I fave the most is spot instances. The spot market is an auction market for unused resources. For literally a penny an hour you can fire up a server to try out something and shut it down when you are done. At a penny an hour you can run it for a long time for whatever reason you want. Pinterest uses this technique for their production load as well.

This weekend an r3.large which comes with 15GB of memory and SSD disks was costing 2 pennies an hour in the us-west-2c availability zone. So when I wanted to experiment around with few database related things it was a no brainer to fire one up on the spot market. Now, it takes all of a minute to zip through the screens on the AWS console to setup the spot request, and once fulfilled another minute or so to start up the machine and you are up and running. A side note, 20 years ago I was managing a super computer at UW-Madison. A grant from IBM for over a $1 million. That big ass machine had less memory than the instance I started up for 2 pennies. I was managing the software and another person was managing the hardware. It took up a large room of space with special cooling system. The fact that 20 years later I can fire up a machine with more processing power and more memory in 2 minutes for 2 pennies is incredible, a testament to Moore’s law and to AWS redefining the modern computing landscape.

At any rate, since I am trying out new things today; including firebase, but that’s a story for another post, I decided I want to fire up the instance from the command line and I want to script it so I can re-use it later.

I use the official aws cli for my command line needs. It works very well on my ubuntu instances as well as on my Macbook running OSX. So I figured I will look up the command line switches in couple of minutes and be on my way. Things did not turn out that well. The documentation for requesting a spot instance is lacking. They have couple of bland examples that did not really fit my needs. For one thing, if you run their example you will end up with 5 running instances that you cannot connect to remotely via ssh

aws ec2 request-spot-instances --spot-price "0.050" --instance-count 5 --type "one-time" --launch-specification "{"ImageId":"ami-a43909e1","InstanceType":
"m1.small","SubnetId":"subnet-d50bfebc"}"

because they do not tell you to specify the Key. That’s fine if you do not need to connect to the machine and if the instance is going to bootstrap itself to do whatever function it needs. But how do you do that when you are not specifying user data or cloud-init data? Unless you had already baked the startup into your AMI. But then they do not tell you that, and that kindda defeats the whole purpose of me starting up an instance in two seconds for trying out stuff.

I want my instance to run in a VPC, register itself with Route53, to create and attach an EBS volume, to install Percona MySQL, install latest ubuntu patches, copy down databases from s3 and restore them locally, and have a public IP address. But it is not clear how i can configure what I need from the command line so that my startup scripts run successfully. I already have the shell scripts to do what I need on startup, but how do I pass it to my spot instance?

The documentation was not really helpful with that. I tried to look at the docs for ec2 run-instances which was more thorough but a lot of options did not translate well to the request-spot-instance command. I could not for example figure out how to pass security group by name. Could not figure out how to launch into a VPC and ask for a public IP Address at the same time, the cli complained that the options are mutually exclusive. The tedious format of the nested JSON classes separate by escaped double quotes on a single line was also error prune.

Eventually, through trial and error and using the options for a dry-run and for generate-cli-skeleton i figured most of it out. But I still needed to pass my user data to the instance, and preferably to tag the instance so i can have the amazing cli53 register it for me in route53 with no fuss. I could not find an option to tag the instance from the aws cli, so i ended up hard coding the tags in my user data and then tagging the instance at first boot, which is not ideal but worked.

But what about the user data? One option is to pass it as string on the command line. My user data script is too long to make this possible. As I said, it created and attaches volumes, downloads a bunch of software, restores databases from s3, yada yada… The ec2 run-instances command has an option to specify the user data as a file on the command line such as: file://my_userdata_script, but that did not work. The ec2 request-spot-instances command actually has a separate option to pass all the input json as a file. That worked but did not let me pass my user data in either.

So eventually, I ended up storing the user data in a separate file, base64 encoding it from my request spot instances script and passing it as a bash variable to the ec2 request-spot-instances script.

As explained earlier my user data scripts tags the instance but I did not solve the issue with public IP address. I ended up specifying it on the subnet setup in the VPC instead. If you launch into a default VPC that would be set for you any how. If you are launching into a non default VPC like I am then you need to switch the setting on the subnet to make the public IP address the default setting.

My final script looks like this, note that you will want to use your own account identifiers for the IAM roles, security groups, etc…

[code language=”bash”]
#!/bin/bash
AWS_DEFAULT_OUTPUT="text"
UserData=$(base64 < userdata-current)
region="us-west-2"
price="0.03"
zone="us-west-2c"
key="UseYourKey"
iamrole="UseYourOwnARN"
sg="UseYourSecurityGroupId_NotName"
subnet="YourVPCSubnet"

aws ec2 request-spot-instances –region $region –spot-price $price –launch-specification "{ "KeyName": "$key", "ImageId": "ami-3d50120d" , "UserData": "$UserData", "InstanceType": "r3.large" , "Placement": {"AvailabilityZone": "$zone"}, "IamInstanceProfile": {"Arn": "$iamrole"}, "SecurityGroupIds": ["$sg"],"SubnetId": "$subnet" }"

[/code]

And now my instance is happily busily restoring databases and running my tests scripts, for couple of pennies an hour.

Once you have the instance up and running there are few handy commands to manage things. To check the recent prices:

aws ec2 describe-spot-price-history –start-time $(date -j -u +”%Y%m%dT%H0000″) –product “Linux/UNIX (Amazon VPC)” –instance-type “r3.large” –availability-zone $zone

This will give you the spot prices as of the current hour.

To check on your request status, which you will need to do a lot initially till you figure out all the command switches as you will run into a lot of “failed due to bad parameters” cases:

aws ec2 describe-spot-instance-requests

Finally to cancel your request

aws ec2 cancel-spot-instance-requests –spot-instance-request-ids $requestId

 

Ugly Sweaters

When we set out to build StratEx we did not set out to build the next ADP or Oracle or the next (insert name here) big company. When we started we decided to build an organization based on two correlated tenants:  Customer Happiness and Employee Happiness. We try hard to make sure one causes the other instead of negating it. A positive feedback loop. Not every company out there focuses on employee happiness. Many companies claim to focus on customer satisfaction (which is not the same as customer happiness), most pay lip service to that tenant. Some sacrifice employee happiness in order to achieve customer satisfaction.

At the same time, we are the eternal startup company. No matter our revenue, number of employees, or number of clients. We always want to be nimble. We always want every employee of the company to feel they have an extended family in their StratEx family.

At our headquarters we always had a pool table. If you walk in any time during the day you may see people taking a break and playing a game of pool, or bean bag toss, or watching a college basketball game on one of the big screen TVs. We work very hard at StratEx, but that’s not enough, that’s not at the expense of our employee happiness. We also play hard. That’s one of the reasons why we got nominated for one of the best companies to work for in Chicago, and we are proud of that distinction.

Going into the upcoming year, we are establishing a culture club at the company to make sure we keep refining our company culture and keep it aligned with our tenants of employee happiness and customer happiness.

As part of our culture club initiatives for the coming year we will be giving every employee time off as a reward for community service. An employee goes out and volunteers for service in their local community? we will reward them with paid time off.  We typically do events at the company for the whole staff. now we are going to make sure it happens at least once a month, on the first Monday of every month we are inviting the whole company for lunch.  We want our employees to be healthy. Starting this coming year we will reimburse employees for gym membership. We want everybody to stay healthy and stay happy. If volunteering makes you happy, we will reward you for it, if working out makes you happy we will reimburse you for it, if both make you happy, you are in luck 🙂

We have other nifty things planned as well, next time you are on the phone with a happy StratExian just chat them up.

But wait, there is more 🙂 Starting this year we will be hosting an annual Ugly Sweater contest. So put it on and bring it on. Coolest Ugly sweater take home the secret prize.

Chicago unilaterally raises the minimum wage

The Chicago city council approved raising the city’s minimum wage to $13 by mid 2019 starting in next July when the rate will raise to $10 from the current $8.25. The next raise will be to $10.50 in July of 2016, $11 in July of 2017, $12 in July of 2018 and $13 in July of 2019. Subsequently the minimum wage will increase based on the local consumer price index.

There has been a lot of chatter at the state level about increasing the minimum wage state wide but nothing materialized thus far.

There are several ways in eStratEx to increase the employee’s pay rate to match the minimum wage as laws change. It is a little trickier to handle tipped employees whose tipped wages for a day put them below the minimum wage. eStratEx automatically looks up the Federal minimum wage and the State minimum wage and creates an FLSA adjustment when running a payroll to make sure such employees are paid the minimum legal wage.

In the near future we will be extending the functionality to specific localities to accommodate cases like the city of Chicago where the minimum wage will be higher than both the State and Federal minimum wage. This will allow you to process payroll for tipped employees in such localities without having to do any manual adjustments.