Compare the two container-orachestrations Elasticbeanstalk and EC2 Container Services

This post is day 23 Sansan Advent Calendar.

f:id:hrt0kmt:20181224051524j:plain

ElasticBeanstalk (EB)

Have you ever used ElasticBeanstalk (hereinafter, this is called "EB") before?

This service with a fairy name (Beanstalk = Jack and the Beanstalk) create a Paas-like environment by linking each AWS Services. Please Google it (something super simple) yourself.

Just a few clicks, you can create all the elements which necessary for the application (load balancer, EC2 which include provisioning, AutoScaling, RDS, CloudWatch, SNS and so on) and deploy automatically.

EB can deploy apps with Blue-Green Deployment or rolling update (host).

"Application" is top layer, Environment and Version are included in it. "Environment" is environment which application deployed. URL will be issued for each Environment. "Version" is for version management of web applications. Roll back will be executed via this version management.

you can use EB CLI as below.

$ eb init {application_name} \
--verbose \
--region "ap-northeast-1" \
--keyname "xxx"
INFO: Setting only environment "xxx-env" as default
INFO: Pulling down defaults from environment xxx-env
INFO: Setting up .elasticbeanstalk directory
INFO: Setting up ignore file for source control

$ eb create xxx-env \
--elb-type classic \
--database.engine mysql \
--database.version "5.7" \
--database.username "xxx" \
--database.password "xxx" \
--instance_type t2.large \
--tier webserver \
--scale 2 \
--vpc.id "xxx "\
--vpc.ec2subnets "xxx" \
--vpc.publicip "xxx" \
--vpc.seciritygroups "xxx" \
--timeout 60
Creating application version archive "app-xxx".
Uploading: [##################################################] 100% Done...
Environment details for: xxx
  Application name: xxx-app
  Region: ap-northeast-1
  Deployed Version: app-xxx
  Environment ID: xxx
  Platform: arn:aws:elasticbeanstalk:ap-northeast-1::platform/xxx running on 64bit Amazon Linux/2.5.5
  Tier: WebServer-Standard-1.0
  CNAME: UNKNOWN
Printing Status:
INFO: createEnvironment is starting.
INFO: Using elasticbeanstalk-ap-northeast-1-xxx as Amazon S3 storage bucket for environment data.
INFO: Created CloudWatch alarm named: awseb--xxx
INFO: Created CloudWatch alarm named: awseb-xxx
INFO: Added instances [i-xxx, i-xxx] to your environment.
INFO: Environment health has transitioned from Pending to Ok. Initialization completed 20 seconds ago and took 14 minutes.
INFO: Successfully launched environment: xxx

provisioning

if you want to overwrite config files of OS files or execute some specify scripts on host (provisioning), you can set "ebextensions". .ebextensions/*.config folder and files can customize environments. file type is YAML or JSON.

AWSConfigurationTemplateVersion: 1.1.0.0

Tags:
  Cost Center: WebApp Dev

packages:
  yum:
    git: []

commands:
  01-command:
    command: cp /usr/share/zoneinfo/Japan /etc/localtime

option_settings:
  - namespace: aws:elasticbeanstalk:container:php:phpini
    option_name: document_root
    value: /var/www/vhosts/web/htdocs

paclages
- install packeges by yum or rpm
sources
- deploy some archive(ex. tar file) from public.
files
- create some files.
services
- start service and set chkconfig
commands
- set commands or scripts (OS or MW settings) before source deployment.
container_commands
- set commands or scripts after source deployment.
option_settings
- option_settings key can define environment variables.
resources
- define additional resources (SQS queue,table of DynamoDB,CloudWatch Alarm).

When create an environment via Management Console, setting changes of ebextensions will not be refrected. the priority order is as below.

Value set by API (Management Console).
Value set with .ebextensions
default value

Actually try to use EB

I feel difficult to use in EB due to fllowing points.

If deploy fails for some reason, it is painful to debug from deployment logs.
EB cant scale in / out with container unit.
It is necessary to customize a lot things by default according to the specification of EB (ex. notification content of SNS).
longest immobilized name of each resources (SG, EC2, ELB name, and so on) difficult to understand (awseb-e-u-AWSEBloa-xxxxxxxx-...).

EC2 Container Service (ECS)

EC2 Container Service (hereinafter, this is called "ECS") is specialized in container deplyment. Here i refer to both EC2 and Fargate type.

ECS can use dynamic port mapping (ALB).
New features Docker are incorporated faster than EB.
Rolling update with container unit (Task Definitions).
There is no need to worry about the host (Only Fargate).
ECS Service Discovery can connect container directory.

Network Mode

awsvpc

Fargate is only use awsvpc mode.
It cant be used only in Amazon ECS Optimized AMI.
The number of ENI is limited (include primary netwrok interface, for example, t2.medium and large can use 6 ENI
awsvpcConfiguration section on task definitions, specifiable subnet is up to 10, SG is 5.
Compared to other mode, awsvpc is very high network performances.
Dynamic port mapping cant be used.
Supports ALB or NLB only.

bridge

Create network stack for docker0 virtual bridge
Dynamic port mapping is possible.
Can not link with container of other tasks.

none

Can not specify port mapping.
There is no external connection to container.
Network stack is not be created.

host

Use stack of host netwroking.
Do not allow links each containers.
Not compatible with dynamic port mappings.

Task Definitions

task-definition is like docker-compose.yaml
secrets section can set confidential variables.

ex. postfix and data-volume container.

{
  "name": "postfix",
  "hostname": "postfix",
  "image": "${var["ecr_repo"]}/postfix:${var["postfix_tag"]}",
  "network_mode": "bridge",
  "dnsSearchDomains": [],
  "logConfiguration": {
    "logDriver": "fluentd",
    "options": {
      "fluentd-address": "${var["fluentd_addr"]}",
      "tag": "ecs.postfix"
    }
  },
  "entryPoint": [],
  "portMappings": [
    {
      "protocol": "tcp",
      "hostPort": 0,
      "containerPort": 25
    }
  ],
  "command": [],
  "linuxParameters": {
    "capabilities": {
      "add": null,
      "drop": null
    },
    "sharedMemorySize": null,
    "tmpfs": null,
    "devices": null,
    "initProcessEnabled": null
  },
  "secrets": [
    {
      "name": "SMTPPASS",
      "valueFrom": "${var["xxx"]}"
    }
  ],
  "environment": [
    {
      "name": "RELAYHOST",
      "value": "${var["xxx"]}"
    }
  ],
  "ulimits": [],
  "dnsServers": [],
  "mountPoints": [],
  "workingDirectory": null,
  "dockerSecurityOptions": [],
  "cpu": 69,
  "memoryReservation": 69,
  "volumesFrom": [],
  "disableNetworking": false,
  "healthCheck": {
    "command":[ "CMD-SHELL", "ping -c 1 localhost || exit 1" ],
    "interval": 10,
    "startPeriod": 10,
    "timeout": 60,
    "retries": 3
  },
  "essential": true,
  "links": [],
  "extraHosts": [],
  "user": null,
  "readonlyRootFilesystem": false,
  "dockerLabels": {},
  "privileged": false
},
{
  "name": "data-volume",
  "hostname": "data-volume",
  "image": "${var["ecr_repo"]}/data-volume:${var["data_volume_tag"]}",
  "network_mode": "bridge",
  "dnsSearchDomains": [],
  "logConfiguration": {
    "logDriver": "fluentd",
    "options": {
      "fluentd-address": "${var["fluentd_addr"]}",
      "tag": "ecs.data-volume"
    }
  },
  "entryPoint": [],
  "portMappings": [
    {
      "protocol": "tcp",
      "hostPort": 0,
      "containerPort": 8989
    }
  ],
  "command": [
    "/bin/sh",
    "-c",
    "/etc/xxx/xxx.sh"
  ],
  "linuxParameters": {
    "capabilities": {
      "add": null,
      "drop": null
    },
    "sharedMemorySize": null,
    "tmpfs": null,
    "devices": null,
    "initProcessEnabled": null
  },
  "environment": [],
  "ulimits": [],
  "dnsServers": [],
  "mountPoints": [],
  "workingDirectory": null,
  "dockerSecurityOptions": [],
  "cpu": 69,
  "memoryReservation": 128,
  "volumesFrom": [],
  "disableNetworking": false,
  "essential": false,
  "links": [],
  "extraHosts": [],
  "user": null,
  "readonlyRootFilesystem": false,
  "dockerLabels": {},
  "privileged": false
},

Actually try to use ECS

I feel not good enough point in ECS.

After task deploy failed, just looking at Management Console, it is hard to understand what is occurring.
Unable to get each CloudWatch metrics per container (around metrics, i'm concerned about datadog).

summary

As my feeling, EB (Multi Container) is good as the getting started with container or early stage of development. Because it is easy to start but there are several problems. For example, all cluster instances needs same container sets and container logs is hard to see (which instance processing target data), soft memory limits are not supported. Rather than spending a lot of time on .ebextensions and more customize, I want to make the most of this time.

I think that ECS can solve almost all the problems of EB. Actually I have not used Fargate yet, so I'd like to use it next year. Considering the cost comprehensively, I dont think that there is a difference of more than twice as much as EC2 type. But k8s will someday, supersede ECS.

By the way, If I really have to log in to the container, i'm planning to use SSM, but i would like to do something about access control.

tail -f /dev/null

If you haven't had any obstacles lately, you're not challenging. be the worst.