Error creating application autoscaling target: ValidationException: ECS service doesn’t exist: service/AAA

Encountered this type of dependency issue before, but re-run terraform plan/apply usually can fix it, but not lucky this time :-)

After having a quick look, and found out the module depends_on (dependency between terraform modules) only officially supported for Terraform version 0.13 - hashicorp/terraform#10462 - but we’re on version 0.12 unfortunately.

Workaround

Found the workaround here:

The key insight here is that variables are nodes in the dependency graph too, and so can use them as a “hub” for passing dependencies across the module boundary.

Example

e.g.

In module A (e.g. with resource aws_appautoscaling_target)

1. Add the following in variables.yml:

1
2
3
4

variable "mod_depends_on" {
  type    = any
  default = null
}

2. Add depends_on for resource A aws_appautoscaling_target (as depends_on can refer directly to variables in Terraform version 0.12)

1
2
3
4

resource "aws_appautoscaling_target" "target" {
  ...
  depends_on = [var.mod_depends_on]
}

In module B (e.g. with resource aws_ecs_service):

Just need to add the following:

1

mod_depends_on = [module.A]

Reference:

https://discuss.hashicorp.com/t/tips-howto-implement-module-depends-on-emulation/2305

• ElasticSearch Backup (snapshot / restore) on AWS S3

• Steps / Configrations for ES snapshot / restore

• Use elastic curator to manage snapshots (create / remove)

• Docker image for ES Curator to manage Elasticsearch snapshots - davidlu1001/docker-curator

Overview

The purpose of this blog is to investigate the possible solutions to backup and restore ES indices. So that in the event of a failure, the cluster data can be quickly restored and minimized the business impact.

Backup content

Generally the backup content would include:

• Cluster data

• State configuration: includes cluster / index / shard settings

1
2
3
4
5
6
7
8
9
10
11
12

# index settings
curl -X GET "localhost:9200/_all/_settings?pretty"

# cluster settings
curl -s -XGET "http://localhost:9200/_cluster/settings?pretty&flat_settings&filter_path=persistent"
{
  "persistent" : {
    "cluster.routing.allocation.cluster_concurrent_rebalance" : "12",
    "cluster.routing.allocation.node_concurrent_recoveries" : "6",
    "indices.recovery.max_bytes_per_sec" : "320mb"
  }
}

P.S.

• Transient settings are not considered for backup

• Also can capture these cluster settings in a data backup snapshot by specifying the include_global_state: true (default) parameter for the snapshot API.

Snapshot Repository

In order to enable backup for ES cluster data, a snapshot repository must be registered before performing snapshot and restore operations.

Snapshots can be stored in:

• Local shared filesystem: NFS

• Remote repositories: backed by the cloud (AWS / Azure / GCP) or by distributed file systems (HDFS) with repository plugins

Considering that if use shared filesystem NFS, additional setup / configuration is required, and the cluster needs rolling restart to take effect. As all nodes must have access to the shared storage to be able to store the snapshot data. Therefore, from the perspective of complexity and operational safety, will not consider it (even the cluster only uses a little of the disk space).

And luckily we’ve got repository-s3 plugin installed on all nodes in the ES cluster, so rolling restart is not needed if using S3 as snapshot repository.

1
2
3
4

$ ES_PATH_CONF=/etc/elasticsearch /usr/share/elasticsearch/bin/elasticsearch-plugin list

discovery-ec2
repository-s3

So would prefer to use AWS S3 as a remote snapshot repository with existing S3 repository plugin.

Backup retention strategy
Based on backup granularity and retention time, we can choose the following strategies based on different scenario:

• Takes daily snapshots (during the hour we specify) and retains up to 14 of them for 30 days.

• Takes hourly snapshots and retains up to 336 of them for 14 days.

Can carefully start with daily backup (the initial backup may take relatively longer time) and keep monitoring, and then could try hourly backup if it’s feasible, in order to provide more granular recovery points.

Incremental snapshot mechanism

The snapshot functionality will simply remove the data that was only referenced by the deleted snapshot but will leave the data that is still required for other snapshots in the repository.

So it would be safe to delete the first initial snapshot without affecting the latter snapshots.

Ref: https://www.elastic.co/blog/found-elasticsearch-snapshot-and-restore

Backup cluster’s data

Will take snapshot per index, instead of backup the whole cluster, which can potentially bring the flexibility for the restore step.

The snapshots are taken incrementally. This enables us to take frequent snapshots with minimal overhead (the initial backup may take relatively longer time, based on the amount of data). The more frequently we take snapshots, the less time it will take to complete the backup.

The snapshot process is executed in non-blocking fashion. All indexing and searching operations can continue to run against the index that is being snapshotted.

Snapshot Prerequisites

• Check repository-s3 plugin is installed

1
2
3
4

$ ES_PATH_CONF=/etc/elasticsearch /usr/share/elasticsearch/bin/elasticsearch-plugin list

discovery-ec2
repository-s3

• Create an S3 bucket to store the snapshot (e.g. es-backup)

• Check current iam instance profile role on ES cluster

• Update existing IAM role and setup proper S3 Permissions

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:ListBucketVersions"
            ],
            "Resource": [
                "arn:aws:s3:::es-backup"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::es-backup/*"
            ]
        }
    ]
}

• Need to update the role permission for both ES master / data node

Note:

Otherwise will get the repository_verification_exception error when trying to register the snapshot repository

1
2
3
4
5
6
7

"root_cause" : [
  {
    "type" : "repository_verification_exception",
    "reason" : "[snapshot-s3-repo] [[zMHw44eLS7q0ItSfYmh2Ug, 'RemoteTransportException[[elastic-data-warm-0db25600a6f728fe9][172.17.24.56:9300][internal:admin/repository/verify]]; nested: BlobStoreException[Failed to check if blob [master.dat] exists]; nested: NotSerializableExceptionWrapper[amazon_s3_exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 425B663C880BBD13; S3 Extended Request ID: huXmNMJSK+Lh4vW5LIqJsYUfu9s3ZhPa2knSeblOVBbNnoffvYaZXhMz4sgfT8ZGTAAC+3yjUNc=)];']
    ...
    ]"
  }

• Register snapshot repository per index with base_path (prefix) in S3 bucket for ES cluster (one-time operation)

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# use `base_path` under S3 for `service-A`

curl -X PUT "localhost:9200/_snapshot/snapshot-s3-repo?pretty" -H 'Content-Type: application/json' -d'
{
  "type": "s3",
  "settings": {
    "bucket": "es-backup",
    "region": "us-west-2",
    "base_path": "service-A",
    "max_snapshot_bytes_per_sec": "500mb",
    "max_restore_bytes_per_sec": "1gb"
  }
}
'

• Register snapshot repository for each index (with base_path: same S3 bucket, but different directory)

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# e.g. for .elastichq

curl -X PUT "localhost:9200/_snapshot/snapshot-s3-elastichq?pretty" -H 'Content-Type: application/json' -d'
{
  "type": "s3",
  "settings": {
    "bucket": "es-backup",
    "base_path": "elastichq",
    "region": "us-west-2",
    "max_snapshot_bytes_per_sec": "500mb",
    "max_restore_bytes_per_sec": "1gb"
  }
}
'

• Check or Verify snapshot repository

  1	curl localhost:9200/_snapshot?pretty

Backup operation

The snapshots lifecycle management (SLM) feature is introduced and natively supported in ES version 7.5.0, so for lower version need to self-manage the snapshots / retention policy.

Can either use existing tool curator (with cronjob) on EC2 instance (e.g. gateway / master node), or can consider using ECS cronjob (scheduled task) to schedule the backup to avoid single point of failure.

For curator version compatibility, should be safe to choose the latest version 5.8.3 to support ES 6.X.

Backup operation with ES snapshot API:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

# e.g. backup specified indice

curl -X PUT "localhost:9200/_snapshot/snapshot-s3-repo/elastichq-20201125?wait_for_completion=false&pretty" -H 'Content-Type: application/json' -d'
{
  "indices": ".elastichq",
  "ignore_unavailable": true,
  "include_global_state": false
}
'

# check snapshots

curl localhost:9200/_snapshot/snapshot-s3-repo/_all?pretty

# delete snapshot

curl -X DELETE "localhost:9200/_snapshot/snapshot-s3-repo/elastichq-20201125?pretty"

Useful Config for Snapshot / Restore Process

max_restore_bytes_per_sec

Throttles per node restore rate. Defaults to unlimited. Note that restores are also throttled through recovery settings.

max_snapshot_bytes_per_sec

Throttles per node snapshot rate. Defaults to 40mb per second.

chunk_size

Big files can be broken down into chunks during snapshotting if needed. Specify the chunk size as a value and unit, for example: 1GB, 10MB, 5KB, 500B. Defaults to 1GB.

buffer_size

Minimum threshold below which the chunk is uploaded using a single request. Beyond this threshold, the S3 repository will use the AWS Multipart Upload API to split the chunk into several parts, each of buffer_size length, and to upload each part in its own request. Should between 5mb to 5gb. Defaults to the minimum between 100mb and 5% of the heap size.

Restore cluster’s data

Restore Prerequisites

• Confirm the target index has same number of shards as the index in the snapshot

• Close the target index need to be restored

Reference

By default, the cluster state is not restored. To include the global cluster state, need to set include_global_state to true in the restore request body (if include the cluster state in the previous backup operation), but usually we don’t want to backup the cluster state.

An existing index can be only restored if it’s closed and has the same number of shards as the index in the snapshot.

The restore operation automatically opens restored indices if they were closed, and creates new indices if they didn’t exist in the cluster.

Restore operation

In order to restore the index from a snapshot:

• Update index settings to speed up restore process:

1
2
3
4
5

# before restore
curl -XPUT "localhost:9200/{index_name}/_settings" -d' {
        "number_of_replicas" : 0,
        "refresh_interval" : "-1"
}'

• Restore snapshot from S3 (Set the index settings back at the same time)

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# e.g restore from specific snapshot service-A

curl -XPOST "localhost:9200/_snapshot/snapshot-s3-repo/service-A-2020112416/_restore?pretty" -H 'Content-Type: application/json' -d'
{
  "indices": "service-A",
  "ignore_unavailable": true,
  "include_global_state": false,
  "include_aliases": false,
  "index_settings": {
    "index.number_of_replicas": 2,
    "index.refresh_interval": "1s"
  }
}
'

P.S.

• Use the index_settings parameter to override index settings during the restore process

• Set include_aliases to false to prevent aliases from being restored together with associated indices

Curator Usage

use curator_cli

Examples:

Here the variable "${DRY_RUN}" can be either "--dry-run" (for dry-run mode) or "" (empty, means without dry-run)

• create snapshot

1
2
3
4
5
6
7
8
9
10
11

# create snapshot for index(es) in the same repo

/usr/local/bin/curator_cli \
${DRY_RUN} \
--host "${ELASTICSEARCH_HOST}" \
--port 9200 \
snapshot \
--repository "${REPO_NAME}" \
--name "${INDEX_PREFIX}-${curr_date}" \
--wait_for_completion --skip_repo_fs_check \
--filter_list "{\"filtertype\":\"pattern\",\"kind\":\"prefix\",\"value\":\"${INDEX_PREFIX}\"}"

• remove snapshot

  1 2 3 4 5 6 7 8

  /usr/local/bin/curator_cli \ ${DRY_RUN} \ --host "${ELASTICSEARCH_HOST}" \ --port 9200 \ delete_snapshots \ --repository "${REPO_NAME}" \ --ignore_empty_list \ --filter_list "[{\"filtertype\":\"age\",\"source\":\"creation_date\",\"direction\":\"older\",\"unit\":\"${UNIT}\",\"unit_count\":\"${UNIT_COUNT}\"},{\"filtertype\":\"pattern\",\"kind\":\"prefix\",\"value\":\"${INDEX_PREFIX}\"}]"

• restore snapshot

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

# close first

/usr/local/bin/curator_cli \
${DRY_RUN} \
--host "${ELASTICSEARCH_HOST}" \
--port 9200 \
close \
--ignore_empty_list \
--filter_list "{\"filtertype\":\"pattern\",\"kind\":\"prefix\",\"value\":\"${INDEX_PREFIX}\"}"

# restore

/usr/local/bin/curator_cli \
${DRY_RUN} \
--host "${ELASTICSEARCH_HOST}" \
--port 9200 \
restore \
--repository "${REPO_NAME}" \
--wait_for_completion --skip_repo_fs_check --ignore_empty_list \
--filter_list "[{\"filtertype\":\"state\"},{\"filtertype\":\"pattern\",\"kind\":\"prefix\",\"value\":\"${INDEX_PREFIX}\"}]"

use curator

Examples:

Here the variable "${DRY_RUN}" can be either "--dry-run" (for dry-run mode) or "" (empty, means without dry-run)

1

/usr/local/bin/curator --config /etc/curator/config.yml "${DRY_RUN}" /etc/curator/actions.yml

actions.yml for snapshot:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

# snapshot

actions:
  1:
    action: snapshot
    description: >-
      Snapshot company-index-A-* prefixed indices with the default snapshot name pattern of
      'company-index-A-%Y%m%d%H%M'.  Wait for the snapshot to complete.  Skip
      the repository filesystem access check.  Use the other options to create
      the snapshot.
    options:
      repository: snapshot-s3-index-A
      # Leaving name blank will result in the default 'curator-%Y%m%d%H%M'
      name: company-index-A-%Y%m%d%H%M
      ignore_unavailable: False
      include_global_state: False
      partial: False
      wait_for_completion: True
      skip_repo_fs_check: True
      disable_action: False
    filters:
      - filtertype: pattern
        kind: prefix
        value: company-index-A
  2:
    action: delete_snapshots
    description: >-
      Delete snapshots from the selected repository older than 14 days
      (based on creation_date), for 'company-index-A-*' prefixed snapshots.
    options:
      repository: snapshot-s3-index-A
      # Leaving name blank will result in the default 'curator-%Y%m%d%H%M'
      disable_action: False
      ignore_empty_list: True
    filters:
      - filtertype: pattern
        kind: prefix
        value: company-index-A
      - filtertype: age
        source: creation_date
        direction: older
        unit: days
        unit_count: 14

actions.yml for restore:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

actions:
  1:
    action: close
    description: "Close selected indices company-index-A before restoring snapshot"
    options:
      continue_if_exception: True
      ignore_empty_list: True
    filters:
      - filtertype: pattern
        kind: prefix
        value: company-index-A
  2:
    action: restore
    description: >-
      Restore all indices in the most recent company-index-A-* snapshot with state
      SUCCESS.  Wait for the restore to complete before continuing.  Skip
      the repository filesystem access check.  Use the other options to define
      the index/shard settings for the restore.
    options:
      repository: snapshot-s3-index-A
      # Leaving name blank will result in restoring the most recent snapshot by age
      name:
      # Leaving indices blank will result in restoring all indices in the snapshot
      indices:
      include_aliases: False
      ignore_unavailable: False
      include_global_state: False
      partial: False
      extra_settings:
        index_settings:
          number_of_replicas: 2
      wait_for_completion: True
      skip_repo_fs_check: True
      disable_action: False
    filters:
      - filtertype: pattern
        kind: prefix
        value: company-index-A
        exclude:
      - filtertype: state
        state: SUCCESS
        exclude:

docker-curator

The Docker image for ES Curator (to manage ES snapshots) could be found here - davidlu1001/docker-curator

This image keeps up to date with curator releases 5.8.3. It is also based on minimal alpine image.

Features

• Upgrade curator to version 5.8.3
• Add support for snapshot / restore (use curator_cli for single index scenario)
• Add support for snapshot / restore ALL indexes for ES using curator with actions rules. This would be useful when:
  • too many indexes (can not match with prefix / regex pattern) for curator_cli
  • if use different snapshot repository per index
  • for accident recovery scenario to restore ALL indexes
• Add DRY_RUN mode
• Rewrite Dockerfile and use alpine to reduce image size (with python3)

Usage

Image entrypoint is set to customized script, need to pass paremeters to CMD, can also support override ENV

Default ENV value:

1
2
3
4

TYPE=snapshot
INDEX_PREFIX=.kibana
REPO_NAME=snapshot-repo
DRY_RUN=True

e.g.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

# Snapshot single index with DRY_RUN mode, and delete snapshots 14 days ago

docker-compose run --rm es-curator snapshot .monitoring-es-7-2020.12.04 snapshot-repo True


# Restore single index (with latest snapshot) without DRY_RUN mode

docker-compose run --rm es-curator restore .monitoring-es-7-2020.12.04 snapshot-repo False


# Snapshot ALL indexes for ES without DRY_RUN mode, and delete snapshots 14 days ago

docker-compose run --rm es-curator snapshot ALL snapshot-repo False


# Restore ALL indexes for ES (with latest snapshot) without DRY_RUN mode

docker-compose run --rm es-curator restore ALL snapshot-repo False

Pass ENV:

1
2
3
4
5

- ELASTICSEARCH_HOST: default is `elasticsearch`

- UNIT: default is days, support `seconds | minutes | hours | days | weeks | months | years`

- UNIT_COUNT: default is 14

e.g.

1
2
3

# Snapshot single index without DRY_RUN mode, and delete snapshots older than 1 minutes ago

UNIT=minutes UNIT_COUNT=1 docker-compose run --rm es-curator snapshot .monitoring-es-7-2020.12.04 snapshot-repo False

References

• Elastic - Backup Cluster

• S3 repository plugin

• Repository settings

• ES - Snapshot and Restore (incremental snapshot mechanism)

• Elastic Curator

1. Check existing terraform version

1

terraform -v

2. Upgrade to Terraform 0.11 first (if applicable)

If the current version is not 0.11 then upgrade it to 0.11.14 first. If you are on 0.11.x, please make sure you are on 0.11.14.

3. Pre-upgrade Checklist

Terraform v0.11.14 introduced a temporary helper command terraform 0.12checklist, which analyses the configuration to detect any required steps that will be easier to perform before upgrading.

4. Initialisation in 0.11

1

terraform init

5. Plan and make sure no errors are shown

1

terraform plan

6. Apply

to ensure that your real infrastructure and Terraform state are consistent with the current configuration.

1

terraform apply

7. Do the checklist check

to see if there are any pre-upgrade steps in the checklist

1

terraform 0.12checklist

8. Resolve suggestions from checklist

Depending upon the suggestions above, take the steps & change the tf scripts.

If you are using templates, then you need to update the template provider as well.

Run above command until there are no suggestions.

9. Switch to terraform 0.12

Switch to terraform 0.12 (choose one of the following methods, ranked in order of recommendation)

• Using tfswitch (recommend) / tfenv to switch to 0.12

• Using docker-terraform

• Unpin the old 0.11 version (if applicable) and upgrade to Terraform 0.12: brew upgrade terraform

10. Initialisation in 0.12

For those repos that ref terraform-modules, need to:

• Make sure use git module to refer terraform-modules:

  e.g.

  1	source = "git@github.com:{user}/terraform-modules.git//{module_name}?ref={branch_name}"

• replace branch name to refer testing branch in terraform-modules (if applicable)

• update .gitmodules first to use testing branch:

  e.g.

  1 2 3 4 5 6 7 8 9

  cat .gitmodules [submodule "common/modules"] path = common/modules url = git@github.com:{user}/terraform-modules.git branch = {branch_name}

Then run command to update submodules:

1

git submodule update --recursive --remote

P.S.

Re-running init with modules already installed will install the sources for any modules that were added to configuration since the last init, but will not change any already-installed modules.

Use -upgrade to override this behavior, updating all modules to the latest available source code..

So use this command when there’re changes from terraform-modules:

1

terraform init -upgrade

11. Check for errors

1

terraform validate

12. Auto update the code

with below command. Terraform v0.12 includes a new command terraform 0.12upgrade that will read the configuration files for a module written for Terraform 0.11 and update them in-place to use the cleaner Terraform 0.12 syntax and also adjust for use of features that have changed behaviour in the 0.12 Terraform language.

1

terraform 0.12upgrade

12.1. Symlink

If there is a symlink under the directory, the directory of the actual file pointed by the symlink needs to be upgraded in the last, to avoid the situation where 0.11/0.12 codes coexist. This will terminate the execution of the 0.12upgrade command.

And also need to delete symlink first, before running 0.12upgrade, then restore the symlink files.

The logic for each directory would be something like this:

e.g.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

# init
terraform init -upgrade

# clean
rm -rf .terraform

# record symlink
find . -type l -name "*.tf" -ls | awk -F' ./' '{print $NF}' | awk -F' -> ' '{print "ln -s "$2,$1}' > /tmp/all_ln.txt

# remove
find . -type l -name "*.tf" | sed -e "s#\./##g" | xargs echo "rm" | bash

# upgrade
terraform 0.12upgrade -yes

# restore symlink
cat /tmp/all_ln.txt | bash

# plan
terraform plan

12.2. Error on batch upgrade

According to the official doc, run the following command for batch upgrade:

1
2
3

gfind . -name '*.tf' -printf "%h\n" | sort | uniq | xargs -n1 terraform init

gfind . -name '*.tf' -printf "%h\n" | sort | uniq | xargs -n1 terraform 0.12upgrade -yes

but got bunch of errors:

1
2
3
4
5

Error: error resolving providers:

- provider.template: no suitable version installed
  version requirements: "(any version)"
  versions installed: none

Turned out the init command will overwrite the .terraform/plugins dir each time:

e.g.

First init module ./A:

1
2

ls -rthl .terraform/plugins/darwin_amd64/
lock.json*                              terraform-provider-aws_v2.63.0_x4*      terraform-provider-template_v2.1.2_x4*

Then init module ./B:

1
2

ls -rthl .terraform/plugins/darwin_amd64/
lock.json*                          terraform-provider-aws_v2.63.0_x4*

So need to create script to process each directory one by one (first init then upgrade) instead of using xargs for batch upgrade.

13. Fix syntax issues

Might need to manual fix syntax issues (that can not be auto-upgraded):

• Invalid expression value: number required

1
2
3
4
5
6

Error: Incorrect value type
 
  on .terraform/modules/service-crayola/service/iam_db_auth.tf line 3, in data "template_file" "iam-db-auth-policy":
   3:   count    = var.enable_iam_db_auth
 
Invalid expression value: number required.

Use a conditional expression to select the count based on the variable:

1

count = var.enable_iam_db_auth ? 1 : 0

• Incorrect attribute value type

1
2
3
4
5
6
7
8
9

Error: Incorrect attribute value type
 
  on .terraform/modules/container-crayola/container/main.tf line 38, in resource "aws_iam_policy_attachment" "ci-attach":
  38:   users      = [var.username, var.extra_ci_users]
    |----------------
    | var.extra_ci_users is empty tuple
    | var.username is "ci-travis-crayola"
 
Inappropriate value for attribute "users": element 1: string required.

Referring to https://www.terraform.io/upgrade-guides/0-12.html#referring-to-list-variables

The solution is just remove the redundant list brackets or add a flatten function.

• Unsupported attribute for lifecycle/ignore_changes

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

Error: Unsupported attribute
 
  on .terraform/modules/balances-api_alb_target_group/alb-target-group/target-group.tf line 46, in resource "aws_alb_target_group" "main":
  46:       healthy_threshold,
 
This object has no argument, nested block, or exported attribute named
"healthy_threshold".
 
 
Error: Unsupported attribute
 
  on .terraform/modules/balances-api_task_definition/task-def-volumeless/main.tf line 8, in resource "aws_ecs_task_definition" "main":
   8:     ignore_changes        = [image]
 
This object has no argument, nested block, or exported attribute named
"image".

Ref: https://www.terraform.io/docs/configuration/resources.html#lifecycle-lifecycle-customizations

ignore_changes can only support list of attribute names, but image is the sub-attribute name for container_definitions

• Deperacated condition for lb_listener_rule

1
2
3
4
5
6
7
8
9
10
11
12
13

Warning: "condition.0.values": [DEPRECATED] use 'host_header' or 'path_pattern' attribute instead
 
  on crafty-microservice-main.tf line 30, in resource "aws_lb_listener_rule" "crafty_rule_0":
  30: resource "aws_lb_listener_rule" "crafty_rule_0" {
 
(and one more similar warning elsewhere)
 
Warning: "condition.0.field": [DEPRECATED] use 'host_header' or 'path_pattern' attribute instead
 
  on crafty-microservice-main.tf line 30, in resource "aws_lb_listener_rule" "crafty_rule_0":
  30: resource "aws_lb_listener_rule" "crafty_rule_0" {
 
(and one more similar warning elsewhere)

Just need to switch the format as follows:

From:

1
2
3
4

condition {
  field  = "path-pattern"
  values = ["*"]
}

To:

1
2
3
4
5

condition {
  path_pattern {
    values = ["*"]
  }
}

From:

1
2
3
4

condition {
  field  = "host-header"
  values = ["abc.hq.com"]
}

To:

1
2
3
4
5

condition {
  host_header {
    values = ["abc.hq.com"]
  }
}

This was a change introduced in terraform-providers/terraform-provider-aws#8268, which was released in AWS Provider 2.42.0.

Ref: https://github.com/brikis98/terraform-up-and-running-code/issues/43#issuecomment-569441777

Doc: https://www.terraform.io/docs/providers/aws/r/lb_listener_rule.html

• After fix
  After fix syntax issues in terraform-modules, need to update modules and re-run init / upgrade when necessary:

1
2
3
4
5

# git update submodules
git submodule update --recursive --remote
 
# upgrade modules to local .terraform/modules
terraform init -upgrade

14. Plan

1

terraform plan

Be aware of using correct aws role to execute plan / apply for different directories.

And might also need to fix syntax errors manually for those files can not be upgraded automatically (e.g. templates)

15. Apply

1

terraform apply

16.Post Actions

IDE support (VSCode)

When using vscode extension for Terraform (v1.x), it might show syntax error as follow:

1
2
3
4

Unknown token: 2:26 IDENT var.cidr
 
Peek Problem (⌥F8)
No quick fixes available

Need to add the following configs in VSCode settings.json to enable support for terraform 0.12

1
2
3
4
5
6
7
8

"terraform.languageServer": {
        "enabled": true,
        "args": []
    },
"terraform.indexing": {
    "enabled": false,
    "liveIndexing": false
  }

Unfortunately after installing the latest VSCdoe extension (e.g. 2.0.1 - which officially support Terraform 0.12) it seems NOT working properly.

Found bunch of errors:

• Workspace not initialized
• Nothing works after update to 2.x
• “Server return 404” when trying to install an old version

The old version 1.40 used to work, so the workaround:

• Remove 2.0.1 extension from VS Code
  Download 1.4.0 of the vsix from here
• Manually install it: code --install-extension mauve.terraform-v1.4.0.vsix
• Restart VS Code
• When prompted install language server, choose: v.0.0.11-beta2
• If no prompt can install manually: ⇧⌘P, then type Terraform: Enable/Disable Language Server and install v.0.0.11-beta2

Reference

• Upgrade Guilds
• Useful 0.12 code examples
• Terraform v0.12 - Syntax check broken: Unknown token IDENT
• Fix syntax issues
  • syntax issue while upgrading from 11 to 12
  • map var no longer merge when overriden
  • refering to list var

AWS Savings Plans is launched in November 2019, and allows customers to save up to 72% on Amazon EC2 / AWS Fargate in exchange for making a commitment (on how much they will spend per hour) to a consistent amount of compute usage for a 1 or 3-year term.

Customers can choose how much they wish to commit to (minimum $0.001 per hour) and layer Savings Plans on top of one another.

The major difference between Reserved Instances and AWS Savings Plans are that, rather than committing to a specific instance type in return for a discount, you are committing to a specific spend per hour.

It’s important to note that at this time, AWS doesn’t allow customers to change their Savings Plans contract once purchased or sell unused discounts in the AWS Marketplace. Once customers commit to a Savings Plan price, they are locked in for the one or three years they committed to.

There are two types of AWS Savings Plans:

• EC2 Instance Saving Plans: like Standard RIs

• Compute Savings Plans: shares attributes with Convertible RIs with the added bonus that discounts can be applied to the Fargate container service.

These two types provide the choice between maximising financial benefit and sacrificing flexibility or maximising flexibility while benefiting from a smaller discount.

Here’s a quick comparison of the two types:

EC2 Instance Savings Plans

The main differences between EC2 Instance Saving Plans and Standard RIs is that Savings Plan discounts are applied automatically to any EC2 instance (within the same Family and Region) regardless of the Operating System or Tenancy.

At this stage it’s important to note AWS Savings Plans cannot yet be applied to RDS instances, AWS Redshift, or ElastiCache services. Customers using these services will have to continue using Reserved Instances.

Compute Savings Plans

Commit to and pay for these in the same way as EC2 Instance Savings Plans, but can only get discounts equal to those offered by Convertible Reserved Instances.

The big selling point for Compute Savings Plans is that the discounts are automatically applied to EC2 instances of any Family, Size, AZ, Region, OS or Tenancy, and also apply to Fargate and Lambda usage.

A diagram says more than a thousand words.

How it works

Findings

• There is no direct financial benefit of purchasing AWS Savings Plans to replace fully-optimized and effectively-managed RIs when reservations expire. However, it is anticipated many customers will opt for the new discount program due to its increased flexibility and reduced management overhead.

• Similarly to Reservations, unused discounts do not accumulate. Therefore it’s critical that need to keep track of any Savings Plan waste in order to minimize it in the future.

• In order to calculate whether or not layering a Savings Plan on top of an RI is financially viable, it’s important to have complete visibility of utilisation metrics (or accurately forecast demand) to avoid wasted Savings Plans discounts.

• Every hour, AWS will assess the possible discount programs that usage qualifies for, and will apply them in the following order: Standard Reservation, Convertible Reservation, Savings Plan.

So we can choose:

• either use the recommendations directly from AWS Cost Exploreer (e.g, a commitment of $6.80/hour)

• or as the minimum commitment is $0.001 per hour. This means only have to spend $8.76 per year ($0.001 x 24 hours x 365 days) in order to qualify for a discount - which might be safer? And if that works as expected then can add another AWS Savings Plans on top of it (due to the discount products applied in the following order: RIs -> Savings Plans -> On Demand).

FAQ

Can I continue to purchase EC2 RIs?

Yes. You can continue purchasing RIs to maintain compatibility with your existing cost management processes, and your RIs will work along-side Savings Plans to reduce your overall bill. However as your RIs expire we encourage you to sign up for Savings Plans as they offer the same savings as RIs, but with additional flexibility.

Do I have to choose between the two types of savings plans?

In the same way as customers can purchase both Standard and Convertible RIs, it’s possible to purchase both EC2 Instance Savings Plans and Compute Savings Plans.

What happens if I don’t spend my minimum monetary commitment?

If you don’t meet the minimum monetary commitment in any given hour, that Savings Plan benefit is forfeited. For example, if you commit to $10 per hour, but only consumes $8 of services in any hour, you’ll lose the remaining $2. However, if you’re using linked or consolidated accounts, it’s likely that the unused Savings Plan commitment would have “floated” to benefit another account.

References

• AWS - New Savings Plans for AWS compute services

• AWS Savings Plans - Pricing

• AWS Savings Plans - FAQ

• AWS re:Invent 2019: Dive deep on how to save with AWS Savings Plans

Generally DogStatsD creates a message that contains information about your metric, event, or service check and sends it to a locally installed Agent as a collector. The destination IP address is 127.0.0.1 and the collector port over UDP is 8125.

Based on the official doc of Datadog, here’s the raw datagram format for metrics, events, and service checks that DogStatsD accepts:

1

<METRIC_NAME>:<VALUE>|<TYPE>|@<SAMPLE_RATE>|#<TAG_KEY_1>:<TAG_VALUE_1>,<TAG_2>

So could use nc and socat on indivial host:

e.g.

1
2
3

# use nc

echo -n "kafka.partition_size:123|g|#topic_name:test,partition_number:1,broker_id:28041,hostname:kafka-12345" | nc -w 1 -cu localhost 8125

To speed up the send period, can reduce the -w timeout to 0 for nc command, but sometimes found that nc -w 0 hung, so started to pipe into socat as an alternative:

1
2
3

# use socat

echo "${METRICS_NAME}:${PARTITION_SIZE}|g|#topic_name:${TOPIC_NAME},partition_number:${PARTITION_NUMBER},broker_id:${BROKER_ID},hostname:${HOSTNAME}" | socat -t 0 STDIN UDP:localhost:8125

Code example

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

# function to send metric with puppet tags

function send_dd_metric() {
    METRICS_NAME="$1"
    value="$2"

    HOSTNAME=$(hostname -f)

    # Pool
    if [[ -f /etc/pool ]]; then
        PUPPET_POOL=$(cat /etc/pool)
    else
        PUPPET_POOL='unknown'
    fi

    # Role
    if [[ -f /etc/role ]]; then
        PUPPET_ROLE=$(cat /etc/role)
    else
        PUPPET_ROLE='unknown'
    fi

    # Datacenter
    if [[ -f /etc/dc ]]; then
        PUPPET_DC=$(cat /etc/dc)
    else
        PUPPET_DC='unknown'
    fi

    AMI_ID=$(curl --connect-timeout 2 --max-time 4 --silent http://169.254.169.254/latest/meta-data/ami-id)

    echo "${METRICS_NAME}:${value}|g|host:${HOSTNAME},puppet_pool:${PUPPET_POOL},puppet_role:${PUPPET_ROLE},puppet_dc:${PUPPET_DC},ami_id:${AMI_ID}" | socat -t 0 STDIN UDP:localhost:8125
}

# send metric to datadog 
send_dd_metric "test.service" 1

# remember also to send metric when no data
send_dd_metric "test.service" 0

Check from Datadog

When metric reporting successfully, we can open Datadog console:

and then search for the metric:

After that we can also create Dashboard and Monitors (with Datadog API or existing Github repo barkdog) when necessary.

Reference

• Datadog - sending-metrics

• https://gist.github.com/nstielau/966835

Updated on 01/May/2020

There are other two areas where it is possible to reduce the AWS cost:

1. Considering migrate from Classic Load Balancer to Application Load Balancer (technical debt):

   • ALB is about 10% cheaper than ELB (AWS Load Balancing pricing)
   • Convenient to perform Canary Depoyments in CI / CD

2. Using AWS Savings Plans: please refer to my article about AWS Savings Plans Overview

Rather than write a big, manual-style cost optimization guide, I’d like to share a few pits I’ve encountered during the process.

Tools

Common tools for AWS cost optimization are as follows:

1. AWS Cost Explorer
2. Cost Reports in S3
3. AWS Trusted Advisor - Cost Optimization

Strategies

Based on AWS Cost Optimization Best Practice, the main measures are probably the following aspects:

1. Right Sizing: Use a more appropriate (convenient) Instance Type / Family (for EC2 / RDS)
2. Price models: leverage Reserved Instances (RI) and Spot Instances (SI)
3. Delete / Stop unused resources: e.g. EBS Volume / Snapshot, EC2 / RDS / EIP / ELB, etc.
4. Storage Tier / Backup Policy: Move cold data to cheaper storage tiers like Glacier; Review EBS Snapshot / RDS backup policy
5. Right Tagging: Enforce allocation tagging, while improving Tag coverage and accuracy
6. Scheduling On / Off times: Review existing Auto Scaling policies; Stop instances used in Dev and Prod when not in use and start them again when needed

Corresponding to the specific situation of the company, mainly want to talk about 3 and 4 mentioned above.

About Backup

AWS Backup

At first, the company used Lambda Function (Python script) to handle EBS / RDS backup. Later, because of the release of AWS Backup Service, it was decided to migrate backup management to AWS Backup.

However, for Database backup, because AWS Backup Service does not support Aurora, so we still need to keep the original Lambda.

AWS Backup currently supports all Amazon RDS database engines except Amazon Aurora.

For AWS Backup, it can support multiple AWS Services / Resources, such as:

• Amazon EFS

• DynamoDB

• Amazon EBS Snapshots

• Amazon RDS DB Instances

• AWS Storage Gateway

Unfortunately, AWS Backup does not support the Wildcard (*) to match resources.

For example:

Internally we use Terraform as IaC (Infrastructure as code) to automate the management of resources on AWS.

Examples of corresponding AWS Backup codes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

  resources                       = [
    # RDS
    "arn:aws:rds:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:db:redash",
    # DynamoDB
    "arn:aws:dynamodb:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:table/crm",
    # EBS
    "arn:aws:ec2:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:volume/
vol-054cfe133afc1dc10",
    # EFS
    "arn:aws:elasticfilesystem:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:file-system/${filesystem-id}",
    # StorageGateway
    "arn:aws:storagegateway:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:gateway/${gateway-id}/volume/${volume-id}",
  ]

  tags = [
  {
    type  = "STRINGEQUALS"
    key   = "Environment"
    value = "Production"
  },
  {
    type  = "STRINGEQUALS"
    key   = "Cost Center"
    value = "Platform"
  },
  {
    type  = "STRINGEQUALS"
    key   = "MakeSnapshot"
    value = "True"
  },
  ]

Therefore, the corresponding Backup solution is:

• Either specify the specific Resource ID (use resources to specify, but cannot support something like arn:aws:rds:${region}:${account_id}:db:*)

• Either use Tags to mark the resources to be backed up by AWS Backup Service

Of course, the former is certainly not realistic, which means:

The feature of AWS Backup determins (well, I believe this is not a bug), in fact you can only use Tags, and need to set Resources to empty [] (by default it’s for all Resource types supported by AWS Backup). Thus laying a hidden danger.

Note: The relationship between Tags is OR, so as long as one Tag get matched, AWS Backup will back it up.

The foreplay is over, and the pit is coming!

At the beginning of March, it was found that the AWS bill was obviously abnormal. After analysis, it was found that the cost of EBS snapshots increased too fast.

As you can see in the chart, it started to grow slowly from around January 15th, so what happened at this time?

The internal search was fruitless. After googling, I found a suspicious What’s New message from AWS official website.

Hmmmmmm, perfectly matched…

Okay! Looks like AWS help me to back up all the EC2 instances by default… Thank you!

According to the documentation, for AWS Backup, in addition to EBS Snapshot itself, it will also make an AMI copy for each EC2 instance, so that’s why there are a lot of AMIs.

When backing up an Amazon EC2 instance, AWS Backup takes a snapshot of the root Amazon EBS storage volume, the launch configurations, and all associated EBS volumes. AWS Backup stores certain configuration parameters of the EC2 instance, including instance type, security groups, Amazon VPC, monitoring configuration, and tags. The backup data is stored as an Amazon EBS volume-backed AMI (Amazon Machine Image).

According to the previous AWS Backup execution logic, for all supported types (including newly supported EC2 on January 13), as long as the tag is matched, the backup will be performed.

So, for EC2 AMI / EBS snapshot, how Tags are added by default?

Checked the Terraform code:

1
2
3
4
5
6
7
8
9

resource "aws_autoscaling_group" "main" {
...
  tag {
    key                 = "Name"
    value               = "Value"
    propagate_at_launch = true
  }
...
}

Since propagate_at_launch = true is used, the tags of ASG will be copied when EC2 launches.

In addition, after EC2 is started, the initialization script will be run through Puppet to copy the ASG / EC2 tags to EBS Volumes. Code example as follows:

1
2
3
4
5
6
7
8
9

# Retrieve the tags attached to the instance if no auto-scaling group exists.
# Add all tags retrieved from the auto-scaling group to all EBS volumes attached to the current EC2 instance.

if [ "$asg" != "" ]; then
    aws ec2 create-tags --region "${region_id}" --resources ${volumes} --tags "${tags}"
else
    tags=$(aws ec2 describe-tags --region "${region_id}" --filters "Name=resource-id,Values="${instance_id}"" --query 'Tags[*].{Key:Key,Value:Value}')
    aws ec2 create-tags --region "${region_id}" --resources ${volumes} --tags "${tags}"
fi

So the current Tags replication chain:

ASG-> EC2-> EBS Volume

As for the solution, just to differentiate Tags between EC2 and EBS, and delete the tags that will be matched by AWS Backup (here is MakeSnapshot*), to achieve the purpose of “Backing up EBS only, not EC2 instance”:

1
2
3
4
5
6
7
8

if [ "$asg" != "" ]; then
    aws ec2 create-tags --region "${region_id}" --resources ${volumes} --tags "${tags}"
    aws ec2 delete-tags --resources "${instance_id}" --tags Key=MakeSnapshot Key=MakeSnapshotShortTerm Key=MakeSnapshotLongTerm
else
    tags=$(aws ec2 describe-tags --region "${region_id}" --filters "Name=resource-id,Values="${instance_id}"" --query 'Tags[*].{Key:Key,Value:Value}')
    aws ec2 create-tags --region "${region_id}" --resources ${volumes} --tags "${tags}"
    aws ec2 delete-tags --resources "${instance_id}" --tags Key=MakeSnapshot Key=MakeSnapshotShortTerm Key=MakeSnapshotLongTerm
fi

The effect after cleaning:

After this battle, I completely felt the fear of being dominated by AWS Bills :-(

You can’t plant heels in the same place, so just add monitoring for news from AWS :-)

About DB storage

RDS backup storage excess free allocation

Potential increase to your Amazon Aurora bill [AWS Account: XXXXXXXXX]

Starting March 1, 2020, we will start charging for backup storage in excess of the free allocation. There will not be be any back-dated or retroactive charges for use of Aurora backups before March 1, 2020.

You can review your current backup retention policy and snapshots on the RDS Management Console. You can lower your monthly bill by reducing your backup retention window or by deleting unnecessary snapshots.

Usually, in our SRE team who is on duty will also pay attention to daily BAU such as consultation / email etc., but on that day, the unlucky guy was too busy to ignore the AWS email when received it, and therefore didn’t create a JIRA ticket, but (the interesting thing is) everyone in the team missed it as well, which is pretty strange.

As a result, we didn’t realize until reviewing the AWS Cost, and the price is not cheap actually:

$ 0.095 per additional GB-month of backup storage exceeding free allocation

Currently the company’s default setting of backup_retention_period for RDS is30 days, and this time, of course, we need to reduce it.

The effect is as follows, better than nothing:

Can see a 70% drop for USW2-Aurora:BackupUsage after reducing retention period

About Data Transfer

One of the more interesting things during this period is that, I found there is something called EC2-others in the AWS Cost Report. Here is the official statement:

The EC2-Other category includes multiple service-related usage types, tracking costs associated:

• Amazon EBS volumes and snapshots
• Elastic IP addresses
• NAT gateways
• Data transfer

Through ithe Billing Report, I found that the following is the culprit:

How to locate

Using Cost Explorer, we can locate the specific item with the following settings:

Finally, the major part of Data Transfer is located, which comes from a (business) Load Balancer, followed by the Network Load Balancer (NLB) that for Logging.

How to reduce data transmission costs

A picture is worth a thousand words:

There’re some general strategies, such as:

• Limit data transfer to the same AZ as much as possible, or between AZs or the same Region.

• Use Private IP whenever possible.

For example:

• EC2 that needs to communicate in a development or test environment should be in the same Availability Zone (AZ) to avoid data transfer costs.

• EC2 can use CloudFront (CDN) if it needs to transfer data such as pictures / videos to Public.

• For resources in different Regions or multiple Accounts, can use VPC Peering or VPC Sharing to further reduce transfer costs.

In reality, a multi-pronged strategy has been adopted.

After locating certain LB problems, the method of “directly reducing the transmitted data” was adopted, e.g.:

• Turn off log_subrequest config in OpenResty(Nginx)

  disables logging of subrequests into access_log

• Add filter strategy in Logstash to discard some useless data

• In Rsyslog, enable the compression mode for omfwd module. Generally, a medium compression level should be fine, for example, ZipLevel can be set to 3-5, and compression.mode can be set to single, which means each message will be evaluated and only the larger message will be compressed. And here in order to save costs, a more aggressive strategy is adopted, which will introduce delay but it’s acceptable.

  1 2 3

  ZipLevel = "9" compression.mode = "stream: always" compression.stream.flushOnTXEnd = "off"

• In addition, can also consider adjusting the index.codec configuration of ElasticSearch, using best_compression to replace the default compression algorithm LZ4 to achieve a higher compression ratio, in order to reduce data transfer across AZ (such as shard allocation)

Well, for the time being, it is predicted that Cost Optimization will be a protracted battle, especially in the current environment of global economic impact.

Therefore, we really need to pay more attention to cost optimization, not only from the beginning of architecture / solution design, development and daily maintenance, but also need to create a lean cost-centric culture.

References

• AWS Cost Management

• Cost Optimization-AWS Well-Architected Framework

• AWS Backup: working-with-other-services

• AWS Load Balancing pricing

• AWS Savings Plans - Overview

ssh into any master node (pro-tip: master instances are the ones within the master autoscaling group)

e.g.

1

ssh elastic-master-<instance-id>.<aws-region>.x.y.com

Handy aliases

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

alias tail-es='tail -500f /var/log/elasticsearch/<ES_NAME>/<ES_Name>.log'
alias cat-nodes='curl -s -XGET http://localhost:9200/_cat/nodes?v'
alias cat-shards='curl -s -XGET http://localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason'
alias cat-allocation='curl -s -XGET http://localhost:9200/_cat/allocation?v'
alias cat-indices='curl -s -XGET http://localhost:9200/_cat/indices?v'
alias get-templates='curl -s -XGET http://localhost:9200/_template?pretty'
alias cat-settings='curl -s -XGET http://localhost:9200/_cluster/settings/?pretty'
alias cat-settings-all='curl -s -XGET '\''http://localhost:9200/_cluster/settings?include_defaults=true&pretty'\'' | jq .'
alias cluster-status='curl -s -XGET http://localhost:9200/_cluster/health?pretty'
alias check-status='while true; do sleep 5; cluster-status | grep status; done'
alias disable-allocation='curl -XPUT localhost:9200/_cluster/settings -d '\''{"transient" : \{"cluster.routing.allocation.enable" : "none"}}'\'''
alias enable-allocation='curl -XPUT localhost:9200/_cluster/settings -d '\''{"transient" : \{"cluster.routing.allocation.enable" : "all"}}'\'''
alias cat-shards-unassigned='curl -s -XGET http://localhost:9200/_cat/shards | grep UNASSIGNED | awk '\''{print $1}'\'''
alias del-shards-unassigned='curl -s -XGET http://localhost:9200/_cat/shards | grep UNASSIGNED | awk '\''{print $1}'\'' | xargs -i curl -s -XDELETE "http://localhost:9200/{}"'
alias cat-snapshot='curl -s -XGET localhost:9200/_snapshot/<S3_BUCKET_FOR_SNAPSHOT_REPOSITORY>/_all?pretty | jq .snapshots[-1]'

Cheatsheet

https://github.com/sematext/cheatsheets/blob/master/elasticsearch-devops-cheatsheet.md

Node types

Elasticsearch nodes can take one ore more roles. Here we are using the following node types:

• Master

The master node is responsible for lightweight cluster-wide actions such as creating or deleting an index, tracking which nodes are part of the cluster, and deciding which shards to allocate to which nodes. It is important for cluster health to have a stable master node. Any master-eligible node (all nodes by default) may be elected to become the master node by the master election process.

• Data

Data nodes hold the shards that contain the documents you have indexed. Data nodes handle data related operations like CRUD, search, and aggregations. These operations are I/O-, memory-, and CPU-intensive. It is important to monitor these resources and to add more data nodes if they are overloaded.

Can be subdivided into warm and hot nodes if wanting to Implement a Hot-Warm-Cold Architecture for ES

• Client

Can only route requests, handle the search reduce phase, and distribute bulk indexing. Essentially, coordinating (aka client) only nodes behave as smart load balancers.

Cluster health

To check the overall cluster health (green, yellow or red) and keep polling for status just type cluster-status.

1
2
3
4

elastic-master:~$ check-status
  "status" : "green",
  "status" : "green",
  "status" : "green",

Tailing logs

Elasticsearch logs live in /var/log/elasticsearch/<cluster-name>/<cluster-name>.log. See the alias tail-es above.

Listing allocations

To list the allocations per host, just type cat-allocation.
This will show things like:

• shards per host
• used disk
• available disk
• indice size
• total disk size
• disk usage percent
  • This is an important metric. as a rule of thumb, once disk usage reaches 85% things starts to go wild. ES will not allocate new shards to nodes once they have more than 85% disk used.

1
2
3
4
5
6
7
8
9
10
11

elastic-master:~$ cat-allocation
shards disk.indices disk.used disk.avail disk.total disk.percent host          ip            node
   927      434.5gb   472.8gb    265.2gb    738.1gb           64 172.21.5.57   172.21.5.57   elastic-data-096596c27d5ea41b8-production
   928      440.8gb   479.1gb    258.9gb    738.1gb           64 172.21.22.28  172.21.22.28  elastic-data-015b67ccc6632775b-production
   927      496.4gb   534.7gb    203.3gb    738.1gb           72 172.21.21.209 172.21.21.209 elastic-data-02b5fb2c632eacc9f-production
   927      463.8gb     502gb      236gb    738.1gb           68 172.21.5.138  172.21.5.138  elastic-data-07aa318c82d04e61c-production
   927      436.2gb     475gb      263gb    738.1gb           64 172.21.39.147 172.21.39.147 elastic-data-0a7381067cc43e698-production
   928      457.5gb   495.8gb    242.2gb    738.1gb           67 172.21.5.32   172.21.5.32   elastic-data-05a9cd4cea5328486-production
   927      419.1gb   457.3gb    280.7gb    738.1gb           61 172.21.21.160 172.21.21.160 elastic-data-0abf96b0106bcba98-production
   928      447.2gb     485gb      253gb    738.1gb           65 172.21.38.167 172.21.38.167 elastic-data-0b00da1daa6c3e055-production
   927      472.8gb   510.6gb    227.4gb    738.1gb           69 172.21.39.249 172.21.39.249 elastic-data-09512dff9c057e62b-production

Deleting corrupted / lost shards

Sometimes shards can get corrupted or entirely lost. This will put the cluster in a RED/YELLOW state. If no other node have the unassigned shard and index, then the only option will be to delete them. To do so, run the alias del-shards-unassigned mentioned above or the command below:

1

curl -s -XGET http://localhost:9200/_cat/shards | grep UNASSIGNED | awk {'print $1'} | xargs -i curl -s -XDELETE "http://localhost:9200/{}"

Listing nodes

To list the nodes participating in a cluster, just type cat-nodes. The node with a star (*) is the actual master within that cluster.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

elastic-master:~$ cat-nodes
host          ip            heap.percent ram.percent load node.role master name
172.21.38.167 172.21.38.167           67          99 8.84 d         -      elastic-data-0b00da1daa6c3e055-production
172.21.5.57   172.21.5.57             22          99 4.27 d         -      elastic-data-096596c27d5ea41b8-production
172.21.39.60  172.21.39.60            48          91 0.00 -         m      elastic-master-033de9aa1c6f03b8a-production
172.21.22.28  172.21.22.28            74          99 6.41 d         -      elastic-data-015b67ccc6632775b-production
172.21.39.147 172.21.39.147           21          99 5.07 d         -      elastic-data-0a7381067cc43e698-production
172.21.36.186 172.21.36.186           25          98 0.06 -         -      elastic-client-0bdfaa8f29b51ee5c-production
172.21.6.250  172.21.6.250            27          87 0.02 -         -      elastic-client-0579bf60b8746b365-production
172.21.20.96  172.21.20.96             9          84 0.01 -         -      elastic-client-0a25e84d941f3446f-production
172.21.5.138  172.21.5.138            62          99 6.83 d         -      elastic-data-07aa318c82d04e61c-production
172.21.7.188  172.21.7.188            20          93 0.00 -         *      elastic-master-049a20b4645a557eb-production
172.21.21.160 172.21.21.160           17          99 7.49 d         -      elastic-data-0abf96b0106bcba98-production
172.21.5.32   172.21.5.32             66          99 6.33 d         -      elastic-data-05a9cd4cea5328486-production
172.21.22.253 172.21.22.253           38          85 0.00 -         m      elastic-master-0644394056fa488a8-production
172.21.21.209 172.21.21.209           19          99 5.47 d         -      elastic-data-02b5fb2c632eacc9f-production
172.21.39.249 172.21.39.249           61          99 4.11 d         -      elastic-data-09512dff9c057e62b-production

Stucked nodes

We have come across an issue where one or more nodes get stucked, preventing the master from reallocating shards properly. This has happened multiple times. To find out which nodes are stucked you can either search for the exception ReceiveTimeoutTransportException from logs or search in Kibana if applicable.

Decomissioning Node

Node decomission is something you do when you “empty” a node, a.k.a, force shard reallocation. To do that just use one of the commands bellow, depending whether you want to exclude a node by ip or by name. Using name is the preferred way.

Both attributes, ip or name, can be retrieved from the cat-nodes alias mentioned above. You usually want to do this for one host at a time.

1
2
3
4
5
6
7
8
9
10
11
12
13

# exclude by IP
curl -XPUT localhost:9200/_cluster/settings -d '{
  "transient" :{
      "cluster.routing.allocation.exclude._ip" : "172.19.22.9"
   }
}'

# exclude by NAME
curl -XPUT localhost:9200/_cluster/settings -d '{
  "transient" :{
      "cluster.routing.allocation.exclude._name" : "elastic-data-05a4f739b4b5fc4ab"
   }
}'

More info could be found: ElasticSearch - cluster-level shard allocation-filtering

Rolling restart

1. Disable shard allocation

1
2
3
4
5

curl -XPUT localhost:9200/_cluster/settings -d '{
"transient" : {
"cluster.routing.allocation.enable" : "none"
}
}'

2. Stop node (check the alias command on the host to stop a node)

3. Perform maintenance / upgrade

4. Restart the node, and confirm that it joined the cluster.

5. Re-enable shard allocation as follows:

1
2
3
4
5

curl -XPUT localhost:9200/_cluster/settings -d '{
"transient" : {
"cluster.routing.allocation.enable" : "all"
}
}'

The above steps can also be done from ansible playbook as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103

---
- name: Perform a rolling restart of an Elasticsearch cluster
  gather_facts: no
  hosts: elastic-client-es7:elastic-master-es7:elastic-data-warm-es7:elastic-data-hot-es7
  serial: 1
  become: yes

  vars_prompt:
    - name: "service"
      prompt: "Enter the cluster name (eg. A|B|C)"
      private: no
      default: "unknown"

  vars:
    es_disable_allocation: '{"transient":{"cluster.routing.allocation.enable": "none"}}'
    es_enable_allocation: '{"transient":{"cluster.routing.allocation.enable": "all"}}'
    es_http_port: 9200
    es_transport_port: 9300

  tasks:
    - name: Ensure Elasticsearch is running on node
      service: name=elasticsearch-{{ service }} enabled=yes state=started
      register: response

    - name: Wait for node to come back up if it was stopped
      wait_for: port={{ es_transport_port }} delay=45
      when: response.changed == true

    # the ansible the uri action needs httplib2
    - name: ensure python-httplib2 is installed
      apt: name=python-httplib2 state=present

    - name: Check current version
      uri: url=http://localhost:{{ es_http_port }} method=GET
      register: version_found
      retries: 10
      delay: 10

    - name: Display current Elasticsearch version
      debug: var=version_found.json.version.number

    # enable first - in case the shards were disabled before for some reason.
    # it's important because the next task will wait for the cluster to become green.
    # If the shard allocation is disabled the cluster will stay yellow and the tasks will hang until it times out.
    - name: Enable shard allocation for the {{ service }} cluster
      uri:
        url: http://localhost:{{ es_http_port }}/_cluster/settings
        method: PUT
        body_format: json
        body: "{{ es_enable_allocation }}"

    - name: Wait for cluster health to return to green
      uri:
        url: http://localhost:{{ es_http_port }}/_cluster/health
        method: GET
      register: response
      until: "response.json.status == 'green'"
      retries: 500
      delay: 15

    - name: Disable shard allocation for the {{ service }} cluster
      uri:
        url: http://localhost:{{ es_http_port }}/_cluster/settings
        method: PUT
        body_format: json
        body: "{{ es_disable_allocation }}"

    - name: Restart elasticsearch-{{ service }} on node
      service: name=elasticsearch-{{ service }} enabled=yes state=restarted

    - name: Wait for node to come back up
      wait_for: port={{ es_transport_port }} delay=35

    - name: Wait for http to come back up
      wait_for: port={{ es_http_port }} delay=5

    - name: Wait for cluster health to return to yellow or green
      uri: url=http://localhost:{{ es_http_port }}/_cluster/health method=GET
      register: response
      until: "response.json.status == 'yellow' or response.json.status == 'green'"
      retries: 500
      delay: 15

    - name: Re-enable shard allocation for the {{ service }} cluster
      uri:
        url: http://localhost:{{ es_http_port }}/_cluster/settings
        method: PUT
        body_format: json
        body: "{{ es_enable_allocation }}"
      register: response
      until: "response.json.acknowledged == true"
      retries: 10
      delay: 15

    - name: Wait for the node to recover
      uri:
        url: http://localhost:{{ es_http_port }}/_cat/health
        method: GET
        return_content: yes
      register: response
      until: "'green' in response.content"
      retries: 500
      delay: 30

Then we can run the ansible playbook to perform the rolling restart for Elasticsearch

1
2
3
4

ansible-playbook \
-i inventory/aws/production/a-us-west-2 \
--ask-become-pass \
playbooks/restart_elasticsearch_es7.yml

This blog describes performance testing results of the continuous delivery (CD) workflow. Testing was performed to answer two questions:

1. Which Docker storage engine is optimal for the continuous delivery workflow?
2. Which EC2 instance types perform best in terms of price and performance?

Test Conditions

All testing was performed with the following conditions:

• Ubuntu 15.10
• Docker 1.10.3
• Docker Compose 1.6.2
• All required Docker images cached in a local registry mirror to minimise network variations
• All required Python wheels cached in a local Devpi mirror to minimise network variations
• Empty docker data volume (/var/lib/docker) as starting state for each test

Testing was performed for the internal web application. The test stage that was executed includes the following tasks:

• Pull base image
• Build development image - this includes installing OS packages, building Python wheels and installing Python wheels
• Run unit and integration tests

Docker Storage Engine Comparison

The Docker storage engine comparison was performed on a single EC2 instance with the following configuration:

• c4.large instance
• 2 x 16GB SSD disks attached as instance stores configure in a software RAID0 configuration unless otherwise noted. These disks were used for Docker data storage (i.e. mounted at /var/lib/docker)
• ext4 file system used unless otherwise noted
• 30GB EBS volume attached as main OS volume

Test Scenarios

The following table describes each scenario that was tested.

It should be noted that some storage engines are more complex to setup than others. A low complexity rating means minimal configuration is required to enable the storage engine, whereas a high complexity rating means significant configuration is required to enable the storage engine (e.g. requires separate block storage)

P.S.

The original Docker storage engine, AUFS, was not considered given it is not supported in mainline Linux kernel.

Test Results

The following chart illustrates the results:

Issues

Initial testing was performed using the xfs file system, however this caused a failure during Docker image building when using the Overlay storage driver. This triggered a decision to use ext4 instead as the standard file system.

EC2 Instance Type Comparison

Testing was performed on various EC2 instance types to determine which EC2 instance types perform the best, both in terms of raw performance and price / performance.

The price / performance comparison was calculated by computing the base cost of how much each run (in terms of time taken) cost on each instance type. Additional storage costs were not included in the calculation.

Test Scenarios

The following table describes each scenario that was tested.

Test Results (Performance)

The following chart illustrates the results in terms of performance:

Test Results (Price / Performance)

The following chart illustrates the results in terms of price / performance:

Conclusions

Storage Engine

The overlay storage engine appears to provide the best performance for the tested continuous delivery workflow.

Given the overlay storage engine has low complexity and is easy to configure out of the box, this is the recommended storage engine for continuous delivery workflows.

An interesting observation was the performance of DeviceMapper direct-lvm mode (this is the storage engine configuration for AWS Linux ECS-optimised instances) vs loop mode. In principle, direct-lvm mode should offer better performance but this was not observed.

EC2 Instance Type Price / Performance

The testing clearly discounted m3.medium and r3.large instance types as not suitable for the Continuous Delivery workflow. The m3.medium instance took twice as long to execute, whilst the r3.large performance was comparable to the t2.medium performance at 2.5x the price.

Interestingly, the use of SSD instance stores did not result in any performance improvements and the best performance result was achieved on a c4.large with EBS storage.

The use of dedicated and/or multiple EBS volumes configured in a RAID0 configuration also did not result in any measurable increase in performance.

The t2.medium instance type offers the best value for money of the instance types tested, however offers burst CPU performance that can be exhausted, and the impact of running continuous delivery workflows in terms of CPU credits must be further understood.

The c4.large instance type offers the best performance of the instance types and was second most cost effective. This instance type is tailored towards CPU intensive workloads, offering better peak performance than the t2.medium (8 ECUs vs 6.5 ECUs) and does not incur the risk of exhausting CPU credits. The c4.large also is EBS optimised with 500Mbps dedicated EBS bandwidth and support enhanced networking (t2.medium does not support either of these enhancements).

End-to-End Workflow Testing

Based upon t2.medium and c4.large being identified as the preferred instance types, additional testing of the workflow was conducted for these instance types. All testing was performed using the overlay storage driver.

The end-to-end workflow and results are described in the table below. Note the make tag and make publish steps were excluded as these steps are dependent on network performance to Docker Hub.

Overview

Cluster settings for ES

Could use Kibana Dev Tool to update cluster settings

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

{
  "persistent" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "cluster_concurrent_rebalance" : "12",
          "node_concurrent_recoveries" : "6",
          "disk" : {
            "watermark" : {
              "high" : "85%"
            }
          },
          "enable" : "all"
        }
      }
    },
    "indices" : {
      "recovery" : {
        "max_bytes_per_sec" : "640mb"
      }
    }
  },
  "transient" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "cluster_concurrent_rebalance" : "12",
          "include" : {
            "_name" : "elastic-data-*",
            "_ip" : ""
          },
          "node_concurrent_recoveries" : "6",
          "balance" : {
            "threshold" : "1.0f"
          },
          "enable" : "all"
        }
      }
    },
    "indices" : {
      "recovery" : {
        "max_bytes_per_sec" : "640mb"
      }
    }
  }
}

indices.recovery.max_bytes_per_sec

Defaults to 40mb. Increasing to 320mb increases throughput

1
2
3
4
5
6

curl -X PUT "localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'
{
    "persistent" : {
        "indices.recovery.max_bytes_per_sec" : "320mb"
    }
}'

cluster.routing.allocation.cluster_concurrent_rebalance

By default ES will only move 2 shards at any given time.

To increase recovery speeds and shard rebalancing we should bump this up.

1
2
3
4
5

curl -XPUT localhost:9200/_cluster/settings?pretty -H 'Content-Type: application/json' -d '{
  "persistent" : {
    "cluster.routing.allocation.cluster_concurrent_rebalance": 12
  }
}'

sample policy granting access to a database instance

1
2
3
4
5
6
7
8
9
10
11
12
13
14

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Action": [
             "rds-db:connect"
         ],
         "Resource": [
             "arn:aws:rds-db:us-west-2:123456789012:dbuser:db-12ABC34DEFG5HIJ6KLMNOP78QR/david_lu"
         ]
      }
   ]
}

sample policy granting access to a cluster

1
2
3
4
5
6
7
8
9
10
11
12
13
14

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Action": [
             "rds-db:connect"
         ],
         "Resource": [
             "arn:aws:rds-db:us-west-2:123456789012:dbuser:cluster-CO4FHMOYDKJ7CVBEJS2UWDQX7I/david_lu"
         ]
      }
   ]
}

Resource takes the form arn:aws:rds-db:region:account-id:dbuser:dbi-resource-id/database-user-name

database-user-name is the name of the MySQL database account to associate with IAM authentication. In the example policy, the database account is david_lu.

With IAM database authentication, we don’t need to assign database passwords to the MySQL user accounts we create. Instead, authentication is handled by AWSAuthenticationPlugin—an AWS-provided plugin that works seamlessly with IAM to authenticate our IAM users.

For MySQL

To create a database account for MySQL, connect to the DB instance or DB cluster and issue the CREATE USER statement, as shown in the following example. You can additionally narrow the grants down a bit.

1
2
3

CREATE USER 'iam_sre' IDENTIFIED WITH AWSAuthenticationPlugin as 'RDS';
GRANT SELECT ON sre.* TO 'iam_sre'@'%';
FLUSH PRIVILEGES;

The IDENTIFIED WITH clause allows MySQL to use the AWSAuthenticationPlugin to authenticate the database account (david_lu). The AS ‘RDS’ clause maps the jane_doe database account to the corresponding IAM user or role.

For Postgres

To create a database account for Postgres, connect to the DB instance or DB cluster and issue the CREATE USER / Role statement, as shown in the following example. You can additionally narrow the grants down a bit.

1
2

CREATE ROLE iam_sre WITH LOGIN;
GRANT rds_iam TO iam_sre;

To connect the new created DB instance from CLI:

1
2
3
4
5

export RDSHOST="sre-testing.xxxxxxx.ap-southeast-2.rds.amazonaws.com"
 
export PGPASSWORD="$(aws rds generate-db-auth-token --hostname $RDSHOST --port 5432 --region ap-southeast-2 --username iam_sre)"
 
psql "host=$RDSHOST port=5432 dbname=sre user=iam_sre"

With IAM database authentication, we use an authentication token when we connect to a DB instance or DB cluster. An authentication token is a string of characters that we use instead of a password. Once we generate an authentication token, it’s valid for 15 minutes before it expires. If we try to connect using an expired token, the connection request is denied.

Every authentication token must be accompanied by a valid signature, using AWS signature version 4

We can connect from the command line to an RDS DB instance or Aurora DB cluster with the AWS CLI and mysql command line tool as described following.

References

http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.Enabling.html
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.IAMPolicy.html
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.DBAccounts.html
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.Connecting.html

We need to perform the ES benchmark for the following purposes:

• Experiment with some configurations of ES for tuning

• Compare the performance differences between different versions of ES

• Capacity planning for the ES cluster, e.g.

  1. how many nodes does your online ES cluster need?

  2. What is the configuration of each node?

  3. What is the write performance limit of this cluster?

  4. What about read performance?

Tool

Several tools are available for performance testing, e.g. esperf, elasticsearch-stress-test or JMeter.

Here we’ll use Elastic official tool esrally, which is a macrobenchmarking framework for Elasticsearch.

Usage

Perform testing

1
2
3

# Locally - use "geonames" data, test against existing ES, overwrite ES config for benchmarks, and export result as CSV format

$ esrally --track=geonames --target-hosts=localhost:9204 --pipeline=benchmark-only --challenge=append-no-conflicts --report-format=csv --report-file=~/es_benchmarks/result.csv --user-tag="shards:6" --track-params="cluster_health:'yellow',number_of_shards:6,bulk_size:5000"

List testing

1
2
3
4
5
6
7

$ esrally list races

Race ID                               Race Timestamp    Track     Track Parameters                                            Challenge            Car       User Tags    Track Revision    Team Revision
------------------------------------  ----------------  --------  ----------------------------------------------------------  -------------------  --------  -----------  ----------------  ---------------
31163590-ddbb-40e8-a2a8-e7c2d5fe59da  20200120T195012Z  geonames  bulk_size=5000, cluster_health=yellow, number_of_shards=16  append-no-conflicts  external               ec69176
2da942c3-6b1b-4056-93ba-b617819b4e49  20200120T021551Z  geonames  cluster_health=yellow, number_of_shards=16                  append-no-conflicts  external               ec69176
b7d492eb-37f3-4734-8ab5-e9f23f87341b  20200120T003713Z  geonames  number_of_shards=6                                          append-no-conflicts  external               ec69176

Compare results

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

$ esrally compare --baseline=2da942c3-6b1b-4056-93ba-b617819b4e49 --contender=31163590-ddbb-40e8-a2a8-e7c2d5fe59da

|                                              Metric |         Task |    Baseline |  Contender |     Diff |   Unit |
| --------------------------------------------------: | -----------: | ----------: | ---------: | -------: | -----: |
|          Cumulative indexing time of primary shards |              |     39.8328 |    42.3733 |  2.54053 |    min |
| Cumulative indexing throttle time of primary shards |              |           0 |          0 |        0 |    min |
|             Cumulative merge time of primary shards |              |     9.66415 |    10.4054 |  0.74123 |    min |
|            Cumulative merge count of primary shards |              |          56 |        134 |       78 |        |
|    Cumulative merge throttle time of primary shards |              |   0.0255833 |    0.01175 | -0.01383 |    min |
|           Cumulative refresh time of primary shards |              |     9.38755 |    10.4285 |  1.04098 |    min |
|          Cumulative refresh count of primary shards |              |         905 |       1672 |      767 |        |
|             Cumulative flush time of primary shards |              |     0.01605 | 0.00281667 | -0.01323 |    min |
|            Cumulative flush count of primary shards |              |          21 |         22 |        1 |        |
|                                  Total Young Gen GC |              |      67.529 |     71.123 |    3.594 |      s |
|                                    Total Old Gen GC |              |       6.193 |      7.579 |    1.386 |      s |
|                                          Store size |              |     3.34169 |    3.33626 | -0.00543 |     GB |
|                                       Translog size |              | 9.73232e-07 |  0.0140035 |    0.014 |     GB |
|                              Heap used for segments |              |     4.85796 |    4.75913 | -0.09883 |     MB |
|                            Heap used for doc values |              |    0.164585 |   0.368065 |  0.20348 |     MB |
|                                 Heap used for terms |              |     3.38647 |    3.11659 | -0.26988 |     MB |
|                                 Heap used for norms |              |    0.213257 |    0.17749 | -0.03577 |     MB |
|                                Heap used for points |              |    0.291598 |   0.292653 |  0.00105 |     MB |
|                         Heap used for stored fields |              |    0.802048 |   0.804329 |  0.00228 |     MB |
|                                       Segment count |              |         279 |        256 |      -23 |        |
|                                      Min Throughput | index-append |     21224.1 |    20200.1 | -1023.95 | docs/s |
|                                   Median Throughput | index-append |     22499.3 |    21105.1 | -1394.22 | docs/s |
|                                      Max Throughput | index-append |     22846.2 |    22496.8 | -349.333 | docs/s |
|                             50th percentile latency | index-append |     1625.08 |    1669.35 |  44.2758 |     ms |
|                             90th percentile latency | index-append |     2536.77 |    2698.41 |  161.644 |     ms |
|                             99th percentile latency | index-append |     5182.57 |    6829.08 |  1646.51 |     ms |
|                           99.9th percentile latency | index-append |     20297.2 |    9961.53 | -10335.7 |     ms |
|                            100th percentile latency | index-append |       21013 |    10312.9 | -10700.1 |     ms |
|                                          error rate | index-append |           0 |          0 |        0 |      % |

We can also specify another ES to store the result, and compare / visualize in Kibana.

More usage info could be found here.

Monitoring

Key Metrics

• Throughput: The throughput of each operation, such as index, search, etc.

• Latency: response time for each operation

• Heap used for X

More info for metrics: summary report

General Metrics from Kibana

Plan

• Performance testing for ES7 with different settings - Locally (with Docker)

• Performance testing for ES7 together with logstash filter (sending pre-configured percentage of traffic) - on AWS

Results will be stored in CSV format, or can be setup and compare in Kibana for better visualization.

Testing

The testing benchmark configurations are coming from Tune for indexing speed and Important System Configrations

References

Rally official doc

Performance Tuning Practice at Ebay

Tune for indexing speed

Important System Configrations

Using Rally to benchmark Elasticsearch queries

Prerequisites

• Terraform work complete to create Zookeeper Auto Scaling Group (ASG)

Autoscaling

Full autoscaling Zookeeper (actually adding new nodes to the Zookeeper ensemble automatically) is a bit tricky currently as a full rolling restart of the Zookeeper cluster would have to be orchestrated without ever losing quorum. Zookeeper 3.5 introduced dynamic reconfiguration which would most likely make this significantly easier. There are some other options for autoscaling Zookeeper 3.4 but we’d need to weigh up whether that additional complexity is really worth it.

Puppet

Using the puppet module: deric/puppet-zookeeper

Updating Hiera / Puppet

When a new node starts within the ASG Puppet will run but Zookeeper will not be started automatically. Hiera will need to be updated and Puppet run again for Zookeeper to start.

“zookeeper::servers” - Needs to be set to a hash of the complete list of servers in the Zookeeper ensemble. This is typically set at the “dc” level in the hierarchy, e.g.

1
2
3
4
5
6
7

puppet/hieradata-masterless/dc

cat test-us-west-2.yaml
zookeeper::servers:
  100: zookeeper-0cdf08e0d09dd3530.oregon.test.xyz.org
  101: zookeeper-01ff482cf77636240.oregon.test.xyz.org
  102: zookeeper-0749d84c7dfd4d798.oregon.test.xyz.org

“zookeeper::id” - Needs to be set to the unique ID of the node within the Zookeeper cluster. This is set at the node level (as it’s always specific to a particular node) and needs to match with one (and only one) entry in “zookeeper::servers”. The corresponding node configuration for the above “zookeeper::servers” configuration would be

1
2
3
4
5
6
7
8
9
10

puppet/hieradata-masterless/nodes

cat zookeeper-0cdf08e0d09dd3530.yaml
zookeeper::id: 100
 
cat zookeeper-01ff482cf77636240.yaml
zookeeper::id: 101
 
cat zookeeper-0749d84c7dfd4d798.yaml
zookeeper::id: 102

Migration steps

• Adding additional nodes to expand the existing ensemble. Must be repeated once for each new node
  • Start Zookeeper on one of the replacement nodes. The node should join the ensemble as a follower.
  • Execute a Puppet run on each of the existing nodes one at a time. Restart the nodes in order with the lowest ZKID first but restart the leader last.
  • Ensure the nodes are all in sync
• Once the ensemble has been expanded and is in sync the original Trusty nodes can be removed. Starting with an ensemble of six nodes:
  • Stop Zookeeper on one of the original nodes
  • Update the list of servers on the remaining nodes to remove the server that has been stopped.
  • Rolling restart of the cluster with the nodes in order from lowest to highest ZKID. The leader should be restarted last. At this point the ensemble should consist of five nodes.
  • Stop Zookeeper on the final two original Trusty nodes. The three replacement nodes left on line can still form a quorum and should continue to work as expected.
  • Update the list of servers on the remaining nodes to remove the servers that have been stopped.
  • Rolling restart of the cluster with the nodes in order from lowest to highest ZKID. The leader should be restarted last. At this point the ensemble should consist of three nodes.

Tidy up

once we’re certain that we’re never going to revert to the original nodes

• Ensure replacement nodes are being monitored by Datadog
• Ensure replacement nodes are sending logs to the Elastic lifesupport cluster
• Ensure bash aliases are updated
• Terraform work to decommission original nodes

Let’s say there’s a legacy platform called XXX, and it still poses an operational cost for SRE to manage the monitoring / alerting / upgrade etc.

At present, XXX’s ElasticSearch architecture consists of 3 hosts (m4.xlarge) acting as master, client and data nodes, for XXX’s searching functionality.

All of them are using Ubuntu Trusty (14.04) image, which is no longer supported and need to be migrated to Xenial (16.04). Its usage seems pretty low and cost wise so we would like to migrate the XXX ES to AWS managed ElasticSearch Service to reduce operational overhead.

This doc will cover the project plan / checklist for migration.

Solution overview

Elasticsearch (ES) indexes can be migrated with following steps:

• Create baseline indexes

  • Create a snapshot repository and associate it to an AWS S3 Bucket.
  • Create the first snapshot of the indexes to be migrated, which is a full snapshot on EC2 - The snapshot will be automatically stored in the AWS S3 bucket created in the first step.
  • Restore this full snapshot to the AWS ES.

• Periodic incremental snapshots

  • Repeat serval incremental snapshot and restore.

• Final snapshot and service switchover

  • Stop services which can modify index data.
  • Create a final incremental snapshot of the EC2.
  • Perform service switchover to the AWS ES.

Checklist

Here’s a list of what will we do in detail:

Assess and analyse current data

The ES data is around 30G, setting number_of_replicas to 1, then according to the simplified version of calculation:

1

Source Data * (1 + Number of Replicas) * 1.45 = Minimum Storage Requirement

So the minimum storage for AWS managed ES is about 90G.

1
2
3
4
5
6
7
8
9
10
11

curl $es/_cat/indices?v
health status index              pri rep docs.count docs.deleted store.size pri.store.size
green  open   .marvel-2019.10.12   1   1     115081            0      413mb        206.3mb
green  open   site-1.0           200   1     937559          884      1.6gb        848.7mb
green  open   site_v1            200   1     324638           33    335.8mb        167.9mb
green  open   .marvel-2019.10.14   1   1     115009            0    424.5mb        212.2mb
green  open   .marvel-2019.10.15   1   1     114962            0    424.7mb          212mb
green  open   .marvel-2019.10.16   1   1      14290            0     56.3mb         28.3mb
green  open   .marvel-2019.10.13   1   1     115104            0    410.5mb          205mb
green  open   search             200   1   20829799      6167280       24gb           12gb
green  open   .marvel-kibana       1   1          1            0      6.4kb          3.2kb