freeeの開発情報ポータルサイト

AWS EKSのingress controllerがupdateされてAWS Load Balancer Controllerという名前に変わったので試してみた

この記事は freee Developers Advent Calendar の 14 日目です。

どうも id:renjikariです。
10月にEKSのingress controllerが2.0.0になり名前がAWS Load Balancer Controllerになって登場しました。 ぶっちゃけ私はingressにも、もとのalb-ingress-controllerにも全然詳しくないんですがこの機会に新旧のControllerを構築して試して(遊んで)見ようと思います。
できるだけだれでもできるように目指します。構築は基本的にはAWSの公式に則ります。

Contents

  1. aws-load-balancer-controllerの紹介
  2. aws-alb-ingress-controllerでingressを作ってみる
  3. aws-load-balancer-controllerでingressを作ってみる
  4. 触ってみてわかった差分とその所感

aws-load-balancer-controllerの紹介

aws-load-balancer-controllerの構成画像へのリンクを貼ってみます。これだけだと難しくてよくわからないので、解説のDocも読みました https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/assets/images/controller-design.png

https://aws.amazon.com/jp/blogs/containers/introducing-aws-load-balancer-controller/ 曰く目玉は以下の通り

  • Kubernetesサービス用のネットワークロードバランサー(NLB)
  • 複数のingress ruleでALBを共有することができるように
  • 新しいTargetGroupBinding CRDの導入
  • 完全にプライベートなクラスターのサポート

この記事ではALBを共有する話に触れます。

  • 今まで1ingress = 1ALBだったのが複数のingressをまとめれるようになりました。
  • 実際にどうなるかやってみます。

This project was formerly known as "AWS ALB Ingress Controller", we rebranded it to be "AWS Load Balancer Controller".

また、公式のDocやgithubのREADMEにこんなことが書いてあったのでやる気を感じました。

aws-alb-ingress-controllerでingressを作ってみる

まずはv1系(いままでのやつ)で構築してみます。
※注意 先述したように、https://docs.aws.amazon.com/ja_jp/eks/latest/userguide/alb-ingress.html を参考にしていて、これはaws-load-balancer-controllerの手順なので若干齟齬があり、最小の手順じゃなくなってる可能性が高いです。

  • まずはclusterを作る
~ ❯ eksctl create cluster \
--name sandbox-renjikari-1 \
--version 1.18 \
--region ap-northeast-1 \
--nodegroup-name linux-nodes \
--nodes 3 \
--nodes-min 1 \
--nodes-max 4 \
--managed
  • ALB用のIAMとServiceAccountを作る
# たぶんoidcはいらない(IRSAしてなさそう)
~ ❯ eksctl utils associate-iam-oidc-provider \
    --region ap-northeast-1 \
    --cluster sandbox-renjikari-1 \
    --approve
[ℹ]  eksctl version 0.33.0
[ℹ]  using region ap-northeast-1
[ℹ]  will create IAM Open ID Connect provider for cluster "sandbox-renjikari-1" in "ap-northeast-1"
[✔]  created IAM Open ID Connect provider for cluster "sandbox-renjikari-1" in "ap-northeast-1"

~ ❯ curl -o iam-policy-ingress-controller.json https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/master/docs/examples/iam-policy.json

~ ❯ aws iam create-policy \
    --policy-name LBControllerPolicyRenjikari-ingress \
    --policy-document file://iam-policy-ingress-controller.json

~ ❯ aws iam attach-role-policy --region=ap-northeast-1 --role-name=$NODE_ROLE_NAME --policy-arn=LBControllerPolicyRenjikari-ingress
    
~ ❯ eksctl create iamserviceaccount \
  --cluster=sandbox-renjikari-1 \
  --namespace=kube-system \
  --name=alb-ingress-controller \
  --attach-policy-arn=arn:aws:iam::*******:policy/LBControllerPolicyRenjikari-ingress \
  --override-existing-serviceaccounts \
  --approve
~ ❯ curl -sS "https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.0.0/docs/examples/rbac-role.yaml" > rbac-role.yaml
k apply -f rbac-role.yaml
  • ingress controllerのdeploy
    • cluster nameとserviceAccountName/serviceAccountの項目を修正
~ ❯ curl -sS "https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.0.0/docs/examples/alb-ingress-controller.yaml" > alb-ingress-controller.yaml

kubectl apply -f alb-ingress-controller.yaml
  • 作られたリソースの確認
~ ❯ k get sa -n kube-system |grep ingress
alb-ingress-controller               1         3d10h
~ ❯ 
~ ❯  k describe sa alb-ingress-controller -n kube-system
Name:                alb-ingress-controller
Namespace:           kube-system
Labels:              <none>
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::**********:role/eksctl-sandbox-renjikari-1-addon-iamservicea-Role1-1IKRVPID7SU75
Image pull secrets:  <none>
Mountable secrets:   alb-ingress-controller-token-wkwd2
Tokens:              alb-ingress-controller-token-wkwd2
Events:              <none>

~ ❯ k get pod -n kube-system
NAME                                      READY   STATUS    RESTARTS   AGE
alb-ingress-controller-55bd445656-jgk45   1/1     Running   0          7m51s
  • アプリのデプロイ
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/examples/2048/2048_full.yaml

作られたLBのDNS経由でアクセスすると2048が遊べた

  • http://*****-game2048-ingress2-*****-********.ap-northeast-1.elb.amazonaws.com/ 2048のゲーム画面の画像

  • logを確認しつつ作られたリソースの確認

    • logからもv1.0.0である主張がめっちゃある
    • LBを作成
    • TargetGroupを作成
    • LBのtagの変更(というか付与)
    • (LBの)Listerの作成
    • Listener Ruleの作成
    • LB用のSGを作ってrule変えてLBに付与
    • Instance(K8s Node)用のSGを作ってrule変えてnodeに付与
-------------------------------------------------------------------------------
AWS ALB Ingress controller
  Release:    v1.0.0
  Build:      git-c25bc6c5
  Repository: https://github.com/kubernetes-sigs/aws-alb-ingress-controller
-------------------------------------------------------------------------------

I1210 21:18:58.566632       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "Controller"="alb-ingress-controller" "Source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"loadBalancer":{}}}}
I1210 21:18:58.566859       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "Controller"="alb-ingress-controller" "Source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"loadBalancer":{}}}}
I1210 21:18:58.566984       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "Controller"="alb-ingress-controller" "Source"={"Type":{"metadata":{"creationTimestamp":null}}}
I1210 21:18:58.567219       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "Controller"="alb-ingress-controller" "Source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"daemonEndpoints":{"kubeletEndpoint":{"Port":0}},"nodeInfo":{"machineID":"","systemUUID":"","bootID":"","kernelVersion":"","osImage":"","containerRuntimeVersion":"","kubeletVersion":"","kubeProxyVersion":"","operatingSystem":"","architecture":""}}}}
I1210 21:18:58.567448       1 leaderelection.go:185] attempting to acquire leader lease  kube-system/ingress-controller-leader-alb...
I1210 21:50:20.000945       1 :0] kubebuilder/controller "level"=0 "msg"="Starting Controller"  "Controller"="alb-ingress-controller"
I1210 21:50:20.101096       1 :0] kubebuilder/controller "level"=0 "msg"="Starting workers"  "Controller"="alb-ingress-controller" "WorkerCount"=1

I1210 22:12:18.387156       1 loadbalancer.go:185] game-2048/ingress-2048: creating LoadBalancer ********-game2048-ingress2-3738
I1210 22:12:19.442761       1 loadbalancer.go:201] game-2048/ingress-2048: LoadBalancer ********-game2048-ingress2-3738 created, ARN: arn:aws:elasticloadbalancing:ap-northeast-1:**********:loadbalancer/app/********-game2048-ingress2-3738/************
I1210 22:12:19.534395       1 targetgroup.go:108] game-2048/ingress-2048: creating target group ********-************
I1210 22:12:19.710479       1 targetgroup.go:127] game-2048/ingress-2048: target group ********-************ created: arn:aws:elasticloadbalancing:ap-northeast-1:**********:targetgroup/********-************/************
I1210 22:12:19.733015       1 tags.go:43] game-2048/ingress-2048: modifying tags {  kubernetes.io/cluster/sandbox-renjikari-1: "owned",  kubernetes.io/namespace: "game-2048",  kubernetes.io/ingress-name: "ingress-2048",  kubernetes.io/service-name: "service-2048",  kubernetes.io/service-port: "80"} on arn:aws:elasticloadbalancing:ap-northeast-1:**********:targetgroup/********-************/************
I1210 22:12:19.785763       1 targets.go:73] game-2048/ingress-2048: Adding targets to arn:aws:elasticloadbalancing:ap-northeast-1:**********:targetgroup/********-************/************: 192.168.35.42:80, 192.168.5.138:80, 192.168.58.15:80, 192.168.76.112:80, 192.168.88.182:80
I1210 22:12:20.045637       1 listener.go:83] game-2048/ingress-2048: creating listener 80
I1210 22:12:20.105254       1 rules.go:59] game-2048/ingress-2048: creating rule 1 on arn:aws:elasticloadbalancing:ap-northeast-1:**********:listener/app/********-game2048-ingress2-3738/************/97bba4db08bf92a9
I1210 22:12:20.154569       1 rules.go:76] game-2048/ingress-2048: rule 1 created with conditions [{    Field: "path-pattern",    Values: ["/*"]  }]

I1210 22:12:20.243404       1 association.go:224] game-2048/ingress-2048: creating securityGroup ********-game2048-ingress2-3738:managed LoadBalancer securityGroup by ALB Ingress Controller
I1210 22:12:20.368442       1 tags.go:69] game-2048/ingress-2048: modifying tags {  kubernetes.io/ingress-name: "ingress-2048",  kubernetes.io/cluster/sandbox-renjikari-1: "owned",  kubernetes.io/namespace: "game-2048"} on sg-************
I1210 22:12:20.517525       1 security_group.go:50] game-2048/ingress-2048: granting inbound permissions to securityGroup sg-************: [{    FromPort: 80,    IpProtocol: "tcp",    IpRanges: [{        CidrIp: "0.0.0.0/0",        Description: "Allow ingress on port 80 from 0.0.0.0/0"      }],    ToPort: 80  }]
I1210 22:12:20.792115       1 lb_attachment.go:30] game-2048/ingress-2048: modify securityGroup on LoadBalancer arn:aws:elasticloadbalancing:ap-northeast-1:**********:loadbalancer/app/********-game2048-ingress2-3738/************ to be [sg-************]

I1210 22:12:21.207053       1 association.go:224] game-2048/ingress-2048: creating securityGroup instance-********-game2048-ingress2-3738:managed instance securityGroup by ALB Ingress Controller
I1210 22:12:21.372176       1 tags.go:69] game-2048/ingress-2048: modifying tags {  kubernetes.io/cluster/sandbox-renjikari-1: "owned",  kubernetes.io/namespace: "game-2048",  kubernetes.io/ingress-name: "ingress-2048"} on sg-************
I1210 22:12:21.498486       1 security_group.go:50] game-2048/ingress-2048: granting inbound permissions to securityGroup sg-************: [{    FromPort: 0,    IpProtocol: "tcp",    ToPort: 65535,    UserIdGroupPairs: [{        GroupId: "sg-************"      }]  }]
I1210 22:12:21.955031       1 instance_attachment.go:87] game-2048/ingress-2048: attaching securityGroup sg-************ to ENI eni-*****************
I1210 22:12:22.425565       1 instance_attachment.go:87] game-2048/ingress-2048: attaching securityGroup sg-************ to ENI eni-*****************
I1210 22:12:22.960393       1 instance_attachment.go:87] game-2048/ingress-2048: attaching securityGroup sg-************ to ENI eni-*****************

aws-load-balancer-controllerでingressを作ってみる

次に、新しく出たaws-load-balancer-controllerで構築してみます。 わかりやすさ重視のため別クラスタを立てています。構築については差分だけ書きます。

  • ALB用のIAMとServiceAccountを作る
~ ❯ eksctl utils associate-iam-oidc-provider \
    --region ap-northeast-1 \
    --cluster sandbox-renjikari-2 \
    --approve
[ℹ]  eksctl version 0.33.0
[ℹ]  using region ap-northeast-1
[ℹ]  will create IAM Open ID Connect provider for cluster "sandbox-renjikari-1" in "ap-northeast-1"
[✔]  created IAM Open ID Connect provider for cluster "sandbox-renjikari-1" in "ap-northeast-1"

~ ❯ curl -o iam-policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/install/iam_policy.json

~ ❯ aws iam create-policy \
    --policy-name LBControllerPolicyRenjikari \
    --policy-document file://iam-policy-ingress-controller.json

~ ❯ eksctl create iamserviceaccount \
  --cluster=sandbox-renjikari-2 \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --attach-policy-arn=arn:aws:iam::**********:policy/LBControllerPolicyRenjikari \
  --override-existing-serviceaccounts \
  --approve
  • ingress controllerのdeploy
    • helmでいれられるらしいのでhelmで
~ ❯ kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller//crds?ref=master"
~ ❯ helm3 repo add eks https://aws.github.io/eks-charts
~ ❯ helm3 upgrade -i aws-load-balancer-controller eks/aws-load-balancer-controller \
  --set clusterName=sandbox-renjikari-2 \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  -n kube-system
  
~ ❯ helm3 ls -n kube-system


NAME                            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                                   APP VERSION
~ ❯ aws-load-balancer-controller    kube-system     3               2020-12-11 08:18:31.666122 +0900 JST    deployed        aws-load-balancer-controller-1.1.0      v2.1.0
  • サンプルアプリケーションのデプロイ
~ ❯ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/examples/2048/2048_full.yaml
  • log
    • フォーマットがv1系と違っている
    • やっていることの流れはほぼ同じっぽいが、targetgroupにIPをattachするところはTargetGroupBindigs経由でやっているのが相違点っぽい
{"level":"info","ts":1607642923.4134078,"msg":"version","GitVersion":"v2.1.0","GitCommit":"f95827224a66cae20a1af999d8ef1d46ec462547","BuildDate":"2020-12-01T19:27:50+0000"}
{"level":"info","ts":1607642923.4581616,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1607642923.4603312,"logger":"setup","msg":"adding health check for controller"}
{"level":"info","ts":1607642923.4604735,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/mutate-v1-pod"}
{"level":"info","ts":1607642923.4605618,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/mutate-elbv2-k8s-aws-v1beta1-targetgroupbinding"}
{"level":"info","ts":1607642923.460624,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validate-elbv2-k8s-aws-v1beta1-targetgroupbinding"}
{"level":"info","ts":1607642923.460945,"logger":"setup","msg":"starting podInfo repo"}
I1210 23:28:45.461095       1 leaderelection.go:242] attempting to acquire leader lease  kube-system/aws-load-balancer-controller-leader...
{"level":"info","ts":1607642925.4611657,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
I1210 23:28:45.480647       1 leaderelection.go:252] successfully acquired lease kube-system/aws-load-balancer-controller-leader
{"level":"info","ts":1607642925.5614917,"logger":"controller-runtime.webhook.webhooks","msg":"starting webhook server"}
{"level":"info","ts":1607642925.5615087,"logger":"controller","msg":"Starting EventSource","controller":"service","source":"kind source: /, Kind="}
{"level":"info","ts":1607642925.5615797,"logger":"controller","msg":"Starting Controller","controller":"service"}
{"level":"info","ts":1607642925.5615995,"logger":"controller","msg":"Starting workers","controller":"service","worker count":3}
{"level":"info","ts":1607642925.5616424,"logger":"controller","msg":"Starting EventSource","controller":"ingress","source":"channel source: 0xc0002eaa00"}
{"level":"info","ts":1607642925.561678,"logger":"controller","msg":"Starting EventSource","controller":"ingress","source":"channel source: 0xc0002eaa50"}
{"level":"info","ts":1607642925.5616996,"logger":"controller","msg":"Starting EventSource","controller":"ingress","source":"kind source: /, Kind="}
{"level":"info","ts":1607642925.5617182,"logger":"controller","msg":"Starting EventSource","controller":"ingress","source":"kind source: /, Kind="}
{"level":"info","ts":1607642925.561738,"logger":"controller","msg":"Starting EventSource","controller":"ingress","source":"kind source: /, Kind="}
{"level":"info","ts":1607642925.5614593,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"elbv2.k8s.aws","reconcilerKind":"TargetGroupBinding","controller":"targetGroupBinding","source":"kind source: /, Kind="}
{"level":"info","ts":1607642925.5621867,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"elbv2.k8s.aws","reconcilerKind":"TargetGroupBinding","controller":"targetGroupBinding","source":"kind source: /, Kind="}
{"level":"info","ts":1607642925.5622096,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"elbv2.k8s.aws","reconcilerKind":"TargetGroupBinding","controller":"targetGroupBinding","source":"kind source: /, Kind="}
{"level":"info","ts":1607642925.5628521,"logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":1607642925.5629292,"logger":"controller-runtime.webhook","msg":"serving webhook server","host":"","port":9443}
{"level":"info","ts":1607642925.5630434,"logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"}
{"level":"info","ts":1607642925.661992,"logger":"controller","msg":"Starting Controller","controller":"ingress"}
{"level":"info","ts":1607642925.6621237,"logger":"controller","msg":"Starting workers","controller":"ingress","worker count":3}
-- ここまで立ち上がっときのlogっぽい
{"level":"info","ts":1607642925.6624649,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"elbv2.k8s.aws","reconcilerKind":"TargetGroupBinding","controller":"targetGroupBinding","source":"kind source: /, Kind="}
{"level":"info","ts":1607642925.7627265,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"elbv2.k8s.aws","reconcilerKind":"TargetGroupBinding","controller":"targetGroupBinding"}
{"level":"info","ts":1607642925.7627664,"logger":"controller","msg":"Starting workers","reconcilerGroup":"elbv2.k8s.aws","reconcilerKind":"TargetGroupBinding","controller":"targetGroupBinding","worker count":3}

{"level":"info","ts":1607643258.6043127,"logger":"controllers.ingress","msg":"successfully built model","model":"{\"id\":\"game-2048/ingress-2048\",\"resources\":{\"AWS::EC2::SecurityGroup\":{\"ManagedLBSecurityGroup\":{\"spec\":{\"groupName\":\"k8s-game2048-ingress2-********\",\"description\":\"[k8s] Managed SecurityGroup for LoadBalancer\",\"ingress\":[{\"ipProtocol\":\"tcp\",\"fromPort\":80,\"toPort\":80,\"ipRanges\":[{\"cidrIP\":\"0.0.0.0/0\"}]}]}}},\"AWS::ElasticLoadBalancingV2::Listener\":{\"80\":{\"spec\":{\"loadBalancerARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::LoadBalancer/LoadBalancer/status/loadBalancerARN\"},\"port\":80,\"protocol\":\"HTTP\",\"defaultActions\":[{\"type\":\"fixed-response\",\"fixedResponseConfig\":{\"contentType\":\"text/plain\",\"statusCode\":\"404\"}}]}}},\"AWS::ElasticLoadBalancingV2::ListenerRule\":{\"80:1\":{\"spec\":{\"listenerARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::Listener/80/status/listenerARN\"},\"priority\":1,\"actions\":[{\"type\":\"forward\",\"forwardConfig\":{\"targetGroups\":[{\"targetGroupARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/game-2048/ingress-2048-service-2048:80/status/targetGroupARN\"}}]}}],\"conditions\":[{\"field\":\"path-pattern\",\"pathPatternConfig\":{\"values\":[\"/*\"]}}]}}},\"AWS::ElasticLoadBalancingV2::LoadBalancer\":{\"LoadBalancer\":{\"spec\":{\"name\":\"k8s-game2048-ingress2-********\",\"type\":\"application\",\"scheme\":\"internet-facing\",\"ipAddressType\":\"ipv4\",\"subnetMapping\":[{\"subnetID\":\"subnet-0bedd9f3cc8fdd18c\"},{\"subnetID\":\"subnet-0242ee825e50cccc5\"},{\"subnetID\":\"subnet-********\"}],\"securityGroups\":[{\"$ref\":\"#/resources/AWS::EC2::SecurityGroup/ManagedLBSecurityGroup/status/groupID\"}]}}},\"AWS::ElasticLoadBalancingV2::TargetGroup\":{\"game-2048/ingress-2048-service-2048:80\":{\"spec\":{\"name\":\"k8s-game2048-service2-********\",\"targetType\":\"ip\",\"port\":80,\"protocol\":\"HTTP\",\"protocolVersion\":\"HTTP1\",\"healthCheckConfig\":{\"port\":\"traffic-port\",\"protocol\":\"HTTP\",\"path\":\"/\",\"matcher\":{\"httpCode\":\"200\"},\"intervalSeconds\":15,\"timeoutSeconds\":5,\"healthyThresholdCount\":2,\"unhealthyThresholdCount\":2}}}},\"K8S::ElasticLoadBalancingV2::TargetGroupBinding\":{\"game-2048/ingress-2048-service-2048:80\":{\"spec\":{\"template\":{\"metadata\":{\"name\":\"k8s-game2048-service2-********\",\"namespace\":\"game-2048\",\"creationTimestamp\":null},\"spec\":{\"targetGroupARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/game-2048/ingress-2048-service-2048:80/status/targetGroupARN\"},\"targetType\":\"ip\",\"serviceRef\":{\"name\":\"service-2048\",\"port\":80},\"networking\":{\"ingress\":[{\"from\":[{\"securityGroup\":{\"groupID\":{\"$ref\":\"#/resources/AWS::EC2::SecurityGroup/ManagedLBSecurityGroup/status/groupID\"}}}],\"ports\":[{\"protocol\":\"TCP\"}]}]}}}}}}}}"}

{"level":"info","ts":1607643259.5002263,"logger":"controllers.ingress","msg":"creating securityGroup","resourceID":"ManagedLBSecurityGroup"}
{"level":"info","ts":1607643259.7226052,"logger":"controllers.ingress","msg":"created securityGroup","resourceID":"ManagedLBSecurityGroup","securityGroupID":"sg-********"}
{"level":"info","ts":1607643259.9485986,"msg":"authorizing securityGroup ingress","securityGroupID":"sg-********","permission":[{"FromPort":80,"IpProtocol":"tcp","IpRanges":[{"CidrIp":"0.0.0.0/0","Description":""}],"Ipv6Ranges":null,"PrefixListIds":null,"ToPort":80,"UserIdGroupPairs":null}]}
{"level":"info","ts":1607643260.110359,"msg":"authorized securityGroup ingress","securityGroupID":"sg-********"}
{"level":"info","ts":1607643264.718656,"logger":"controllers.ingress","msg":"creating targetGroup","stackID":"game-2048/ingress-2048","resourceID":"game-2048/ingress-2048-service-2048:80"}
{"level":"info","ts":1607643265.120149,"logger":"controllers.ingress","msg":"created targetGroup","stackID":"game-2048/ingress-2048","resourceID":"game-2048/ingress-2048-service-2048:80","arn":"arn:aws:elasticloadbalancing:ap-northeast-1:**********:targetgroup/k8s-game2048-service2-********/********"}
{"level":"info","ts":1607643266.3804967,"logger":"controllers.ingress","msg":"creating loadBalancer","stackID":"game-2048/ingress-2048","resourceID":"LoadBalancer"}
{"level":"info","ts":1607643267.251105,"logger":"controllers.ingress","msg":"created loadBalancer","stackID":"game-2048/ingress-2048","resourceID":"LoadBalancer","arn":"arn:aws:elasticloadbalancing:ap-northeast-1:**********:loadbalancer/app/k8s-game2048-ingress2-********/********"}
{"level":"info","ts":1607643267.2861407,"logger":"controllers.ingress","msg":"creating listener","stackID":"game-2048/ingress-2048","resourceID":"80"}
{"level":"info","ts":1607643267.3420944,"logger":"controllers.ingress","msg":"created listener","stackID":"game-2048/ingress-2048","resourceID":"80","arn":"arn:aws:elasticloadbalancing:ap-northeast-1:**********:listener/app/k8s-game2048-ingress2-********/********/********"}
{"level":"info","ts":1607643267.3545017,"logger":"controllers.ingress","msg":"creating listener rule","stackID":"game-2048/ingress-2048","resourceID":"80:1"}
{"level":"info","ts":1607643267.4122574,"logger":"controllers.ingress","msg":"created listener rule","stackID":"game-2048/ingress-2048","resourceID":"80:1","arn":"arn:aws:elasticloadbalancing:ap-northeast-1:**********:listener-rule/app/k8s-game2048-ingress2-********/********/********/********"}
{"level":"info","ts":1607643267.4123323,"logger":"controllers.ingress","msg":"creating targetGroupBinding","stackID":"game-2048/ingress-2048","resourceID":"game-2048/ingress-2048-service-2048:80"}
{"level":"info","ts":1607643267.4465287,"logger":"controllers.ingress","msg":"created targetGroupBinding","stackID":"game-2048/ingress-2048","resourceID":"game-2048/ingress-2048-service-2048:80","targetGroupBinding":{"namespace":"game-2048","name":"k8s-game2048-service2-********"}}
{"level":"info","ts":1607643267.5947661,"logger":"controllers.ingress","msg":"successfully deployed model","ingressGroup":"game-2048/ingress-2048"}
{"level":"info","ts":1607643267.7672768,"msg":"authorizing securityGroup ingress","securityGroupID":"sg-********","permission":[{"FromPort":0,"IpProtocol":"tcp","IpRanges":null,"Ipv6Ranges":null,"PrefixListIds":null,"ToPort":65535,"UserIdGroupPairs":[{"Description":"elbv2.k8s.aws/targetGroupBinding=shared","GroupId":"sg-********","GroupName":null,"PeeringStatus":null,"UserId":null,"VpcId":null,"VpcPeeringConnectionId":null}]}]}
{"level":"info","ts":1607643267.9896986,"msg":"authorized securityGroup ingress","securityGroupID":"sg-********"}
{"level":"info","ts":1607643268.2313712,"msg":"registering targets","arn":"arn:aws:elasticloadbalancing:ap-northeast-1:**********:targetgroup/k8s-game2048-service2-********/********","targets":[{"AvailabilityZone":null,"Id":"192.168.19.30","Port":80},{"AvailabilityZone":null,"Id":"192.168.4.120","Port":80},{"AvailabilityZone":null,"Id":"192.168.66.91","Port":80},{"AvailabilityZone":null,"Id":"192.168.9.200","Port":80},{"AvailabilityZone":null,"Id":"192.168.90.202","Port":80}]}
{"level":"info","ts":1607643268.5573752,"msg":"registered targets","arn":"arn:aws:elasticloadbalancing:ap-northeast-1:**********:targetgroup/k8s-game2048-service2-********/********"}

2つ目のアプリをデプロイ

~ ❯ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.0.0/docs/examples/echoservice/echoserver-namespace.yaml &&\
~ ❯ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.0.0/docs/examples/echoservice/echoserver-service.yaml &&\
~ ❯ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.0.0/docs/examples/echoservice/echoserver-deployment.yaml

# docには変更してapplyしろって書いてあったけどそのままで動いた
~ ❯ wget https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.0.0/docs/examples/echoservice/echoserver-ingress.yaml
~ ❯ kubectl apply -f echoserver-ingress.yaml

# 疎通確認(Host指定しないとListenerがうまくいかんので超注意(1時間以上はまった))
curl -H "Host:echoserver.example.com" ${LBのDNS}
  • aws-load-balancer-controllerの新機能の一つである1LBで複数サービスを共有する機能を試す
    • 2つのingressリソースにおもむろに以下のannotationを付け足す
metadata:
  annotations:
    alb.ingress.kubernetes.io/group.name: test-group
  • これだけでもともとのechoserverとgame-2048のLBが削除され、test-groupという名前のLBが生成されて、疎通できるようになった
    • なにこれすごい

触ってみた差分とその所感

  • 構築時
    • aws-load-balancer-controllerの中で必要な権限はIRSAを利用しているので、Node Roleに直接policyをattachする必要がなくなっている
    • helmでinstallできるようになっている
    • updateの記事で、kubectlとawsコマンドを行き来する回数が減っている、みたいなことが書いてあって構築時もほとんどeksctlとhelmで完結している
      • aws iam create-policy が唯一の残りっぽさ
  • アプリのデプロイ
    • user操作は変わっていない
    • logの出力がjsonになった
    • 結果的に少しわかりづらくはなった気がする
  • 全体的な感想
    • aws-load-balancer-controller、かなり良さそうに感じたので使っていきたい気持ちになった

明日の12月15日の記事はQAチームの(@mishizuka99)さんの担当です!