Tungsten Fabric Primer
最近 Tungsten Fabric Primer という文書を書いている。
https://github.com/tnaganawa/tungstenfabric-docs/blob/master/TungstenFabricPrimer.md
インストール、状態確認、設定変更、アップデート、監視ツールとの連携、等、Tungsten Fabric の構築・運用に必要な情報が一通りまとまっているはず (多分)
JNCIA-DevOps, JNCIS-DevOps 取得記
先日、JNCIA-DevOps, JNCIS-DevOps を取得したので、その時のメモとなる。
https://www.juniper.net/jp/jp/training/certification/certification-tracks/devops?tab=jnciadevops
https://www.juniper.net/jp/jp/training/certification/certification-tracks/devops?tab=jncis-devops
こちらの資格は、 Juniper Networks から提供されている DevOps Track の資格で、Junos に特化した Automation の試験となっている。
基本的には、 operational command, configuration command をon-box (SLAX, Python など), off-box (PyEZ, ansible など) で実行するための仕組みが対象となっているが、それ以外にも YANG によるコンフィグ拡張、スクリプト使用時に必要な Junos コンフィグ、等もかなり多く出題されるので、Junos の試験だと思った方が実態に近いかもしれない。
出題範囲は、サイトに記載されている通りで、JNCIA-DevOps / JNCIS-DevOps 共に、Python, ansible 等が出題の中心となり、JNCIA/JNCISで、出題範囲そのものはあまり変わらない印象だった。
ただ、 JNCIS の方では、実際のスクリプトが問題内で提供され、出力内容・エラー内容を選択する、など、実機である程度使っていないと答えられない問題が増えており、より実機経験が求められる試験になっているようである。
内容としては、XML, JSON, YAML, Python, ansible など、一般的な話題も多いため、これらの事前経験があれば、入りやすいかもしれない。
ただ、Junos 特化の内容も多いため、ある程度実機で試してから受ける方が、より確実と思われる。(vMX, vSRX 等でも試せる範囲)
※ JNCIA では Junos の REST API explorer の設定方法、JNCIS では SLAX 全般、op / commit / event / snmp script のコンフィグ方法、等が範囲に含まれる
これ以外にも、 JNCIS では Ruby, RubyEZ, JET, JSNAPy 等も範囲になるので、マニュアルを一読しつつ、可能であれば、一度は実機で試しておいた方がよいかもしれない。
公式トレーニングもあるので、可能であれば、受講するのが確実と思われる。ただ、 JNCIS-Cloud 等と比べると、比較的、自習はしやすい印象を受けた。
こちらも、自習する場合、Junos Genius で、模擬問題が手に入るので、事前に目を通しておくことをおすすめする。
※ JNCIA-DevOps については、無料で取得できるが、JNCIS-DevOps については、有料、かつ期間制限ありとなるので、注意
https://www.juniper.net/jp/jp/training/junos-genius/
追記:
- 比較的多かった出題として、 NETCONF の transport, messages, operations, content の違い、があったので、チェックしていくとよいかもしれない
- mgd, jsd の用途、も多かった
JNCIA-Cloud, JNCIS-Cloud 取得記
先日、JNCIA-Cloud, JNCIS-Cloud を取得したので、その時のメモとなる。
https://www.juniper.net/jp/jp/training/certification/certification-tracks/cloud-track?tab=jncia-cloud
https://www.juniper.net/jp/jp/training/certification/certification-tracks/cloud-track?tab=jncis-cloud
こちらの資格は、 Juniper Networks から提供されている Cloud Track の資格で、大まかには Contrail Series を中心とした、 SDN の話題を扱っている。
※ ただし、(特に JNCIA-Cloudでは) Northstar 等、 MPLS-Core 用 SDN の問題も出るので、範囲そのものはかなり広い
基本的に出題範囲は、サイトに記載されている通りで、JNCIA-Cloud では、Contrail Networking, Openstack (nova とは何か? 等), NorthStar, Contrail Service Orchestration 等が出題される。少し意外だったのだが、これ以外に vSRX, Security Director, SkyATP, vMX 等も範囲になっており、問題もそれなりの数が出題されてくるので、このあたりもある程度調べていった方がよいかもしれない。
※ 選択式なので、難易度そのものはさほど高くないが、知らないと回答できない問題も、一定数は出てくる
JNCIS-Cloud は、ほぼ完全に Contrail Networking の試験となっており、SDN とは何か?, この場合のopenstack構成方法は?, サービスチェインの設定方法?, AnalyzerVM の設定方法?, vRouter の動作? 等、Contrail Networking の動作・設定方法について、かなり細かい出題が行われる。こちらも選択式だが、ある程度実機で確認を行ってから受験するか、可能であれば、公式トレーニングを受けてからの方がよいかもしれない。
自習する場合、Junos Genius で、模擬問題が手に入る。
※ JNCIA-Cloud については、無料で取得できるが、JNCIS-Cloud については、有料、かつ期間制限ありとなるので、注意
https://www.juniper.net/jp/jp/training/junos-genius/
juju charm による openstack / Tungsten Fabric インストール
以下の記述に従い、juju charm による openstack / Tungsten Fabric インストールを試してみている。
https://github.com/Juniper/contrail-charms/blob/R5/README.md
https://github.com/Juniper/contrail-charms/blob/R5/manual-deploy.md
環境としては、ubuntu xenial (AMI-ID: ami-06c43a7df16e8213c) 4台 (juju node, openstack controller, Tungsten Fabric Controller, openstack compute) を使用した。
※ 4vCPU, 15GB mem, 60 GB disk
openstack のバージョンとしては、ocata を使用した。
※ queens 等も試してみたのだが、なぜか Tungsten Fabric からの keystone v3 接続がうまくいかなかったため、今回は ocata で試してみている
実行したコマンドは以下となる。
(juju node 上で実施)
# apt-get update
# apt-get install juju
# juju add-cloud
Select cloud type: manual
Enter a name for your manual cloud: manual-cloud-1
Enter the controller's hostname or IP address: (juju node の ip を記載)
# ssh-keygen
# cd .ssh
# cat id_rsa.pub >> authorized_keys
# cd
※ 他のノードの /root/.ssh/authorized_keys にも、上記の公開鍵を追加しておく
# juju bootstrap manual-cloud-1
-> 2分程度必要
# git clone https://github.com/Juniper/contrail-charms -b R5
# juju add-machine ssh:root@(openstack-controllerのip)
# juju add-machine ssh:root@(openstack-computeのip)
# juju add-machine ssh:root@(TungstenFabric-controllerのip)
※ それぞれ 2分程度必要
# vi set-juju.sh
juju deploy cs:xenial/ntp
juju deploy cs:xenial/rabbitmq-server --to lxd:0
juju deploy cs:xenial/percona-cluster mysql --config root-password=contrail123 --config max-connections=1500 --to lxd:0
juju deploy cs:xenial/openstack-dashboard --config openstack-origin=cloud:xenial-ocata --to lxd:0
juju deploy cs:xenial/nova-cloud-controller --config console-access-protocol=novnc --config openstack-origin=cloud:xenial-ocata --config network-manager=Neutron --to lxd:0
juju deploy cs:xenial/neutron-api --config manage-neutron-plugin-legacy-mode=false --config openstack-origin=cloud:xenial-ocata --config neutron-security-groups=true --to lxd:0
juju deploy cs:xenial/glance --config openstack-origin=cloud:xenial-ocata --to lxd:0
juju deploy cs:xenial/keystone --config admin-password=contrail123 --config admin-role=admin --config openstack-origin=cloud:xenial-ocata --to lxd:0
juju deploy cs:xenial/nova-compute --config ./nova-compute-config.yaml --to 1
CHARMS_DIRECTORY=/root
juju deploy --series=xenial $CHARMS_DIRECTORY/contrail-charms/contrail-keystone-auth --to 2
juju deploy --series=xenial $CHARMS_DIRECTORY/contrail-charms/contrail-controller --config auth-mode=rbac --config cassandra-minimum-diskgb=4 --config cassandra-jvm-extra-opts="-Xms1g -Xmx2g" --to 2
juju deploy --series=xenial $CHARMS_DIRECTORY/contrail-charms/contrail-analyticsdb --config cassandra-minimum-diskgb=4 --config cassandra-jvm-extra-opts="-Xms1g -Xmx2g" --to 2
juju deploy --series=xenial $CHARMS_DIRECTORY/contrail-charms/contrail-analytics --to 2
juju deploy --series=xenial $CHARMS_DIRECTORY/contrail-charms/contrail-openstack
juju deploy --series=xenial $CHARMS_DIRECTORY/contrail-charms/contrail-agent
juju expose openstack-dashboard
juju expose nova-cloud-controller
juju expose neutron-api
juju expose glance
juju expose keystone
juju expose contrail-controller
juju expose contrail-analytics
juju add-relation keystone:shared-db mysql:shared-db
juju add-relation glance:shared-db mysql:shared-db
juju add-relation keystone:identity-service glance:identity-service
juju add-relation nova-cloud-controller:image-service glance:image-service
juju add-relation nova-cloud-controller:identity-service keystone:identity-service
juju add-relation nova-cloud-controller:cloud-compute nova-compute:cloud-compute
juju add-relation nova-compute:image-service glance:image-service
juju add-relation nova-compute:amqp rabbitmq-server:amqp
juju add-relation nova-cloud-controller:shared-db mysql:shared-db
juju add-relation nova-cloud-controller:amqp rabbitmq-server:amqp
juju add-relation openstack-dashboard:identity-service keystone
juju add-relation neutron-api:shared-db mysql:shared-db
juju add-relation neutron-api:neutron-api nova-cloud-controller:neutron-api
juju add-relation neutron-api:identity-service keystone:identity-service
juju add-relation neutron-api:amqp rabbitmq-server:amqp
juju add-relation contrail-controller ntp
juju add-relation nova-compute:juju-info ntp:juju-info
juju add-relation contrail-controller contrail-keystone-auth
juju add-relation contrail-keystone-auth keystone
juju add-relation contrail-controller contrail-analytics
juju add-relation contrail-controller contrail-analyticsdb
juju add-relation contrail-analytics contrail-analyticsdb
juju add-relation contrail-openstack neutron-api
juju add-relation contrail-openstack nova-compute
juju add-relation contrail-openstack contrail-controller
juju add-relation contrail-agent:juju-info nova-compute:juju-info
juju add-relation contrail-agent contrail-controller
# vi nova-compute-config.yaml
nova-compute:
openstack-origin: cloud:xenial-ocata
virt-type: qemu
enable-resize: True
enable-live-migration: True
migration-auth-type: ssh
# bash set-juju.sh
以下、完了まで定期的に status を確認 (20分程度必要となった)
# juju status
# tail -f /var/log/juju/*log | grep -v -w DEBUG注意点として、以下の二点が必要となった。
1. openstack-controller では LXD が使用されており、こちらと、Tungsten Fabric controller が直接疎通できる必要があった。このため、VPC の route table に LXD 用の /24 route を追加 (openstack controller の instance に紐付け) し、openstack controller 用 instance の送信元/送信先のチェック、を無効化している。
2. LXD コンテナ内で、 docker が起動出来ない (Tungsten Fabric の neutron-init で必要) 事象が発生したため、以下で LXD の設定を行っている。
juju ssh 0 ## openstack controller にログイン sudo su - lxc list ## neutron 用 LXD の id を確認 lxc config set juju-cb8047-0-lxd-4 security.nesting true lxc config show juju-cb8047-0-lxd-4
上手くインストールが完了すると、以下のように openstack / Tungsten Fabric の組み合わせが使用できるようになるはずである。
root@ip-172-31-19-222:~# juju status
Model Controller Cloud/Region Version SLA
default manual-cloud-1 manual-cloud-1 2.3.7 unsupported
App Version Status Scale Charm Store Rev OS Notes
contrail-agent 5.1.0-708.el7 active 1 contrail-agent local 0 ubuntu
contrail-analytics 5.1.0-708.el7 active 1 contrail-analytics local 0 ubuntu exposed
contrail-analyticsdb 5.1.0-708.el7 active 1 contrail-analyticsdb local 0 ubuntu
contrail-controller 5.1.0-708.el7 active 1 contrail-controller local 0 ubuntu exposed
contrail-keystone-auth active 1 contrail-keystone-auth local 0 ubuntu
contrail-openstack 5.1.0-708.el7 active 2 contrail-openstack local 0 ubuntu
glance 14.0.1 active 1 glance jujucharms 278 ubuntu exposed
keystone 11.0.4 active 1 keystone jujucharms 298 ubuntu exposed
mysql 5.6.37-26.21 active 1 percona-cluster jujucharms 275 ubuntu
neutron-api 10.0.7 active 1 neutron-api jujucharms 272 ubuntu exposed
nova-cloud-controller 15.1.5 active 1 nova-cloud-controller jujucharms 327 ubuntu exposed
nova-compute 15.1.5 active 1 nova-compute jujucharms 299 ubuntu
ntp 4.2.8p4+dfsg active 2 ntp jujucharms 32 ubuntu
openstack-dashboard 11.0.4 active 1 openstack-dashboard jujucharms 280 ubuntu exposed
rabbitmq-server 3.5.7 active 1 rabbitmq-server jujucharms 88 ubuntu
Unit Workload Agent Machine Public address Ports Message
contrail-analytics/0* active idle 2 172.31.35.214 Unit is ready
contrail-analyticsdb/0* active idle 2 172.31.35.214 Unit is ready
contrail-controller/0* active idle 2 172.31.35.214 8080/tcp,8082/tcp,8143/tcp Unit is ready
ntp/0* active idle 172.31.35.214 123/udp ntp: Ready
contrail-keystone-auth/0* active idle 2 172.31.35.214 Unit is ready
glance/0* active idle 0/lxd/5 10.0.206.248 9292/tcp Unit is ready
keystone/0* active idle 0/lxd/6 10.0.206.215 5000/tcp Unit is ready
mysql/0* active idle 0/lxd/1 10.0.206.124 3306/tcp Unit is ready
neutron-api/0* active idle 0/lxd/4 10.0.206.164 9696/tcp Unit is ready
contrail-openstack/1 active idle 10.0.206.164 Unit is ready
nova-cloud-controller/0* active idle 0/lxd/3 10.0.206.157 8774/tcp,8778/tcp Unit is ready
nova-compute/0* active idle 1 13.112.122.142 Unit is ready
contrail-agent/0* active idle 13.112.122.142 Unit is ready
contrail-openstack/0* active idle 13.112.122.142 Unit is ready
ntp/1 active idle 13.112.122.142 123/udp ntp: Ready
openstack-dashboard/0* active idle 0/lxd/2 10.0.206.82 80/tcp,443/tcp Unit is ready
rabbitmq-server/0* active idle 0/lxd/0 10.0.206.50 5672/tcp Unit is ready
Machine State DNS Inst id Series AZ Message
0 started 172.31.6.145 manual:172.31.6.145 xenial Manually provisioned machine
0/lxd/0 started 10.0.206.50 juju-cb8047-0-lxd-0 xenial Container started
0/lxd/1 started 10.0.206.124 juju-cb8047-0-lxd-1 xenial Container started
0/lxd/2 started 10.0.206.82 juju-cb8047-0-lxd-2 xenial Container started
0/lxd/3 started 10.0.206.157 juju-cb8047-0-lxd-3 xenial Container started
0/lxd/4 started 10.0.206.164 juju-cb8047-0-lxd-4 xenial Container started
0/lxd/5 started 10.0.206.248 juju-cb8047-0-lxd-5 xenial Container started
0/lxd/6 started 10.0.206.215 juju-cb8047-0-lxd-6 xenial Container started
1 started 13.112.122.142 manual:13.112.122.142 xenial Manually provisioned machine
2 started 172.31.35.214 manual:172.31.35.214 xenial Manually provisioned machine
Relation provider Requirer Interface Type Message
contrail-analytics:analytics-cluster contrail-analytics:analytics-cluster contrail-analytics-cluster peer
contrail-analytics:contrail-analytics contrail-controller:contrail-analytics contrail-analytics regular
contrail-analyticsdb:analyticsdb-cluster contrail-analyticsdb:analyticsdb-cluster contrail-analyticsdb-cluster peer
contrail-analyticsdb:contrail-analyticsdb contrail-analytics:contrail-analyticsdb contrail-analyticsdb regular
contrail-analyticsdb:contrail-analyticsdb contrail-controller:contrail-analyticsdb contrail-analyticsdb regular
contrail-controller:contrail-controller contrail-agent:contrail-controller contrail-controller regular
contrail-controller:contrail-controller contrail-openstack:contrail-controller contrail-controller regular
contrail-controller:controller-cluster contrail-controller:controller-cluster contrail-controller-cluster peer
contrail-controller:juju-info ntp:juju-info juju-info subordinate
contrail-keystone-auth:contrail-auth contrail-controller:contrail-auth contrail-auth regular
contrail-openstack:cluster contrail-openstack:cluster contrail-openstack-cluster peer
contrail-openstack:neutron-api neutron-api:neutron-plugin-api-subordinate neutron-plugin-api-subordinate subordinate
contrail-openstack:nova-compute nova-compute:neutron-plugin neutron-plugin subordinate
glance:cluster glance:cluster glance-ha peer
glance:image-service nova-cloud-controller:image-service glance regular
glance:image-service nova-compute:image-service glance regular
keystone:cluster keystone:cluster keystone-ha peer
keystone:identity-admin contrail-keystone-auth:identity-admin keystone-admin regular
keystone:identity-service glance:identity-service keystone regular
keystone:identity-service neutron-api:identity-service keystone regular
keystone:identity-service nova-cloud-controller:identity-service keystone regular
keystone:identity-service openstack-dashboard:identity-service keystone regular
mysql:cluster mysql:cluster percona-cluster peer
mysql:shared-db glance:shared-db mysql-shared regular
mysql:shared-db keystone:shared-db mysql-shared regular
mysql:shared-db neutron-api:shared-db mysql-shared regular
mysql:shared-db nova-cloud-controller:shared-db mysql-shared regular
neutron-api:cluster neutron-api:cluster neutron-api-ha peer
neutron-api:neutron-api nova-cloud-controller:neutron-api neutron-api regular
nova-cloud-controller:cluster nova-cloud-controller:cluster nova-ha peer
nova-compute:cloud-compute nova-cloud-controller:cloud-compute nova-compute regular
nova-compute:compute-peer nova-compute:compute-peer nova peer
nova-compute:juju-info contrail-agent:juju-info juju-info subordinate
nova-compute:juju-info ntp:juju-info juju-info subordinate
ntp:ntp-peers ntp:ntp-peers ntp peer
openstack-dashboard:cluster openstack-dashboard:cluster openstack-dashboard-ha peer
rabbitmq-server:amqp neutron-api:amqp rabbitmq regular
rabbitmq-server:amqp nova-cloud-controller:amqp rabbitmq regular
rabbitmq-server:amqp nova-compute:amqp rabbitmq regular
rabbitmq-server:cluster rabbitmq-server:cluster rabbitmq-ha peer
root@ip-172-31-19-222:~#
root@ip-172-31-35-214:~# contrail-status
Pod Service Original Name State Id Status
redis contrail-external-redis running d4d57d26cadf Up 8 minutes
analytics api contrail-analytics-api running da9de5110f9f Up 8 minutes
analytics collector contrail-analytics-collector running ac04930bc5c1 Up 8 minutes
analytics nodemgr contrail-nodemgr running a48717a004c2 Up 8 minutes
analytics-alarm alarm-gen contrail-analytics-alarm-gen running 9fe1da20a9e8 Up 8 minutes
analytics-alarm kafka contrail-external-kafka running f7e964a49cd7 Up 8 minutes
analytics-alarm nodemgr contrail-nodemgr running 607f2ef09c5d Up 8 minutes
analytics-snmp nodemgr contrail-nodemgr running 10bbff7fe1b1 Up 8 minutes
analytics-snmp snmp-collector contrail-analytics-snmp-collector running 082f6ebcbd37 Up 8 minutes
analytics-snmp topology contrail-analytics-snmp-topology running cd3b563f3bbb Up 8 minutes
config api contrail-controller-config-api running 3631e5abe9b6 Up 8 minutes
config device-manager contrail-controller-config-devicemgr running 8eaedcd070ae Up 8 minutes
config nodemgr contrail-nodemgr running 07203da0a748 Up 8 minutes
config schema contrail-controller-config-schema running 8c6a339dd6d0 Up 8 minutes
config svc-monitor contrail-controller-config-svcmonitor running 44856f8ea9bc Up 8 minutes
config-database cassandra contrail-external-cassandra running 22483d05229e Up 8 minutes
config-database nodemgr contrail-nodemgr running f7658b9c04af Up 8 minutes
config-database rabbitmq contrail-external-rabbitmq running 0225630978a7 Up 8 minutes
config-database zookeeper contrail-external-zookeeper running 4e3d96385f92 Up 8 minutes
control control contrail-controller-control-control running 382be60341ce Up 8 minutes
control dns contrail-controller-control-dns running 14cb5dda1dc3 Up 8 minutes
control named contrail-controller-control-named running 67279cdc5385 Up 8 minutes
control nodemgr contrail-nodemgr running 0456c3f4ade4 Up 8 minutes
database cassandra contrail-external-cassandra running de289b60d667 Up 8 minutes
database nodemgr contrail-nodemgr running 8289c2002bca Up 8 minutes
database query-engine contrail-analytics-query-engine running b6fe0b3f6ef4 Up 8 minutes
webui job contrail-controller-webui-job running 4f4a5c07e1fb Up 6 minutes
webui web contrail-controller-webui-web running f56ff61fef1f Up 6 minutes
== Contrail control ==
control: active
nodemgr: active
named: active
dns: active
== Contrail analytics-alarm ==
nodemgr: active
kafka: active
alarm-gen: active
== Contrail database ==
nodemgr: active
query-engine: active
cassandra: active
== Contrail analytics ==
nodemgr: active
api: active
collector: active
== Contrail config-database ==
nodemgr: active
zookeeper: active
rabbitmq: active
cassandra: active
== Contrail webui ==
web: active
job: active
== Contrail analytics-snmp ==
snmp-collector: active
nodemgr: active
topology: active
== Contrail config ==
svc-monitor: active
nodemgr: active
device-manager: active
api: active
schema: active
root@ip-172-31-35-214:~#
root@ip-172-31-4-230:~# contrail-status
Pod Service Original Name State Id Status
vrouter agent contrail-vrouter-agent running b30c790ac0f1 Up 8 minutes
vrouter nodemgr contrail-nodemgr running 47be0b238f30 Up 7 minutes
vrouter kernel module is PRESENT
== Contrail vrouter ==
nodemgr: active
agent: active
root@ip-172-31-4-230:~#
root@ip-172-31-4-230:~# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
opencontrailnightly/contrail-vrouter-kernel-build-init latest 9717147e05b3 18 hours ago 255MB
opencontrailnightly/contrail-vrouter-agent latest 4b4f4651d8b7 18 hours ago 1.41GB
opencontrailnightly/contrail-status latest fa3a147f3236 18 hours ago 1GB
opencontrailnightly/contrail-openstack-compute-init latest ba1e85fdb5bb 18 hours ago 1GB
opencontrailnightly/contrail-nodemgr latest fd743b6a284f 18 hours ago 1.01GB
opencontrailnightly/contrail-node-init latest 868186c43bf5 18 hours ago 1GB
opencontrailnightly/contrail-base latest d85a1c331fa3 18 hours ago 979MB
root@ip-172-31-4-230:~#
root@ip-172-31-35-214:~# cat openstackrc
export OS_USERNAME=admin
export OS_PASSWORD=contrail123
export OS_TENANT_NAME=admin
export OS_REGION_NAME=RegionOne
export OS_AUTH_URL=http://10.0.206.215:5000/v2.0
root@ip-172-31-35-214:~#
pip install python-openstackclient
source openstackrc
root@ip-172-31-35-214:~# openstack network list
+--------------------------------------+-------------------------+---------+
| ID | Name | Subnets |
+--------------------------------------+-------------------------+---------+
| 6d4589ca-eb25-4182-812c-f47f53d0b9d8 | __link_local__ | |
| cd9b79f0-9b05-4820-865a-fe1ab9446f88 | ip-fabric | |
| cf4871f6-35be-4f02-8ad7-04dc21e95440 | default-virtual-network | |
| 1d36fa0d-90be-42c2-b651-cc147969d152 | dci-network | |
+--------------------------------------+-------------------------+---------+
root@ip-172-31-35-214:~#
root@ip-172-31-35-214:~# ./contrail-introspect-cli/ist.py ctr route summary
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
| name | prefixes | paths | primary_paths | secondary_paths | infeasible_paths |
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
| default-domain:default- | 0 | 0 | 0 | 0 | 0 |
| project:__link_local__:__link_local__.inet.0 | | | | | |
| default-domain:default-project:dci- | 0 | 0 | 0 | 0 | 0 |
| network:__default__.inet.0 | | | | | |
| default-domain:default-project:dci-network:dci- | 0 | 0 | 0 | 0 | 0 |
| network.inet.0 | | | | | |
| default-domain:default-project:default-virtual- | 0 | 0 | 0 | 0 | 0 |
| network:default-virtual-network.inet.0 | | | | | |
| inet.0 | 0 | 0 | 0 | 0 | 0 |
| default-domain:default-project:ip-fabric:ip- | 1 | 1 | 1 | 0 | 0 |
| fabric.inet.0 | | | | | |
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
root@ip-172-31-35-214:~#
curl -O http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img
openstack image create cirros --disk-format qcow2 --public --container-format bare --file cirros-0.4.0-x86_64-disk.img
openstack flavor create --ram 512 --disk 1 --vcpus 1 m1.tiny
openstack network create testvn
openstack subnet create --subnet-range 192.168.100.0/24 --network testvn subnet1
NET_ID=`openstack network list | grep testvn | awk -F '|' '{print $2}' | tr -d ' '`
openstack server create --flavor m1.tiny --image cirros --nic net-id=${NET_ID} vm1
openstack server create --flavor m1.tiny --image cirros --nic net-id=${NET_ID} vm2
root@ip-172-31-35-214:~# openstack server list
+--------------------------------------+------+--------+----------------------+--------+---------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+------+--------+----------------------+--------+---------+
| 36970673-a7b7-4248-8ea8-207bfc808beb | vm2 | ACTIVE | testvn=192.168.100.4 | cirros | m1.tiny |
| 7e222583-e37b-4570-a5a8-fda4d2ca7d5b | vm1 | ACTIVE | testvn=192.168.100.3 | cirros | m1.tiny |
+--------------------------------------+------+--------+----------------------+--------+---------+
root@ip-172-31-35-214:~#
root@ip-172-31-35-214:~# ./contrail-introspect-cli/ist.py ctr route summary
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
| name | prefixes | paths | primary_paths | secondary_paths | infeasible_paths |
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
| default-domain:admin:testvn:testvn.inet.0 | 2 | 2 | 2 | 0 | 0 |
| default-domain:default- | 0 | 0 | 0 | 0 | 0 |
| project:__link_local__:__link_local__.inet.0 | | | | | |
| default-domain:default-project:dci- | 0 | 0 | 0 | 0 | 0 |
| network:__default__.inet.0 | | | | | |
| default-domain:default-project:dci-network:dci- | 0 | 0 | 0 | 0 | 0 |
| network.inet.0 | | | | | |
| default-domain:default-project:default-virtual- | 0 | 0 | 0 | 0 | 0 |
| network:default-virtual-network.inet.0 | | | | | |
| inet.0 | 0 | 0 | 0 | 0 | 0 |
| default-domain:default-project:ip-fabric:ip- | 1 | 1 | 1 | 0 | 0 |
| fabric.inet.0 | | | | | |
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
root@ip-172-31-35-214:~# ./contrail-introspect-cli/ist.py ctr route show -t default-domain:admin:testvn:testvn.inet.0
default-domain:admin:testvn:testvn.inet.0: 2 destinations, 2 routes (2 primary, 0 secondary, 0 infeasible)
192.168.100.3/32, age: 0:00:43.784175, last_modified: 2019-May-04 08:35:34.135843
[XMPP (interface)|ip-172-31-4-230.ap-northeast-1.compute.internal] age: 0:00:43.787824, localpref: 200, nh: 172.31.4.230, encap: ['gre', 'udp'], label: 25, AS path: None
192.168.100.4/32, age: 0:00:25.368270, last_modified: 2019-May-04 08:35:52.551748
[XMPP (interface)|ip-172-31-4-230.ap-northeast-1.compute.internal] age: 0:00:25.372239, localpref: 200, nh: 172.31.4.230, encap: ['gre', 'udp'], label: 30, AS path: None
root@ip-172-31-35-214:~#
ubuntu@ip-172-31-4-230:~$ ip route
default via 172.31.0.1 dev vhost0
169.254.0.1 dev vhost0 proto 109 scope link
169.254.0.3 dev vhost0 proto 109 scope link
169.254.0.4 dev vhost0 proto 109 scope link
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.31.0.0/20 dev vhost0 proto kernel scope link src 172.31.4.230
ubuntu@ip-172-31-4-230:~$
ubuntu@ip-172-31-4-230:~$ ssh ^C
ubuntu@ip-172-31-4-230:~$
ubuntu@ip-172-31-4-230:~$ ssh cirros@169.254.0.3
The authenticity of host '169.254.0.3 (169.254.0.3)' can't be established.
ECDSA key fingerprint is SHA256:+dk0gBCbyj52tmf1QHD4J6Lem39S25dqfoIPw1VCzJs.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '169.254.0.3' (ECDSA) to the list of known hosts.
cirros@169.254.0.3's password:
$
$ ip -o a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1\ link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
1: lo inet 127.0.0.1/8 scope host lo\ valid_lft forever preferred_lft forever
1: lo inet6 ::1/128 scope host \ valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000\ link/ether 02:ac:c9:3a:e7:8e brd ff:ff:ff:ff:ff:ff
2: eth0 inet 192.168.100.3/24 brd 192.168.100.255 scope global eth0\ valid_lft forever preferred_lft forever
2: eth0 inet6 fe80::ac:c9ff:fe3a:e78e/64 scope link \ valid_lft forever preferred_lft forever
$ ping 192.168.100.4
PING 192.168.100.4 (192.168.100.4): 56 data bytes
64 bytes from 192.168.100.4: seq=0 ttl=64 time=4.563 ms
64 bytes from 192.168.100.4: seq=1 ttl=64 time=0.857 ms
^C
--- 192.168.100.4 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.857/2.710/4.563 ms
$
2 kubernetes クラスタ間の名前解決
TungstenFabric の kubernetes クラスタ2組で、他のクラスタ内の svc / pod の名前解決、および ping 疎通が出来るか、を確認してみている。
環境としては、AWS 上の CentOS7.5 4台 (ami-3185744e, t2.medium) を使用した。
ansible-deployer でのインストールだと、 kubernetes クラスタが持つ ip subnet が重複してしまうため、今回は、 kubeadm を使って、kubernetes のインストールを行っている。
この際、 kubernetes で使用する subnet / service-dns-domain を変更したかったため、kubeadm init 実行時に以下のコマンドを使用している。
クラスタ0: kubeadm init --pod-network-cidr=10.32.0.0/24 --service-cidr=10.96.0.0/24 クラスタ1: kubeadm init --pod-network-cidr=10.32.1.0/24 --service-cidr=10.96.1.0/24 --service-dns-domain=cluster1.local
また、クラスタ1については、 coredns 用の svc ip も変更している (subnet の変更と合わせるため)
# cat /etc/sysconfig/kubelet -KUBELET_EXTRA_ARGS= +KUBELET_EXTRA_ARGS="--cluster-dns=10.96.1.10" # systemctl restart kubelet
TungstenFabric のインストール方法は、以下とほぼ同じだが、今回は、 TungstenFabric controller も、kubernetes 上で稼働させてみている。
http://aaabbb-200904.hatenablog.jp/entry/2019/03/17/222320
このため、TungstenFabric デプロイ時に使用する yaml が変わっている。
- # ./resolve-manifest.sh contrail-non-nested-kubernetes.yaml > cni-vrouter.yaml + # ./resolve-manifest.sh contrail-standalone-kubernetes.yaml > cni-vrouter.yaml
他に、 cni-vrouter.yaml の編集時、および反映後に、以下を実施している。
cni-vrouter.yaml に以下を追記 (subnet, AS番号は、クラスタごとに重複しない値を指定):
KUBERNETES_POD_SUBNETS: 10.32.1.0/24
KUBERNETES_IP_FABRIC_SUBNETS: 10.64.1.0/24
KUBERNETES_SERVICE_SUBNETS: 10.96.1.0/24
JVM_EXTRA_OPTS: "-Xms128m -Xmx1g"
BGP_ASN: "64513"
※ VROUTER_GATEWAY の行を削除 (こちらが残っていると、適用後に vRouter に疎通が取れなくなる)
# vi set-label.sh
masternode=$(kubectl get node | grep -w master | awk '{print $1}')
agentnodes=$(kubectl get node | grep -v -w -e master -e NAME | awk '{print $1}')
for i in config configdb analytics webui control
do
kubectl label node ${masternode} node-role.opencontrail.org/${i}=
done
for i in ${agentnodes}
do
kubectl label node ${i} node-role.opencontrail.org/agent=
done
# bash set-label.sh
※ controller, vrouter に、それぞれの role 割り当てを実施controller, vrouter が上がってきたら、各クラスタの webui にアクセス出来ることを確認した後、1. k8s-pod-network, k8s-service-network に、route-target: 64512:11 を設定 2. controller 間で bgp peer を設定 を実施し、各クラスタの pod / svc 間で疎通が取れることを確認している。
http://aaabbb-200904.hatenablog.jp/entry/2017/11/06/011959
この後、coredns の設定を行うのだが、 coredns の deployment の状態を確認したところ、 pod が認識されていない状態だったため、以下のコマンドで、livenessProbe, readinessProbe の削除を行い、pod が認識されたことを確認している。(この作業を行わないと、coredns のpodが Service からの割り振り対象にならない)
# kubectl edit deployment -n kube-system coredns
また、1. 名前解決に時間がかかる事象の解消、2. service-dns-domain を元に 他クラスタへのforward、を実施するために、coredns の設定で、以下の変更を実施している。
# kubectl edit -n kube-system configmap coredns
1.
- forward . /etc/resolv.conf
+ forward . 10.32.0.253
の変更を実施 (forward 先は、k8s-pod-network の service-ip に設定)
2.
cluster1.local:53 {
errors
cache 30
forward . 10.96.1.10
}
を追記 (domain と forward 先が一致するように設定)上記を実施することで、以下のように、クラスタ0, クラスタ1から、他のクラスタ内の pod の名前解決 / ping 疎通ができることを確認できた。
cluster0 -> cluster1: / # nslookup 10-32-1-249.default.pod.cluster1.local Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local Name: 10-32-1-249.default.pod.cluster1.local Address 1: 10.32.1.249 ip-10-32-1-249.ap-northeast-1.compute.internal / # / # ping 10-32-1-249.default.pod.cluster1.local PING 10-32-1-249.default.pod.cluster1.local (10.32.1.249): 56 data bytes 64 bytes from 10.32.1.249: seq=0 ttl=63 time=1.025 ms 64 bytes from 10.32.1.249: seq=1 ttl=63 time=0.598 ms ^C --- 10-32-1-249.default.pod.cluster1.local ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.598/0.811/1.025 ms / # / # ip -o a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000\ link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 1: lo inet 127.0.0.1/8 scope host lo\ valid_lft forever preferred_lft forever 1: lo inet6 ::1/128 scope host \ valid_lft forever preferred_lft forever 15: eth0@if16: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue \ link/ether 02:10:48:88:da:59 brd ff:ff:ff:ff:ff:ff 15: eth0 inet 10.32.0.252/24 scope global eth0\ valid_lft forever preferred_lft forever 15: eth0 inet6 fe80::501c:63ff:fe7e:6166/64 scope link \ valid_lft forever preferred_lft forever / # cluster1 -> cluster0: / # nslookup 10-32-0-252.default.pod.cluster.local Server: 10.96.1.10 Address 1: 10.96.1.10 kube-dns.kube-system.svc.cluster1.local Name: 10-32-0-252.default.pod.cluster.local Address 1: 10.32.0.252 ip-10-32-0-252.ap-northeast-1.compute.internal / # / # / # ping 10-32-0-252.default.pod.cluster.local PING 10-32-0-252.default.pod.cluster.local (10.32.0.252): 56 data bytes 64 bytes from 10.32.0.252: seq=0 ttl=63 time=0.900 ms 64 bytes from 10.32.0.252: seq=1 ttl=63 time=0.535 ms ^C --- 10-32-0-252.default.pod.cluster.local ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.535/0.717/0.900 ms / # / # ip -o a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000\ link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 1: lo inet 127.0.0.1/8 scope host lo\ valid_lft forever preferred_lft forever 1: lo inet6 ::1/128 scope host \ valid_lft forever preferred_lft forever 9: eth0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue \ link/ether 02:74:65:28:34:59 brd ff:ff:ff:ff:ff:ff 9: eth0 inet 10.32.1.249/24 scope global eth0\ valid_lft forever preferred_lft forever 9: eth0 inet6 fe80::2c59:7bff:fe92:114c/64 scope link \ valid_lft forever preferred_lft forever / #
仮にクラスタが複数に分かれている場合も、 TungstenFabric 内で、かつ fqdn を使用すれば、あまりクラスタの違いを意識すること無く疎通が出来そうなことが分かった。
複数のクラスタを運用する場合は、適用してみてもよいのではなかろうか。
introspect-cli
TungstenFabric の control には多数のルートが登録されており、これらを cli で確認する方法を探していたのだが、以下のツールで実施することが出来たので、出力例を記載しておく。
https://github.com/vcheny/contrail-introspect-cli
特に
./ist.py ctr nei ./ist.py ctr route summary ./ist.py ctr route tables ./ist.py ctr route show [-t table] [-r] [prefix] ./ist.py vr xmpp ./ist.py vr vn ./ist.py vr vrf ./ist.py vr route ./ist.py (対応するコンポーネント) status
あたりは、troubleshoot に活用できそうである。
インストール方法
※ controller ノード上で実施 pip install lxml prettytable git clone https://github.com/vcheny/contrail-introspect-cli.git cd contrail-introspect-cli
出力結果
共通:
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py -h
usage: ist [-h] [--version] [--debug] [--host HOST] [--port PORT]
{alarm_gen,analytics,cfg_api,cfg_disc,cfg_schema,cfg_svcmon,collector,ctr,dm,dns,nodemgr_analytics,nodemgr_cfg,nodemgr_ctr,nodemgr_db,nodemgr_vr,qe,vr}
...
A script to make Contrail Introspect output CLI friendly.
positional arguments:
{alarm_gen,analytics,cfg_api,cfg_disc,cfg_schema,cfg_svcmon,collector,ctr,dm,dns,nodemgr_analytics,nodemgr_cfg,nodemgr_ctr,nodemgr_db,nodemgr_vr,qe,vr}
alarm_gen contrail-alarm-gen
analytics contrail-analytics-api
cfg_api contrail-api
cfg_disc contrail-discovery
cfg_schema contrail-schema
cfg_svcmon contrail-svc-monitor
collector contrail-collector
ctr contrail-control
dm contrail-device-manager
dns contrail-dns
nodemgr_analytics contrail-analytics-nodemgr
nodemgr_cfg contrail-config-nodemgr
nodemgr_ctr contrail-control-nodemgr
nodemgr_db contrail-database-nodemgr
nodemgr_vr contrail-vrouter-nodemgr
qe contrail-query-engine
vr contrail-vrouter-agent
optional arguments:
-h, --help show this help message and exit
--version Script version
--debug Verbose mode
--host HOST Introspect host address. Default: localhost
--port PORT Introspect port number
[root@ip-172-31-42-64 contrail-introspect-cli]#
control:
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py ctr -h
usage: ist ctr [-h]
{status,cpu,trace,uve,nei,ri,route,mcast,bgp_stats,xmpp,ifmap,sc,config,rt}
...
positional arguments:
{status,cpu,trace,uve,nei,ri,route,mcast,bgp_stats,xmpp,ifmap,sc,config,rt}
status Node/component status
cpu CPU load info
trace Sandesh trace buffer
uve Sandesh UVE cache
nei Show BGP/XMPPP neighbors
ri Show routing instances
route Show route info
mcast Show multicast managers
bgp_stats Show BGP server stats
xmpp Show XMPP info
ifmap Show IFMAP info
sc Show ServiceChain info
config Show related config info
rt Show RtGroup info
optional arguments:
-h, --help show this help message and exit
[root@ip-172-31-42-64 contrail-introspect-cli]#
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py ctr status
module_id: contrail-control
state: Functional
description
+-----------+-----------+---------------------+--------+----------------------------------+
| type | name | server_addrs | status | description |
+-----------+-----------+---------------------+--------+----------------------------------+
| Collector | n/a | 172.31.42.64:8086 | Up | Established |
| Database | Cassandra | 172.31.42.64:9041 | Up | Established Cassandra connection |
| Database | RabbitMQ | 172.31.42.64:5673 | Up | RabbitMQ connection established |
+-----------+-----------+---------------------+--------+----------------------------------+
[root@ip-172-31-42-64 contrail-introspect-cli]#
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py ctr nei
+------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+
| peer | peer_address | peer_asn | encoding | peer_type | state | send_state | flap_count | flap_time |
+------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+
| ip-172-31-18-221.local | 172.31.18.221 | 0 | XMPP | internal | Established | in sync | 0 | n/a |
| ip-172-31-4-246.local | 172.31.4.246 | 0 | XMPP | internal | Established | in sync | 0 | n/a |
+------------------------+---------------+----------+----------+-----------+-------------+------------+------------+-----------+
[root@ip-172-31-42-64 contrail-introspect-cli]#
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py ctr ri
+--------------------------------------+----------+----------+-------------------------+------------------------+------------------+
| name | vn_index | vxlan_id | import_target | export_target | routing_policies |
+--------------------------------------+----------+----------+-------------------------+------------------------+------------------+
| default-domain:default-project:__lin | 3 | 0 | target:64512:7999999 | | |
| k_local__:__link_local__ | | | target:172.31.42.64:4 | | |
| default-domain:default-project:dci- | 4 | 0 | target:64512:7999999 | target:64512:8000001 | |
| network:__default__ | | | target:64512:8000001 | | |
| | | | target:172.31.42.64:1 | | |
| default-domain:default-project:dci- | 4 | 0 | target:64512:7999999 | target:64512:8000003 | |
| network:dci-network | | | target:64512:8000003 | | |
| | | | target:172.31.42.64:5 | | |
| default-domain:default-project | 1 | 0 | target:64512:7999999 | target:64512:8000000 | |
| :default-virtual-network:default- | | | target:64512:8000000 | | |
| virtual-network | | | target:172.31.42.64:2 | | |
| default-domain:default-project:ip- | 2 | 0 | | | |
| fabric:__default__ | | | | | |
| default-domain:default-project:ip- | 2 | 0 | target:64512:7999999 | target:64512:8000002 | |
| fabric:ip-fabric | | | target:64512:8000002 | | |
| | | | target:64512:8000004 | | |
| | | | target:64512:8000005 | | |
| | | | target:172.31.42.64:3 | | |
| default-domain:k8s-default:k8s- | 5 | 0 | target:64512:7999999 | target:64512:8000004 | |
| default-pod-network:k8s-default-pod- | | | target:64512:8000002 | | |
| network | | | target:64512:8000004 | | |
| | | | target:64512:8000005 | | |
| | | | target:172.31.42.64:6 | | |
| default-domain:k8s-default:k8s- | 6 | 0 | target:64512:7999999 | target:64512:8000005 | |
| default-service-network:k8s-default- | | | target:64512:8000002 | | |
| service-network | | | target:64512:8000004 | | |
| | | | target:64512:8000005 | | |
| | | | target:172.31.42.64:7 | | |
+--------------------------------------+----------+----------+-------------------------+------------------------+------------------+
[root@ip-172-31-42-64 contrail-introspect-cli]#
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py ctr route -h
usage: ist ctr route [-h] {summary,tables,show,static,aggregate} ...
positional arguments:
{summary,tables,show,static,aggregate}
summary Show route summary
tables List route table names
show Show route
static Show static routes
aggregate Show aggregate routes
optional arguments:
-h, --help show this help message and exit
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py ctr route summary
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
| name | prefixes | paths | primary_paths | secondary_paths | infeasible_paths |
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
| default-domain:default- | 0 | 0 | 0 | 0 | 0 |
| project:__link_local__:__link_local__.inet.0 | | | | | |
| default-domain:default-project:dci- | 0 | 0 | 0 | 0 | 0 |
| network:__default__.inet.0 | | | | | |
| default-domain:default-project:dci-network:dci- | 0 | 0 | 0 | 0 | 0 |
| network.inet.0 | | | | | |
| default-domain:default-project:default-virtual- | 0 | 0 | 0 | 0 | 0 |
| network:default-virtual-network.inet.0 | | | | | |
| inet.0 | 0 | 0 | 0 | 0 | 0 |
| default-domain:default-project:ip-fabric:ip- | 5 | 5 | 2 | 3 | 0 |
| fabric.inet.0 | | | | | |
| default-domain:k8s-default:k8s-default-pod-network | 5 | 5 | 2 | 3 | 0 |
| :k8s-default-pod-network.inet.0 | | | | | |
| default-domain:k8s-default:k8s-default-service- | 5 | 5 | 1 | 4 | 0 |
| network:k8s-default-service-network.inet.0 | | | | | |
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
[root@ip-172-31-42-64 contrail-introspect-cli]#
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py ctr route tables
name: default-domain:default-project:__link_local__:__link_local__.inet.0
name: default-domain:default-project:dci-network:__default__.inet.0
name: default-domain:default-project:dci-network:dci-network.inet.0
name: default-domain:default-project:default-virtual-network:default-virtual-network.inet.0
name: inet.0
name: default-domain:default-project:ip-fabric:ip-fabric.inet.0
name: default-domain:k8s-default:k8s-default-pod-network:k8s-default-pod-network.inet.0
name: default-domain:k8s-default:k8s-default-service-network:k8s-default-service-network.inet.0
[root@ip-172-31-42-64 contrail-introspect-cli]#
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py ctr route show -h
usage: ist ctr route show [-h]
[-f {inet,inet6,evpn,ermvpn,rtarget,inetvpn,l3vpn}]
[-l LAST] [-d] [-r]
[-p {BGP,XMPP,local,ServiceChain,Static}] [-v VRF]
[-s SOURCE] [-t TABLE] [--longer_match]
[--shorter_match]
[prefix]
positional arguments:
prefix Show routes matching given prefix
optional arguments:
-h, --help show this help message and exit
-f {inet,inet6,evpn,ermvpn,rtarget,inetvpn,l3vpn}, --family {inet,inet6,evpn,ermvpn,rtarget,inetvpn,l3vpn}
Show routes for given family.
-l LAST, --last LAST Show routes modified during last time period (e.g.
10s, 5m, 2h, or 5d)
-d, --detail Display detailed output
-r, --raw Display raw output in text
-p {BGP,XMPP,local,ServiceChain,Static}, --protocol {BGP,XMPP,local,ServiceChain,Static}
Show routes learned from given protocol
-v VRF, --vrf VRF Show routes in given routing instance specified as fqn
-s SOURCE, --source SOURCE
Show routes learned from given source
-t TABLE, --table TABLE
Show routes in given table
--longer_match Shows more specific routes
--shorter_match Shows less specific routes
[root@ip-172-31-42-64 contrail-introspect-cli]#
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py ctr route show -t default-domain:k8s-default:k8s-default-pod-network:k8s-default-pod-network.inet.0
default-domain:k8s-default:k8s-default-pod-network:k8s-default-pod-network.inet.0: 5 destinations, 5 routes (2 primary, 3 secondary, 0 infeasible)
10.47.255.251/32, age: 0:05:08.042661, last_modified: 2019-Apr-07 10:22:37.597451
[XMPP (interface)|ip-172-31-4-246.local] age: 0:05:08.045915, localpref: 200, nh: 172.31.4.246, encap: ['gre', 'udp'], label: 30, AS path: None
10.47.255.252/32, age: 0:05:11.002858, last_modified: 2019-Apr-07 10:22:34.637254
[XMPP (interface)|ip-172-31-4-246.local] age: 0:05:11.006508, localpref: 200, nh: 172.31.4.246, encap: ['gre', 'udp'], label: 25, AS path: None
10.96.0.10/32, age: 0:05:08.042742, last_modified: 2019-Apr-07 10:22:37.597370
[XMPP (interface)|ip-172-31-4-246.local] age: 0:05:08.046665, localpref: 200, nh: 172.31.4.246, encap: ['gre', 'udp'], label: 37, AS path: None
172.31.4.246/32, age: 0:06:28.376773, last_modified: 2019-Apr-07 10:21:17.263339
[XMPP (interface)|ip-172-31-4-246.local] age: 0:06:28.380937, localpref: 200, nh: 172.31.4.246, encap: ['gre', 'udp', 'native'], label: 16, AS path: None
172.31.18.221/32, age: 0:06:27.287767, last_modified: 2019-Apr-07 10:21:18.352345
[XMPP (interface)|ip-172-31-18-221.local] age: 0:06:27.292165, localpref: 200, nh: 172.31.18.221, encap: ['gre', 'udp', 'native'], label: 16, AS path: None
[root@ip-172-31-42-64 contrail-introspect-cli]#
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py ctr xmpp conn
+------------------------+---------+---------------------+-------------------+-------------+-----------------------+------------+-----------------------------+-----------+------------------+------------+
| name | deleted | remote_endpoint | local_endpoint | state | last_event | last_state | last_state_at | receivers | server_auth_type | dscp_value |
+------------------------+---------+---------------------+-------------------+-------------+-----------------------+------------+-----------------------------+-----------+------------------+------------+
| ip-172-31-4-246.local | false | 172.31.4.246:34576 | 172.31.42.64:5269 | Established | xmsm::EvXmppKeepalive | Active | 2019-Apr-07 10:21:17.161634 | IFMap | NIL | 0 |
| | | | | | | | | BGP | | |
| ip-172-31-18-221.local | false | 172.31.18.221:39769 | 172.31.42.64:5269 | Established | xmsm::EvXmppKeepalive | Active | 2019-Apr-07 10:21:18.252562 | IFMap | NIL | 0 |
| | | | | | | | | BGP | | |
+------------------------+---------+---------------------+-------------------+-------------+-----------------------+------------+-----------------------------+-----------+------------------+------------+
vrouter:
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py --host 172.31.4.246 vr -h
Introspect Host: 172.31.4.246
usage: ist vr [-h]
{status,cpu,trace,uve,intf,vn,vrf,route,sg,acl,hc,ifmap,baas,xmpp,xmpp-dns,stats,service,si,nh,vm,mpls,vrfassign,linklocal,vxlan,mirror}
...
positional arguments:
{status,cpu,trace,uve,intf,vn,vrf,route,sg,acl,hc,ifmap,baas,xmpp,xmpp-dns,stats,service,si,nh,vm,mpls,vrfassign,linklocal,vxlan,mirror}
status Node/component status
cpu CPU load info
trace Sandesh trace buffer
uve Sandesh UVE cache
intf Show vRouter interfaces
vn Show Virtual Network
vrf Show VRF
route Show routes
sg Show Security Groups
acl Show ACL info
hc Health Check info
ifmap IFMAP info
baas Bgp As A Service info
xmpp Show Agent XMPP connections (route&config) status
xmpp-dns Show Agent XMPP connections (dns) status
stats Show Agent stats
service Service related info
si Service instance info
nh NextHop info
vm VM info
mpls MPLS info
vrfassign VrfAssign info
linklocal LinkLocal service info
vxlan vxlan info
mirror mirror info
optional arguments:
-h, --help show this help message and exit
[root@ip-172-31-42-64 contrail-introspect-cli]#
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py --host 172.31.4.246 vr status
Introspect Host: 172.31.4.246
module_id: contrail-vrouter-agent
state: Functional
description
+-----------+---------------------------+---------------------+--------+-------------+
| type | name | server_addrs | status | description |
+-----------+---------------------------+---------------------+--------+-------------+
| XMPP | control-node:172.31.42.64 | 172.31.42.64:5269 | Up | OpenSent |
| XMPP | dns-server:172.31.42.64 | 172.31.42.64:53 | Up | OpenSent |
| Collector | n/a | 172.31.42.64:8086 | Up | Established |
+-----------+---------------------------+---------------------+--------+-------------+
[root@ip-172-31-42-64 contrail-introspect-cli]#
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py --host 172.31.4.246 vr intf
Introspect Host: 172.31.4.246
+-------+----------------+--------+-------------------+---------------+---------------+---------+--------------------------------------+
| index | name | active | mac_addr | ip_addr | mdata_ip_addr | vm_name | vn_name |
+-------+----------------+--------+-------------------+---------------+---------------+---------+--------------------------------------+
| 0 | eth0 | Active | n/a | n/a | n/a | n/a | n/a |
| 1 | vhost0 | Active | 06:c2:b8:cd:fe:fc | 172.31.4.246 | 169.254.0.1 | n/a | default-domain:default-project:ip- |
| | | | | | | | fabric |
| 3 | tapeth0-1a3aed | Active | 02:c7:14:2f:38:59 | 10.47.255.252 | 169.254.0.3 | n/a | default-domain:k8s-default:k8s- |
| | | | | | | | default-pod-network |
| 4 | tapeth0-1a3bbd | Active | 02:c7:53:a3:fc:59 | 10.47.255.251 | 169.254.0.4 | n/a | default-domain:k8s-default:k8s- |
| | | | | | | | default-pod-network |
| 2 | pkt0 | Active | n/a | n/a | n/a | n/a | n/a |
+-------+----------------+--------+-------------------+---------------+---------------+---------+--------------------------------------+
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py --host 172.31.4.246 vr vn
Introspect Host: 172.31.4.246
+--------------------------------------+--------------------------------------+-------------------+-----------------+------------+----------+
| name | uuid | layer2_forwarding | ipv4_forwarding | enable_rpf | bridging |
+--------------------------------------+--------------------------------------+-------------------+-----------------+------------+----------+
| default-domain:k8s-default:k8s- | 1ca95bc7-2c74-492f-9aa9-05e755752ee5 | false | true | true | false |
| default-service-network | | | | | |
| default-domain:k8s-default:k8s- | ab5a4cc8-1bce-4e68-a24a-72a0053cb711 | false | true | true | false |
| default-pod-network | | | | | |
+--------------------------------------+--------------------------------------+-------------------+-----------------+------------+----------+
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py --host 172.31.4.246 vr vrf
Introspect Host: 172.31.4.246
+--------------------------------------+---------+---------+---------+-----------+----------+--------------------------------------+
| name | ucindex | mcindex | brindex | evpnindex | vxlan_id | vn |
+--------------------------------------+---------+---------+---------+-----------+----------+--------------------------------------+
| default-domain:default-project:ip- | 0 | 0 | 0 | 0 | 0 | N/A |
| fabric:__default__ | | | | | | |
| default-domain:default-project:ip- | 1 | 1 | 1 | 1 | 2 | default-domain:default-project:ip- |
| fabric:ip-fabric | | | | | | fabric |
| default-domain:k8s-default:k8s- | 2 | 2 | 2 | 2 | 5 | default-domain:k8s-default:k8s- |
| default-pod-network:k8s-default-pod- | | | | | | default-pod-network |
| network | | | | | | |
| default-domain:k8s-default:k8s- | 3 | 3 | 3 | 3 | 6 | default-domain:k8s-default:k8s- |
| default-service-network:k8s-default- | | | | | | default-service-network |
| service-network | | | | | | |
+--------------------------------------+---------+---------+---------+-----------+----------+--------------------------------------+
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py --host 172.31.4.246 vr route ## -v 0 (ip-fabric:__default__ の route が表示されている)
Introspect Host: 172.31.4.246
0.0.0.0/0
[Local] pref:100
nh_index:0 , nh_type:None, nh_policy:, active_label:-1, vxlan_id:0
169.254.0.3/32
[LinkLocal] pref:100
to 2:c7:14:2f:38:59 via tapeth0-1a3aed, assigned_label:29, nh_index:26 , nh_type:interface, nh_policy:enabled, active_label:29, vxlan_id:0
169.254.0.4/32
[LinkLocal] pref:100
to 2:c7:53:a3:fc:59 via tapeth0-1a3bbd, assigned_label:21, nh_index:16 , nh_type:interface, nh_policy:enabled, active_label:21, vxlan_id:0
172.31.0.0/20
[LocalVmPort] pref:100
nh_index:14 , nh_type:resolve, nh_policy:disabled, active_label:-1, vxlan_id:0
172.31.0.1/32
[Local] pref:100
via 6:8f:fa:85:cf:16, nh_index:15 , nh_type:arp, nh_policy:disabled, active_label:-1, vxlan_id:0
172.31.0.2/32
[Local] pref:100
via 6:8f:fa:85:cf:16, nh_index:39 , nh_type:arp, nh_policy:disabled, active_label:-1, vxlan_id:0
172.31.4.246/32
[FabricRouteExport] pref:100
via vhost0, nh_index:10 , nh_type:receive, nh_policy:disabled, active_label:0, vxlan_id:0
172.31.18.221/32
[Local] pref:100
nh_index:0 , nh_type:None, nh_policy:, active_label:0, vxlan_id:0
224.0.0.0/8
[Local] pref:100
via vhost0, nh_index:11 , nh_type:receive, nh_policy:enabled, active_label:0, vxlan_id:0
[root@ip-172-31-42-64 contrail-introspect-cli]#
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py --host 172.31.4.246 vr route -v 2 ## k8s-pod-network を表示
Introspect Host: 172.31.4.246
10.32.0.0/12
[Local] pref:100
nh_index:1 , nh_type:discard, nh_policy:disabled, active_label:-1, vxlan_id:0
10.47.255.251/32
[172.31.42.64] pref:200
to 2:c7:53:a3:fc:59 via tapeth0-1a3bbd, assigned_label:21, nh_index:16 , nh_type:interface, nh_policy:enabled, active_label:21, vxlan_id:0
[LocalVmPort] pref:200
to 2:c7:53:a3:fc:59 via tapeth0-1a3bbd, assigned_label:21, nh_index:16 , nh_type:interface, nh_policy:enabled, active_label:21, vxlan_id:0
10.47.255.252/32
[172.31.42.64] pref:200
to 2:c7:14:2f:38:59 via tapeth0-1a3aed, assigned_label:29, nh_index:26 , nh_type:interface, nh_policy:enabled, active_label:29, vxlan_id:0
[LocalVmPort] pref:200
to 2:c7:14:2f:38:59 via tapeth0-1a3aed, assigned_label:29, nh_index:26 , nh_type:interface, nh_policy:enabled, active_label:29, vxlan_id:0
10.47.255.253/32
[Local] pref:100
to 0:0:0:0:0:1 via pkt0, assigned_label:-1, nh_index:13 , nh_type:interface, nh_policy:enabled, active_label:-1, vxlan_id:0
10.47.255.254/32
[Local] pref:100
to 0:0:0:0:0:1 via pkt0, assigned_label:-1, nh_index:13 , nh_type:interface, nh_policy:enabled, active_label:-1, vxlan_id:0
10.96.0.1/32
[LinkLocal] pref:100
via vhost0, nh_index:11 , nh_type:receive, nh_policy:enabled, active_label:0, vxlan_id:0
10.96.0.10/32
[172.31.42.64] pref:200
via ['tapeth0-1a3bbd', 'tapeth0-1a3aed'], nh_index:45 , nh_type:ECMP Composite sub nh count: 2, nh_policy:enabled, active_label:-1, vxlan_id:0
172.31.4.246/32
[172.31.42.64] pref:200
to 6:c2:b8:cd:fe:fc via vhost0, assigned_label:16, nh_index:5 , nh_type:interface, nh_policy:enabled, active_label:16, vxlan_id:0
172.31.18.221/32
[172.31.42.64] pref:200
to 6:8f:fa:85:cf:16 via MPLSoUDP dip:172.31.18.221 sip:172.31.4.246 label:16, nh_index:35 , nh_type:tunnel, nh_policy:disabled, active_label:16, vxlan_id:0
[root@ip-172-31-42-64 contrail-introspect-cli]#
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py --host 172.31.4.246 vr sg
Introspect Host: 172.31.4.246
+-----------+---------+--------------------------------------+----------+--------------------------------------+--------------------------------------+
| ref_count | sg_id | sg_uuid | acl_uuid | egress_acl_uuid | ingress_acl_uuid |
+-----------+---------+--------------------------------------+----------+--------------------------------------+--------------------------------------+
| 2 | 8000005 | 20bb4785-6cd2-43c2-8160-7fbfb1c18e1d | n/a | 2d7ab4e6-2758-441b-8743-2df5d9eb4ab8 | 024deaeb-5f79-4268-82b0-595e609d5c28 |
+-----------+---------+--------------------------------------+----------+--------------------------------------+--------------------------------------+
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py --host 172.31.4.246 vr acl
Introspect Host: 172.31.4.246
+--------------------------------------+--------------------------------------+-------------+
| uuid | name | dynamic_acl |
+--------------------------------------+--------------------------------------+-------------+
| 024deaeb-5f79-4268-82b0-595e609d5c28 | default-domain:k8s-kube-system:k8s- | false |
| | kube-system-default-sg:ingress- | |
| | access-control-list | |
| 11d8294f-e049-42b9-a0e6-e64eb036fd5f | default-domain:k8s-default:k8s- | false |
| | default-service-network:k8s-default- | |
| | service-network | |
| 21deedf2-2c26-4897-b5a7-b5a0ca060532 | default-domain:k8s-default:k8s- | false |
| | default-pod-network:k8s-default-pod- | |
| | network | |
| 2d7ab4e6-2758-441b-8743-2df5d9eb4ab8 | default-domain:k8s-kube-system:k8s- | false |
| | kube-system-default-sg:egress- | |
| | access-control-list | |
| b4e48fd4-e75d-4989-bc25-c55a99a998a8 | default-policy-management:k8s- | false |
| | denyall | |
| c5552c5f-f588-41f9-bcfd-62799e8483b0 | default-policy-management:k8s- | false |
| | Ingress | |
| edc2d263-d0f1-4f0d-ad39-0570153bc674 | default-policy-management:k8s- | false |
| | allowall | |
| f527d50b-5f0a-4aa3-8607-7514cb96b30f | default-domain:default-project:ip- | false |
| | fabric:ip-fabric | |
+--------------------------------------+--------------------------------------+-------------+
[root@ip-172-31-42-64 contrail-introspect-cli]#
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py --host 172.31.4.246 vr xmpp
Introspect Host: 172.31.4.246
+---------------+-------------+-------------------------------------+-------------------+----------------+------------+-----------+
| controller_ip | state | peer_name | peer_address | cfg_controller | flap_count | flap_time |
+---------------+-------------+-------------------------------------+-------------------+----------------+------------+-----------+
| 172.31.42.64 | Established | network-control@contrailsystems.com | 172.31.42.64:5269 | Yes | 0 | n/a |
+---------------+-------------+-------------------------------------+-------------------+----------------+------------+-----------+
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py --host 172.31.4.246 vr xmpp-dns
Introspect Host: 172.31.4.246
+-------------------+-------------+---------------------------------+-------------------+------------+-----------------------------+
| dns_controller_ip | state | peer_name | peer_address | flap_count | flap_time |
+-------------------+-------------+---------------------------------+-------------------+------------+-----------------------------+
| 172.31.42.64 | Established | network-dns@contrailsystems.com | 172.31.42.64:8093 | 0 | 1970-Jan-01 00:00:54.080512 |
+-------------------+-------------+---------------------------------+-------------------+------------+-----------------------------+
[root@ip-172-31-42-64 contrail-introspect-cli]#
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py --host 172.31.4.246 vr stats
Introspect Host: 172.31.4.246
IpcStatsResp
ipc_in_msgs: 0
ipc_out_msgs: 0
PktTrapStatsResp
exceptions: 1175
invalid_agent_hdr: 0
invalid_interface: 8
no_handler: 0
pkt_dropped: 8
pkt_fragments_dropped: 0
FlowStatsResp
flow_active: 60
flow_created: 1079
flow_aged: 1019
flow_drop_due_to_max_limit: 0
flow_drop_due_to_linklocal_limit: 0
flow_max_system_flows: 629760
flow_max_vm_flows: 0
XmppStatsInfo
ip: 172.31.42.64
in_msgs: 43
out_msgs: 75
reconnect: 1
config_in_msgs: 22
SandeshStatsResp
sandesh_in_msgs: 0
sandesh_out_msgs: 0
sandesh_http_sessions: 0
sandesh_reconnects: 0
ShowIFMapAgentStatsResp
node_updates_processed: 75
node_deletes_processed: 0
link_updates_processed: 88
link_deletes_processed: 0
node_update_parse_errors: 0
link_update_parse_errors: 0
node_delete_parse_errors: 0
link_delete_parse_errors: 0
[root@ip-172-31-42-64 contrail-introspect-cli]#
[root@ip-172-31-42-64 contrail-introspect-cli]#
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py --host 172.31.4.246 vr service
Introspect Host: 172.31.4.246
PktStats
total_rcvd: 1183
dhcp_rcvd: 0
arp_rcvd: 626
dns_rcvd: 4
icmp_rcvd: 0
flow_rcvd: 545
dropped: 0
total_sent: 952
dhcp_sent: 0
arp_sent: 948
dns_sent: 4
icmp_sent: 0
dhcp_q_threshold_exceeded: 0
arp_q_threshold_exceeded: 0
dns_q_threshold_exceeded: 0
icmp_q_threshold_exceeded: 0
flow_q_threshold_exceeded: 0
mac_learning_msg_rcvd: 0
DhcpStats
dhcp_discover: 0
dhcp_request: 0
dhcp_inform: 0
dhcp_decline: 0
dhcp_other: 0
dhcp_errors: 0
offers_sent: 0
acks_sent: 0
nacks_sent: 0
relay_request: 0
relay_response: 0
ArpStats
arp_entries: 2
arp_requests: 5
arp_replies: 624
arp_gratuitous: 0
arp_resolved: 2
arp_max_retries_exceeded: 0
arp_errors: 0
arp_invalid_packets: 0
arp_invalid_interface: 0
arp_invalid_vrf: 0
arp_invalid_address: 0
DnsStats
dns_resolver
172.31.42.64
dscp: 0
dns_requests: 4
dns_resolved: 0
dns_retransmit_reqs: 0
dns_unsupported: 0
dns_failures: 4
dns_drops: 0
IcmpStats
icmp_gw_ping: 0
icmp_gw_ping_err: 0
icmp_drop: 0
MetadataResponse
metadata_server_port: 8097
metadata_requests: 0
metadata_responses: 0
metadata_proxy_sessions: 0
metadata_internal_errors: 0
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py --host 172.31.4.246 vr vm
Introspect Host: 172.31.4.246
+--------------------------------------+----------------+
| uuid | drop_new_flows |
+--------------------------------------+----------------+
| 1a3aedb4-591e-11e9-9fb1-0e78d1b55f1c | false |
| 1a3bbd2e-591e-11e9-9fb1-0e78d1b55f1c | false |
+--------------------------------------+----------------+
[root@ip-172-31-42-64 contrail-introspect-cli]#
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py --host 172.31.4.246 vr linklocal
Introspect Host: 172.31.4.246
+--------------------------------------+----------------------+------------------------+-------------------+----------------+---------------+
| linklocal_service_name | linklocal_service_ip | linklocal_service_port | ipfabric_dns_name | ipfabric_ip | ipfabric_port |
+--------------------------------------+----------------------+------------------------+-------------------+----------------+---------------+
| default-domain-k8s-default- | 10.96.0.1 | 443 | n/a | 172.31.42.64 | 6443 |
| kubernetes-443 | | | | | |
+--------------------------------------+----------------------+------------------------+-------------------+----------------+---------------+
[root@ip-172-31-42-64 contrail-introspect-cli]#
※ nh,mpls,vrfassign,vxlan,mirror は、対応する cli とほぼ同じだったので、割愛
その他 (collector, schema-transformer, svc-monitor 以外は status, cpu, trace, uve で共通なので、省略):
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py collector -h
usage: ist collector [-h] {status,cpu,trace,uve,server,redis} ...
positional arguments:
{status,cpu,trace,uve,server,redis}
status Node/component status
cpu CPU load info
trace Sandesh trace buffer
uve Sandesh UVE cache
server Show collector server info
redis Show redis server UVE info
optional arguments:
-h, --help show this help message and exit
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py cfg_schema -h
usage: ist cfg_schema [-h] {status,cpu,trace,uve,vn,ri,sc,object} ...
positional arguments:
{status,cpu,trace,uve,vn,ri,sc,object}
status Node/component status
cpu CPU load info
trace Sandesh trace buffer
uve Sandesh UVE cache
vn List Virtual Networks
ri List Routing Instances
sc List Service Chains
object List Schema-transformer Ojbects
optional arguments:
-h, --help show this help message and exit
[root@ip-172-31-42-64 contrail-introspect-cli]#
[root@ip-172-31-42-64 contrail-introspect-cli]# ./ist.py cfg_svcmon -h
usage: ist cfg_svcmon [-h] {status,cpu,trace,uve,si} ...
positional arguments:
{status,cpu,trace,uve,si}
status Node/component status
cpu CPU load info
trace Sandesh trace buffer
uve Sandesh UVE cache
si List service instances
optional arguments:
-h, --help show this help message and exit
[root@ip-172-31-42-64 contrail-introspect-cli]#
4,872ノードでの負荷状況
前回に続いて、4,872ノードでの負荷状況を確認してみている。
http://aaabbb-200904.hatenablog.jp/entry/2019/03/17/222320
※ 本来は kubernetes クラスタの最大数である、5,000ノードで検証したかったのだが、実機で試したときは このノード数しか起動できなかった、、
https://kubernetes.io/ja/docs/setup/cluster-large/
環境は GCP を使い、インスタンスイメージとしては、CentOS7 (centos-7-v20190312, CentOS7.6) を使用している。
controller兼analytics, k8s master を1台ずつ用意し、インスタンスタイプとしては、n1-highcpu-64 (64vCPU, 58GM mem, 30GB disk)を使用した。
vRouter としては、n1-standard-1 (1vCPU, 3.75GB mem, 10GB disk) を使用した。
手順は基本的に前回と同じだが、変更点として、global ip の数を節約するため、今回は controller/analytics, k8s master の2台にのみ、global ip を割り当て、vRouter のノードについては、private ip のみを割り当てる構成とした。 (default のサブネットは /20 となっており、5,000 ip が入りきらないため、別のVPC を作成し、10.0.0.0/9 を割り当てている) ただし、vRouterノードも、モジュールインストールのためにインターネットにアクセスする必要があるため、CloudNAT (ネットワークサービス > CloudNAT) を追加で作成するようにしている。
また、元々の設定だと、途中で、cassandra がスローダウンする動作となったため、以下のように heap size の最大値を20GBに変更し、事象を回避している。
JVM_EXTRA_OPTS: "-Xms128m -Xmx20g"
他に、前回と比べて追加したコマンドを列記しておく。
# kubectl label node instance-group-2-m2cq node-role.opencontrail.org/config=
cni-vrouter.yaml の適用後、contrail-kube-manager を起動するために実施 (instance-group-2-m2cq には k8s master の node名を入力する)
※ upstream の変更に追随するため
# pip install google-cloud
$ gcloud init
$ gcloud auth login
$ gcloud --format="value(networkInterfaces[0].networkIP)" compute instances list
GCP instances の ip をダンプするために使用
※ parallel -j 5000 にすると、実行ノードのメモリが枯渇したため、-j 3000, -j 2000 の2回に分けて実施した
ipの差分は以下で取得:
$ cat (インスタンスipをダンプしたファイルを全て列記) | sort | uniq -c | grep ' 1 ' | awk '{print $2}'起動後、以下のように、4,872台の vRouter が登録される動作となった。
※ interface数は本来、4,878 (vRouter ごとに1, coredns x 2, この時起動していた cirros x2, default で作成される k8s service: kubernetes API, kube-dns) となるはずだったのだが、確認時は、なぜかこの値から変化しなかった、、(analytics-api の応答では、正しく4,878で出力されている、後述)
負荷状況としては、controller兼analytics では、以下のように control が最も多くの cpu / mem を使用する動きとなった。
特に、メモリ使用量は前回と比べて大きく上昇しており、30GB を使用する動作となっている。
この状態でも、cirros への ip 払いだし、等は、問題なく実施できていたので、基本的な動作は継続できていたようである。
top - 16:01:05 up 1:17, 2 users, load average: 62.04, 44.99, 35.31
Tasks: 572 total, 2 running, 570 sleeping, 0 stopped, 0 zombie
%Cpu(s): 65.6 us, 6.5 sy, 0.0 ni, 27.4 id, 0.0 wa, 0.0 hi, 0.4 si, 0.0 st
KiB Mem : 59192668 total, 11975852 free, 42433520 used, 4783296 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 15865188 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19347 root 20 0 35.2g 31.1g 13688 S 3199 55.1 505:42.00 contrail-contro
21052 root 20 0 7336100 2.2g 10960 S 1020 4.0 336:58.27 contrail-collec
19339 root 20 0 5990856 562944 12160 S 286.5 1.0 110:07.13 contrail-dns
21051 root 20 0 559792 259616 6464 R 92.7 0.4 10:36.47 python
10429 polkitd 20 0 890380 854872 1668 S 52.8 1.4 9:47.83 redis-server
13024 polkitd 20 0 34.5g 161112 3816 S 18.2 0.3 22:18.44 beam.smp
9538 root 20 0 3179672 113380 35224 S 7.6 0.2 4:27.13 dockerd
19290 root 20 0 246400 40248 5284 S 2.3 0.1 0:42.96 python
21044 root 20 0 246404 40192 5284 S 2.3 0.1 0:40.39 python
$ free -h
total used free shared buff/cache available
Mem: 56G 40G 11G 9.8M 4.6G 15G
Swap: 0B 0B 0B
$ df -h .
ファイルシス サイズ 使用 残り 使用% マウント位置
/dev/sda1 30G 5.2G 25G 18% /
$ curl 172.16.1.18:8081/analytics/uves/vrouters | python -m json.tool | grep -w href | wc -l
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1065k 100 1065k 0 0 3268k 0 --:--:-- --:--:-- --:--:-- 3279k
4872
$ curl 172.16.1.18:8081/analytics/uves/virtual-machines | python -m json.tool | grep -w href | wc -l
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 768 100 768 0 0 230k 0 --:--:-- --:--:-- --:--:-- 375k
4
$ curl 172.16.1.18:8081/analytics/uves/virtual-machine-interfaces | python -m json.tool | grep -w href | wc -l
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1495k 100 1495k 0 0 6018k 0 --:--:-- --:--:-- --:--:-- 6006k
4878
※ 4872(vRouter vhost0)+4(k8s pod: coredns, cirros x 2)+2(defaultで作成される k8s service: kubernetes, kube-dns)k8s master は、前回と同じく、 kube-apiserver/etcd が最も多くの cpu / mem を使用する動作となった。
top - 15:55:01 up 1:11, 2 users, load average: 27.01, 24.74, 21.02
Tasks: 610 total, 2 running, 608 sleeping, 0 stopped, 0 zombie
%Cpu(s): 31.2 us, 2.5 sy, 0.0 ni, 65.3 id, 0.2 wa, 0.0 hi, 0.8 si, 0.0 st
KiB Mem : 59192676 total, 41605700 free, 15398320 used, 2188656 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 42949840 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20248 root 20 0 19.0g 10.6g 39724 S 1653 18.7 292:06.02 kube-apiserver
9460 root 20 0 11.0g 1.6g 620208 S 359.3 2.8 54:45.80 etcd
20705 root 20 0 1406768 1.1g 30552 S 245.4 2.0 32:05.66 kube-controller
20410 root 20 0 385024 105376 5992 S 12.6 0.2 2:20.30 python
20257 root 20 0 635832 555136 15836 S 8.9 0.9 8:46.33 kube-scheduler
9107 root 20 0 5875912 92168 17240 S 3.0 0.2 4:51.77 kubelet
3285 root 0 -20 0 0 0 S 1.0 0.0 0:11.79 kworker/0:1H
# free -h
total used free shared buff/cache available
Mem: 56G 14G 39G 66M 2.1G 40G
Swap: 0B 0B 0B
# df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 30G 4.7G 26G 16% /
# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
cirros1 1/1 Running 0 77s 10.47.255.250 instance-group-2-4197 <none> <none>
cirros2 1/1 Running 0 73s 10.47.255.249 instance-group-2-k7sr <none> <none>今回、かなり多くの台数を controller/analytics に追加してみたのだが、実際のところ、ここまで多くのノードを1クラスタにおさめる必要があるのか、という議論はあり、管理を分離する、という意味では、アプリケーションごとに kubernetes クラスタを立てた方がよいかもしれない。
ただ、kubernetes クラスタを複数立てると、その間で連携を行うような操作が難しくなる。
この場合、TungstenFabric のように、多数のノードを1つのクラスタにいれておき、必要に応じて、アプリケーションごとのネットワーク分離の有効・無効を切り替える (policy 等も活用可能)、という動作の方がよいかもしれない。
