juju charm による openstack / Tungsten Fabric インストール

以下の記述に従い、juju charm による openstack / Tungsten Fabric インストールを試してみている。
https://github.com/Juniper/contrail-charms/blob/R5/README.md
https://github.com/Juniper/contrail-charms/blob/R5/manual-deploy.md

環境としては、ubuntu xenial (AMI-ID: ami-06c43a7df16e8213c) 4台 (juju node, openstack controller, Tungsten Fabric Controller, openstack compute) を使用した。
※ 4vCPU, 15GB mem, 60 GB disk

openstack のバージョンとしては、ocata を使用した。
※ queens 等も試してみたのだが、なぜか Tungsten Fabric からの keystone v3 接続がうまくいかなかったため、今回は ocata で試してみている

実行したコマンドは以下となる。

(juju node 上で実施)
# apt-get update
# apt-get install juju

# juju add-cloud

Select cloud type: manual
Enter a name for your manual cloud: manual-cloud-1
Enter the controller's hostname or IP address: (juju node の ip を記載)

# ssh-keygen
# cd .ssh
# cat id_rsa.pub >> authorized_keys
# cd
※ 他のノードの /root/.ssh/authorized_keys にも、上記の公開鍵を追加しておく

# juju bootstrap manual-cloud-1
 -> 2分程度必要

# git clone https://github.com/Juniper/contrail-charms -b R5

# juju add-machine ssh:root@(openstack-controllerのip)
# juju add-machine ssh:root@(openstack-computeのip)
# juju add-machine ssh:root@(TungstenFabric-controllerのip)
※ それぞれ 2分程度必要


# vi set-juju.sh 
juju deploy cs:xenial/ntp
juju deploy cs:xenial/rabbitmq-server --to lxd:0
juju deploy cs:xenial/percona-cluster mysql --config root-password=contrail123 --config max-connections=1500 --to lxd:0
juju deploy cs:xenial/openstack-dashboard --config openstack-origin=cloud:xenial-ocata --to lxd:0
juju deploy cs:xenial/nova-cloud-controller --config console-access-protocol=novnc --config openstack-origin=cloud:xenial-ocata --config network-manager=Neutron --to lxd:0
juju deploy cs:xenial/neutron-api --config manage-neutron-plugin-legacy-mode=false --config openstack-origin=cloud:xenial-ocata --config neutron-security-groups=true --to lxd:0
juju deploy cs:xenial/glance --config openstack-origin=cloud:xenial-ocata --to lxd:0
juju deploy cs:xenial/keystone --config admin-password=contrail123 --config admin-role=admin --config openstack-origin=cloud:xenial-ocata --to lxd:0

juju deploy cs:xenial/nova-compute --config ./nova-compute-config.yaml --to 1

CHARMS_DIRECTORY=/root
juju deploy --series=xenial $CHARMS_DIRECTORY/contrail-charms/contrail-keystone-auth --to 2
juju deploy --series=xenial $CHARMS_DIRECTORY/contrail-charms/contrail-controller --config auth-mode=rbac --config cassandra-minimum-diskgb=4 --config cassandra-jvm-extra-opts="-Xms1g -Xmx2g" --to 2
juju deploy --series=xenial $CHARMS_DIRECTORY/contrail-charms/contrail-analyticsdb --config cassandra-minimum-diskgb=4 --config cassandra-jvm-extra-opts="-Xms1g -Xmx2g" --to 2
juju deploy --series=xenial $CHARMS_DIRECTORY/contrail-charms/contrail-analytics --to 2
juju deploy --series=xenial $CHARMS_DIRECTORY/contrail-charms/contrail-openstack
juju deploy --series=xenial $CHARMS_DIRECTORY/contrail-charms/contrail-agent

juju expose openstack-dashboard
juju expose nova-cloud-controller
juju expose neutron-api
juju expose glance
juju expose keystone

juju expose contrail-controller
juju expose contrail-analytics

juju add-relation keystone:shared-db mysql:shared-db
juju add-relation glance:shared-db mysql:shared-db
juju add-relation keystone:identity-service glance:identity-service
juju add-relation nova-cloud-controller:image-service glance:image-service
juju add-relation nova-cloud-controller:identity-service keystone:identity-service
juju add-relation nova-cloud-controller:cloud-compute nova-compute:cloud-compute
juju add-relation nova-compute:image-service glance:image-service
juju add-relation nova-compute:amqp rabbitmq-server:amqp
juju add-relation nova-cloud-controller:shared-db mysql:shared-db
juju add-relation nova-cloud-controller:amqp rabbitmq-server:amqp
juju add-relation openstack-dashboard:identity-service keystone

juju add-relation neutron-api:shared-db mysql:shared-db
juju add-relation neutron-api:neutron-api nova-cloud-controller:neutron-api
juju add-relation neutron-api:identity-service keystone:identity-service
juju add-relation neutron-api:amqp rabbitmq-server:amqp

juju add-relation contrail-controller ntp
juju add-relation nova-compute:juju-info ntp:juju-info

juju add-relation contrail-controller contrail-keystone-auth
juju add-relation contrail-keystone-auth keystone
juju add-relation contrail-controller contrail-analytics
juju add-relation contrail-controller contrail-analyticsdb
juju add-relation contrail-analytics contrail-analyticsdb

juju add-relation contrail-openstack neutron-api
juju add-relation contrail-openstack nova-compute
juju add-relation contrail-openstack contrail-controller

juju add-relation contrail-agent:juju-info nova-compute:juju-info
juju add-relation contrail-agent contrail-controller

# vi nova-compute-config.yaml 
nova-compute:
    openstack-origin: cloud:xenial-ocata
    virt-type: qemu 
    enable-resize: True
    enable-live-migration: True
    migration-auth-type: ssh

# bash set-juju.sh

以下、完了まで定期的に status を確認 (20分程度必要となった)
# juju status
# tail -f /var/log/juju/*log | grep -v -w DEBUG

注意点として、以下の二点が必要となった。
1. openstack-controller では LXD が使用されており、こちらと、Tungsten Fabric controller が直接疎通できる必要があった。このため、VPC の route table に LXD 用の /24 route を追加 (openstack controller の instance に紐付け) し、openstack controller 用 instance の送信元/送信先のチェック、を無効化している。
2. LXD コンテナ内で、 docker が起動出来ない (Tungsten Fabric の neutron-init で必要) 事象が発生したため、以下で LXD の設定を行っている。

juju ssh 0 ## openstack controller にログイン
  sudo su -
  lxc list ## neutron 用 LXD の id を確認
  lxc config set juju-cb8047-0-lxd-4 security.nesting true
  lxc config show juju-cb8047-0-lxd-4

上手くインストールが完了すると、以下のように openstack / Tungsten Fabric の組み合わせが使用できるようになるはずである。

root@ip-172-31-19-222:~# juju status
Model    Controller      Cloud/Region    Version  SLA
default  manual-cloud-1  manual-cloud-1  2.3.7    unsupported

App                     Version        Status  Scale  Charm                   Store       Rev  OS      Notes
contrail-agent          5.1.0-708.el7  active      1  contrail-agent          local         0  ubuntu  
contrail-analytics      5.1.0-708.el7  active      1  contrail-analytics      local         0  ubuntu  exposed
contrail-analyticsdb    5.1.0-708.el7  active      1  contrail-analyticsdb    local         0  ubuntu  
contrail-controller     5.1.0-708.el7  active      1  contrail-controller     local         0  ubuntu  exposed
contrail-keystone-auth                 active      1  contrail-keystone-auth  local         0  ubuntu  
contrail-openstack      5.1.0-708.el7  active      2  contrail-openstack      local         0  ubuntu  
glance                  14.0.1         active      1  glance                  jujucharms  278  ubuntu  exposed
keystone                11.0.4         active      1  keystone                jujucharms  298  ubuntu  exposed
mysql                   5.6.37-26.21   active      1  percona-cluster         jujucharms  275  ubuntu  
neutron-api             10.0.7         active      1  neutron-api             jujucharms  272  ubuntu  exposed
nova-cloud-controller   15.1.5         active      1  nova-cloud-controller   jujucharms  327  ubuntu  exposed
nova-compute            15.1.5         active      1  nova-compute            jujucharms  299  ubuntu  
ntp                     4.2.8p4+dfsg   active      2  ntp                     jujucharms   32  ubuntu  
openstack-dashboard     11.0.4         active      1  openstack-dashboard     jujucharms  280  ubuntu  exposed
rabbitmq-server         3.5.7          active      1  rabbitmq-server         jujucharms   88  ubuntu  

Unit                       Workload  Agent  Machine  Public address  Ports                       Message
contrail-analytics/0*      active    idle   2        172.31.35.214                               Unit is ready
contrail-analyticsdb/0*    active    idle   2        172.31.35.214                               Unit is ready
contrail-controller/0*     active    idle   2        172.31.35.214   8080/tcp,8082/tcp,8143/tcp  Unit is ready
  ntp/0*                   active    idle            172.31.35.214   123/udp                     ntp: Ready
contrail-keystone-auth/0*  active    idle   2        172.31.35.214                               Unit is ready
glance/0*                  active    idle   0/lxd/5  10.0.206.248    9292/tcp                    Unit is ready
keystone/0*                active    idle   0/lxd/6  10.0.206.215    5000/tcp                    Unit is ready
mysql/0*                   active    idle   0/lxd/1  10.0.206.124    3306/tcp                    Unit is ready
neutron-api/0*             active    idle   0/lxd/4  10.0.206.164    9696/tcp                    Unit is ready
  contrail-openstack/1     active    idle            10.0.206.164                                Unit is ready
nova-cloud-controller/0*   active    idle   0/lxd/3  10.0.206.157    8774/tcp,8778/tcp           Unit is ready
nova-compute/0*            active    idle   1        13.112.122.142                              Unit is ready
  contrail-agent/0*        active    idle            13.112.122.142                              Unit is ready
  contrail-openstack/0*    active    idle            13.112.122.142                              Unit is ready
  ntp/1                    active    idle            13.112.122.142  123/udp                     ntp: Ready
openstack-dashboard/0*     active    idle   0/lxd/2  10.0.206.82     80/tcp,443/tcp              Unit is ready
rabbitmq-server/0*         active    idle   0/lxd/0  10.0.206.50     5672/tcp                    Unit is ready

Machine  State    DNS             Inst id                Series  AZ  Message
0        started  172.31.6.145    manual:172.31.6.145    xenial      Manually provisioned machine
0/lxd/0  started  10.0.206.50     juju-cb8047-0-lxd-0    xenial      Container started
0/lxd/1  started  10.0.206.124    juju-cb8047-0-lxd-1    xenial      Container started
0/lxd/2  started  10.0.206.82     juju-cb8047-0-lxd-2    xenial      Container started
0/lxd/3  started  10.0.206.157    juju-cb8047-0-lxd-3    xenial      Container started
0/lxd/4  started  10.0.206.164    juju-cb8047-0-lxd-4    xenial      Container started
0/lxd/5  started  10.0.206.248    juju-cb8047-0-lxd-5    xenial      Container started
0/lxd/6  started  10.0.206.215    juju-cb8047-0-lxd-6    xenial      Container started
1        started  13.112.122.142  manual:13.112.122.142  xenial      Manually provisioned machine
2        started  172.31.35.214   manual:172.31.35.214   xenial      Manually provisioned machine

Relation provider                          Requirer                                    Interface                       Type         Message
contrail-analytics:analytics-cluster       contrail-analytics:analytics-cluster        contrail-analytics-cluster      peer         
contrail-analytics:contrail-analytics      contrail-controller:contrail-analytics      contrail-analytics              regular      
contrail-analyticsdb:analyticsdb-cluster   contrail-analyticsdb:analyticsdb-cluster    contrail-analyticsdb-cluster    peer         
contrail-analyticsdb:contrail-analyticsdb  contrail-analytics:contrail-analyticsdb     contrail-analyticsdb            regular      
contrail-analyticsdb:contrail-analyticsdb  contrail-controller:contrail-analyticsdb    contrail-analyticsdb            regular      
contrail-controller:contrail-controller    contrail-agent:contrail-controller          contrail-controller             regular      
contrail-controller:contrail-controller    contrail-openstack:contrail-controller      contrail-controller             regular      
contrail-controller:controller-cluster     contrail-controller:controller-cluster      contrail-controller-cluster     peer         
contrail-controller:juju-info              ntp:juju-info                               juju-info                       subordinate  
contrail-keystone-auth:contrail-auth       contrail-controller:contrail-auth           contrail-auth                   regular      
contrail-openstack:cluster                 contrail-openstack:cluster                  contrail-openstack-cluster      peer         
contrail-openstack:neutron-api             neutron-api:neutron-plugin-api-subordinate  neutron-plugin-api-subordinate  subordinate  
contrail-openstack:nova-compute            nova-compute:neutron-plugin                 neutron-plugin                  subordinate  
glance:cluster                             glance:cluster                              glance-ha                       peer         
glance:image-service                       nova-cloud-controller:image-service         glance                          regular      
glance:image-service                       nova-compute:image-service                  glance                          regular      
keystone:cluster                           keystone:cluster                            keystone-ha                     peer         
keystone:identity-admin                    contrail-keystone-auth:identity-admin       keystone-admin                  regular      
keystone:identity-service                  glance:identity-service                     keystone                        regular      
keystone:identity-service                  neutron-api:identity-service                keystone                        regular      
keystone:identity-service                  nova-cloud-controller:identity-service      keystone                        regular      
keystone:identity-service                  openstack-dashboard:identity-service        keystone                        regular      
mysql:cluster                              mysql:cluster                               percona-cluster                 peer         
mysql:shared-db                            glance:shared-db                            mysql-shared                    regular      
mysql:shared-db                            keystone:shared-db                          mysql-shared                    regular      
mysql:shared-db                            neutron-api:shared-db                       mysql-shared                    regular      
mysql:shared-db                            nova-cloud-controller:shared-db             mysql-shared                    regular      
neutron-api:cluster                        neutron-api:cluster                         neutron-api-ha                  peer         
neutron-api:neutron-api                    nova-cloud-controller:neutron-api           neutron-api                     regular      
nova-cloud-controller:cluster              nova-cloud-controller:cluster               nova-ha                         peer         
nova-compute:cloud-compute                 nova-cloud-controller:cloud-compute         nova-compute                    regular      
nova-compute:compute-peer                  nova-compute:compute-peer                   nova                            peer         
nova-compute:juju-info                     contrail-agent:juju-info                    juju-info                       subordinate  
nova-compute:juju-info                     ntp:juju-info                               juju-info                       subordinate  
ntp:ntp-peers                              ntp:ntp-peers                               ntp                             peer         
openstack-dashboard:cluster                openstack-dashboard:cluster                 openstack-dashboard-ha          peer         
rabbitmq-server:amqp                       neutron-api:amqp                            rabbitmq                        regular      
rabbitmq-server:amqp                       nova-cloud-controller:amqp                  rabbitmq                        regular      
rabbitmq-server:amqp                       nova-compute:amqp                           rabbitmq                        regular      
rabbitmq-server:cluster                    rabbitmq-server:cluster                     rabbitmq-ha                     peer         

root@ip-172-31-19-222:~# 

root@ip-172-31-35-214:~# contrail-status 
Pod              Service         Original Name                          State    Id            Status        
                 redis           contrail-external-redis                running  d4d57d26cadf  Up 8 minutes  
analytics        api             contrail-analytics-api                 running  da9de5110f9f  Up 8 minutes  
analytics        collector       contrail-analytics-collector           running  ac04930bc5c1  Up 8 minutes  
analytics        nodemgr         contrail-nodemgr                       running  a48717a004c2  Up 8 minutes  
analytics-alarm  alarm-gen       contrail-analytics-alarm-gen           running  9fe1da20a9e8  Up 8 minutes  
analytics-alarm  kafka           contrail-external-kafka                running  f7e964a49cd7  Up 8 minutes  
analytics-alarm  nodemgr         contrail-nodemgr                       running  607f2ef09c5d  Up 8 minutes  
analytics-snmp   nodemgr         contrail-nodemgr                       running  10bbff7fe1b1  Up 8 minutes  
analytics-snmp   snmp-collector  contrail-analytics-snmp-collector      running  082f6ebcbd37  Up 8 minutes  
analytics-snmp   topology        contrail-analytics-snmp-topology       running  cd3b563f3bbb  Up 8 minutes  
config           api             contrail-controller-config-api         running  3631e5abe9b6  Up 8 minutes  
config           device-manager  contrail-controller-config-devicemgr   running  8eaedcd070ae  Up 8 minutes  
config           nodemgr         contrail-nodemgr                       running  07203da0a748  Up 8 minutes  
config           schema          contrail-controller-config-schema      running  8c6a339dd6d0  Up 8 minutes  
config           svc-monitor     contrail-controller-config-svcmonitor  running  44856f8ea9bc  Up 8 minutes  
config-database  cassandra       contrail-external-cassandra            running  22483d05229e  Up 8 minutes  
config-database  nodemgr         contrail-nodemgr                       running  f7658b9c04af  Up 8 minutes  
config-database  rabbitmq        contrail-external-rabbitmq             running  0225630978a7  Up 8 minutes  
config-database  zookeeper       contrail-external-zookeeper            running  4e3d96385f92  Up 8 minutes  
control          control         contrail-controller-control-control    running  382be60341ce  Up 8 minutes  
control          dns             contrail-controller-control-dns        running  14cb5dda1dc3  Up 8 minutes  
control          named           contrail-controller-control-named      running  67279cdc5385  Up 8 minutes  
control          nodemgr         contrail-nodemgr                       running  0456c3f4ade4  Up 8 minutes  
database         cassandra       contrail-external-cassandra            running  de289b60d667  Up 8 minutes  
database         nodemgr         contrail-nodemgr                       running  8289c2002bca  Up 8 minutes  
database         query-engine    contrail-analytics-query-engine        running  b6fe0b3f6ef4  Up 8 minutes  
webui            job             contrail-controller-webui-job          running  4f4a5c07e1fb  Up 6 minutes  
webui            web             contrail-controller-webui-web          running  f56ff61fef1f  Up 6 minutes  

== Contrail control ==
control: active
nodemgr: active
named: active
dns: active

== Contrail analytics-alarm ==
nodemgr: active
kafka: active
alarm-gen: active

== Contrail database ==
nodemgr: active
query-engine: active
cassandra: active

== Contrail analytics ==
nodemgr: active
api: active
collector: active

== Contrail config-database ==
nodemgr: active
zookeeper: active
rabbitmq: active
cassandra: active

== Contrail webui ==
web: active
job: active

== Contrail analytics-snmp ==
snmp-collector: active
nodemgr: active
topology: active

== Contrail config ==
svc-monitor: active
nodemgr: active
device-manager: active
api: active
schema: active

root@ip-172-31-35-214:~# 

root@ip-172-31-4-230:~# contrail-status 
Pod      Service  Original Name           State    Id            Status        
vrouter  agent    contrail-vrouter-agent  running  b30c790ac0f1  Up 8 minutes  
vrouter  nodemgr  contrail-nodemgr        running  47be0b238f30  Up 7 minutes  

vrouter kernel module is PRESENT
== Contrail vrouter ==
nodemgr: active
agent: active

root@ip-172-31-4-230:~# 
root@ip-172-31-4-230:~# docker images
REPOSITORY                                               TAG                 IMAGE ID            CREATED             SIZE
opencontrailnightly/contrail-vrouter-kernel-build-init   latest              9717147e05b3        18 hours ago        255MB
opencontrailnightly/contrail-vrouter-agent               latest              4b4f4651d8b7        18 hours ago        1.41GB
opencontrailnightly/contrail-status                      latest              fa3a147f3236        18 hours ago        1GB
opencontrailnightly/contrail-openstack-compute-init      latest              ba1e85fdb5bb        18 hours ago        1GB
opencontrailnightly/contrail-nodemgr                     latest              fd743b6a284f        18 hours ago        1.01GB
opencontrailnightly/contrail-node-init                   latest              868186c43bf5        18 hours ago        1GB
opencontrailnightly/contrail-base                        latest              d85a1c331fa3        18 hours ago        979MB
root@ip-172-31-4-230:~# 

root@ip-172-31-35-214:~# cat openstackrc 
export OS_USERNAME=admin
export OS_PASSWORD=contrail123
export OS_TENANT_NAME=admin
export OS_REGION_NAME=RegionOne
export OS_AUTH_URL=http://10.0.206.215:5000/v2.0
root@ip-172-31-35-214:~# 

pip install python-openstackclient
source openstackrc

root@ip-172-31-35-214:~# openstack network list
+--------------------------------------+-------------------------+---------+
| ID                                   | Name                    | Subnets |
+--------------------------------------+-------------------------+---------+
| 6d4589ca-eb25-4182-812c-f47f53d0b9d8 | __link_local__          |         |
| cd9b79f0-9b05-4820-865a-fe1ab9446f88 | ip-fabric               |         |
| cf4871f6-35be-4f02-8ad7-04dc21e95440 | default-virtual-network |         |
| 1d36fa0d-90be-42c2-b651-cc147969d152 | dci-network             |         |
+--------------------------------------+-------------------------+---------+
root@ip-172-31-35-214:~# 

root@ip-172-31-35-214:~# ./contrail-introspect-cli/ist.py ctr route summary
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
| name                                               | prefixes | paths | primary_paths | secondary_paths | infeasible_paths |
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
| default-domain:default-                            | 0        | 0     | 0             | 0               | 0                |
| project:__link_local__:__link_local__.inet.0       |          |       |               |                 |                  |
| default-domain:default-project:dci-                | 0        | 0     | 0             | 0               | 0                |
| network:__default__.inet.0                         |          |       |               |                 |                  |
| default-domain:default-project:dci-network:dci-    | 0        | 0     | 0             | 0               | 0                |
| network.inet.0                                     |          |       |               |                 |                  |
| default-domain:default-project:default-virtual-    | 0        | 0     | 0             | 0               | 0                |
| network:default-virtual-network.inet.0             |          |       |               |                 |                  |
| inet.0                                             | 0        | 0     | 0             | 0               | 0                |
| default-domain:default-project:ip-fabric:ip-       | 1        | 1     | 1             | 0               | 0                |
| fabric.inet.0                                      |          |       |               |                 |                  |
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
root@ip-172-31-35-214:~# 

curl -O http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img
openstack image create cirros --disk-format qcow2 --public --container-format bare --file cirros-0.4.0-x86_64-disk.img
openstack flavor create --ram 512 --disk 1 --vcpus 1 m1.tiny
openstack network create testvn
openstack subnet create --subnet-range 192.168.100.0/24 --network testvn subnet1
NET_ID=`openstack network list | grep testvn | awk -F '|' '{print $2}' | tr -d ' '`
openstack server create --flavor m1.tiny --image cirros --nic net-id=${NET_ID} vm1
openstack server create --flavor m1.tiny --image cirros --nic net-id=${NET_ID} vm2

root@ip-172-31-35-214:~# openstack server list
+--------------------------------------+------+--------+----------------------+--------+---------+
| ID                                   | Name | Status | Networks             | Image  | Flavor  |
+--------------------------------------+------+--------+----------------------+--------+---------+
| 36970673-a7b7-4248-8ea8-207bfc808beb | vm2  | ACTIVE | testvn=192.168.100.4 | cirros | m1.tiny |
| 7e222583-e37b-4570-a5a8-fda4d2ca7d5b | vm1  | ACTIVE | testvn=192.168.100.3 | cirros | m1.tiny |
+--------------------------------------+------+--------+----------------------+--------+---------+
root@ip-172-31-35-214:~# 

root@ip-172-31-35-214:~# ./contrail-introspect-cli/ist.py ctr route summary
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
| name                                               | prefixes | paths | primary_paths | secondary_paths | infeasible_paths |
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
| default-domain:admin:testvn:testvn.inet.0          | 2        | 2     | 2             | 0               | 0                |
| default-domain:default-                            | 0        | 0     | 0             | 0               | 0                |
| project:__link_local__:__link_local__.inet.0       |          |       |               |                 |                  |
| default-domain:default-project:dci-                | 0        | 0     | 0             | 0               | 0                |
| network:__default__.inet.0                         |          |       |               |                 |                  |
| default-domain:default-project:dci-network:dci-    | 0        | 0     | 0             | 0               | 0                |
| network.inet.0                                     |          |       |               |                 |                  |
| default-domain:default-project:default-virtual-    | 0        | 0     | 0             | 0               | 0                |
| network:default-virtual-network.inet.0             |          |       |               |                 |                  |
| inet.0                                             | 0        | 0     | 0             | 0               | 0                |
| default-domain:default-project:ip-fabric:ip-       | 1        | 1     | 1             | 0               | 0                |
| fabric.inet.0                                      |          |       |               |                 |                  |
+----------------------------------------------------+----------+-------+---------------+-----------------+------------------+
root@ip-172-31-35-214:~# ./contrail-introspect-cli/ist.py ctr route show -t default-domain:admin:testvn:testvn.inet.0

default-domain:admin:testvn:testvn.inet.0: 2 destinations, 2 routes (2 primary, 0 secondary, 0 infeasible)

192.168.100.3/32, age: 0:00:43.784175, last_modified: 2019-May-04 08:35:34.135843
    [XMPP (interface)|ip-172-31-4-230.ap-northeast-1.compute.internal] age: 0:00:43.787824, localpref: 200, nh: 172.31.4.230, encap: ['gre', 'udp'], label: 25, AS path: None

192.168.100.4/32, age: 0:00:25.368270, last_modified: 2019-May-04 08:35:52.551748
    [XMPP (interface)|ip-172-31-4-230.ap-northeast-1.compute.internal] age: 0:00:25.372239, localpref: 200, nh: 172.31.4.230, encap: ['gre', 'udp'], label: 30, AS path: None
root@ip-172-31-35-214:~#

ubuntu@ip-172-31-4-230:~$ ip route
default via 172.31.0.1 dev vhost0 
169.254.0.1 dev vhost0  proto 109  scope link 
169.254.0.3 dev vhost0  proto 109  scope link 
169.254.0.4 dev vhost0  proto 109  scope link 
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1 linkdown 
172.31.0.0/20 dev vhost0  proto kernel  scope link  src 172.31.4.230 
ubuntu@ip-172-31-4-230:~$ 
ubuntu@ip-172-31-4-230:~$ ssh ^C
ubuntu@ip-172-31-4-230:~$ 
ubuntu@ip-172-31-4-230:~$ ssh cirros@169.254.0.3
The authenticity of host '169.254.0.3 (169.254.0.3)' can't be established.
ECDSA key fingerprint is SHA256:+dk0gBCbyj52tmf1QHD4J6Lem39S25dqfoIPw1VCzJs.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '169.254.0.3' (ECDSA) to the list of known hosts.
cirros@169.254.0.3's password: 
$ 
$ ip -o a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1\    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
1: lo    inet 127.0.0.1/8 scope host lo\       valid_lft forever preferred_lft forever
1: lo    inet6 ::1/128 scope host \       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000\    link/ether 02:ac:c9:3a:e7:8e brd ff:ff:ff:ff:ff:ff
2: eth0    inet 192.168.100.3/24 brd 192.168.100.255 scope global eth0\       valid_lft forever preferred_lft forever
2: eth0    inet6 fe80::ac:c9ff:fe3a:e78e/64 scope link \       valid_lft forever preferred_lft forever
$ ping 192.168.100.4
PING 192.168.100.4 (192.168.100.4): 56 data bytes
64 bytes from 192.168.100.4: seq=0 ttl=64 time=4.563 ms
64 bytes from 192.168.100.4: seq=1 ttl=64 time=0.857 ms
^C
--- 192.168.100.4 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.857/2.710/4.563 ms
$