configdbのデータ冗長の動作

Tungsten Fabric (以下、TF) ではデータベースとしてcassandra を使っているが、この部分の冗長の動作を確認してみている。

環境としては、こちらのリンク (http://aaabbb-200904.hatenablog.jp/entry/2018/04/28/215922) の k8s+TF をベースに、TF controller 部分を3台に増やしている。

provider_config:
  bms:
   ssh_user: root
   ssh_public_key: /root/.ssh/id_rsa.pub
   ssh_private_key: /root/.ssh/id_rsa
   domainsuffix: local
instances:
  bms1:
   provider: bms
   roles:
      config_database:
      config:
      control:
      analytics_database:
      analytics:
      webui:
      k8s_master:
      kubemanager:
   ip: 172.31.9.60
  bms2:
   provider: bms
   roles:
      config_database:
      config:
      control:
      analytics_database:
      analytics:
      webui:
      kubemanager:
   ip: 172.31.5.117
  bms3:
   provider: bms
   roles:
      config_database:
      config:
      control:
      analytics_database:
      analytics:
      webui:
      kubemanager:
   ip: 172.31.2.97
  bms11:
   provider: bms
   roles:
     vrouter:
     k8s_node:
   ip: 172.31.7.122
contrail_configuration:
  CONTAINER_REGISTRY: opencontrailnightly
  CONTRAIL_VERSION: latest
  KUBERNETES_CLUSTER_PROJECT: {}

cassandra のデータは、作成・変更時、ノード間でレプリケートされる動作になるのだが、その際のレプリカの数は KEYSPACE で定義されたreplication-factor によって決まる。
http://cassandra.apache.org/doc/latest/architecture/dynamo.html#replication-strategy

cqlsh で確認してみたところ、config では replication-factor 3 となっており、ノード数分のデータが保持されるようになっていた。
このため、万一 ディスクエラー等で データが破損しても、別のノードのデータからローカルのデータを復旧できるようになっている。

※ config (一部keyspace のみ抜粋)

# docker exec -it config_database_cassandra_1 bash
root@ip-172-31-9-60:/# cqlsh 127.0.0.1 9041
Connected to contrail_database at 127.0.0.1:9041.
[cqlsh 5.0.1 | Cassandra 3.11.2 | CQL spec 3.4.4 | Native protocol v4]
Use HELP for help.
cqlsh> DESCRIBE KEYSPACES 

system_schema  config_db_uuid      system_traces         dm_keyspace
system_auth    to_bgp_keyspace     useragent           
system         system_distributed  svc_monitor_keyspace

cqlsh> 
cqlsh> DESCRIBE to_bgp_keyspace

CREATE KEYSPACE to_bgp_keyspace WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'}  AND durable_writes = true;
(snip)

上記 cqlsh は、cassandra のデータの確認・変更を行うコマンドだが、 cassandra のプロセスの状態については、nodetool というコマンドで確認できる。
nodetool info, nodetool status 等を実行することで、現在のデータ容量/ヒープ使用量、クラスタ内の各ノードの状態、等が確認出来る。
※ 7201 は configdb 用の管理ポートに対応

root@ip-172-31-9-60:/# nodetool -p 7201 info
ID                     : c1c03c07-7553-419a-a902-a3ccacd69ea0
Gossip active          : true
Thrift active          : true
Native Transport active: true
Load                   : 100.94 KiB   <-- DBのデータ容量が 約 100KB であることを確認可能
Generation No          : 1529829846
Uptime (seconds)       : 1307
Heap Memory (MB)       : 87.37 / 1936.00   <-- java heap の使用量が 87MB 程度であることを確認可能
Off Heap Memory (MB)   : 0.00
Data Center            : datacenter1
Rack                   : rack1
Exceptions             : 0
Key Cache              : entries 31, size 2.46 KiB, capacity 96 MiB, 417 hits, 523 requests, 0.797 recent hit rate, 14400 save period in seconds
Row Cache              : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Counter Cache          : entries 0, size 0 bytes, capacity 48 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
Chunk Cache            : entries 16, size 1 MiB, capacity 452 MiB, 178 misses, 1651 requests, 0.892 recent hit rate, NaN microseconds miss latency
Percent Repaired       : 100.0%
Token                  : (invoke with -T/--tokens to see all 256 tokens)
root@ip-172-31-9-60:/#

root@ip-172-31-9-60:/# nodetool -p 7201 status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens       Owns (effective)  Host ID                               Rack
UN  172.31.9.60   578.4 KiB  256          100.0%            c1c03c07-7553-419a-a902-a3ccacd69ea0  rack1
UN  172.31.5.117  434.21 KiB  256          100.0%            65f5c795-1f98-4e65-9d5e-10bacf2af7a4  rack1
UN  172.31.2.97   578.68 KiB  256          100.0%            ba2b4ece-740b-4cd2-a157-fb9c2f1e0d65  rack1
root@ip-172-31-9-60:/#
 -> 3台ともUp/Normal の状態であることが確認可能

cassandra が 3台とも動作している状態であれば、webui, API等からconfigに変更が行われた際に、各DB間で自動的にレプリケートが行われるため、特に追加の作業は発生しない。
ただし、以下の状況の際には、レプリケートが自動で行われないため、レプリケートの実施のために、手動で、nodetool repair を発行する必要がある。
1. cassandra のうちの1台が 3時間 (デフォルト値: http://cassandra.apache.org/doc/latest/configuration/cassandra_config_file.html#max-hint-window-in-ms ) 以上停止し、その間に別のノードで書き込みが行われた場合
2. ディスクの破損等によって、cassandra のデータが消失した場合

※ 以下も参照
http://cassandra.apache.org/doc/latest/operating/repair.html


nodetool repair 実行時には以下のような出力が発生し、DBの修復が行われる
※ データ量によるが、インストール直後のconfigdb であれば10秒程度で完了する

root@ip-172-31-9-60:/# nodetool -p 7201 repair --full
[2018-06-24 09:06:30,116] Starting repair command #1 (e204ce60-778d-11e8-a576-1d0021e0e5fc), repairing keyspace to_bgp_keyspace with repair options (parallelism: parallel, primary range: false, incremental: false, job threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], # of ranges: 768, pull repair: false)
[2018-06-24 09:06:33,428] Repair session e24559d0-778d-11e8-a576-1d0021e0e5fc for range [(-5608140120549516339,-5575501037168442187], (-8319274023993233504,-8299210256863649616], (-4245142859867217208,-4240769915265534326], (1027014965654704718,1029169142596609729], (5022264611528068477,5035402145262803026], (-2571074354323857941,-2554982761970724031], (-4470575950908750827,-4449301038271372254], (-5384933651707468057,-5373862792440986216], (-5352184587049184101,-5350028384852015586], (-3332727924868644218,-3300828413730832178], (327426618458049693,333131064481674807], (-1962725683988
(snip)

また、データ破損時の repair の動作を確認するため、以下のようなコマンドで1台分のconfigdb の削除を行い、削除されたノード上で、nodetool repair を発行してみた。

(3台で実施)# docker-compose -f /etc/contrail/config_database/docker-compose.yaml down
(1台で実施)# rm -rf /var/lib/docker/volumes/config_database_config_cassandra/_data/*
(3台で実施)# docker-compose -f /etc/contrail/config_database/docker-compose.yaml up -d

この際、nodetool info の出力上は、削除後に repair 率が一旦下がり、repair 後に100%に戻る動作となった。

※ 削除直後
root@ip-172-31-9-60:/# nodetool -p 7201 info
ID                     : c1c03c07-7553-419a-a902-a3ccacd69ea0
Gossip active          : true
Thrift active          : true
Native Transport active: true
Load                   : 578.4 KiB
Generation No          : 1529832761
Uptime (seconds)       : 349
Heap Memory (MB)       : 183.52 / 1936.00
Off Heap Memory (MB)   : 0.00
Data Center            : datacenter1
Rack                   : rack1
Exceptions             : 0
Key Cache              : entries 144, size 13.6 KiB, capacity 96 MiB, 4798 hits, 4947 requests, 0.970 recent hit rate, 14400 save period in seconds
Row Cache              : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Counter Cache          : entries 0, size 0 bytes, capacity 48 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
Chunk Cache            : entries 25, size 1.56 MiB, capacity 452 MiB, 39 misses, 5127 requests, 0.992 recent hit rate, NaN microseconds miss latency
Percent Repaired       : 93.39964148074145%   <-- repair 率が低下
Token                  : (invoke with -T/--tokens to see all 256 tokens)
root@ip-172-31-9-60:/#

※nodetool repair後
root@ip-172-31-9-60:/# nodetool -p 7201 info
ID                     : c1c03c07-7553-419a-a902-a3ccacd69ea0
Gossip active          : true
Thrift active          : true
Native Transport active: true
Load                   : 598.03 KiB
Generation No          : 1529832761
Uptime (seconds)       : 460
Heap Memory (MB)       : 91.59 / 1936.00
Off Heap Memory (MB)   : 0.00
Data Center            : datacenter1
Rack                   : rack1
Exceptions             : 0
Key Cache              : entries 150, size 14.2 KiB, capacity 96 MiB, 4816 hits, 4965 requests, 0.970 recent hit rate, 14400 save period in seconds
Row Cache              : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Counter Cache          : entries 0, size 0 bytes, capacity 48 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
Chunk Cache            : entries 29, size 1.81 MiB, capacity 452 MiB, 43 misses, 5186 requests, 0.992 recent hit rate, 3371.628 microseconds miss latency
Percent Repaired       : 100.0%   <-- repair 率が回復
Token                  : (invoke with -T/--tokens to see all 256 tokens)
root@ip-172-31-9-60:/#

また、上記出力だけだと、どの部分が修復されたのか、等が確認できなかったため、再度同じ環境を作成して、cassandra のログを確認してみたところ、
以下のようにログ出力が行われており、nodetool repair 発行時にデータレプリケートが実施されていることが確認できた。
※ 05:49, 05:54 の2回 nodetool repair を実行したのだが、1度目の実行では、out of sync, performing streaming repair 等のログが発生して、データ修復が発生しており、2回目では修復が行われていないことがわかる

# docker exec -it config_database_cassandra_1 bash

root@ip-172-31-3-118:/# grep 'RepairJobTask' /var/log/cassandra/system.log | grep -v RepairRunnable
INFO  [RepairJobTask:2] 2018-07-01 05:49:24,671 SyncTask.java:73 - [repair #806c9650-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 have 4 range(s) out of sync for route_target_table
INFO  [RepairJobTask:3] 2018-07-01 05:49:24,673 SyncTask.java:73 - [repair #806c9650-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 have 4 range(s) out of sync for route_target_table
INFO  [RepairJobTask:3] 2018-07-01 05:49:24,674 LocalSyncTask.java:71 - [repair #806c9650-7cf2-11e8-ad88-a1b725c94e34] Performing streaming repair of 4 ranges with /172.31.3.108
INFO  [RepairJobTask:4] 2018-07-01 05:49:24,675 SyncTask.java:66 - [repair #806c9650-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for route_target_table
INFO  [RepairJobTask:2] 2018-07-01 05:49:24,675 LocalSyncTask.java:71 - [repair #806c9650-7cf2-11e8-ad88-a1b725c94e34] Performing streaming repair of 4 ranges with /172.31.15.136
INFO  [RepairJobTask:1] 2018-07-01 05:49:24,707 SyncTask.java:66 - [repair #806c9650-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for service_chain_table
INFO  [RepairJobTask:5] 2018-07-01 05:49:24,707 SyncTask.java:66 - [repair #806c9650-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for service_chain_table
INFO  [RepairJobTask:2] 2018-07-01 05:49:24,716 StreamResultFuture.java:90 - [Stream #826ab040-7cf2-11e8-ad88-a1b725c94e34] Executing streaming plan for Repair
INFO  [RepairJobTask:3] 2018-07-01 05:49:24,719 StreamResultFuture.java:90 - [Stream #826afe60-7cf2-11e8-ad88-a1b725c94e34] Executing streaming plan for Repair
INFO  [RepairJobTask:6] 2018-07-01 05:49:24,721 SyncTask.java:66 - [repair #806c9650-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for service_chain_table
INFO  [RepairJobTask:5] 2018-07-01 05:49:24,729 RepairJob.java:143 - [repair #806c9650-7cf2-11e8-ad88-a1b725c94e34] service_chain_table is fully synced
INFO  [RepairJobTask:5] 2018-07-01 05:49:24,743 SyncTask.java:66 - [repair #806c9650-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for service_chain_uuid_table
INFO  [RepairJobTask:1] 2018-07-01 05:49:24,744 SyncTask.java:66 - [repair #806c9650-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for service_chain_uuid_table
INFO  [RepairJobTask:4] 2018-07-01 05:49:24,745 SyncTask.java:66 - [repair #806c9650-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for service_chain_uuid_table
INFO  [RepairJobTask:1] 2018-07-01 05:49:24,745 RepairJob.java:143 - [repair #806c9650-7cf2-11e8-ad88-a1b725c94e34] service_chain_uuid_table is fully synced
INFO  [RepairJobTask:4] 2018-07-01 05:49:24,750 SyncTask.java:66 - [repair #806c9650-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for service_chain_ip_address_table
INFO  [RepairJobTask:5] 2018-07-01 05:49:24,751 SyncTask.java:66 - [repair #806c9650-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for service_chain_ip_address_table
INFO  [RepairJobTask:6] 2018-07-01 05:49:24,752 SyncTask.java:66 - [repair #806c9650-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for service_chain_ip_address_table
INFO  [RepairJobTask:5] 2018-07-01 05:49:24,752 RepairJob.java:143 - [repair #806c9650-7cf2-11e8-ad88-a1b725c94e34] service_chain_ip_address_table is fully synced
INFO  [RepairJobTask:3] 2018-07-01 05:49:24,972 RepairJob.java:143 - [repair #806c9650-7cf2-11e8-ad88-a1b725c94e34] route_target_table is fully synced
INFO  [RepairJobTask:3] 2018-07-01 05:49:24,974 RepairSession.java:270 - [repair #806c9650-7cf2-11e8-ad88-a1b725c94e34] Session completed successfully
INFO  [RepairJobTask:3] 2018-07-01 05:49:27,364 SyncTask.java:66 - [repair #82d5ce20-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for dm_pnf_resource_table
INFO  [RepairJobTask:2] 2018-07-01 05:49:27,365 SyncTask.java:66 - [repair #82d5ce20-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for dm_pnf_resource_table
INFO  [RepairJobTask:1] 2018-07-01 05:49:27,366 SyncTask.java:66 - [repair #82d5ce20-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for dm_pnf_resource_table
INFO  [RepairJobTask:2] 2018-07-01 05:49:27,366 RepairJob.java:143 - [repair #82d5ce20-7cf2-11e8-ad88-a1b725c94e34] dm_pnf_resource_table is fully synced
INFO  [RepairJobTask:1] 2018-07-01 05:49:27,405 SyncTask.java:66 - [repair #82d5ce20-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for dm_pr_vn_ip_table
INFO  [RepairJobTask:3] 2018-07-01 05:49:27,406 SyncTask.java:66 - [repair #82d5ce20-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for dm_pr_vn_ip_table
INFO  [RepairJobTask:4] 2018-07-01 05:49:27,407 SyncTask.java:66 - [repair #82d5ce20-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for dm_pr_vn_ip_table
INFO  [RepairJobTask:3] 2018-07-01 05:49:27,407 RepairJob.java:143 - [repair #82d5ce20-7cf2-11e8-ad88-a1b725c94e34] dm_pr_vn_ip_table is fully synced
INFO  [RepairJobTask:4] 2018-07-01 05:49:27,423 SyncTask.java:66 - [repair #82d5ce20-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for dm_pr_ae_id_table
INFO  [RepairJobTask:1] 2018-07-01 05:49:27,424 SyncTask.java:66 - [repair #82d5ce20-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for dm_pr_ae_id_table
INFO  [RepairJobTask:2] 2018-07-01 05:49:27,425 SyncTask.java:66 - [repair #82d5ce20-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for dm_pr_ae_id_table
INFO  [RepairJobTask:1] 2018-07-01 05:49:27,425 RepairJob.java:143 - [repair #82d5ce20-7cf2-11e8-ad88-a1b725c94e34] dm_pr_ae_id_table is fully synced
INFO  [RepairJobTask:1] 2018-07-01 05:49:27,426 RepairSession.java:270 - [repair #82d5ce20-7cf2-11e8-ad88-a1b725c94e34] Session completed successfully
INFO  [RepairJobTask:2] 2018-07-01 05:49:28,374 SyncTask.java:66 - [repair #8439f200-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for useragent_keyval_table
INFO  [RepairJobTask:1] 2018-07-01 05:49:28,375 SyncTask.java:66 - [repair #8439f200-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for useragent_keyval_table
INFO  [RepairJobTask:4] 2018-07-01 05:49:28,380 SyncTask.java:66 - [repair #8439f200-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for useragent_keyval_table
INFO  [RepairJobTask:3] 2018-07-01 05:49:28,381 RepairJob.java:143 - [repair #8439f200-7cf2-11e8-ad88-a1b725c94e34] useragent_keyval_table is fully synced
INFO  [RepairJobTask:3] 2018-07-01 05:49:28,382 RepairSession.java:270 - [repair #8439f200-7cf2-11e8-ad88-a1b725c94e34] Session completed successfully
INFO  [RepairJobTask:4] 2018-07-01 05:49:30,803 SyncTask.java:66 - [repair #84ca9940-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for service_instance_table
INFO  [RepairJobTask:3] 2018-07-01 05:49:30,803 SyncTask.java:66 - [repair #84ca9940-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for service_instance_table
INFO  [RepairJobTask:2] 2018-07-01 05:49:30,807 SyncTask.java:66 - [repair #84ca9940-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for service_instance_table
INFO  [RepairJobTask:3] 2018-07-01 05:49:30,808 RepairJob.java:143 - [repair #84ca9940-7cf2-11e8-ad88-a1b725c94e34] service_instance_table is fully synced
INFO  [RepairJobTask:2] 2018-07-01 05:49:30,893 SyncTask.java:66 - [repair #84ca9940-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for healthmonitor_table
INFO  [RepairJobTask:4] 2018-07-01 05:49:30,904 SyncTask.java:66 - [repair #84ca9940-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for healthmonitor_table
INFO  [RepairJobTask:5] 2018-07-01 05:49:30,905 SyncTask.java:66 - [repair #84ca9940-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for healthmonitor_table
INFO  [RepairJobTask:4] 2018-07-01 05:49:30,905 RepairJob.java:143 - [repair #84ca9940-7cf2-11e8-ad88-a1b725c94e34] healthmonitor_table is fully synced
INFO  [RepairJobTask:5] 2018-07-01 05:49:31,007 SyncTask.java:66 - [repair #84ca9940-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for pool_table
INFO  [RepairJobTask:2] 2018-07-01 05:49:31,007 SyncTask.java:66 - [repair #84ca9940-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for pool_table
INFO  [RepairJobTask:3] 2018-07-01 05:49:31,008 SyncTask.java:66 - [repair #84ca9940-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for pool_table
INFO  [RepairJobTask:2] 2018-07-01 05:49:31,008 RepairJob.java:143 - [repair #84ca9940-7cf2-11e8-ad88-a1b725c94e34] pool_table is fully synced
INFO  [RepairJobTask:3] 2018-07-01 05:49:31,071 SyncTask.java:66 - [repair #84ca9940-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for loadbalancer_table
INFO  [RepairJobTask:5] 2018-07-01 05:49:31,071 SyncTask.java:73 - [repair #84ca9940-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 have 3 range(s) out of sync for loadbalancer_table
INFO  [RepairJobTask:5] 2018-07-01 05:49:31,072 LocalSyncTask.java:71 - [repair #84ca9940-7cf2-11e8-ad88-a1b725c94e34] Performing streaming repair of 3 ranges with /172.31.3.108
INFO  [RepairJobTask:4] 2018-07-01 05:49:31,073 SyncTask.java:73 - [repair #84ca9940-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 have 3 range(s) out of sync for loadbalancer_table
INFO  [RepairJobTask:4] 2018-07-01 05:49:31,073 LocalSyncTask.java:71 - [repair #84ca9940-7cf2-11e8-ad88-a1b725c94e34] Performing streaming repair of 3 ranges with /172.31.15.136
INFO  [RepairJobTask:4] 2018-07-01 05:49:31,075 StreamResultFuture.java:90 - [Stream #863acb10-7cf2-11e8-ad88-a1b725c94e34] Executing streaming plan for Repair
INFO  [RepairJobTask:5] 2018-07-01 05:49:31,075 StreamResultFuture.java:90 - [Stream #863aa400-7cf2-11e8-ad88-a1b725c94e34] Executing streaming plan for Repair
INFO  [RepairJobTask:5] 2018-07-01 05:49:31,188 RepairJob.java:143 - [repair #84ca9940-7cf2-11e8-ad88-a1b725c94e34] loadbalancer_table is fully synced
INFO  [RepairJobTask:5] 2018-07-01 05:49:31,189 RepairSession.java:270 - [repair #84ca9940-7cf2-11e8-ad88-a1b725c94e34] Session completed successfully
INFO  [RepairJobTask:4] 2018-07-01 05:49:33,099 SyncTask.java:66 - [repair #866bc620-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for obj_fq_name_table
INFO  [RepairJobTask:1] 2018-07-01 05:49:33,100 SyncTask.java:73 - [repair #866bc620-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 have 32 range(s) out of sync for obj_fq_name_table
INFO  [RepairJobTask:1] 2018-07-01 05:49:33,101 LocalSyncTask.java:71 - [repair #866bc620-7cf2-11e8-ad88-a1b725c94e34] Performing streaming repair of 32 ranges with /172.31.15.136
INFO  [RepairJobTask:2] 2018-07-01 05:49:33,105 SyncTask.java:73 - [repair #866bc620-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 have 32 range(s) out of sync for obj_fq_name_table
INFO  [RepairJobTask:1] 2018-07-01 05:49:33,105 StreamResultFuture.java:90 - [Stream #87703dd0-7cf2-11e8-ad88-a1b725c94e34] Executing streaming plan for Repair
INFO  [RepairJobTask:2] 2018-07-01 05:49:33,105 LocalSyncTask.java:71 - [repair #866bc620-7cf2-11e8-ad88-a1b725c94e34] Performing streaming repair of 32 ranges with /172.31.3.108
INFO  [RepairJobTask:2] 2018-07-01 05:49:33,106 StreamResultFuture.java:90 - [Stream #8770da10-7cf2-11e8-ad88-a1b725c94e34] Executing streaming plan for Repair
INFO  [RepairJobTask:1] 2018-07-01 05:49:33,169 SyncTask.java:66 - [repair #866bc620-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for obj_uuid_table
INFO  [RepairJobTask:4] 2018-07-01 05:49:33,177 SyncTask.java:73 - [repair #866bc620-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 have 120 range(s) out of sync for obj_uuid_table
INFO  [RepairJobTask:4] 2018-07-01 05:49:33,177 LocalSyncTask.java:71 - [repair #866bc620-7cf2-11e8-ad88-a1b725c94e34] Performing streaming repair of 120 ranges with /172.31.3.108
INFO  [RepairJobTask:3] 2018-07-01 05:49:33,179 SyncTask.java:73 - [repair #866bc620-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 have 120 range(s) out of sync for obj_uuid_table
INFO  [RepairJobTask:3] 2018-07-01 05:49:33,179 LocalSyncTask.java:71 - [repair #866bc620-7cf2-11e8-ad88-a1b725c94e34] Performing streaming repair of 120 ranges with /172.31.15.136
INFO  [RepairJobTask:3] 2018-07-01 05:49:33,182 StreamResultFuture.java:90 - [Stream #877c24b0-7cf2-11e8-ad88-a1b725c94e34] Executing streaming plan for Repair
INFO  [RepairJobTask:4] 2018-07-01 05:49:33,184 StreamResultFuture.java:90 - [Stream #877bd690-7cf2-11e8-ad88-a1b725c94e34] Executing streaming plan for Repair
INFO  [RepairJobTask:4] 2018-07-01 05:49:33,348 SyncTask.java:66 - [repair #866bc620-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for obj_shared_table
INFO  [RepairJobTask:1] 2018-07-01 05:49:33,349 SyncTask.java:73 - [repair #866bc620-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 have 3 range(s) out of sync for obj_shared_table
INFO  [RepairJobTask:1] 2018-07-01 05:49:33,349 LocalSyncTask.java:71 - [repair #866bc620-7cf2-11e8-ad88-a1b725c94e34] Performing streaming repair of 3 ranges with /172.31.3.108
INFO  [RepairJobTask:1] 2018-07-01 05:49:33,350 StreamResultFuture.java:90 - [Stream #87961550-7cf2-11e8-ad88-a1b725c94e34] Executing streaming plan for Repair
INFO  [RepairJobTask:2] 2018-07-01 05:49:33,350 SyncTask.java:73 - [repair #866bc620-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 have 3 range(s) out of sync for obj_shared_table
INFO  [RepairJobTask:2] 2018-07-01 05:49:33,350 LocalSyncTask.java:71 - [repair #866bc620-7cf2-11e8-ad88-a1b725c94e34] Performing streaming repair of 3 ranges with /172.31.15.136
INFO  [RepairJobTask:2] 2018-07-01 05:49:33,351 StreamResultFuture.java:90 - [Stream #87963c60-7cf2-11e8-ad88-a1b725c94e34] Executing streaming plan for Repair
INFO  [RepairJobTask:2] 2018-07-01 05:49:33,457 RepairJob.java:143 - [repair #866bc620-7cf2-11e8-ad88-a1b725c94e34] obj_fq_name_table is fully synced
INFO  [RepairJobTask:2] 2018-07-01 05:49:33,488 RepairJob.java:143 - [repair #866bc620-7cf2-11e8-ad88-a1b725c94e34] obj_uuid_table is fully synced
INFO  [RepairJobTask:2] 2018-07-01 05:49:33,526 RepairJob.java:143 - [repair #866bc620-7cf2-11e8-ad88-a1b725c94e34] obj_shared_table is fully synced
INFO  [RepairJobTask:2] 2018-07-01 05:49:33,528 RepairSession.java:270 - [repair #866bc620-7cf2-11e8-ad88-a1b725c94e34] Session completed successfully
INFO  [RepairJobTask:1] 2018-07-01 05:49:34,098 SyncTask.java:66 - [repair #87cf74d0-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for events
INFO  [RepairJobTask:2] 2018-07-01 05:49:34,098 RepairJob.java:143 - [repair #87cf74d0-7cf2-11e8-ad88-a1b725c94e34] events is fully synced
INFO  [RepairJobTask:2] 2018-07-01 05:49:34,110 SyncTask.java:66 - [repair #87cf74d0-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for sessions
INFO  [RepairJobTask:3] 2018-07-01 05:49:34,111 RepairJob.java:143 - [repair #87cf74d0-7cf2-11e8-ad88-a1b725c94e34] sessions is fully synced
INFO  [RepairJobTask:3] 2018-07-01 05:49:34,113 RepairSession.java:270 - [repair #87cf74d0-7cf2-11e8-ad88-a1b725c94e34] Session completed successfully
INFO  [RepairJobTask:3] 2018-07-01 05:49:34,419 SyncTask.java:66 - [repair #8802e0e0-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for sessions
INFO  [RepairJobTask:2] 2018-07-01 05:49:34,420 RepairJob.java:143 - [repair #8802e0e0-7cf2-11e8-ad88-a1b725c94e34] sessions is fully synced
INFO  [RepairJobTask:2] 2018-07-01 05:49:34,424 SyncTask.java:66 - [repair #8802e0e0-7cf2-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for events
INFO  [RepairJobTask:3] 2018-07-01 05:49:34,424 RepairJob.java:143 - [repair #8802e0e0-7cf2-11e8-ad88-a1b725c94e34] events is fully synced
INFO  [RepairJobTask:3] 2018-07-01 05:49:34,425 RepairSession.java:270 - [repair #8802e0e0-7cf2-11e8-ad88-a1b725c94e34] Session completed successfully
INFO  [RepairJobTask:5] 2018-07-01 05:54:42,625 SyncTask.java:66 - [repair #3edda390-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for service_chain_ip_address_table
INFO  [RepairJobTask:3] 2018-07-01 05:54:42,625 SyncTask.java:66 - [repair #3edda390-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for service_chain_ip_address_table
INFO  [RepairJobTask:2] 2018-07-01 05:54:42,626 SyncTask.java:66 - [repair #3edda390-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for service_chain_ip_address_table
INFO  [RepairJobTask:3] 2018-07-01 05:54:42,632 RepairJob.java:143 - [repair #3edda390-7cf3-11e8-ad88-a1b725c94e34] service_chain_ip_address_table is fully synced
INFO  [RepairJobTask:2] 2018-07-01 05:54:42,748 SyncTask.java:66 - [repair #3edda390-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for service_chain_table
INFO  [RepairJobTask:5] 2018-07-01 05:54:42,749 SyncTask.java:66 - [repair #3edda390-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for service_chain_table
INFO  [RepairJobTask:4] 2018-07-01 05:54:42,749 SyncTask.java:66 - [repair #3edda390-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for service_chain_table
INFO  [RepairJobTask:5] 2018-07-01 05:54:42,750 RepairJob.java:143 - [repair #3edda390-7cf3-11e8-ad88-a1b725c94e34] service_chain_table is fully synced
INFO  [RepairJobTask:4] 2018-07-01 05:54:42,796 SyncTask.java:66 - [repair #3edda390-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for service_chain_uuid_table
INFO  [RepairJobTask:2] 2018-07-01 05:54:42,797 SyncTask.java:66 - [repair #3edda390-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for service_chain_uuid_table
INFO  [RepairJobTask:3] 2018-07-01 05:54:42,798 SyncTask.java:66 - [repair #3edda390-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for service_chain_uuid_table
INFO  [RepairJobTask:2] 2018-07-01 05:54:42,798 RepairJob.java:143 - [repair #3edda390-7cf3-11e8-ad88-a1b725c94e34] service_chain_uuid_table is fully synced
INFO  [RepairJobTask:3] 2018-07-01 05:54:42,833 SyncTask.java:66 - [repair #3edda390-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for route_target_table
INFO  [RepairJobTask:4] 2018-07-01 05:54:42,833 SyncTask.java:66 - [repair #3edda390-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for route_target_table
INFO  [RepairJobTask:5] 2018-07-01 05:54:42,834 SyncTask.java:66 - [repair #3edda390-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for route_target_table
INFO  [RepairJobTask:4] 2018-07-01 05:54:42,835 RepairJob.java:143 - [repair #3edda390-7cf3-11e8-ad88-a1b725c94e34] route_target_table is fully synced
INFO  [RepairJobTask:4] 2018-07-01 05:54:42,837 RepairSession.java:270 - [repair #3edda390-7cf3-11e8-ad88-a1b725c94e34] Session completed successfully
INFO  [RepairJobTask:1] 2018-07-01 05:54:44,466 SyncTask.java:66 - [repair #4030b070-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for dm_pr_vn_ip_table
INFO  [RepairJobTask:2] 2018-07-01 05:54:44,467 SyncTask.java:66 - [repair #4030b070-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for dm_pr_vn_ip_table
INFO  [RepairJobTask:5] 2018-07-01 05:54:44,471 SyncTask.java:66 - [repair #4030b070-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for dm_pr_vn_ip_table
INFO  [RepairJobTask:4] 2018-07-01 05:54:44,472 RepairJob.java:143 - [repair #4030b070-7cf3-11e8-ad88-a1b725c94e34] dm_pr_vn_ip_table is fully synced
INFO  [RepairJobTask:2] 2018-07-01 05:54:44,477 SyncTask.java:66 - [repair #4030b070-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for dm_pnf_resource_table
INFO  [RepairJobTask:5] 2018-07-01 05:54:44,478 SyncTask.java:66 - [repair #4030b070-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for dm_pnf_resource_table
INFO  [RepairJobTask:7] 2018-07-01 05:54:44,478 SyncTask.java:66 - [repair #4030b070-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for dm_pnf_resource_table
INFO  [RepairJobTask:5] 2018-07-01 05:54:44,480 RepairJob.java:143 - [repair #4030b070-7cf3-11e8-ad88-a1b725c94e34] dm_pnf_resource_table is fully synced
INFO  [RepairJobTask:6] 2018-07-01 05:54:44,495 SyncTask.java:66 - [repair #4030b070-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for dm_pr_ae_id_table
INFO  [RepairJobTask:5] 2018-07-01 05:54:44,496 SyncTask.java:66 - [repair #4030b070-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for dm_pr_ae_id_table
INFO  [RepairJobTask:7] 2018-07-01 05:54:44,497 SyncTask.java:66 - [repair #4030b070-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for dm_pr_ae_id_table
INFO  [RepairJobTask:5] 2018-07-01 05:54:44,499 RepairJob.java:143 - [repair #4030b070-7cf3-11e8-ad88-a1b725c94e34] dm_pr_ae_id_table is fully synced
INFO  [RepairJobTask:5] 2018-07-01 05:54:44,505 RepairSession.java:270 - [repair #4030b070-7cf3-11e8-ad88-a1b725c94e34] Session completed successfully
INFO  [RepairJobTask:1] 2018-07-01 05:54:45,183 SyncTask.java:66 - [repair #4134b2f0-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for useragent_keyval_table
INFO  [RepairJobTask:1] 2018-07-01 05:54:45,186 SyncTask.java:66 - [repair #4134b2f0-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for useragent_keyval_table
INFO  [RepairJobTask:3] 2018-07-01 05:54:45,189 SyncTask.java:66 - [repair #4134b2f0-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for useragent_keyval_table
INFO  [RepairJobTask:1] 2018-07-01 05:54:45,190 RepairJob.java:143 - [repair #4134b2f0-7cf3-11e8-ad88-a1b725c94e34] useragent_keyval_table is fully synced
INFO  [RepairJobTask:1] 2018-07-01 05:54:45,191 RepairSession.java:270 - [repair #4134b2f0-7cf3-11e8-ad88-a1b725c94e34] Session completed successfully
INFO  [RepairJobTask:1] 2018-07-01 05:54:47,073 SyncTask.java:66 - [repair #41948630-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for healthmonitor_table
INFO  [RepairJobTask:4] 2018-07-01 05:54:47,073 SyncTask.java:66 - [repair #41948630-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for healthmonitor_table
INFO  [RepairJobTask:2] 2018-07-01 05:54:47,074 SyncTask.java:66 - [repair #41948630-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for healthmonitor_table
INFO  [RepairJobTask:4] 2018-07-01 05:54:47,077 RepairJob.java:143 - [repair #41948630-7cf3-11e8-ad88-a1b725c94e34] healthmonitor_table is fully synced
INFO  [RepairJobTask:1] 2018-07-01 05:54:47,114 SyncTask.java:66 - [repair #41948630-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for service_instance_table
INFO  [RepairJobTask:5] 2018-07-01 05:54:47,115 SyncTask.java:66 - [repair #41948630-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for service_instance_table
INFO  [RepairJobTask:2] 2018-07-01 05:54:47,115 SyncTask.java:66 - [repair #41948630-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for service_instance_table
INFO  [RepairJobTask:3] 2018-07-01 05:54:47,116 SyncTask.java:66 - [repair #41948630-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for pool_table
INFO  [RepairJobTask:5] 2018-07-01 05:54:47,119 RepairJob.java:143 - [repair #41948630-7cf3-11e8-ad88-a1b725c94e34] service_instance_table is fully synced
INFO  [RepairJobTask:5] 2018-07-01 05:54:47,120 SyncTask.java:66 - [repair #41948630-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for pool_table
INFO  [RepairJobTask:6] 2018-07-01 05:54:47,121 SyncTask.java:66 - [repair #41948630-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for pool_table
INFO  [RepairJobTask:5] 2018-07-01 05:54:47,121 RepairJob.java:143 - [repair #41948630-7cf3-11e8-ad88-a1b725c94e34] pool_table is fully synced
INFO  [RepairJobTask:6] 2018-07-01 05:54:47,182 SyncTask.java:66 - [repair #41948630-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for loadbalancer_table
INFO  [RepairJobTask:4] 2018-07-01 05:54:47,183 SyncTask.java:66 - [repair #41948630-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for loadbalancer_table
INFO  [RepairJobTask:3] 2018-07-01 05:54:47,183 SyncTask.java:66 - [repair #41948630-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for loadbalancer_table
INFO  [RepairJobTask:4] 2018-07-01 05:54:47,184 RepairJob.java:143 - [repair #41948630-7cf3-11e8-ad88-a1b725c94e34] loadbalancer_table is fully synced
INFO  [RepairJobTask:4] 2018-07-01 05:54:47,185 RepairSession.java:270 - [repair #41948630-7cf3-11e8-ad88-a1b725c94e34] Session completed successfully
INFO  [RepairJobTask:4] 2018-07-01 05:54:48,708 SyncTask.java:66 - [repair #42d06190-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for obj_shared_table
INFO  [RepairJobTask:1] 2018-07-01 05:54:48,709 SyncTask.java:66 - [repair #42d06190-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for obj_shared_table
INFO  [RepairJobTask:2] 2018-07-01 05:54:48,711 SyncTask.java:66 - [repair #42d06190-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for obj_shared_table
INFO  [RepairJobTask:1] 2018-07-01 05:54:48,712 RepairJob.java:143 - [repair #42d06190-7cf3-11e8-ad88-a1b725c94e34] obj_shared_table is fully synced
INFO  [RepairJobTask:2] 2018-07-01 05:54:48,748 SyncTask.java:66 - [repair #42d06190-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for obj_fq_name_table
INFO  [RepairJobTask:4] 2018-07-01 05:54:48,749 SyncTask.java:66 - [repair #42d06190-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for obj_fq_name_table
INFO  [RepairJobTask:3] 2018-07-01 05:54:48,750 SyncTask.java:66 - [repair #42d06190-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for obj_fq_name_table
INFO  [RepairJobTask:4] 2018-07-01 05:54:48,751 RepairJob.java:143 - [repair #42d06190-7cf3-11e8-ad88-a1b725c94e34] obj_fq_name_table is fully synced
INFO  [RepairJobTask:3] 2018-07-01 05:54:48,880 SyncTask.java:66 - [repair #42d06190-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.15.136 are consistent for obj_uuid_table
INFO  [RepairJobTask:2] 2018-07-01 05:54:48,880 SyncTask.java:66 - [repair #42d06190-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for obj_uuid_table
INFO  [RepairJobTask:1] 2018-07-01 05:54:48,881 SyncTask.java:66 - [repair #42d06190-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for obj_uuid_table
INFO  [RepairJobTask:2] 2018-07-01 05:54:48,881 RepairJob.java:143 - [repair #42d06190-7cf3-11e8-ad88-a1b725c94e34] obj_uuid_table is fully synced
INFO  [RepairJobTask:2] 2018-07-01 05:54:48,883 RepairSession.java:270 - [repair #42d06190-7cf3-11e8-ad88-a1b725c94e34] Session completed successfully
INFO  [RepairJobTask:1] 2018-07-01 05:54:49,330 SyncTask.java:66 - [repair #43c2b0d0-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for sessions
INFO  [RepairJobTask:3] 2018-07-01 05:54:49,330 RepairJob.java:143 - [repair #43c2b0d0-7cf3-11e8-ad88-a1b725c94e34] sessions is fully synced
INFO  [RepairJobTask:1] 2018-07-01 05:54:49,335 SyncTask.java:66 - [repair #43c2b0d0-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.3.108 and /172.31.3.118 are consistent for events
INFO  [RepairJobTask:3] 2018-07-01 05:54:49,335 RepairJob.java:143 - [repair #43c2b0d0-7cf3-11e8-ad88-a1b725c94e34] events is fully synced
INFO  [RepairJobTask:3] 2018-07-01 05:54:49,336 RepairSession.java:270 - [repair #43c2b0d0-7cf3-11e8-ad88-a1b725c94e34] Session completed successfully
INFO  [RepairJobTask:1] 2018-07-01 05:54:49,639 SyncTask.java:66 - [repair #43e61752-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for events
INFO  [RepairJobTask:2] 2018-07-01 05:54:49,639 RepairJob.java:143 - [repair #43e61752-7cf3-11e8-ad88-a1b725c94e34] events is fully synced
INFO  [RepairJobTask:2] 2018-07-01 05:54:49,640 SyncTask.java:66 - [repair #43e61752-7cf3-11e8-ad88-a1b725c94e34] Endpoints /172.31.15.136 and /172.31.3.118 are consistent for sessions
INFO  [RepairJobTask:3] 2018-07-01 05:54:49,641 RepairJob.java:143 - [repair #43e61752-7cf3-11e8-ad88-a1b725c94e34] sessions is fully synced
INFO  [RepairJobTask:3] 2018-07-01 05:54:49,641 RepairSession.java:270 - [repair #43e61752-7cf3-11e8-ad88-a1b725c94e34] Session completed successfully
root@ip-172-31-3-118:/# 

定期的に nodetool repair を実施しておけば (3台同時にデータ破損、等の場合を除いて) コンフィグデータは複数ノード上にレプリカが保持される、といってよいのではなかろうか。