Troubleshooting Tools
Checking Deployment Status of Each Instance
After deployment, the environment contains the following components:
- 1
deployment/smartagent-operator(full-stack deployment mode) - 1
deployment/bonree-webhook(application monitoring only mode) - 1
daemonset/bonree-smartagent-csi-driver(optional, application monitoring only mode) - 1
daemonset/{smartkube-name}-smartagent(full-stack deployment mode only)
[root@lll-arm-openeuler ~]# kubectl get all -n bonree-agent
NAME READY STATUS RESTARTS AGE
pod/bonree-smartagent-csi-driver-5gzdh 3/3 Running 3 5h11m
pod/bonree-webhook-8454b78bf6-tfhwz 1/1 Running 1 5h11m
pod/smartagent-operator-dd4994b47-kg4kv 1/1 Running 1 5h11m
pod/smartkube-smartagent-hvfs5 1/1 Running 25 4h29m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/bonree-webhook ClusterIP 10.96.168.8 <none> 443/TCP 5h11m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/bonree-smartagent-csi-driver 1 1 1 1 1 <none> 5h11m
daemonset.apps/smartkube-smartagent 1 1 1 1 1 <none> 4h42m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/bonree-webhook 1/1 1 1 5h11m
deployment.apps/smartagent-operator 1/1 1 1 5h11m
NAME DESIRED CURRENT READY AGE
replicaset.apps/bonree-webhook-8454b78bf6 1 1 1 5h11m
replicaset.apps/smartagent-operator-dd4994b47 1 1 1 5h11m
Listing SmartKube Instances
Use the following command to view all SmartKube instances:
[root@lll-arm-openeuler ~]# kubectl get sk -A
NAMESPACE NAME APIURL STATUS AGE
bonree-agent smartkube https://one.bonree.com/rest/apm/agentDown Deploying 15h
Viewing SmartKube Instance Status
Use the describe command for detailed SmartKube instance status:
[root@lll-arm-openeuler ~]# kubectl describe sk smartkube -n bonree-agent
Name: smartkube
Namespace: bonree-agent
Labels: <none>
Annotations: <none>
API Version: bonree.com/v1alpha1
Kind: SmartKube
Metadata:
Creation Timestamp: 2023-12-04T07:11:49Z
Generation: 1
Resource Version: 3573037
UID: 52416042-1178-47e3-bb7d-06fbbe79aa2f
Spec:
API URL: https://one.bonree.com/rest/apm/agentDown
Classic Full Stack:
Enabled: true
Tolerations:
Effect: NoExecute
Key: node-role.kubernetes.io/master
Operator: Exists
Controller Info:
Account GUID: xxxxxxxxxxxxxxxxx
Controller URL: https://oneupload.bonree.com/APM
Log Controller URL: https://onelog.joinbr.com/controller/logs
Status:
Communication Host For Client:
Host: one.bonree.com
Port: 443
Protocol: https
Conditions:
Last Transition Time: 2023-12-04T07:13:09Z
Message: Ready
Reason: TokenReady
Status: True
Type: PaaSToken # PaaSToken status
Connection Info:
Account GUID: xxxxxxxxxxxxxxxxx
Communication Hosts:
Host: oneupload.bonree.com
Port: 443
Protocol: https
Kube System UUID: 7a921db9-631d-4032-a828-07f9f10bf3c3
Last PaaS Token Probe Timestamp: 2023-12-04T11:35:22Z
Latest Agent Version Unix Default: 7.4.2 # Latest SmartAgent version detected
Latest Agent Version Unix Paas: 7.4.2 # Latest SmartAgent PaaS version detected
Phase: Running # Deployment status
Smart Agent:
Instances: # SmartAgent full-stack deployed Pod instance info
Kind - Control - Plane: # Node name
Ip Address: 172.18.0.2 # Node IP
Pod Name: smartkube-smartagent-hvfs5 # Pod name
Last Hosts Request Timestamp: 2023-12-04T11:35:22Z
Last Update Probe Timestamp: 2023-12-04T11:30:22Z
Version: 7.4.2 # Installed version
Tokens: smartkube # Secret used
Updated Timestamp: 2023-12-04T11:35:22Z
Events: <none> # Deployment exceptions, webhook serverless agent events, etc.
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Inject 9m53s (x18 over 93m) bonree-webhook Injecting the necessary info into pod nginx-deployment-56c88b7477- in namespace default
Viewing Component Logs
# Operator logs
kubectl -n bonree-agent logs -f deployment/smartagent-operator
# Webhook logs
kubectl -n bonree-agent logs -f deployment/bonree-webhook
# CSI-driver logs
kubectl -n bonree-agent logs -f daemonset/bonree-smartagent-csi-driver bonree-driver
kubectl -n bonree-agent logs -f daemonset/bonree-smartagent-csi-driver bonree-registrar
kubectl -n bonree-agent logs -f daemonset/bonree-smartagent-csi-driver bonree-liveness-probe
# SmartAgent Pod logs
kubectl -n bonree-agent logs -f {smartkube-instance-name}-smartagent-xxxxx
# install-smartagent sidecar container logs in application monitoring only mode
kubectl logs target-pod -c install-smartagent
CSI Driver cached SmartAgent Directory Structure
tree /var/lib/kubelet/plugins/csi.smartagent.bonree.com -L 4
.
|-- csi.sock
`-- data
|-- 6505b917-b1b4-11ed-a56c-a2f32a921294 // accountGUID
| |-- bin // Stores smartagent directories by version
| | `-- 8.5.0
| |-- gc // Mount directories per version
| | `-- 8.5.0
| |-- run // Runtime mount directories for each application Pod; changes inside container do not persist globally unless bin files are updated
| | |-- csi-44ff4cb776e917743698aa198f060f2ab7db617f9c614cdc66f621a07b61ee4e
| | |-- csi-a7703e2d35dab0a20b15eac8427022dcb86a5c82ae5fcf22d163fa63b3952641
| | |-- csi-e3ec6b87ef258fae36704029abab2b926a6f7f1e7ba737d5cddbbb466b37bb93
| | |-- csi-e8ead32dce314443e287c59c491cc4dc4cf4947c652c59276eb2bc8aaeadd300
| | `-- csi-edd52ef3ef48aecc52014b8d074feccc8ab2376a72fb502d557c9b4e6ad2e2bd
| `-- version (current version)
|-- gc
| |-- csi-a7703e2d35dab0a20b15eac8427022dcb86a5c82ae5fcf22d163fa63b3952641
| |-- csi-e3ec6b87ef258fae36704029abab2b926a6f7f1e7ba737d5cddbbb466b37bb93
| |-- csi-e8ead32dce314443e287c59c491cc4dc4cf4947c652c59276eb2bc8aaeadd300
| `-- csi-edd52ef3ef48aecc52014b8d074feccc8ab2376a72fb502d557c9b4e6ad2e2bd
`-- tenant-smartkube // Currently active accountGUID
13 directories, 7 files
Viewing Deployment and Injection Events
[root@lll-arm-openeuler ~]# kubectl get event -n bonree-agent
Events: <none> # Deployment exceptions, webhook serverless agent events, etc.
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Inject 9m53s (x18 over 93m) bonree-webhook Injecting the necessary info into pod nginx-deployment-56c88b7477- in namespace default