skywalking-java/docs/en/Collector-Table-Description.md

Collector Table Description

This document describes the usage of tables and their columns, based on elasticsearch storage implementation.

Metric table time bucket

Date format

• second: yyyyMMddHHmmss
• minute: yyyyMMddHHmm
• hour: yyyyMMddHH
• day: yyyyMMdd
• month: yyyyMM

Tables of Register related

Application

• Table name: application
• Get or create a database record by "application_code".

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, the value same as application_id
application_code	ac	Keyword	The application name, see agent.config
application_id	ai	Integer	Auto increment, is a signed integer
is_address	ia	Integer	Is a boolean data. True(1), False(0)
address_id	ni	Integer	A foreign key reference by network_address table

• Column is_address
  • false. A real application, which has a custom application_code. At the same time, the address_id column value must to be 0.
  • true. A conjunction application based on IP address. address_id is registered in network_address table.

Instance

• Table name: instance
• Create a instance by application_id and agent_uuid

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, the value same as instance_id
application_id	ai	Integer	Owner application id
application_code	ac	Text	Owner application code
agent_uuid	iau	Keyword	Uniquely identifies each server monitored by agent
register_time	irt	Long	First register time
instance_id	ii	Integer	Auto increment, is a unsigned integer
heartbeat_time	iht	Long	Represent server is alive
os_info	ioi	Text	A Json data.
is_address	iia	Integer	Is a boolean data. True(1), False(0)
address_id	ni	Integer	A foreign key reference by network_address table

• Column os_info
  • For example: {"osName":"MacOS X","hostName":peng-yongsheng","processId":1000,"ipv4s":["10.0.0.1","10.0.0.2"]}
• Column heartbeat_time
  • Updated by agent heart beat [1]
  • Updated by JVM metric data [2]
  • Updated by trace segment data. [3]
  • Priority: [1] > [2] > [3]

NetworkAddress

• Table name: network_address
• Create a network address record by "network_address" and "span_layer".

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, the value same as address_id
address_id	ni	Integer	Auto increment, is a signed integer
network_address	na	Keyword	Host name or IP address
src_span_layer	ssl	Integer	Register by client or server side
server_type	st	Integer	Such as component id, used for topology.

ServiceName

• Table name: service_name
• Create a service record by "service_name_keyword", "application_id" and "src_span_type".

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, the value same as service_id
service_id	si	Integer	Auto increment, is a signed integer
service_name	sn	Text	Operation name, used for fuzzy matching
service_name_keyword	snk	Keyword	Operation name, used for full matching
application_id	ai	Integer	Owner application id
src_span_type	sst	Integer	Register from client or server side based on src_span_type

• See src_span_type in protocol doc

Table of Trace Metric related

ApplicationComponent

• Table name: application_component_TimeDimension
• TimeDimension contains minute, hour, day, month
• It is primarily used for the view of node type in application topology.

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, time_bucket_metric_id
metric_id	mi	Keyword	application_id_component_id
component_id	ci	Integer	Component id
application_id	ai	Integer	Owner application id
time_bucket	tb	Long	Date format

ApplicationMapping

• Table name: application_mapping_TimeDimension
• TimeDimension contains minute, hour, day, month
• For example: Application A calls Application B, collector generates the following two metrics::
  • From the caller's trace data: A application -> B application's IP address (Topology will use this metric when B application is not monitored by agent)
  • From the callee's trace data: A application -> B application (Topology will use this metric when B application is monitored by agent)

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, time_bucket_metric_id
metric_id	mi	Keyword	application_id_mapping_application_id
application_id	ai	Integer	Registered at server side.
mapping_application_id	mai	Integer	Registered at client side with the server's IP address.
time_bucket	tb	Long	Date format

ApplicationMetric

• Table name: application_metric_TimeDimension
• TimeDimension contains minute, hour, day, month

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, time_bucket_metric_id
metric_id	mi	Keyword	application_id_source_value
application_id	ai	Integer	Owner application id
source_value	sv	Integer	Caller(0), Callee(1)
transaction_calls	t1	Long	The total number of calls based on time_bucket
transaction_error_calls	t2	Long	The total number of error calls, sums values aggregate by time_bucket
transaction_duration_sum	t3	Long	The total duration of calls based on time_bucket
transaction_error_duration_sum	t4	Long	The total duration of error calls based on time_bucket
transaction_average_duration	t5	Long	The average duration of all calls, used for order by this column in database.
business_transaction_calls	b1	Long
business_transaction_error_calls	b2	Long
business_transaction_duration_sum	b3	Long
business_transaction_error_duration_sum	b4	Long
business_transaction_average_duration	b5	Long
mq_transaction_calls	m1	Long
mq_transaction_error_calls	m2	Long
mq_transaction_duration_sum	m3	Long
mq_transaction_error_duration_sum	m4	Long
mq_transaction_average_duration	m5	Long
satisfied_count	a1	Long	The formula
tolerating_count	a2	Long
frustrated_count	a3	Long
time_bucket	tb	Long	Date format

ApplicationReferenceMetric

• Table name: application_reference_metric_TimeDimension
• TimeDimension contains minute, hour, day, month

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, time_bucket_metric_id
metric_id	mi	Keyword	front_application_idbehind_application_idsource_value
front_application_id	fai	Integer
behind_application_id	bai	Integer
source_value	sv	Integer	Caller(0), Callee(1)
transaction_calls	t1	Long	The total number of calls based on time_bucket
transaction_error_calls	t2	Long	The total number of error calls, sums values aggregate by time_bucket
transaction_duration_sum	t3	Long	The total duration of calls based on time_bucket
transaction_error_duration_sum	t4	Long	The total duration of error calls based on time_bucket
transaction_average_duration	t5	Long	The average duration of all calls, used for order by this column in database.
business_transaction_calls	b1	Long
business_transaction_error_calls	b2	Long
business_transaction_duration_sum	b3	Long
business_transaction_error_duration_sum	b4	Long
business_transaction_average_duration	b5	Long
mq_transaction_calls	m1	Long
mq_transaction_error_calls	m2	Long
mq_transaction_duration_sum	m3	Long
mq_transaction_error_duration_sum	m4	Long
mq_transaction_average_duration	m5	Long
satisfied_count	a1	Long	The formula
tolerating_count	a2	Long
frustrated_count	a3	Long
time_bucket	tb	Long	Date format

InstanceMapping

• Table name: instance_mapping_TimeDimension
• TimeDimension contains minute, hour, day, month

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, time_bucket_metric_id
metric_id	mi	Keyword	instance_id_address_id
application_id	ai	Integer	The instance_id's owner application id.
instance_id	ii	Integer	Registered at server side.
address_id	ni	Integer	Registered at client side with the server's IP address.
time_bucket	tb	Long	Date format

InstanceMetric

• Table name: instance_metric_TimeDimension
• TimeDimension contains minute, hour, day, month

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, time_bucket_metric_id
metric_id	mi	Keyword	instance_id_source_value
application_id	ai	Integer
instance_id	ii	Integer
source_value	sv	Integer	Caller(0), Callee(1)
transaction_calls	t1	Long	The total number of calls based on time_bucket
transaction_error_calls	t2	Long	The total number of error calls, sums values aggregate by time_bucket
transaction_duration_sum	t3	Long	The total duration of calls based on time_bucket
transaction_error_duration_sum	t4	Long	The total duration of error calls based on time_bucket
transaction_average_duration	t5	Long	The average duration of all calls, used for order by this column in database.
business_transaction_calls	b1	Long
business_transaction_error_calls	b2	Long
business_transaction_duration_sum	b3	Long
business_transaction_error_duration_sum	b4	Long
business_transaction_average_duration	b5	Long
mq_transaction_calls	m1	Long
mq_transaction_error_calls	m2	Long
mq_transaction_duration_sum	m3	Long
mq_transaction_error_duration_sum	m4	Long
mq_transaction_average_duration	m5	Long
time_bucket	tb	Long	Date format

InstanceReferenceMetric

• Table name: instance_reference_metric_TimeDimension
• TimeDimension contains minute, hour, day, month

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, time_bucket_metric_id
metric_id	mi	Keyword	front_instance_idbehind_instance_idsource_value
front_application_id	fai	Integer
behind_application_id	bai	Integer
front_instance_id	fii	Integer
behind_instance_id	bii	Integer
source_value	sv	Integer	Caller(0), Callee(1)
transaction_calls	t1	Long	The total number of calls based on time_bucket
transaction_error_calls	t2	Long	The total number of error calls, sums values aggregate by time_bucket
transaction_duration_sum	t3	Long	The total duration of calls based on time_bucket
transaction_error_duration_sum	t4	Long	The total duration of error calls based on time_bucket
transaction_average_duration	t5	Long	The average duration of all calls, used for order by this column in database.
business_transaction_calls	b1	Long
business_transaction_error_calls	b2	Long
business_transaction_duration_sum	b3	Long
business_transaction_error_duration_sum	b4	Long
business_transaction_average_duration	b5	Long
mq_transaction_calls	m1	Long
mq_transaction_error_calls	m2	Long
mq_transaction_duration_sum	m3	Long
mq_transaction_error_duration_sum	m4	Long
mq_transaction_average_duration	m5	Long
time_bucket	tb	Long	Date format

ServiceMetric

• Table name: service_metric_TimeDimension
• TimeDimension contains minute, hour, day, month

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, time_bucket_metric_id
metric_id	mi	Keyword	service_id_source_value
application_id	ai	Integer
instance_id	ii	Integer
service_id	si	Integer
source_value	sv	Integer	Caller(0), Callee(1)
transaction_calls	t1	Long	The total number of calls based on time_bucket
transaction_error_calls	t2	Long	The total number of error calls, sums values aggregate by time_bucket
transaction_duration_sum	t3	Long	The total duration of calls based on time_bucket
transaction_error_duration_sum	t4	Long	The total duration of error calls based on time_bucket
transaction_average_duration	t5	Long	The average duration of all calls, used for order by this column in database.
business_transaction_calls	b1	Long
business_transaction_error_calls	b2	Long
business_transaction_duration_sum	b3	Long
business_transaction_error_duration_sum	b4	Long
business_transaction_average_duration	b5	Long
mq_transaction_calls	m1	Long
mq_transaction_error_calls	m2	Long
mq_transaction_duration_sum	m3	Long
mq_transaction_error_duration_sum	m4	Long
mq_transaction_average_duration	m5	Long
time_bucket	tb	Long	Date format

ServiceReferenceMetric

• Table name: service_reference_metric_TimeDimension
• TimeDimension contains minute, hour, day, month

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, time_bucket_metric_id
metric_id	mi	Keyword	front_service_idbehind_service_idsource_value
front_application_id	fai	Integer
front_instance_id	fii	Integer
front_service_id	fsi	Integer
behind_application_id	bai	Integer
behind_instance_id	bii	Integer
behind_service_id	bsi	Integer
source_value	sv	Integer	Caller(0), Callee(1)
transaction_calls	t1	Long	The total number of calls based on time_bucket
transaction_error_calls	t2	Long	The total number of error calls, sums values aggregate by time_bucket
transaction_duration_sum	t3	Long	The total duration of calls based on time_bucket
transaction_error_duration_sum	t4	Long	The total duration of error calls based on time_bucket
transaction_average_duration	t5	Long	The average duration of all calls, used for order by this column in database.
business_transaction_calls	b1	Long
business_transaction_error_calls	b2	Long
business_transaction_duration_sum	b3	Long
business_transaction_error_duration_sum	b4	Long
business_transaction_average_duration	b5	Long
mq_transaction_calls	m1	Long
mq_transaction_error_calls	m2	Long
mq_transaction_duration_sum	m3	Long
mq_transaction_error_duration_sum	m4	Long
mq_transaction_average_duration	m5	Long
time_bucket	tb	Long	Date format

GlobalTrace

• Table name: global_trace
• The relationship between trace id and segment id is many to many.

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, global_trace_id_segment_id
segment_id	sgi	Keyword
trace_id	ti	Keyword
time_bucket	tb	Long	Second date format, Date format

SegmentDuration

• Table name: segment_duration
• The relationship between trace id and segment id is many to many.

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, global_trace_id_segment_id
segment_id	sgi	Keyword
trace_id	ti	Keyword
application_id	ai	Integer	Owner of the segment
service_name	sn	Text	The entry span's operation name in this segment
duration	ddt	Long	The cost duration of this segment
start_time	dst	Long
end_time	det	Long
is_error	die	Long	Is a boolean data. True(1), False(0)
time_bucket	tb	Long	Second date format, Date format

Segment

• Table name: segment

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, the value same as segment_id
data_binary	sdb	Binary	The protobuf segment binary encode by base64 scheme.
time_bucket	tb	Long	Second date format, Date format

Tables of JVM Metric related

CpuMetric

• Table name: cpu_metric_TimeDimension
• TimeDimension contains second, minute, hour, day, month

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, the value is time_bucket_metric_id
metric_id	mi	Keyword	the value is instance_id
instance_id	ii	Integer	Owner instance id
usage_percent	up	Double	Cpu usage percent, sums values aggregate by time_bucket
times	t	Long	The records received times in this time bucket
time_bucket	tb	Long	Date format

GCMetric

• Table name: gc_metric_TimeDimension
• TimeDimension contains second, minute, hour, day, month

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, the value is time_bucket_metric_id
metric_id	mi	Keyword	the value is instance_id_phrase
instance_id	ii	Integer	Owner instance id
phrase	p	Integer	GCPhrase
count	c	Long	GC count, sums values aggregate by time_bucket
times	t	Long	The records received times in this time bucket
duration	d	Long	GC duration count, sums values aggregate by time_bucket
time_bucket	tb	Long	A formatted date

MemoryMetric

• Table name: memory_metric_TimeDimension
• TimeDimension contains second, minute, hour, day, month

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, the value is time_bucket_metric_id
metric_id	mi	Keyword	the value is instance_id_is_heap
instance_id	ii	Integer	Owner instance id
is_heap	ih	Integer	Is a boolean data. True(1), False(0)
init	init	Long	Sums values aggregate by time_bucket
max	max	Long	Sums values aggregate by time_bucket
used	used	Long	Sums values aggregate by time_bucket
committed	cd	Long	Sums values aggregate by time_bucket
times	t	Long	The records received times in this time bucket
time_bucket	tb	Long	A formatted date

MemoryPoolMetric

• Table name: memory_pool_metric_TimeDimension
• TimeDimension contains second, minute, hour, day, month

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, the value is time_bucket_metric_id
metric_id	mi	Keyword	the value is instance_id_pool_type
instance_id	ii	Integer	Owner instance id
pool_type	pt	Integer	Owner instance id
init	init	Long	Sums values aggregate by time_bucket
max	max	Long	Sums values aggregate by time_bucket
used	used	Long	Sums values aggregate by time_bucket
committed	ct	Long	Sums values aggregate by time_bucket
times	t	Long	The records received times in this time bucket
time_bucket	tb	Long	A formatted date

Table of Alarm metric related

ApplicationAlarm

• Table name: application_alarm

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, the value is source_valuealarm_typeapplication_id
application_id	ai	Integer	Owner application id
source_value	sv	Integer	Caller(0), Callee(1)
alarm_type	aat	Integer	ERROR_RATE(0), SLOW_RTT(1)
alarm_content	aac	Text
last_time_bucket	ltb	Long	Second date format. A formatted date

ApplicationAlarmList

• Table name: application_alarm_list
• TimeDimension contains minute, hour, day, month

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, the value is time_bucket_metric_id
metric_id	mi	Keyword	source_valuealarm_typeapplication_id
alarm_content	aac	Text
application_id	ai	Integer	Owner application id
source_value	sv	Integer	Caller(0), Callee(1)
alarm_type	aat	Integer	ERROR_RATE(0), SLOW_RTT(1)
time_bucket	tb	Long	A formatted date

ApplicationReferenceAlarm

• Table name: application_reference_alarm

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, the value is source_valuealarm_typeapplication_id
front_application_id	fai	Integer	Owner application id
behind_application_id	bai	Integer	Owner application id
source_value	sv	Integer	Caller(0), Callee(1)
alarm_type	aat	Integer	ERROR_RATE(0), SLOW_RTT(1)
alarm_content	aac	Text
last_time_bucket	ltb	Long	Second date format. A formatted date

ApplicationReferenceAlarmList

• Table name: application_reference_alarm

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, the value is source_valuealarm_typeapplication_id
front_application_id	fai	Integer	Owner application id
behind_application_id	bai	Integer	Owner application id
source_value	sv	Integer	Caller(0), Callee(1)
alarm_type	aat	Integer	ERROR_RATE(0), SLOW_RTT(1)
alarm_content	aac	Text
time_bucket	tb	Long	Second date format. A formatted date

InstanceAlarm

• Table name: application_alarm

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, the value is source_valuealarm_typeinstance_id
application_id	ai	Integer	Owner application id
instance_id	ii	Integer	Owner instance id
source_value	sv	Integer	Caller(0), Callee(1)
alarm_type	aat	Integer	ERROR_RATE(0), SLOW_RTT(1)
alarm_content	aac	Text
last_time_bucket	ltb	Long	Second date format. A formatted date

InstanceAlarmList

• Table name: instance_alarm_list

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, the value is source_valuealarm_typeinstance_id
application_id	ai	Integer	Owner application id
instance_id	ii	Integer	Owner instance id
source_value	sv	Integer	Caller(0), Callee(1)
alarm_type	aat	Integer	ERROR_RATE(0), SLOW_RTT(1)
alarm_content	aac	Text
time_bucket	tb	Long	Second date format. A formatted date

InstanceReferenceAlarm

• Table name: instance_reference_alarm

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, the value is source_valuealarm_typeinstance_id
front_application_id	fai	Integer	Owner application id
front_instance_id	fii	Integer	Owner instance id
behind_application_id	bai	Integer	Owner instance id
behind_instance_id	bii	Integer	Owner instance id
source_value	sv	Integer	Caller(0), Callee(1)
alarm_type	aat	Integer	ERROR_RATE(0), SLOW_RTT(1)
alarm_content	aac	Text
last_time_bucket	ltb	Long	Second date format. A formatted date

InstanceReferenceAlarmList

• Table name: instance_reference_alarm_list

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, the value is source_valuealarm_typeinstance_id
front_application_id	fai	Integer	Owner application id
front_instance_id	fii	Integer	Owner instance id
behind_application_id	bai	Integer	Owner instance id
behind_instance_id	bii	Integer	Owner instance id
source_value	sv	Integer	Caller(0), Callee(1)
alarm_type	aat	Integer	ERROR_RATE(0), SLOW_RTT(1)
alarm_content	aac	Text
time_bucket	tb	Long	Second date format. A formatted date

ServiceAlarm

• Table name: service_alarm

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, the value is source_valuealarm_typeservice_id
application_id	ai	Integer	Owner application id
instance_id	ii	Integer	Owner instance id
service_id	si	Integer	Owner service id
source_value	sv	Integer	Caller(0), Callee(1)
alarm_type	aat	Integer	ERROR_RATE(0), SLOW_RTT(1)
alarm_content	aac	Text
last_time_bucket	ltb	Long	Second date format. A formatted date

ServiceAlarmList

• Table name: service_alarm_list

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, the value is source_valuealarm_typeservice_id
application_id	ai	Integer	Owner application id
instance_id	ii	Integer	Owner instance id
service_id	si	Integer	Owner instance id
source_value	sv	Integer	Caller(0), Callee(1)
alarm_type	aat	Integer	ERROR_RATE(0), SLOW_RTT(1)
alarm_content	aac	Text
time_bucket	tb	Long	Second date format. A formatted date

ServiceReferenceAlarm

• Table name: service_reference_alarm

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, the value is source_valuealarm_typeservice_id
front_application_id	fai	Integer	Owner application id
front_instance_id	fii	Integer	Owner instance id
front_service_id	fsi	Integer	Owner service id
behind_application_id	bai	Integer	Owner service id
behind_instance_id	bii	Integer	Owner service id
behind_service_id	bsi	Integer	Owner service id
source_value	sv	Integer	Caller(0), Callee(1)
alarm_type	aat	Integer	ERROR_RATE(0), SLOW_RTT(1)
alarm_content	aac	Text
last_time_bucket	ltb	Long	Second date format. A formatted date

ServiceReferenceAlarmList

• Table name: service_reference_alarm_list

Column Name	Short Name	Data Type	Description
_id	_id	Keyword	primary key, es speciality, the value is time_bucketsource_valuealarm_type_service_id
front_application_id	fai	Integer	Owner application id
front_instance_id	fii	Integer	Owner instance id
front_service_id	fsi	Integer	Owner service id
behind_application_id	bai	Integer	Owner service id
behind_instance_id	bii	Integer	Owner service id
behind_service_id	bsi	Integer	Owner service id
source_value	sv	Integer	Caller(0), Callee(1)
alarm_type	aat	Integer	ERROR_RATE(0), SLOW_RTT(1)
alarm_content	aac	Text
time_bucket	tb	Long	Second date format. A formatted date