User Guides

In the Grafana monitoring tool, a list of all dashboards is available, but data access on some of them may be restricted depending on the user's permissions. In this case, the graphs will display "No data" and hovering over the alert icon in the top left corner of the graph will indicate "Forbidden".

Storage & Dashboards

Logs are stored in database kosmos. There are three tables:

• kosmos.kosmos_logs: stores logs from applications and workloads running in kubernetes
• kosmos.kosmos_audit: stores security related logs from applications and workloads running in kubernetes.
• kosmos.audit_iaas: stores logs from hosts/VM

Tables schema

For tables kosmos.kosmos_logs and kosmos.kosmos_audit

Column	Description
kubernetes_container_image	Name of the container image
kubernetes_container_image_id	The id of the image
kubernetes_container_name	The name of the container. Acts as the name of the application
kubernetes_pod_name	The name of the pod
kubernetes_pod_namespace	The name of the namespace the pod is deployed in
kubernetes_pod_node_name	The name of the node running the pod
kubernetes_pod_uid	UUI of the pod
kubernetes_pod_labels	Labels of the pods
timestamp	The timestamp of the log
log_level	Always set to AUDIT. The actual log level is in the message column
tenant_id	The id of the tenant
message	The content of the actual log. Most of the time, it is a JSON string, but can also be an unstructured string

For table kosmos.audit_iaas

Column	Description
host	Name of the host the logs come from
source	The source of the logs: file path, journald, etc. Example: /var/log/audit/audit.log
source_type	Deprecated column. To be removed in the future
timestamp	The timestamp of the log. Most of the time, it is the scrapping time, not the generation time
log_level	Always set to AUDIT
message	Structured or Unstructured string. It's the content of the actual log

How to create dashboards

The dashboard that come with the system can not be edited. If needed, new dashboards can be added to meet the needs. There are two ways to do so:

• Solution 1: Export an existing dashboard. Then import it back while changing the dashboard name and Id. Then edit the dashboards as needed. That's the easiest and recommended way to proceeded.
• Solution 2: Add a new dashboard from scratch. Go down this path only if you are comfortable with how grafana works. Even in this case, looking at existing dashboards will still provide useful insights on how to proceed.

Sample of queries

Query to create a Pie Chart visualization for Falco Events by priority

SELECT
  JSONExtractString(message, 'priority') as priority,
  count(*)  as `OTHER`
FROM
  kosmos.audit_iaas
WHERE
    $timeFilter AND
    message ILIKE '%${search:raw}%' AND
    message ILIKE '%${search2:raw}%' AND
    ( $host == 'All' OR host IN (${host:sqlstring}) )AND
    ( $rule == 'All' OR JSONExtractString(message, 'rule') IN (${rule:sqlstring})) AND
    source='/var/log/falco/falco.log' AND
    ( $user == 'All' OR JSONExtractString(message, 'output_fields', 'user.name') IN (${user:sqlstring}) ) AND
    ( $process == 'All' OR JSONExtractString(message, 'output_fields', 'proc.name') IN (${process:sqlstring}))  AND
    ( $tag == 'All' OR arrayExists(
      tag -> tag IN (${tag:sqlstring}),
      JSONExtractArrayRaw(message, 'tags')

{search:raw} , {rule:sqlstring}, etc. are variables.

Query to create a time series visualisation for App Armor events

Select allOperationTable.operation as `Tout`, timeTable.time
  FROM(
WITH
  toStartOfHour(toDateTime($from)) AS start,
  toStartOfHour(timestamp_add(toDateTime($to), INTERVAL 1 HOUR)) AS end
SELECT
  0 as operation,
  arrayJoin(arrayMap(x -> toDateTime(x, $__timezone), range(toUInt32(start), toUInt32(end), 3600))) as time
) as timeTable
left join
(
  SELECT
  count() AS operation,
  toStartOfHour(timestamp) AS time
  FROM
    kosmos.audit_iaas
  WHERE
    $timeFilter AND
    source='/var/log/audit/audit.log' AND
    ('All' IN (${host}) OR host IN (${host:sqlstring}) ) AND
    ('All' IN (${type}) OR extractKeyValuePairs(message, '=', ', ')['type'] IN (${type:sqlstring})) AND
    ('All' IN (${status}) OR extract(message, 'apparmor="([^"]+)"') IN (${status:sqlstring}) ) AND
    message ILIKE '%apparmor%' AND
    message ILIKE '%${search:raw}%' AND
    message ILIKE '%${search2:raw}%'
  GROUP BY time
) AS allOperationTable
ON timeTable.time = allOperationTable.time

The first part of the query ( before left join) makes sure that 0 is used as value is no event is found for the given interval (One Hour in this case).

The second part of the query computes the actual amount of events per interval.

Query to create a log visualisation for gitlab events

SELECT
  timestamp, message
FROM
  kosmos.kosmos_audit
where
  $timeFilter AND
  kubernetes_container_name == 'webservice' AND
  message ILIKE '%${search:raw}%' AND
  ('$user' == 'Tout' OR message LIKE '%"user":"$user"%') AND
  ('$action' ==  'Tout' OR message LIKE '%"action":"$action"%' ) AND
  ('$object' ==  'Tout' OR message ILIKE '%${object}Controller"%' ) AND
  ('$remote_ip' ==  'Tout' OR message LIKE '%"remote_ip":"$remote_ip"%' )
LIMIT 1000