feat(log-analyser): transfered docker logs from docker containers and updated readme (#1884)

* feat(log-analyser): docker logs transfered and parsed from docker containers * feat(log-analyser): fixed regex patterns and updated Readme file
3 months ago · 1e589f3b71
parent 1f056b4c9d
commit 1e589f3b71
4 changed files with 96 additions and 59 deletions
--- a/docker-compose.yml
+++ b/docker-compose.yml
@ -12,6 +12,8 @@ services:
            - ${CONFIG}/web:/config:Z
            - ${CONFIG}/web/crontabs:/var/spool/cron/crontabs:Z
            - ${CONFIG}/transcripts:/usr/share/jitsi-meet/transcripts:Z
+        labels:
+            service: "jitsi-web"
        environment:
            - AMPLITUDE_ID
            - ANALYTICS_SCRIPT_URLS
@ -187,6 +189,8 @@ services:
            - '${PROSODY_S2S_PORT:-5269}'
            - '5347'
            - '${PROSODY_HTTP_PORT:-5280}'
+        labels:
+            service: "jitsi-prosody"
        volumes:
            - ${CONFIG}/prosody/config:/config:Z
            - ${CONFIG}/prosody/prosody-plugins-custom:/prosody-plugins-custom:Z
@ -322,6 +326,8 @@ services:
            - '127.0.0.1:${JICOFO_REST_PORT:-8888}:8888'
        volumes:
            - ${CONFIG}/jicofo:/config:Z
+        labels:
+            service: "jitsi-jicofo"
        environment:
            - AUTH_TYPE
            - BRIDGE_AVG_PARTICIPANT_STRESS
@ -406,6 +412,8 @@ services:
            - '127.0.0.1:${JVB_COLIBRI_PORT:-8080}:8080'
        volumes:
            - ${CONFIG}/jvb:/config:Z
+        labels:
+            service: "jitsi-jvb"
        environment:
            - AUTOSCALER_SIDECAR_KEY_FILE
            - AUTOSCALER_SIDECAR_KEY_ID
--- a/log-analyser.yml
+++ b/log-analyser.yml
@ -18,9 +18,11 @@ services:
  otel-collector:
    container_name: otel
    image: otel/opentelemetry-collector-contrib
+    user: "0" # required for reading docker container logs
    volumes:
      - ./log-analyser/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
      - ./log-analyser/jitsi-logs/:/tmp/jitsi-logs/
+      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      meet.jitsi:
--- a/log-analyser/README.md
+++ b/log-analyser/README.md
@ -5,13 +5,13 @@ Currently this is an in-progress GSoC Summer of Code project and so the instruct

 ## Overview

-This project demonstrates how to configure and use Grafana Loki with OpenTelemetry to collect, parse, and visualize log data. It includes:
+This project demonstrates how to configure and use Grafana Loki with OpenTelemetry to collect, parse, and visualize log data from Jitsi Meet components. It includes:

 - A Docker Compose setup (`log-analyser.yml`) for Loki and OpenTelemetry Collector.
 - A Docker Compose setup (`grafana.yml`) for Grafana.
- Configuration files for log parsing and exporting.
+- A unified Docker Compose command to start all services.
 - Instructions to set up and access Grafana with Loki as a data source.
- 
+
 ## Getting Started

 ### Prerequisites
@ -23,39 +23,39 @@ This project demonstrates how to configure and use Grafana Loki with OpenTelemet

 1. **Clone the repository:**

-    ```bash
-    git clone https://github.com/jitsi/docker-jitsi-meet.git
-    ```
-
-### Log Analyser
+```bash
+git clone https://github.com/jitsi/docker-jitsi-meet.git
+```

-1. **Add your log files:**
+2. **Update Jitsi Meet Docker Compose Configuration:**

-   Place your log file in the `log-analyser/jitsi-logs` directory. Update the `otel-collector-config.yaml` file with the correct file path to start ingesting the logs. This setup allows OpenTelemetry to read logs from the file and forward them to Loki.
+To enable log collection and analysis, you need to modify the `docker-compose.yml` file for Jitsi Meet components. Add the following configuration to each Jitsi service within the `docker-compose.yml` file:

-2. **Update the otel-collector-config.yaml file:**
+```yaml
+    logging:
+      driver: "json-file"
+      options:
+        labels: "service"
+```

-   Update the file path to point to your log file for ingestion by OpenTelemetry.
+This configuration ensures that logs are collected in JSON format and tagged with service labels, which is essential for Loki to properly ingest and index the logs.

 3. **Start the Docker containers:**

-   ```bash
-   docker-compose -f docker-compose.yml -f log-analyser.yml up -d
-    ```
-
-### Grafana
+   To start all necessary services, including Jitsi Meet components, Grafana, Loki, and OpenTelemetry, run:

-1. **Start the Docker container:**
-
-   ```bash
-   docker-compose -f docker-compose.yml -f grafana.yml up -d
+    ```bash
+    docker-compose -f docker-compose.yml -f log-analyser.yml -f grafana.yml up -d
    ```

-2. **Access Grafana:**
+   - This command will start the Jitsi Meet components from `docker-compose.yml`, the log analysis tools from `log-analyser.yml`, and Grafana from `grafana.yml`. The logs from Jitsi Meet components will automatically be sent to Grafana through Loki.
+   - **Note:** To use only Grafana for visualization without log analysis, you can use just `grafana.yml` alone. However, for the complete log analysis project, you need both `log-analyser.yml` and `grafana.yml`.
+
+### Access Grafana

-   Open your web browser and navigate to [http://localhost:3000](http://localhost:3000).
+1. **Open your web browser and navigate to [http://localhost:3000](http://localhost:3000).**

-3. **Log in to Grafana:**
+2. **Log in to Grafana:**

   Use the default credentials:

@ -64,16 +64,17 @@ This project demonstrates how to configure and use Grafana Loki with OpenTelemet
    Password: admin
    ```

-### Dashboard Setup
-
-The dashboard setups are available as JSON files in the `log-analyser/grafana-dashboards` directory. You can import these JSON files into Grafana to use the pre-configured dashboards. In the future, we plan to automate this import process.
+3. **Dashboard Setup:**

+   The dashboards for Jitsi Meet components are pre-configured and will be automatically available in Grafana. You can explore these dashboards to view and analyze logs.

 ## Usage

- **Log Parsing and Visualization:** After setting up, you can use Grafana to explore and visualize your logs. Check our dashboards and panels to monitor log data effectively.
-
+- **Log Parsing and Visualization:** After setting up, use Grafana to explore and visualize your logs. Check the pre-configured dashboards and panels to monitor and analyze log data from Jitsi Meet components effectively.

 ## Acknowledgements

-Thanks for checking out this project! If you have any questions or need further assistance, don't hesitate to reach out.
+Thank you for exploring this project!
+For detailed documentation, follow the [Jitsi Handbook](https://jitsi.github.io/handbook/docs/intro), you can follow the Docker and Log-Analyser guides under Self-Hosting Guide > Deployment guide.
+
+If you have any questions or need further assistance, please feel free to reach out.
--- a/log-analyser/otel-collector-config.yaml
+++ b/log-analyser/otel-collector-config.yaml
@ -4,26 +4,64 @@ receivers:
      http:
      grpc:
        endpoint: 0.0.0.0:4317
-  filelog/jicofo:
-    include: ['/tmp/jitsi-logs/jicofo.log']
+  filelog/jitsi-containers:
+    include: ['/var/lib/docker/containers/*/*.log']
+    encoding: utf-8
    operators:
+      - type: json_parser
+        id: parser-docker
+        output: filter_non_tagged_containers
+        timestamp:
+          parse_from: attributes.time
+          layout: '%Y-%m-%dT%H:%M:%S.%LZ'
+      - type: filter
+        id: filter_non_tagged_containers
+        expr: |
+          (attributes?.attrs?.service != "jitsi-web" and 
+          attributes?.attrs?.service != "jitsi-jicofo" and 
+          attributes?.attrs?.service != "jitsi-jvb" and
+          attributes?.attrs?.service != "jitsi-prosody")
+        output: regex_parser_choice
+      - type: router
+        id: regex_parser_choice
+        routes:
+          - expr: 'attributes.attrs.service == "jitsi-web"'
+            output: jitsi_web_parser
+          - expr: 'attributes.attrs.service == "jitsi-jicofo"'
+            output: jitsi_jicofo_parser
+          - expr: 'attributes.attrs.service == "jitsi-jvb"'
+            output: jitsi_jvb_parser
+          - expr: 'attributes.attrs.service == "jitsi-prosody"'
+            output: jitsi_prosody_parser
      - type: regex_parser
-        regex: "^(?P<app>Jicofo) (?P<timestamp>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}\\.\\d{3}) (?P<level>\\w+): \\[(?P<pid>\\d+)\\] (?P<codefile>[\\w\\.]+)#(?P<codeline>\\d+): (?P<message>[\\S\\s]*)$"
-  filelog/jvb:
-    include: ['/tmp/jitsi-logs/jvb.log']
-    operators:
+        id: jitsi_web_parser
+        parse_from: attributes.log
+        regex: "^(?P<message>[\\s\\S]*)$|^(\\[(?P<temp_meta>[\\w\\W]+)\\] (?P<temp_message>[\\S\\s]*)$)"
      - type: regex_parser
-        regex: "^(?P<app>JVB) (?P<timestamp>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}\\.\\d{3}) (?P<level>\\w+): \\[(?P<pid>\\d+)\\] (?P<codefile>[\\w\\.]+)#(?P<codeline>\\d+): (?P<message>[\\S\\s]*)$"
-  filelog/prosody:
-    include: ['/tmp/jitsi-logs/prosody.log']
-    operators:
+        id: jitsi_jicofo_parser
+        parse_from: attributes.log
+        regex: "(^(?P<app>\\w+) (?P<timestamp>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}\\.\\d{3}) (?P<level>\\w+): \\[(?P<pid>\\d+)\\]( \\[(?P<meta>[\\w\\W]+)\\])? (?P<codefile>[\\w\\W]+)(#(?P<codeline>\\d+))?: (?P<message>[\\s\\S]*)$)|^(\\[(?P<temp_meta>[\\w\\W]+)\\] (?P<temp_message>[\\S\\s]*)$)"
+        output: move_meta_key
      - type: regex_parser
-        regex: "^(?P<timestamp>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2})\\s+(?P<service>\\S+)\\s+(?P<level>\\w+)\\s+(?P<message>.+)$"
-  filelog/jitsi-web:
-    include: ['/tmp/jitsi-logs/jitsi-web.log']
-    operators:
+        id: jitsi_jvb_parser
+        parse_from: attributes.log
+        regex: "^(?P<app>JVB)  \\[(?P<pid>\\d+)\\] (?P<codefile>[\\w\\.]+)#(?P<codeline>\\d+): (?P<message>[\\S\\s]*)$"
      - type: regex_parser
-        regex: "^(?P<message>\\[\\w+-\\w+\\.\\w+\\]\\s+(\\w+\\s+)*\\w+:?\\s*.*$)"
+        id: jitsi_prosody_parser
+        parse_from: attributes.log
+        regex: (^(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})\s+(?<service>\S+)\s+(?<level>\w{0,10})(\t)(?P<message>[\s\S]*)$)|^(\[(?P<temp_meta>[\w\W]+)\] (?P<temp_message>[\S\s]*)$)
+        output: move_meta_key
+      - type: move
+        id: move_meta_key
+        if: "attributes.temp_meta != nil and attributes.temp_meta != ''"
+        from: attributes.temp_meta
+        to: attributes.meta
+        output: move_message_key
+      - type: move
+        id: move_message_key
+        if: "attributes.temp_message != nil and attributes.temp_message != ''"
+        from: attributes.temp_message
+        to: attributes.message
  docker_stats:
    endpoint: "unix:///var/run/docker.sock"
 processors:
@ -39,20 +77,8 @@ exporters:

 service:
  pipelines:
-    logs/jicofo:
-      receivers: [otlp, filelog/jicofo]
-      processors: [batch]
-      exporters: [loki]
-    logs/jvb:
-      receivers: [otlp, filelog/jvb]
-      processors: [batch]
-      exporters: [loki]
-    logs/prosody:
-      receivers: [otlp, filelog/prosody]
-      processors: [batch]
-      exporters: [loki]
-    logs/jitsi-web:
-      receivers: [otlp, filelog/jitsi-web]
+    logs:
+      receivers: [otlp, filelog/jitsi-containers]
      processors: [batch]
      exporters: [loki]
    metrics: