Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-3933

Fluentd pods are in CrashLoopBackOff status when the cluster has FIPS enabled.

XMLWordPrintable

    • False
    • None
    • False
    • NEW
    • NEW
    • Before this change, the collector was unable to read private certificate keys on FIPS enabled clusters. This change updates the OpenSSL gem to allow reading private certificates
    • Bug Fix
    • Log Collection - Sprint 234, Log Collection - Sprint 235, Log Collection - Sprint 244, Log Collection - Sprint 245, Log Collection - Sprint 246, Log Collection - Sprint 247, Log Collection - Sprint 250, Log Collection - Sprint 251

      Description of problem:

      Deploy logging 5.6.5 on a cluster which has enabled FIPS, then check pods' status. When using fluentd as the collector, collector pods can't start and stuck in CrashLoopBackOff status:

      $ oc get pod
NAME                                            READY   STATUS             RESTARTS        AGE
cluster-logging-operator-75f94b5648-tr9x9       1/1     Running            0               16m
collector-4rt5r                                 1/2     CrashLoopBackOff   6 (4m29s ago)   14m
collector-g7864                                 1/2     CrashLoopBackOff   7 (2m56s ago)   14m
collector-kdbxr                                 1/2     CrashLoopBackOff   7 (29s ago)     14m
collector-wkgfl                                 1/2     CrashLoopBackOff   7 (2m48s ago)   14m
collector-wtq6x                                 2/2     Running            7 (8m2s ago)    14m
collector-x8k8q                                 1/2     CrashLoopBackOff   7 (2m59s ago)   14m

      And raise below errors:

      2023-04-07 07:35:06 +0000 [warn]: For security reason, setting private_key_passphrase is recommended when cert_path is specified
2023-04-07 07:35:06 +0000 [error]: unexpected error error_class=OpenSSL::PKey::PKeyError error="Could not parse PKey"
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/cert_option.rb:89:in `read'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/cert_option.rb:89:in `cert_option_load'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/cert_option.rb:65:in `cert_option_server_validate!'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/cert_option.rb:27:in `cert_option_create_context'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/http_server/ssl_context_builder.rb:32:in `build'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/http_server.rb:94:in `http_server_create_https_server'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/plugin_helper/http_server.rb:67:in `http_server_create_http_server'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluent-plugin-prometheus-2.0.3/lib/fluent/plugin/in_prometheus.rb:109:in `start'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:203:in `block in start'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:192:in `block (2 levels) in lifecycle'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:191:in `each'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:191:in `block in lifecycle'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:178:in `each'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:178:in `lifecycle'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:202:in `start'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/engine.rb:248:in `start'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/engine.rb:147:in `run'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:720:in `block in run_worker'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:971:in `main_process'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:711:in `run_worker'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:376:in `<top (required)>'
  2023-04-07 07:35:06 +0000 [error]: <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
  2023-04-07 07:35:06 +0000 [error]: <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.14.6/bin/fluentd:15:in `<top (required)>'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/bin/fluentd:25:in `load'
  2023-04-07 07:35:06 +0000 [error]: /usr/local/bin/fluentd:25:in `<main>'
2023-04-07 07:35:06 +0000 [error]: unexpected error error_class=OpenSSL::PKey::PKeyError error="Could not parse PKey"
  2023-04-07 07:35:06 +0000 [error]: suppressed same stacktrace

      Version-Release number of selected component (if applicable):

      cluster-logging.v5.6.5

      How reproducible:

      Always

      Steps to Reproduce:

      1. launch cluster with FIPS enabled
      2. deploy logging 5.6.5, use fluentd as the collector
      3. check pods' status

      Actual results:

      Collector pods are in CrashLoopBackOff status.

      Expected results:

      Collector pods should be ready.

      Additional info:

      No issue when deploy 5.6.4 on the same cluster.

      No issue when deploy 5.6.5 with vector on the same cluster. 

              jcantril@redhat.com Jeffrey Cantrill
              qitang@redhat.com Qiaoling Tang
              Qiaoling Tang Qiaoling Tang
              Votes:
              6 Vote for this issue
              Watchers:
              11 Start watching this issue

                Created:
                Updated:
                Resolved: