Kubernetes MariaDB-Galera Cluster – Fix Bitnami Helm Chart Readiness Probe Failed

I am trying to setup a mariadb-galera cluster through the bitnami helm chart in my kubernetes cluster (1 master, 3 nodes). I have modified the myvalues.yaml to include an existingClaim: dbstorage and the storageClass: "nfs-storage". The image repository is 10.5.9-debian-10-r52, I added a root password and outcommented the accessModes: as well as the size as that was defined in the existing persistence volume claim. I did not define anything in the section db: and left that as the defaults. I also did not define anything under the galera.mariabackup section like password and left the defaults.

As soon as I run the helm chart with helm install helm install mariadb-galera-cluster -f mariadb-galera.values.yaml bitnami/mariadb-galera --namespace database and describe the pod I get the error message

Readiness probe failed: mysqladmin: connect to server at 'localhost' failed error: 'Can't connect to local MySQL server through socket '/opt/bitnami/mariadb/tmp/mysql.sock' (2)' 
Check that mysqld is running and that the socket: '/opt/bitnami/mariadb/tmp/mysql.sock' exists!

When checking the container for that mysql.sock it is true, that it is not available in that location.

I am using a NFS provisioner to provision the persistent storage which works fine. On my nfs server I can see the directory being created and data being stored in it. It is a NFS3 directory that is used by the container.

When I access the container and try to run the scrips "run.sh" or "entrypoint.sh" in the folder /opt/bitnami/scripts/mariadb-galera I am getting an error The mariadb configuration file '/opt/bitnami/mariadb/conf/my.cnf' is not writable or does not exist. Configurations based on environment variables will not be applied for this file. but the file is right in the folder where it should be.
All components, like the stateful set are created and started properly as I can tell just the container, in my case it was obviously called mariadb-galera-cluster-0, is not finishing starting up because of the socket it can't find.

Version of Helm:

version.BuildInfo{Version:"v3.5.4", GitCommit:"1b5edb69df3d3a08df77c9902dc17af864ff05d1", GitTreeState:"clean", GoVersion:"go1.15.11"}

Version of Kubernetes

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:25:06Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}

version of the values.yaml file

## Please, note that this will override the image parameters, including dependencies, configured to use the global v$
## Current available global Docker image parameters: imageRegistry and imagePullSecrets
##
# global:
#   imageRegistry: myRegistryName
#   imagePullSecrets:
#     - myRegistryKeySecretName
#   storageClass: myStorageClass

## Bitnami MariaDB Galera image
## ref: https://hub.docker.com/r/bitnami/mariadb-galera/tags/
##
image:
  registry: docker.io
  repository: bitnami/mariadb-galera
  tag: 10.5.9-debian-10-r52
  ## Specify a imagePullPolicy
  ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'
  ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
  ##
  pullPolicy: IfNotPresent
  ## Optionally specify an array of imagePullSecrets.
  ## Secrets must be manually created in the namespace.
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
  ##
  # pullSecrets:
  #   - myRegistryKeySecretName

  ## Set to true if you would like to see extra information on logs
  ##
  debug: false

## String to partially override common.names.fullname template (will maintain the release name)
##
# nameOverride:

## String to fully override common.names.fullname template
##
# fullnameOverride:

## Use an alternate scheduler, e.g. "stork".
## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
##
# schedulerName:

## Specifies the Kubernetes Cluster's Domain Name.
##
clusterDomain: cluster.local

## StatefulSet controller supports relax its ordering guarantees while preserving its uniqueness and identity guaran$
## ref: https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#pod-management-policy
##
podManagementPolicy: OrderedReady

## Deployment pod host aliases
## https://kubernetes.io/docs/concepts/services-networking/add-entries-to-pod-etc-hosts-with-host-aliases/
hostAliases: []

## MariaDB Gallera K8s svc properties
##
service:
  ## Kubernetes service type and port number
  ##
  type: ClusterIP
  port: 3306
  # clusterIP: None

  ## Specify the nodePort value for the LoadBalancer and NodePort service types.
  ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport
  ##
  # nodePort: 30001

  ## Specify the externalIP value ClusterIP service type.
  ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#external-ips
  ##
  # externalIPs: []

  ## Set the LoadBalancer service type to internal only.
  ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#internal-load-balancer
  ##
  # loadBalancerIP:

  ## Load Balancer sources
  ## https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/#restrict-access-$
  ##
  # loadBalancerSourceRanges:
  # - 10.10.10.0/24

  ## Provide any additional annotations which may be required. This can be used to
  ## set the LoadBalancer service type to internal only.
  ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#internal-load-balancer
  ##
  annotations: {}

  ## Headless service properties
  ##
  headless:
    ## Additional annotations for headless service.
    ## Can be useful in case peer-finder is used in a sidecar,
    ## e.g.: service.alpha.kubernetes.io/tolerate-unready-endpoints="true"
    ##
    annotations: {}

## Pods Service Account
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/
##
serviceAccount:
  ## Specifies whether a ServiceAccount should be created
  ##
  create: false
  ## The name of the ServiceAccount to use.
  ## If not set and create is true, a name is generated using the common.names.fullname template
  ##
  name: ""

## An array to add extra environment variables
## For example:
## extraEnvVars:
##  - name: TZ
##    value: "Europe/Paris"
##
extraEnvVars:

## ConfigMap with extra env vars:
##
extraEnvVarsCM:

## Secret with extra env vars:
##
extraEnvVarsSecret:

## Role Based Access
## Ref: https://kubernetes.io/docs/admin/authorization/rbac/
##
rbac:
  ## Specifies whether RBAC rules should be created
  ##
  create: false

## Pod Security Context
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
##
securityContext:
  enabled: true
  fsGroup: 1001
  runAsUser: 1001

## Database credentials for root (admin) user
##
rootUser:
  ## MariaDB admin user
  ##
  user: root
  ## MariaDB admin password
  ## Password is ignored if existingSecret is specified.
  ## ref: https://github.com/bitnami/bitnami-docker-mariadb-galera#setting-the-root-password-on-first-run
  ##
  password: "ObviouslyIChangedThis"
  ## Option to force users to specify a password. That is required for 'helm upgrade' to work properly.
  ## If it is not force, a random password will be generated.
  ##
  forcePassword: false

## Use existing secret (ignores rootUser.password, db.password, and galera.mariabackup.password)
##
# existingSecret:

## Custom db configuration
##
db:
  ## MariaDB username and password
  ## Password is ignored if existingSecret is specified.
  ## ref: https://github.com/bitnami/bitnami-docker-mariadb-galera#creating-a-database-user-on-first-run
  ##
  user: ""
  password: ""
  ## Database to create
  ## ref: https://github.com/bitnami/bitnami-docker-mariadb-galera#creating-a-database-on-first-run
  ##
  name: my_database
  ## Option to force users to specify a password. That is required for 'helm upgrade' to work properly.
  ## If it is not force, a random password will be generated.
  ##
  forcePassword: false

## Galera configuration
##
galera:
  ## Galera cluster name
  ##
  name: galera

  ## Bootstraping options
  ## ref: https://github.com/bitnami/bitnami-docker-mariadb-galera#bootstraping
  ##
  bootstrap:
    ## Node to bootstrap from, you will need to change this parameter in case you want to bootstrap from other node
    ##
    bootstrapFromNode:
    ## Force safe_to_bootstrap in grastate.date file.
    ## This will set safe_to_bootstrap=1 in the node indicated by bootstrapFromNode.
    ##
    forceSafeToBootstrap: false

  ## Credentials to perform backups
  ##
  mariabackup:
    ## MariaBackup username and password
    ## Password is ignored if existingSecret is specified.
    ## ref: https://github.com/bitnami/bitnami-docker-mariadb-galera#setting-up-a-multi-master-cluster
    ##
    user: mariabackup
    password: ""
    ## Option to force users to specify a password. That is required for 'helm upgrade' to work properly.
    ## If it is not force, a random password will be generated.
    ##
    forcePassword: false

## LDAP configuration
##
ldap:
  ## Enable LDAP support
  ##
  enabled: false
  uri: ""
  base: ""
  binddn: ""
  bindpw: ""
  bslookup:
  filter:
  map:
  nss_initgroups_ignoreusers: root,nslcd
  scope:
  tls_reqcert:

## TLS configuration
##
tls:
  ## Enable TLS
  ##
  enabled: false
  ## Name of the secret that contains the certificates
  ##
  # certificatesSecret:
  ## Certificate filename
  ##
  # certFilename:
  ## Certificate Key filename
  ##
  # certKeyFilename:
  ## CA Certificate filename
  ##
  # certCAFilename:

## Configure MariaDB with a custom my.cnf file
## ref: https://mysql.com/kb/en/mysql/configuring-mysql-with-mycnf/#example-of-configuration-file
## Alternatively, you can put your my.cnf under the files/ directory
mariadbConfiguration: |-
  [client]
  port=3306
  socket=/opt/bitnami/mariadb/tmp/mysql.sock
  plugin_dir=/opt/bitnami/mariadb/plugin
  [mysqld]
  default_storage_engine=InnoDB
  basedir=/opt/bitnami/mariadb
  datadir=/bitnami/mariadb/data
  plugin_dir=/opt/bitnami/mariadb/plugin
  tmpdir=/opt/bitnami/mariadb/tmp
  socket=/opt/bitnami/mariadb/tmp/mysql.sock
  pid_file=/opt/bitnami/mariadb/tmp/mysqld.pid
  bind_address=0.0.0.0
  ## Character set
  ##
  collation_server=utf8_unicode_ci
  init_connect='SET NAMES utf8'
  character_set_server=utf8
  ## MyISAM
  ##
  key_buffer_size=32M
  myisam_recover_options=FORCE,BACKUP
  ## Safety
  ##
  skip_host_cache
  skip_name_resolve
  max_allowed_packet=16M
  max_connect_errors=1000000
  sql_mode=STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_AUTO_VALUE_ON_ZERO,NO_ENGINE_SUBSTI$
  sysdate_is_now=1
  ## Binary Logging
  ##
  log_bin=mysql-bin
  expire_logs_days=14
  # Disabling for performance per http://severalnines.com/blog/9-tips-going-production-galera-cluster-mysql
  sync_binlog=0
  # Required for Galera
  binlog_format=row
  ## Caches and Limits
  ##
  tmp_table_size=32M
  max_heap_table_size=32M
  # Re-enabling as now works with Maria 10.1.2
  query_cache_type=1
  query_cache_limit=4M
  query_cache_size=256M
  max_connections=500
  thread_cache_size=50
  open_files_limit=65535
  table_definition_cache=4096
  table_open_cache=4096
  ## InnoDB
  ##
  innodb=FORCE
  innodb_strict_mode=1
  # Mandatory per https://github.com/codership/documentation/issues/25
  innodb_autoinc_lock_mode=2
  # Per https://www.percona.com/blog/2006/08/04/innodb-double-write/
  innodb_doublewrite=1
  innodb_flush_method=O_DIRECT
  innodb_log_files_in_group=2
  innodb_log_file_size=128M
  innodb_flush_log_at_trx_commit=1
  innodb_file_per_table=1
  # 80% Memory is default reco.
  # Need to re-evaluate when DB size grows
  innodb_buffer_pool_size=2G
  innodb_file_format=Barracuda
  ## Logging
  ##
  log_error=/opt/bitnami/mariadb/logs/mysqld.log
  slow_query_log_file=/opt/bitnami/mariadb/logs/mysqld.log
  log_queries_not_using_indexes=1
  slow_query_log=1
  ## SSL
  ## Use extraVolumes and extraVolumeMounts to mount /certs filesystem
  # ssl_ca=/certs/ca.pem
  # ssl_cert=/certs/server-cert.pem
  # ssl_key=/certs/server-key.pem
  [galera]
  wsrep_on=ON
  wsrep_provider=/opt/bitnami/mariadb/lib/libgalera_smm.so
  wsrep_sst_method=mariabackup
  wsrep_slave_threads=4
  wsrep_cluster_address=gcomm://
  wsrep_cluster_name=galera
  wsrep_sst_auth="root:"
  # Enabled for performance per https://mariadb.com/kb/en/innodb-system-variables/#innodb_flush_log_at_trx_commit
  innodb_flush_log_at_trx_commit=2
  # MYISAM REPLICATION SUPPORT #
  wsrep_replicate_myisam=ON
  [mariadb]
  plugin_load_add=auth_pam
  ## Data-at-Rest Encryption
  ## Use extraVolumes and extraVolumeMounts to mount /encryption filesystem
  # plugin_load_add=file_key_management
  # file_key_management_filename=/encryption/keyfile.enc
  # file_key_management_filekey=FILE:/encryption/keyfile.key
  # file_key_management_encryption_algorithm=AES_CTR
  # encrypt_binlog=ON
  # encrypt_tmp_files=ON
  ## InnoDB/XtraDB Encryption
  # innodb_encrypt_tables=ON
  # innodb_encrypt_temporary_tables=ON
  # innodb_encrypt_log=ON
  # innodb_encryption_threads=4
  # innodb_encryption_rotate_key_age=1
  ## Aria Encryption
  # aria_encrypt_tables=ON
  # encrypt_tmp_disk_tables=ON
## ConfigMap with MariaDB configuration
## NOTE: This will override mariadbConfiguration
##
# configurationConfigMap:

## Specify dictionary of scripts to be run at first boot
## Alternatively, you can put your scripts under the files/docker-entrypoint-initdb.d directory
##
# initdbScripts:
#   my_init_script.sh: |
#      #!/bin/sh
#      echo "Do something."

## ConfigMap with scripts to be run at first boot
## Note: This will override initdbScripts
##
# initdbScriptsConfigMap:

## MariaDB additional command line flags
## Can be used to specify command line flags, for example:
##
## extraFlags: "--max-connect-errors=1000 --max_connections=155"
##

## Desired number of cluster nodes
##
replicaCount: 3

## updateStrategy for MariaDB Master StatefulSet
## ref: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#update-strategies
##
updateStrategy:
  type: RollingUpdate

## Additional labels for MariaDB Galera pods
## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
##
podLabels: {}

## Additional annotations for MariaDB Galera pods
## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
##
podAnnotations: {}

## Pod affinity preset
## ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity
## Allowed values: soft, hard
##
podAffinityPreset: ""

## Pod anti-affinity preset
## Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity
## Allowed values: soft, hard
##
podAntiAffinityPreset: soft

## Node affinity preset
## Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity
## Allowed values: soft, hard
##
nodeAffinityPreset:
  ## Node affinity type
  ## Node affinity type
  ## Allowed values: soft, hard
  ##
  type: ""
  ## Node label key to match
  ## E.g.
  ## key: "kubernetes.io/e2e-az-name"
  ##
  key: ""
  ## Node label values to match
  ## E.g.
  ## values:
  ##   - e2e-az1
  ##   - e2e-az2
  ##
  values: []

## Affinity for pod assignment. Evaluated as a template.
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
## Note: podAffinityPreset, podAntiAffinityPreset, and nodeAffinityPreset will be ignored when it's set
##
affinity: {}

## Node labels for pod assignment. Evaluated as a template.
## ref: https://kubernetes.io/docs/user-guide/node-selection/
##
nodeSelector: {}

## Tolerations for pod assignment. Evaluated as a template.
## ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
tolerations: []

## Enable persistence using Persistent Volume Claims
## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/
##
persistence:
  ## If true, use a Persistent Volume Claim, If false, use emptyDir
  ##
  enabled: true
  # Enable persistence using an existing PVC
  existingClaim: dbstorage
  # Subdirectory of the volume to mount
  # subPath:
  mountPath: /bitnami/mariadb
  ## selector can be used to match an existing PersistentVolume
  ## selector:
  ##   matchLabels:
  ##     app: my-app
  ##
  selector: {}
  ## Persistent Volume Storage Class
  ## If defined, storageClassName: <storageClass>
  ## If set to "-", storageClassName: "", which disables dynamic provisioning
  ## If undefined (the default) or set to null, no storageClassName spec is
  ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
  ##   GKE, AWS & OpenStack)
  ##
  storageClass: "nfs-storage"
  ## Persistent Volume Claim annotations
  ##
  annotations:
  ## Persistent Volume Access Mode
  ##
  #accessModes:
  #  - ReadWriteOnce
  ## Persistent Volume size
  ##
  #size: 8Gi

## Priority Class Name
#
# priorityClassName: 'priorityClass'

## Additional init containers
##
extraInitContainers: []
# - name: do-something
#   image: bitnami/minideb
#   command: ['do', 'something']

## Additional containers
##
extraContainers: []

## extraVolumes and extraVolumeMounts allows you to mount other volumes
## Example Use Cases:
##  mount certificates to enable data-in-transit encryption
##  mount keys for data-at-rest encryption using file plugin
# extraVolumes:
# - name: mariadb-certs
#   secret:
#     defaultMode: 288
#     secretName: mariadb-certs
# - name: mariadb-encryption
#   secret:
#     defaultMode: 288
#     secretName: mariadb-encryption
# extraVolumeMounts:
# - name: mariadb-certs
#   mountPath: /certs
#   readOnly: true
# - name: mariadb-encryption
#   mountPath: /encryption
#   readOnly: true

## MariaDB Galera containers' resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  limits: {}
  #   cpu: 0.5
  #   memory: 256Mi
  requests: {}
  #   cpu: 0.5
  #   memory: 256Mi

## MariaDB Galera containers' liveness and readiness probes
## ref: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes
##
livenessProbe:
  enabled: true
  ## Initializing the database could take some time
  ##
  initialDelaySeconds: 120
  periodSeconds: 10
  timeoutSeconds: 1
  successThreshold: 1
  failureThreshold: 3
readinessProbe:
  enabled: true
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 1
  successThreshold: 1
  failureThreshold: 3
startupProbe:
  enabled: false
  ## Initializing the database could take some time
  ##
  initialDelaySeconds: 120
  periodSeconds: 10
  timeoutSeconds: 1
  successThreshold: 1
  ## Let's wait 600 seconds by default, it should give enough time in any cluster for mysql to init
  ##
  failureThreshold: 48

## Pod disruption budget configuration
##
podDisruptionBudget:
  ## Specifies whether a Pod disruption budget should be created
  ##
  create: false
  minAvailable: 1
  # maxUnavailable: 1

## Prometheus exporter configuration
##
metrics:
  enabled: false
  ## Bitnami MySQL Prometheus exporter image
  ## ref: https://hub.docker.com/r/bitnami/mysqld-exporter/tags/
  ##
  image:
    registry: docker.io
    repository: bitnami/mysqld-exporter
    tag: 0.12.1-debian-10-r416
    pullPolicy: IfNotPresent
    ## Optionally specify an array of imagePullSecrets.
    ## Secrets must be manually created in the namespace.
    ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
    ##
    # pullSecrets:
    #   - myRegistryKeySecretName
  ## MySQL exporter additional command line flags
  ## Can be used to specify command line flags
  ## E.g.:
  ## extraFlags:
  ##   - --collect.binlog_size
  ##
  extraFlags: []
  ## MySQL Prometheus exporter containers' resource requests and limits
  ## ref: http://kubernetes.io/docs/user-guide/compute-resources/
  ##
  resources:
    # We usually recommend not to specify default resources and to leave this as a conscious
    # choice for the user. This also increases chances charts run on environments with little
    # resources, such as Minikube. If you do want to specify resources, uncomment the following
    # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
    limits: {}
    #   cpu: 0.5
    #   memory: 256Mi
    requests: {}
    #   cpu: 0.5
    #   memory: 256Mi
  ## MySQL Prometheus exporter service parameters
  ##
  service:
    type: ClusterIP
    port: 9104
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "9104"

  ## Prometheus Operator ServiceMonitor configuration
  ##
  serviceMonitor:
    enabled: false
    ## Namespace in which Prometheus is running
    ##
    # namespace: monitoring

    ## Interval at which metrics should be scraped.
    ## ref: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#endpoint
    ##
    # interval: 10s

    ## Timeout after which the scrape is ended
    ## ref: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#endpoint
    ##
    # scrapeTimeout: 10s

    ## ServiceMonitor selector labels
    ## ref: https://github.com/bitnami/charts/tree/master/bitnami/prometheus-operator#prometheus-configuration
    ##
    selector:
      prometheus: kube-prometheus
    ## RelabelConfigs to apply to samples before scraping
    ## ref: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#relabelconfig
    ## Value is evalued as a template
    ##
    relabelings: []

    ## MetricRelabelConfigs to apply to samples before ingestion
    ## ref: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#relabelconfig
    ## Value is evalued as a template
    ##
    metricRelabelings: []
    #  - sourceLabels:
    #      - "__name__"
    #    targetLabel: "__name__"
    #    action: replace
    #    regex: '(.*)'
    #    replacement: 'example_prefix_$1'

  ## Prometheus Operator PrometheusRule configuration
  ##
  prometheusRules:
    enabled: false

    ## Additional labels to add to the PrometheusRule so it is picked up by the operator.
    ## If using the [Helm Chart](https://github.com/helm/charts/tree/master/stable/prometheus-operator) this is the $
    ##
    selector:
      app: prometheus-operator
      release: prometheus

    ## Rules as a map.
    ##
    rules: {}
    #  - alert: MariaDB-Down
    #    annotations:
    #      message: 'MariaDB instance {{ $labels.instance }} is down'
    #      summary: MariaDB instance is down
    #    expr: absent(up{job="mariadb-galera"} == 1)
    #    labels:
    #      severity: warning
    #      service: mariadb-galera
    #    for: 5m

EDIT
adding log file from container initialization:

mariadb 13:59:39.44
mariadb 13:59:39.44 Welcome to the Bitnami mariadb-galera container
mariadb 13:59:39.45 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mariadb-galera
mariadb 13:59:39.45 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mariadb-galera/issues
mariadb 13:59:39.46
mariadb 13:59:39.47 INFO  ==> ** Starting MariaDB setup **
mariadb 13:59:40.30 INFO  ==> Validating settings in MYSQL_*/MARIADB_* env vars
mariadb 13:59:40.41 DEBUG ==> Set Galera cluster address to gcomm://
mariadb 13:59:40.42 INFO  ==> Initializing mariadb database
mariadb 13:59:40.43 DEBUG ==> Ensuring expected directories/files exist
mariadb 13:59:40.46 WARN  ==> The mariadb configuration file '/opt/bitnami/mariadb/conf/my.cnf' is not writable or does not exist. Configurations based on environment variables will not be applied for this file.
mariadb 13:59:40.49 DEBUG ==> Cleaning data directory to ensure successfully initialization
Installing MariaDB/MySQL system tables in '/bitnami/mariadb/data' ...

Unfortunately that is it. The container is now in a "CrashLoopBackOff" state. It restarted 7 times already and will not finish. I can see on my NFS server the "data" directory is created and it contains 2 files, "aria_log_control" and "mysql-bin.index". The my.cnf looks like being owned by root but as far as I could find in the documentation the container is a non-root container so the file would not be writable. But it is definitely there. I checked that several times.
If I describe the pod I get the error message mentioned above "Readiness probe failed".

END EDIT

EDIT 2
here are the details on the readiness, liveness probe.

## MariaDB Galera containers' liveness and readiness probes
## ref: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes
##
livenessProbe:
  enabled: true
  ## Initializing the database could take some time
  ##
  initialDelaySeconds: 120
  periodSeconds: 10
  timeoutSeconds: 1
  successThreshold: 1
  failureThreshold: 3
readinessProbe:
  enabled: true
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 1
  successThreshold: 1
  failureThreshold: 3
startupProbe:
  enabled: false
  ## Initializing the database could take some time
  ##
  initialDelaySeconds: 120
  periodSeconds: 10
  timeoutSeconds: 1
  successThreshold: 1
  ## Let's wait 600 seconds by default, it should give enough time in any cluster for mysql to init
  ##
  failureThreshold: 48

END EDIT

EDIT 3
I have no choice but to respond to my original post. I would otherwise exceed the character limit of stackoverflow.
End EDIT

I would be grateful for any help or hint you can provide to get the cluster up and running.

realshadow

So I followed your advise @WytrzymałyWiktor and disabled the readiness and liveness probes. That way all 3 instances are spun up right away. The problem is they are not communicating among each other. Logs from the
So despite having the pods running they are not working and communicating among each other.