IPFS Cluster Configuration

All IPFS Cluster configurations and persistent data can be found, by default, at the ~/.ipfs-cluster folder. For more information about the persistent data in this folder, see the Data, backups and recovery section.

ipfs-cluster-service -c <path> sets the location of the configuration folder. This is also controlled by the IPFS_CLUSTER_PATH environment variable.

The ipfs-cluster-service program uses two main configuration files:

• service.json, containing the cluster peer configuration, usually identical in all cluster peers.
• identity.json, containing the unique identity used by each peer.

identity.json

The identity.json file is auto-generated during ipfs-cluster-service init.  It includes a base64-encoded private key and the public peer ID  associated to it. This peer ID identifies the peer in the Cluster. You  can see an example here.

This file is not overwritten when re-running ipfs-cluster-service -f init. If you wish to generate a new one, you will need to delete it first.

Manual identity generation

When automating a deployment or creating configurations for several  peers, it might be handy to generate peer IDs and private keys manually  beforehand.

You can obtain a valid peer ID and its associated private key in the format expected by the configuration using ipfs-key as follows:

ipfs-key -type ed25519 | base64 -w 0

service.json

The service.json file holds all the configurable options  for the cluster peer and its different components. The configuration  file is divided in sections. Each section represents a component. Each  item inside the section represents an implementation of that component  and contains specific options. A default service.json file with sensible values is created when running ipfs-cluster-service init.

Important: The cluster section of the configuration stores a 32 byte hex-encoded secret which secures communication among all cluster peers. The secret must be shared by all cluster peers. Using an empty secret has security implications (see the Security section).If present, the `CLUSTER_SECRET` environment  value is used when running `ipfs-cluster-service init` to set the  cluster `secret` value.

As an example, this is a default service.json configuration file.

The file looks like:

{
  "source": "url" // a single source field may appear for remote configurations
  "cluster": {...},
  "consensus": {
    "crdt": {...}, // either crdt or raft
    "raft": {...},
  },
  "api": {
    "ipfsproxy": {...},
    "restapi": {...}
  },
  "ipfs_connector": {
    "ipfshttp": {...}
  },
  "pin_tracker": {
    "maptracker": {...},
    "stateless": {...}
  },
  "monitor": {
    "pubsubmon": {...}
  },
  "informer": {
    "disk": {...},
  },
  "observations": {
    "metrics": {...},
    "tracing": {...}
  },
  "datastore": {
    "badger": {...}
  }
}

The different sections and subsections are documented in detail below.

Using environment variables to overwrite configuration values

All the options in the configuration file can be can be overridden by setting environment variables. i.e. CLUSTER_SECRET will overwrite the secretvalue; CLUSTER_LEAVEONSHUTDOWN will overwrite the leave_on_shutdown value; CLUSTER_RESTAPI_CORSALLOWEDORIGINS will overwrite the restapi.cors_allowed_origins value.

In general the environment variable takes the form CLUSTER_<COMPONENTNAME>_KEYWITHOUTUNDERSCORES=value. Environment variables will be applied to the resultant configuration file when generating it with ipfs-cluster-service init.

Remote configurations

Since version 0.11.0, the service.json may be initialized with a single source field containing a URL that points to a standard service.json file. This configuration is read on every start of the peer.

A remote service.json can be used to point all peers the  same configuration file stored in the same location. It is also  possible to use an URL pointing to an file provided through IPFS.

The cluster main section

The main cluster section of the configuration file configures the core component and contains the following keys:

The leave_on_shutdown option allows a peer to remove itself from the peerset when shutting down cleanly. It is most relevant when using raft. This means that, for any subsequent starts, the peer will need to be bootstrapped in order to re-join the Cluster.

Manual secret generation

You can obtain a 32-bit hex encoded random string with:

export CLUSTER_SECRET=$(od  -vN 32 -An -tx1 /dev/urandom | tr -d ' \n')

The consensus section

The consensus contains a single configuration object for the chosen implementations of the consensus component (either crdt or raft, but not both).

crdt

raft

Raft stores and maintains the peerset internally but the cluster  configuration offers the option to manually provide the peerset for the  first start of a peer using the init_peerset key in the raft section of the configuration. For example:

"init_peerset": [
  "QmPQD6NmQkpWPR1ioXdB3oDy8xJVYNGN9JcRVScLAqxkLk",
  "QmcDV6Tfrc4WTTGQEdatXkpyFLresZZSMD8DgrEhvZTtYY",
  "QmWXkDxTf17MBUs41caHVXWJaz1SSAD79FVLbBYMTQSesw",
  "QmWAeBjoGph92ktdDb5iciveKuAX3kQbFpr5wLWnyjtGjb"
]

This will allow you to start a Cluster from scratch with already fixed peerset.

The api section

The api section contains configurations for the  implementations of the API component, which are meant to provide  endpoints for the interaction with Cluster. Removing any of these  sections will disable the component. For example, removing the ipfsproxy section from the configuration will disable the proxy endpoint on the running peer.

ipfsproxy

This component provides the IPFS Proxy Endpoint. This is an API which  mimics the IPFS daemon. Some requests (pin, unpin, add) are hijacked  and handled by Cluster. Others are simply forwarded to the IPFS daemon  specified by node_multiaddress. The component is by default configured to mimic CORS headers configurations as present in the IPFS daemon. For that it triggers accessory requests to them (like CORS preflights).

restapi

This is the component which provides the REST API implementation to interact with Cluster.

The REST API component automatically, and additionally, exposes the  HTTP API as a libp2p service on the main libp2p cluster Host (which  listens on port 9096). Exposing the HTTP API as a libp2p  service allows users to benefit from the channel encryption provided by  libp2p. Alternatively, the API supports specifying a fully separate  libp2p Host by providing id, private_key and libp2p_listen_multiaddress.  When using a separate Host, it is not necessary for an API consumer to  know the cluster secret. Both the HTTP and the libp2p endpoints are  supported by the API Client and by ipfs-cluster-ctl.

The ipfs_connector section

The ipfs_connector section contains configurations for  the implementations of the IPFS Connector component, which are meant to  provide a way for the Cluster peer to interact with an IPFS daemon.

ipfshttp

This is the default and only IPFS Connector implementation. It provides a gateway to the IPFS daemon API and an IPFS HTTP Proxy.

The pin_tracker section

The pin_tracker section contains configurations for the  implementations of the Pin Tracker component, which are meant to ensure  that the content in IPFS matches the allocations as decided by IPFS  Cluster.

maptracker

The maptracker implements a pintracker which keeps the local state in memory.

stateless

The stateless tracker implements a pintracker which relies on ipfs and the shared state, thus reducing the memory usage in comparison to the maptracker.

The monitor section

The monitor section contains configurations for the  implementations of the Peer Monitor component, which are meant to  distribute and collects monitoring information (informer metrics, pings)  to and from other peers, and trigger alerts.

pubsubmon

The pubsubmon implementation collects and broadcasts  metrics using libp2p’s pubsub. This will provide a more efficient and  scalable approach for metric distribution.

The informer section

The informer section contains configuration for  Informers. Informers fetch the metrics which are used to allocate  content to the different peers.

disk

The disk informer collects disk-related metrics at intervals.

numpin

The numpin informer uses the total number of pins as metric, which collects at intervals.

The observations section

The observations section contains configuration for application distributed tracing and metrics collection.

metrics

The metrics component configures the OpenCensus metrics endpoint for scraping of metrics by Prometheus.

tracing

The tracing component configures the Jaeger tracing client for use by OpenCensus.

The datastore section

The datastore section contains configuration for different storage backends (currently only badger).

badger

The badger component configures the BadgerDB backend which is used by the CRDT component.