Building a Twitter Bot using IPFS Cluster 🤖

Till now, we have explored a number of fun examples using InterPlanetary File System (IPFS), InterPlanetary Linked Data (IPLD) & Libp2p. We have built Websites on IPFS, Youtube on IPFS, Online Publication on IPLD & Chat Application on Libp2p.

In this tutorial we are going to build a Twitter Bot(or a Pinbot) using IPFS Cluster.

You can see our Twitter bot in action!

You can try it out yourself.

1) Follow the @simpleaswater_ twitter account so that bot can filter your tweets out.  

2) Just tweet something using the below button and it will reply you with links to the content of the tweet saved on the distributed Web!
P.S. We recommend you to add some Images/Gifs/Videos to your tweet.

Cool, right? Let's see how you can build your own twitter pinbot!

In this tutorial, we are go through:

• An Intro to IPFS Cluster, its significance & use-cases.
• Setting up an IPFS Cluster Network.
• Setting up Twitter Developer Account.
• Building a Twitter bot.

You can find the full code implementation here:

If you get stuck in any part or have any queries/doubts, then feel free to reach us out on our discord channel.

Intro to IPFS Cluster

Before setting up our own IPFS Cluster network, it's good to spend a few minutes to understand why do we need IPFS Cluster and what is it.

If you are  new to InterPlanetary File System (IPFS) then we would recommend you to check out this post first:

If you are familiar with IPFS, then you already know that IPFS aims to make the internet decentralized using content addressing approach.

The way this works is similar to services like BitTorrent where every user does not just consume the data but also serves it to other people in the BitTorrent network. That means you can not only get the data(let's say Star Wars movie) from anyone on the network who has it, but also share this data with anyone who asks for it on the network.

But if all the people who have the Star Wars movie go offline, then you are out of luck.

No movie for you 😔

Now to understand why do we need IPFS Cluster consider the following scenario:

Let's suppose you want to store your collection of favorite songs on the IPFS Network. You upload your playlist on the IPFS network and the people who also love the same songs can get the songs from you.

Now, you may also have other things on your device that you want to keep(your Movies, vacation photos, etc.). So, you decide to remove the playlist from your device. After all, you can get your playlist from the people who downloaded it from you earlier. So, you are not worried about your songs getting lost, and go to sleep happily.

One fine day, you want to get your songs back for your long workout session but are devastated to see that you can't find your playlist on the IPFS Network 😨

But how can this happen? You saved it on the network, right?

It turns out that all the people who saved the playlist are either offline or have deleted the songs, as they got bored with them.

The problem was that our playlist was not stored redundantly on a number of nodes(or devices) so that we could be sure that it would not get lost in the oblivion.

So, how do we ensure data availability and redundancy on IPFS?

Enter IPFS Cluster.

The purpose of IPFS Cluster is to make it easy for you to orchestrate/manage data across several IPFS peers by allocating, replicating and tracking a global pinset(list of saved data) distributed among multiple peers.

This way you can build a network of IPFS peers that redundantly store data for us.

If you want to know more about IPFS Cluster and see how it works under the hood, then head here:

If you have any queries/doubts till now, then feel free to reach out on our discord channel.

Now, as we have explored why we need IPFS Cluster let's set up our own IPFS Cluster network, on which we will save our tweets.

Setting up Your IPFS Peer

As the IPFS Cluster network is a collection of IPFS peers, first we need to install & run an IPFS peer.

You can follow this guide to install IPFS Peer on your device.

After installing the IPFS, you can test your installation using ipfs version

$ ipfs version
ipfs version <VERSION_NUMBER>

$ ipfs help
USAGE:

ipfs - Global p2p merkle-dag filesystem.
...

NOTE: Throughout this tutorial, we use the $ character to indicate your terminal’s shell prompt. When following along, don’t type the $ character, or you’ll get some weird errors.

Now, let's initialize your IPFS peer using ipfs init

$ ipfs init

initializing IPFS node at /home/vasa/.ipfs
generating 2048-bit RSA keypair...done
peer identity: Qmcpo2iLBikrdf1d6QU6vXuNb6P7hwrbNPW9kLAH8eG67z
to get started, enter:

ipfs cat /ipfs/QmYwAPJzv5CZsnA625s3Xf2nemtYgPpHdWEz79ojWnPbdG/readme

Voila! Your IPFS peer setup is complete!

If you are having any problems while setting up your IPFS peer, then feel free to reach us out on our discord channel.

Setting up Your IPFS Cluster Peer

In order to run an IPFS Cluster peer we need to install ipfs-cluster-service.

• First, download the ipfs-cluster-service binary for your Operating System.
• Now, unzip/untar the downloaded binary and you will find a ipfs-cluster-service executable inside the folder. Place this binary where it can be run by a system user(usually usr/local/bin).
• Now run the following command to initialize generate the IPFS Cluster configuration file.  

$ ipfs-cluster-service init

Viola! You have setup your IPFS Cluster Peer.

Making a few changes to IPFS Cluster Config

In order to make the twitter bot example work, we need to change the IPFS Cluster config a bit. If you want to know why we are doing these changes, then you can refer to this discussion thread.

To make the changes, first open your IPFS Cluster config located at $HOME/.ipfs-cluster/service.json.

Now, scroll down to the api JSON object. In the api section, you will find restapi JSON object. You need to add these 3 key-values: libp2p_listen_multiaddress, private_key and id as shown below.

{
...
"api": {
    "ipfsproxy": {
		...
    },
    "restapi": {
      "libp2p_listen_multiaddress": "/ip4/127.0.0.1/tcp/9696",
      "private_key": "CAASqAkwggSkAgEAAoIBAQDLZZcGcbe4urMBVlcHgN0fpBymY+xcr14ewvamG70QZODJ1h9sljlExZ7byLiqRB3SjGbfpZ1FweznwNxWtWpjHkQjTVXeoM4EEgDSNO/Cg7KNlU0EJvgPJXeEPycAZX9qASbVJ6EECQ40VR/7+SuSqsdL1hrmG1phpIju+D64gLyWpw9WEALfzMpH5I/KvdYDW3N4g6zOD2mZNp5y1gHeXINHWzMF596O72/6cxwyiXV1eJ000k1NVnUyrPjXtqWdVLRk5IU1LFpoQoXZU5X1hKj1a2qt/lZfH5eOrF/ramHcwhrYYw1txf8JHXWO/bbNnyemTHAvutZpTNrsWATfAgMBAAECggEAQj0obPnVyjxLFZFnsFLgMHDCv9Fk5V5bOYtmxfvcm50us6ye+T8HEYWGUa9RrGmYiLweuJD34gLgwyzE1RwptHPj3tdNsr4NubefOtXwixlWqdNIjKSgPlaGULQ8YF2tm/kaC2rnfifwz0w1qVqhPReO5fypL+0ShyANVD3WN0Fo2ugzrniCXHUpR2sHXSg6K+2+qWdveyjNWog34b7CgpV73Ln96BWae6ElU8PR5AWdMnRaA9ucA+/HWWJIWB3Fb4+6uwlxhu2L50Ckq1gwYZCtGw63q5L4CglmXMfIKnQAuEzazq9T4YxEkp+XDnVZAOgnQGUBYpetlgMmkkh9qQKBgQDvsEs0ThzFLgnhtC2Jy//ZOrOvIAKAZZf/mS08AqWH3L0/Rjm8ZYbLsRcoWU78sl8UFFwAQhMRDBP9G+RPojWVahBL/B7emdKKnFR1NfwKjFdDVaoX5uNvZEKSl9UubbC4WZJ65u/cd5jEnj+w3ir9G8n+P1gp/0yBz02nZXFgSwKBgQDZPQr4HBxZL7Kx7D49ormIlB7CCn2i7mT11Cppn5ifUTrp7DbFJ2t9e8UNk6tgvbENgCKXvXWsmflSo9gmMxeEOD40AgAkO8Pn2R4OYhrwd89dECiKM34HrVNBzGoB5+YsAno6zGvOzLKbNwMG++2iuNXqXTk4uV9GcI8OnU5ZPQKBgCZUGrKSiyc85XeiSGXwqUkjifhHNh8yH8xPwlwGUFIZimnD4RevZI7OEtXw8iCWpX2gg9XGuyXOuKORAkF5vvfVriV4e7c9Ad4Igbj8mQFWz92EpV6NHXGCpuKqRPzXrZrNOA9PPqwSs+s9IxI1dMpk1zhBCOguWx2m+NP79NVhAoGBAI6WSoTfrpu7ewbdkVzTWgQTdLzYNe6jmxDf2ZbKclrf7lNr/+cYIK2Ud5qZunsdBwFdgVcnu/02czeS42TvVBgs8mcgiQc/Uy7yi4/VROlhOnJTEMjlU2umkGc3zLzDgYiRd7jwRDLQmMrYKNyEr02HFKFn3w8kXSzW5I8rISnhAoGBANhchHVtJd3VMYvxNcQb909FiwTnT9kl9pkjhwivx+f8/K8pDfYCjYSBYCfPTM5Pskv5dXzOdnNuCj6Y2H/9m2SsObukBwF0z5Qijgu1DsxvADVIKZ4rzrGb4uSEmM6200qjJ/9U98fVM7rvOraakrhcf9gRwuspguJQnSO9cLj6",
      "id": "QmcrQZ6RJdpYuGvZqD5QEHAv6qX4BrQLJLQPQUrTrzdcgm",
      ...
      }
  }
  ...
} 

NOTE: The private_key that we are using is just for the demo purpose. If you are building a production bot, you should create your own libp2p private_key and id. You can do that by following this thread.

Starting Your IPFS Cluster Network

Now, as we have installed IPFS and IPFS Cluster peers, we can start our Cluster network.

• Open a terminal window and start the IPFS peer using ipfs daemon

• Open another terminal and start the IPFS Cluster peer using ipfs-cluster-service daemon

If you managed to reach here, then you deserve a pat on your back!

You are now running an IPFS Cluster network with 1 peer. For the purpose of this tutorial, we will proceed with this setup.

If you have any queries/doubts, then feel free to reach out on our discord channel.

Creating a Twitter Developer Account

To build a twitter bot, first we need to setup a developer account on twitter.

• Go to Twitter developers website. Click the "Create an app" button.

• You will be prompted to apply for the twitter developer account.

• Now, select the "Making a bot" option for your primary reason for using twitter developer tools, and click "Next".

• In the next page, you will be asked some personal details. Fill them up and click "Next".
• In the next page you will be asked a few more questions about how will you use twitter's APIs. Fill up the answers by saying that you are creating a bot for fun 😉. After filling all the info, click "Next".
• In the last page, you will be asked to confirm the details that you provided.
• You may have to confirm your email account. And wait for sometime to get your account approved.

After your account has been approved, you can go your twitter apps page, and click "Create an app", fill the app details and hit "Create" button.

Congratulations! You now have a twitter app 🎉

Now, come back to the twitter apps page, where you can see your app. Click on the "details" button.

Navigate to the "keys and tokens" section, where you will find all the credentials we want. Copy the "API key", "API secret key", "Access token" & "Access token secret".

Building the Twitter Bot 🤖

Now, as we have everything we need to build the Twitter Pinbot, let's fire up our code editor!

Installing Golang

We are going to use Golang to build our Twitter bot. You can download Golang from here.

• First you need to download Golang.
• You will be redirected to a page with installation instructions. Follow these instructions to install Golang.

After installing Golang, we are ready to start with the twitter bot code.

Writing the Twitter Bot Code 👩‍💻👨‍💻

We are going to build a twitter bot which works in the following way:

• You need to follow the twitter bot account, so that the bot can filter out your tweets from all the tweets.
• Then you can "pin", "unpin" or "add" data to the IPFS Cluster Network by tweeting to @botHandle with some specific text.

//To pin a CID with a name/label to IPFS Cluster Network
@botHandle !pin <cid> <name>

//To unpin a CID from IPFS Cluster Network
@botHandle !unpin <cid>

//To add a file(via URL) to IPFS Cluster Network
@botHandle !add <url-to-single-file>

//To check out what the bot can do
@botHandle !help

//Adding photos, memes, videos to IPFS Cluster Network
Tweet to @botHandle attaching photos, memes or videos 

Now, let's start by downloading some boilerplate code.

Clone the boilerplate branch of twitter-pinbot repo:  

$ git clone --single-branch --branch boilerplate https://github.com/simpleaswater/twitter-pinbot

Now, first we need to make a few changes in the config.json file.

{
  "twitter_name": "@botHandle",
  "twitter_id": "@botHandle",
  "consumer_key": "API key",
  "consumer_secret": "API secret key",
  "access_key": "Access token",
  "access_secret": "Access token secret",
  "cluster_peer_addr": "cluster peer multiaddress",
  "cluster_username": "",
  "cluster_password": ""
}

You need to add your @botHandle and the credentials that we got from the twitter developer portal.

Then replace the "cluster peer multiaddress" with:

/ip4/127.0.0.1/tcp/9696/ipfs/QmcrQZ6RJdpYuGvZqD5QEHAv6qX4BrQLJLQPQUrTrzdcgm

If you look closely, you can see that this is the same multiaddress which we added above while changing the IPFS Cluster config file.

We do this to interact with the IPFS Cluster Setup that we created above.

You can leave the "cluster_username" & "cluster_password" blank.

Now, as we are done with our config.json file, let's start working on main.go.

package main

import (
	"regexp"
)

// ConfigFile is the path of the default configuration file
var ConfigFile = "config.json"

// Gateway
var IPFSGateway = "https://ipfs.io"

const twittercom = "twitter.com"

type Action string

// Variables containing the different available actions
var (
	// (spaces)(action)whitespaces(arguments)
	actionRegexp = regexp.MustCompile(`^\s*([[:graph:]]+)\s+(.+)`)
	// (cid)whitespaces(name with whitespaces). [:graph:] does not
	// match line breaks or spaces.
	pinRegexp          = regexp.MustCompile(`([[:graph:]]+)\s+([[:graph:]\s]+)`)
	PinAction   Action = "!pin"
	UnpinAction Action = "!unpin"
	AddAction   Action = "!add"
	HelpAction  Action = "!help"
)

func main() {
	//Let's code ☕
}

The main.go file contains a few constants that we will be using as we move forward in the tutorial.

Let's first add a way to read and manage the data from config.json.

// Config is the configuration format for the Twitter Pinbot
type Config struct {
	TwitterID       string `json:"twitter_id"`
	TwitterName     string `json:"twitter_name"`
	AccessKey       string `json:"access_key"`
	AccessSecret    string `json:"access_secret"`
	ConsumerKey     string `json:"consumer_key"`
	ConsumerSecret  string `json:"consumer_secret"`
	ClusterPeerAddr string `json:"cluster_peer_addr"`
	ClusterUsername string `json:"cluster_username"`
	ClusterPassword string `json:"cluster_password"`
}

// Function to read JSON config file
func readConfig(path string) *Config {
	cfg := &Config{}
	cfgFile, err := ioutil.ReadFile(path)
	if err != nil {
		log.Fatal(err)
	}
	err = json.Unmarshal(cfgFile, &cfg)
	if err != nil {
		log.Fatal(err)
	}
	return cfg
}

Here we have Config struct which manages different key-value pairs from our config.json file. Also, we have the function readConfig(path string) which takes the path of the config.json and parses the JSON from the file.

Next, we create the Bot struct, which is used to manage the different functionalities of the twitter pinbot.  

// Bot is a twitter bot which reads a user's timeline
// and performs actions on IPFS Cluster if the tweets
// match, i.e. a tweet with: "@botHandle !pin <cid> <name>"
// will pin something. The users with pin permissions are
// those who follow the bot. Retweets by users who follow
// the bot should also work. The bot will answer
// the tweet with a result.
type Bot struct {
	ctx    context.Context
	cancel context.CancelFunc

	name          string
	id            string
	twClient      *twitter.Client
	clusterClient client.Client

	followedBy sync.Map

	die chan struct{}
}

You will also have to import these 2 modules:

import (
	"github.com/dghubble/go-twitter/twitter"
	"github.com/ipfs/ipfs-cluster/api/rest/client"
)

Now, let's add the function to create a Bot.

import (
	"github.com/multiformats/go-multiaddr"
    "github.com/dghubble/oauth1"
)

// New creates a new Bot with the Config.
func New(cfg *Config) (*Bot, error) {
	ctx, cancel := context.WithCancel(context.Background())

	// Creating Twitter client
	ocfg := oauth1.NewConfig(cfg.ConsumerKey, cfg.ConsumerSecret)
	token := oauth1.NewToken(cfg.AccessKey, cfg.AccessSecret)
	httpClient := ocfg.Client(ctx, token)
	twClient := twitter.NewClient(httpClient)

	// Creating IPFS Cluster client
	peerAddr, err := multiaddr.NewMultiaddr(cfg.ClusterPeerAddr)
	if err != nil {
		cancel()
		return nil, err
	}
	clusterClient, err := client.NewDefaultClient(&client.Config{
		APIAddr:  peerAddr,
		Username: cfg.ClusterUsername,
		Password: cfg.ClusterPassword,
		LogLevel: "info",
	})

	if err != nil {
		cancel()
		return nil, err
	}

	//Creating Bot Object
	bot := &Bot{
		ctx:           ctx,
		cancel:        cancel,
		twClient:      twClient,
		clusterClient: clusterClient,
		name:          cfg.TwitterName,
		id:            cfg.TwitterID,
		die:           make(chan struct{}, 1),
	}

	bot.fetchFollowers()
	go bot.watchFollowers()
	go bot.watchTweets()
	return bot, nil
}

Here we first create twClient which interacts with the twitter APIs. Also, we create clusterClient which interacts with our IPFS Cluster Setup. Then, using these 2, we create our Bot object.

At last, we have 3 methods:

• fetchFollowers(): This method is used to fetch the list of bot followers, so that we can track their tweets, and respond accordingly.
• watchFollowers(): This goroutine is used to watch the list of bot followers every 60 seconds. In case, somebody follows the bot, the list will be updated within 60 minutes, after which we will start filtering the tweets of the recent follower.
• watchTweets(): This goroutine is used to watch & filter the tweets who mention our @botHandle.

We will implement these 3 methods in a while.

Next, we have a few small utility functions.

// Kill destroys this bot.
func (b *Bot) Kill() {
	b.cancel()
}

// Name returns the twitter handle used by the bot
func (b *Bot) Name() string {
	return b.name
}

// ID returns the twitter user ID used by the bot
func (b *Bot) ID() string {
	return b.id
}

As explained in the comments:

• Kill(): destroys the bot.
• Name(): returns the twitter handle used by the bot.
• ID(): returns the twitter user ID used by the bot.

Now, let's implement the above 3 methods.

//Fetching the bot Followers
func (b *Bot) fetchFollowers() {
	var nextCursor int64 = -1
	includeEntities := false
	for nextCursor != 0 {
		//Getting the List of bot followers
		followers, _, err := b.twClient.Followers.List(
			&twitter.FollowerListParams{
				Count:               200, //Number of bot followers we want in our list
				IncludeUserEntities: &includeEntities,
			})

		if err != nil {
			log.Println(err)
		}

		//Parsing and Storing the ID(s) of the bot followers
		for _, u := range followers.Users {
			_, old := b.followedBy.LoadOrStore(u.ID, struct{}{})
			if !old {
				//Printing the bot followers ScreenName
				log.Println("Friend: ", u.ScreenName)
			}
		}
		nextCursor = followers.NextCursor
		time.Sleep(2 * time.Second)
	}
}

The fetchFollowers() method fetches the list of bot followers using twClient. You can specify a maximum cap(Count) on how many latest followers will be allowed to interact with the bot. We will set this to 200. You can change this as per your wish.

We also parse through the followers list to get the ID of our bot followers, and print out their screen-names.

Next, let's implement the watchFollowers() method.

//Watching bot followers every 60 secs.
//This checks the accounts following
//the bot in every 60 seconds, so that
//any new follower can be added to the
//followers list.
func (b *Bot) watchFollowers() {
	for {
		time.Sleep(60 * time.Second)
		select {
		case <-b.ctx.Done():
		default:
			b.fetchFollowers()
		}
	}
}

As the comments suggest, we call the fetchFollowers() method every 60 seconds to check for new bot followers and update our followers list.

Now, let's implement the watchTweets() method.

//Function to watch tweets that mentions the bot
func (b *Bot) watchTweets() {
	log.Println("watching tweets")

	/*
		Filter Streams return Tweets that match one
		or more filtering predicates such as Track,
		Follow, and Locations.

		Here we are filtering our tweets with
		"!pin", "!unpin", "!add", "!help" or "<bot-name>" strings
		in tweet body.
	*/
	params := &twitter.StreamFilterParams{
		Track: []string{
			PinAction.String(),
			UnpinAction.String(),
			HelpAction.String(),
			AddAction.String(),
			b.Name(),
		},
		StallWarnings: twitter.Bool(true),
	}

	stream, err := b.twClient.Streams.Filter(params)
	if err != nil {
		log.Println(err)
	}

	/*
		Receiving messages of type interface{} isn't very nice,
		it means you'll have to type switch and probably filter
		out message types you don't care about.

		For this,we use Demux, which receives messages and type
		switches them to call functions with typed messages.

		For example, say we're only interested in Tweets.
	*/

	demux := twitter.NewSwitchDemux()
	demux.Tweet = func(t *twitter.Tweet) {
		//Processing the tweets
		b.processTweet(t, t)
	}

	//Constantly watching for new filtered tweets
	for {
		select {
		case <-b.ctx.Done():
			return
		case msg := <-stream.Messages:
			//"Handle(msg)" determines the type of a message
			//and calls the corresponding receiver
			//function with the typed message.
			go demux.Handle(msg)
		}
	}
}

func (a Action) String() string {
	return string(a)
}

The watchTweets() method continuously watches for different tweets, by adding a filter on the tweets. We filter for specific strings occurrence in the tweets, so that we don't reply to the every tweet by who mentions the @botHandle.

After filtering the tweets, we process the tweets using processTweet(t, t).

import (
	"github.com/ipfs/go-cid"
    "github.com/ipfs/ipfs-cluster/api"
)

//Process the filetered tweets and handle the tweets according to the
//desired action.
func (b *Bot) processTweet(tweet *twitter.Tweet, srcTweet *twitter.Tweet) {
	if tweet == nil {
		return
	}

	if srcTweet == nil {
		srcTweet = tweet
	}

	// Skip processing our own tweets (written by us)
	// and quotes or retweets we've made (origUser is us)
	// (avoid potential loops)
	if tweet.User.IDStr == b.ID() || srcTweet.User.IDStr == b.ID() {
		return
	}

	action, arguments, urls, err := b.parseTweet(tweet)
	if err != nil {
		b.tweet(err.Error(), tweet, srcTweet, false)
		return
	}

	log.Printf("Parsed: %s, %s, %s\n", action, arguments, urls)

	_, ok := b.followedBy.Load(srcTweet.User.ID)
	if !ok && action.Valid() {
		log.Println("Error: NoFollow")
		b.tweet("Follow me, and try again.", tweet, srcTweet, false)
		return
	}
	if !ok {
		return
	}

	// Process actions
	switch action {
	case PinAction:
		//Pin the Tweet to the IPFS Cluster network
		b.pin(arguments, tweet, srcTweet)
	case UnpinAction:
		//UnPin the Tweet to the IPFS Cluster network
		b.unpin(arguments, tweet, srcTweet)
	case AddAction:
		////Add the Tweet to the IPFS Cluster network
		b.add(arguments, tweet, srcTweet)
	case HelpAction:
		//Tweet a "help tweet" that demostrates how to use the bot
		b.tweetHelp(tweet, srcTweet)
	default:
		//Need to handle default tweets by adding the assets
		log.Println("no handled action for this tweet")
	}

	// Add any media urls
	if len(urls) > 0 {
		log.Println("adding media: ", urls)
		out := make(chan *api.AddedOutput, 1)
		go func() {
			cids := []cid.Cid{}
			for added := range out {
				log.Printf("added %s\n", added.Cid)
				cids = append(cids, added.Cid)
			}
			if len(cids) > 0 {
				b.tweetAdded(cids, tweet, srcTweet)
			}
		}()
		params := api.DefaultAddParams()
		params.Wrap = true
		params.Name = "Tweet-" + tweet.IDStr
		err := b.clusterClient.Add(context.Background(), urls, params, out)
		if err != nil {
			log.Println(err)
		}
	}

	// If the tweet has retweets, process them as if they were
	// from this user.
	retweets := []*twitter.Tweet{tweet.QuotedStatus, tweet.RetweetedStatus}
	for _, rt := range retweets {
		b.processTweet(rt, srcTweet)
	}
}

The processTweet(tweet, tweet) method processes the filtered tweets and handles the tweets according to the desired action.

First, we check if the tweet is originated from the bot's account or not. If the tweet originated from the bot's account then we return, otherwise we will end up in an infinite loop, tweeting our own tweet.

Then we parse the tweets using parseTweet(tweet) to get the action, arguments & urls from the tweet. We will implement the parseTweet(tweet) in a while.

After that we check 2 things. First, we check that the tweets must originate from accounts that follow the bot. Second, we check if the action is a valid action or not.

We can check the validity of the parsed action using the following function.

func (a Action) Valid() bool {
	switch a {
	case PinAction, UnpinAction, AddAction, HelpAction:
		return true
	}
	return false
}

If the tweet is not from someone who follows the bot, then we return.

Then, in the switch statement, we check for action type and perform different actions accordingly. We will implement these different methods(pin, unpin, etc.) in a while.

After that we check for any urls, and use clusterClient to Add the urls to the IPFS Cluster setup, and return the content identifiers (CIDs).

And finally, we handle an edge case in which if the tweet has retweets, then we process them as if they were from the bot follower, and thus calling the processTweet(rt, srcTweet).

Now, let's implement the parseTweet(tweet) method.

// parseTweet returns Action, arguments, media urls, and error
func (b *Bot) parseTweet(tweet *twitter.Tweet) (Action, string, []string, error) {
	// Extended tweet? let's use the entities from the extended tweet then.
	if tweet.ExtendedTweet != nil {
		tweet.Entities = tweet.ExtendedTweet.Entities
		tweet.ExtendedEntities = tweet.ExtendedTweet.ExtendedEntities
		tweet.FullText = tweet.ExtendedTweet.FullText

	}
	text := tweet.FullText
	if text == "" {
		text = tweet.Text
	}

	log.Println("Parsing:", text)

	// remote our username if they started with it
	text = strings.TrimPrefix(text, b.name)
	var action Action
	var arguments string

	if text == " "+string(HelpAction) {
		return HelpAction, "", []string{}, nil
	}

	// match to see if any action
	matches := actionRegexp.FindAllStringSubmatch(text, -1)
	if len(matches) > 0 {
		firstMatch := matches[0]
		action = Action(firstMatch[1]) // first group match
		arguments = firstMatch[2]      // second group match
	}

	urls := extractMediaURLs(tweet)
	return action, arguments, urls, nil
}

Here, we first handle the ExtendedTweet type, which is a enterprise feature of Twitter, in which you can can have tweets with no word limit.

Then we apply regular expression to our tweets to check for action & arguments. For extracting the urls we use extractMediaURLs(tweet).

//Extracting MediaURLs from tweets
func extractMediaURLs(tweet *twitter.Tweet) []string {
	var urls []string

	// Grab any media entities from the tweet
	for _, m := range media(tweet.ExtendedEntities) {
		urls = append(urls, extractMediaURL(&m))
	}

	if len(urls) == 0 {
		// If no extended entitites, try with traditional.
		for _, m := range media(tweet.Entities) {
			urls = append(urls, extractMediaURL(&m))
		}
	}
	return urls
}

// takes *Entities or *MediaEntities
func media(ent interface{}) []twitter.MediaEntity {
	if ent == nil {
		return nil
	}

	switch ent.(type) {
	case *twitter.Entities:
		e := ent.(*twitter.Entities)
		if e != nil {
			return e.Media
		}
	case *twitter.ExtendedEntity:
		e := ent.(*twitter.ExtendedEntity)
		if e != nil {
			return e.Media
		}
	}
	return nil
}

Here we loop through the ExtendedEntities and Entities to grab the media entities. Now, we extract the actual MediaURL using the extractMediaURL(MediaEntity) function.

type byBitrate []twitter.VideoVariant

func (vv byBitrate) Len() int           { return len(vv) }
func (vv byBitrate) Swap(i, j int)      { vv[i], vv[j] = vv[j], vv[i] }
func (vv byBitrate) Less(i, j int) bool { return vv[i].Bitrate < vv[j].Bitrate }

//Extracting the highest bitrate MediaURL from MediaEntity
func extractMediaURL(me *twitter.MediaEntity) string {
	switch me.Type {
	case "video", "animated_gif":
		variants := me.VideoInfo.Variants
		sort.Sort(byBitrate(variants))
		// pick video with highest bitrate
		last := variants[len(variants)-1]
		return last.URL
	default:
		return me.MediaURL
	}
}

extractMediaURL(MediaEntity) extracts the MediaURL with the highest bitrate from the available variants of the video or animated_gif URLs.

Now, as we have parsed all the necessary information from the filtered tweets, we can proceed to implement pin, unpin,and add methods.

//Function to pin a CID to IPFS Cluster
func (b *Bot) pin(args string, tweet, srcTweet *twitter.Tweet) {
	log.Println("pin with ", args)
	pinUsage := fmt.Sprintf("Usage: '%s <cid> <name>'", PinAction)

	matches := pinRegexp.FindAllStringSubmatch(args, -1)
	if len(matches) == 0 {
		b.tweet(pinUsage, srcTweet, nil, false)
		return
	}

	firstMatch := matches[0]
	cidStr := firstMatch[1]
	name := firstMatch[2]
	c, err := cid.Decode(cidStr)
	if err != nil {
		b.tweet(pinUsage+". Make sure your CID is valid.", tweet, srcTweet, false)
		return
	}

	_, err = b.clusterClient.Pin(context.Background(), c, api.PinOptions{Name: name})
	if err != nil {
		log.Println(err)
		b.tweet("An error happened pinning. I will re-start myself. Please retry in a bit.", srcTweet, nil, false)
		b.die <- struct{}{}
		return
	}
	waitParams := client.StatusFilterParams{
		Cid:       c,
		Local:     false,
		Target:    api.TrackerStatusPinned,
		CheckFreq: 10 * time.Second,
	}
	ctx, cancel := context.WithTimeout(b.ctx, 10*time.Minute)
	defer cancel()
	_, err = client.WaitFor(ctx, b.clusterClient, waitParams)
	if err != nil {
		log.Println(err)
		b.tweet("IPFS Cluster has been pinning this for 10 mins. This is normal for big files. Otherwise, make sure there are providers for it. Don't worry, Cluster will keep at it for a week before giving up.", srcTweet, nil, false)
		return
	}

	b.tweet(fmt.Sprintf("Pinned! Check it out at %s/ipfs/%s", IPFSGateway, cidStr), tweet, srcTweet, true)
}

Here we again apply regular expression on args to validate the tweet text and content identifier(cidStr) and name(name) for the pin to be added to IPFS Cluster setup.

We have added a number of error checks for:

• Checking of the CID(cidStr) is valid.
• Checking if some unexpected error happened while pinning the CID to IPFS Cluster setup using clusterClient.Pin.
• Checking if the pinning process takes more that 10 seconds(timeout), which is sufficient for big files.

In case, everything is fine, the bot tweets the IPFSGateway link for the added pin using the tweet method. We will implement the tweet method in a while.

Now, let's see the unpin method.

// Function to unpin a CID from IPFS Cluster network
func (b *Bot) unpin(args string, tweet, srcTweet *twitter.Tweet) {
	log.Println("unpin with ", args)
	unpinUsage := fmt.Sprintf("Usage: '%s <cid>'", UnpinAction)

	c, err := cid.Decode(args)
	if err != nil {
		b.tweet(unpinUsage+". Make sure your CID is valid.", tweet, srcTweet, false)
		return
	}

	_, err = b.clusterClient.Unpin(context.Background(), c)
	if err != nil && !strings.Contains(err.Error(), "uncommited to state") {
		log.Println(err)
		b.tweet("An error happened unpinning. I will re-start myself. Please retry in a bit.", srcTweet, nil, false)
		b.die <- struct{}{}
		return
	}
	waitParams := client.StatusFilterParams{
		Cid:       c,
		Local:     false,
		Target:    api.TrackerStatusUnpinned,
		CheckFreq: 10 * time.Second,
	}
	ctx, cancel := context.WithTimeout(b.ctx, time.Minute)
	defer cancel()
	_, err = client.WaitFor(ctx, b.clusterClient, waitParams)
	if err != nil {
		log.Println(err)
		b.tweet("IPFS Cluster did not manage to unpin the item, but it's trying...", srcTweet, nil, false)
		return
	}

	b.tweet(fmt.Sprintf("Unpinned %s! :'(", args), tweet, srcTweet, false)
}

In the unpin method we use cid.Decode to get the CID(c) from the args. Similar to pin, here also we have a few checks in place to handle errors.

• Making sure that the CID(c) is valid.
• Checking if some unexpected error happened while unpinning the CID from IPFS Cluster setup using clusterClient.Unpin.
• Checking if the unpinning process takes more that 10 seconds(timeout).

In case, everything is fine, the bot tweets the acknowledging that the CID has been unpinned.

Now, let's see the add method.

//Function to add URL to IPFS Cluster network
func (b *Bot) add(arg string, tweet, srcTweet *twitter.Tweet) {
	log.Println("add with ", arg)
	addUsage := fmt.Sprintf("Usage: '%s <http-or-https-url>'")
	url, err := url.Parse(arg)
	if err != nil {
		b.tweet(addUsage+". Make sure you gave a valid url!", srcTweet, nil, false)
		return
	}
	if url.Scheme != "http" && url.Scheme != "https" {
		b.tweet(addUsage+". Not an HTTP(s) url!", srcTweet, nil, false)
		return
	}

	if url.Host == "localhost" || url.Host == "127.0.0.1" || url.Host == "::1" {
		b.tweet("ehem ehem...", srcTweet, nil, false)
		return
	}

	out := make(chan *api.AddedOutput, 1)
	go func() {
		cids := []cid.Cid{}
		for added := range out {
			cids = append(cids, added.Cid)
		}
		if len(cids) > 0 {
			b.tweetAdded(cids, tweet, srcTweet)
		}
	}()

	params := api.DefaultAddParams()
	params.Wrap = true
	params.Name = "Tweet-" + tweet.IDStr
	log.Println([]string{arg})
	err = b.clusterClient.Add(context.Background(), []string{arg}, params, out)
	if err != nil {
		log.Println(err)
		b.tweet("An error happened adding. I will re-start myself. Please retry in a bit.", srcTweet, nil, false)
		b.die <- struct{}{}
		return
	}
}

The add method is used to add data via URLs in the tweet to the IPFS Cluster setup. We parse the url using url.Pars(arg). As the above 2 methods, add also has checks to counter catch the errors.

• Making sure that the URL(c) is valid.
• Checking if the url.Scheme and url.Host are valid.
• Checking if there is an unexpected error while adding files to the IPFS Cluster setup.

In case, everything is fine, the bot tweets CID(s) of the file(via the URL) and the folder-wrap(file wrapped in a folder) using tweetAdded.

//Function for Tweeting for the Add Action
func (b *Bot) tweetAdded(cids []cid.Cid, tweet, srcTweet *twitter.Tweet) {
	msg := "Just added this to #IPFS Cluster!\n\n"
	for i, c := range cids {
		if i != len(cids)-1 {
			msg += fmt.Sprintf("• File: %s/ipfs/%s\n", IPFSGateway, c)
		} else { // last
			msg += fmt.Sprintf("• Folder-wrap: %s/ipfs/%s\n", IPFSGateway, c)
		}
	}
	b.tweet(msg, tweet, srcTweet, true)
}

Here is the implementation of the tweetHelp method which tweets how to use the bot.

//Function for Tweeting for the Help Action
func (b *Bot) tweetHelp(tweet, srcTweet *twitter.Tweet) {
	help := fmt.Sprintf(`Hi! Here's what I can do:

!pin <cid> <name>
!unpin <cid>
!add <url-to-single-file>
!help

You can always prepend these commands mentioning me (%s).

Happy pinning!
`, b.name)
	b.tweet(help, srcTweet, nil, false)
}

Now, let's implement the tweet method which sends tweets quoting or replying to the filtered tweets.

// tweets sends a tweet quoting or replying to the given tweets.
// srcTweet might be nil.
// Otherwise it just posts the message.
func (b *Bot) tweet(msg string, inReplyTo, srcTweet *twitter.Tweet, quote bool) {
	tweetMsg := ""
	params := &twitter.StatusUpdateParams{}
	sameTweets := false

	if inReplyTo == nil {
		tweetMsg = msg
		goto TWEET
	}

	sameTweets = srcTweet == nil || inReplyTo.ID == srcTweet.ID
	params.InReplyToStatusID = inReplyTo.ID

	switch {
	case sameTweets && !quote:
		// @user msg (reply thread)
		tweetMsg = fmt.Sprintf("@%s %s", inReplyTo.User.ScreenName, msg)
	case sameTweets && quote:
		// @user msg <permalink> (quote RT)
		tweetMsg = fmt.Sprintf(".@%s %s %s",
			inReplyTo.User.ScreenName,
			msg,
			permaLink(inReplyTo),
		)
	case !sameTweets && !quote:
		// @user @srcUser msg (reply thread)
		tweetMsg = fmt.Sprintf("@%s @%s %s",
			inReplyTo.User.ScreenName,
			srcTweet.User.ScreenName,
			msg,
		)
	case !sameTweets && quote:
		// @srcuser <replyPermalink> (quote RT mentioning src user)
		tweetMsg = fmt.Sprintf(".@%s %s %s",
			srcTweet.User.ScreenName,
			msg,
			permaLink(inReplyTo),
		)

	}

TWEET:
	log.Println("tweeting:", tweetMsg)
	newTweet, _, err := b.twClient.Statuses.Update(tweetMsg, params)
	if err != nil {
		log.Println(err)
		return
	}
	_ = newTweet
	// if quote { // then retweet my tweet after a minute
	// 	go func() {
	// 		time.Sleep(time.Minute)
	// 		_, _, err := b.twClient.Statuses.Retweet(newTweet.ID, nil)
	// 		log.Println("retweeted: ", tweetMsg)
	// 		if err != nil {
	// 			log.Println(err)
	// 			return
	// 		}
	// 	}()
	// }
	return
}

func permaLink(tweet *twitter.Tweet) string {
	return fmt.Sprintf("https://%s/%s/status/%s", twittercom, tweet.User.ScreenName, tweet.IDStr)
}

Now, finally, let's implement the main function of our pinbot, which will complete our pinbot code.

func main() {
	//Fetching the optional path from command line
	path := flag.String("config", ConfigFile, "path to config file")
	flag.Parse()

	//Reading the config file
	cfg := readConfig(*path)

	//Creating a new bot
	bot, err := New(cfg)
	if err != nil {
		log.Fatal(err)
	}
	log.Println("Bot created:", bot.Name(), bot.ID())

	// Wait for SIGINT and SIGTERM (HIT CTRL-C)
	ch := make(chan os.Signal)
	signal.Notify(ch, syscall.SIGINT, syscall.SIGTERM)
	select {
	case sig := <-ch:
		log.Println(sig)
	case <-bot.die:
	}

	bot.Kill()
}

In the main function, we read the config file from the path using the readConfig function. Then using the config file key-values(cfg) we create our bot using the New(cfg) function.

Also, we wait for SIGINT and SIGTERM so that hitting CTRL-C kills out bot.

Final Showdown

Now, it's time to test our code.

First build the code by executing go build in the directory/folder(twitter-pinbot) where you have saved your main.go file.

This will download all the modules that we have imported. After this is done, you will see an executable file named twitter-pinbot.

Before starting, make sure that both, IPFS dameon and IPFS Cluster daemon are running, that we started above.

Now, you can run the executable file by executing ./twitter-pinbot.

If everything works fine, then you would see a list of your bot followers like this:

Now, we are waiting for your tweets...

If you get stuck in any part or have any queries/doubts, then feel free to reach us out on our discord channel.