How to Build a Blockchain from Scratch with Go

Blockchain technology has taken the world by storm, powering cryptocurrencies like Bitcoin and Ethereum and offering a decentralized, immutable ledger for recording transactions. As a developer, what better way to truly understand blockchain than by coding one yourself from scratch?

In this tutorial, we‘ll do exactly that using the Go programming language. By the end, you‘ll have a working blockchain implementation and a deep understanding of core concepts like blocks, hashes, Merkle trees, consensus algorithms, and more.

Go is an excellent language for blockchain development due to its simplicity, strong typing, memory safety, speed, and built-in support for cryptography and concurrency. It‘s no surprise that many blockchain projects like Ethereum, Hyperledger Fabric, and Cosmos are written in Go.

Components of a Basic Blockchain

At its core, a blockchain is simply a linked list of blocks, with each block containing a list of transactions. The key properties that make a blockchain secure and trustworthy include:

• Immutability – Once a block is added it cannot be changed
• Distributed – The blockchain is shared among many peer nodes
• Cryptographically secure – Hashes ensure the integrity of blocks and transactions

Here are the main components we‘ll need to build our basic blockchain:

• Block – A container data structure that holds transactions
• Transaction – Represents an exchange of value between parties
• Hash – A fixed-length fingerprint of data, generated by a hash function
• Merkle Tree – A tree of transaction hashes allowing efficient verification
• Consensus – The algorithm that enables agreement on the blockchain state (we‘ll use Proof-of-Work)

Step 1: Defining the Core Data Structures

Let‘s start by defining the data structures for Block and Transaction:

type Block struct {
    Hash          []byte
    PrevHash      []byte
    Timestamp     int64 
    Transactions  []*Transaction
    Nonce         int
}

type Transaction struct {
    From    string
    To      string 
    Amount  float64
    Fee     float64
}

A Block contains the hash of the block, the hash of the previous block (to form the chain), a timestamp, a list of transactions, and a nonce (used for Proof-of-Work).

A Transaction contains the sender (From), recipient (To), amount being transferred, and the transaction fee.

Next let‘s define the Blockchain type, which is just a slice of blocks:

type Blockchain struct {
    Blocks []*Block
}

Step 2: Generating Block Hashes

Hashing is a crucial part of blockchains, used to generate a fixed-size "fingerprint" of data that cannot be reversed. We‘ll use the SHA-256 hash function to hash our blocks:

func (b *Block) Hash() []byte {
    data := bytes.Join([][]byte{
        b.PrevHash,
        b.HashTransactions(),
        ToHex(b.Timestamp),
        ToHex(int64(b.Nonce)),
    }, []byte{})

    hash := sha256.Sum256(data)
    return hash[:]
}

This method joins together the previous block hash, the Merkle root hash of the block‘s transactions, the timestamp, and nonce into a byte slice. It then hashes the concatenated data using SHA-256 and returns the hash digest.

Step 3: Creating a Merkle Tree

A Merkle tree is a binary tree of transaction hashes that allows efficient verification of transactions within a block. The root of the tree serves as the representative hash for all transactions.

Here‘s how we can generate a Merkle tree:

func (b *Block) HashTransactions() []byte {
    var txHashes [][]byte
    for _, tx := range b.Transactions {
        txHashes = append(txHashes, tx.Hash())
    }
    tree := NewMerkleTree(txHashes)
    return tree.RootNode.Data
}

func NewMerkleTree(data [][]byte) *MerkleTree {
    var nodes []MerkleNode
    for _, datum := range data {
        nodes = append(nodes, NewMerkleNode(nil, nil, datum))
    }

    for len(nodes) > 1 {
        if len(nodes)%2 != 0 {
            nodes = append(nodes, nodes[len(nodes)-1])
        }

        var level []MerkleNode
        for i := 0; i < len(nodes); i += 2 {
            node := NewMerkleNode(nodes[i], nodes[i+1], nil)
            level = append(level, *node)
        }

        nodes = level
    }

    return NewMerkleTree(nodes[0])
}

We start by hashing each transaction individually. Then we build the tree bottom-up by pairing off nodes, concatenating their hashes, and hashing them again, until there is only one node left – the root.

Step 4: Implementing Proof-of-Work Consensus

Proof-of-Work (PoW) is a consensus algorithm used to determine which peer in the network has the right to add a new block. "Miners" compete to solve a difficult mathematical problem, and the first one to solve gets to add their block.

We can implement a simple PoW algorithm as follows:

const targetBits = 24

func (b *Block) Validate() bool {
    target := big.NewInt(1)
    target.Lsh(target, uint(256-targetBits))

    blockHash := big.NewInt(1).SetBytes(b.Hash())
    return blockHash.Cmp(target) == -1
}

func (b *Block) MineBlock(difficulty int) {
    target := big.NewInt(1)
    target.Lsh(target, uint(256-difficulty))

    for b.Nonce < maxNonce {
        blockHash := big.NewInt(1).SetBytes(b.Hash())
        if blockHash.Cmp(target) == -1 {
            break
        }
        b.Nonce++
    }
}

The "mathematical problem" in PoW essentially requires finding a hash that is numerically smaller than a set target value. The smaller the target, the more difficult it is to find a valid hash. Miners keep trying different nonces until they find one that produces a valid hash.

Step 5: Adding and Verifying Blocks

To add a new block to the blockchain:

1. Gather new transactions into a block
2. Mine the block (PoW) to obtain a valid hash
3. Append the block to the existing chain

func (bc *Blockchain) AddBlock(txs []*Transaction) {
    prevBlock := bc.Blocks[len(bc.Blocks)-1]
    newBlock := NewBlock(txs, prevBlock.Hash)

    newBlock.MineBlock(targetBits)
    bc.Blocks = append(bc.Blocks, newBlock)
}

To verify that an existing blockchain is valid, we need to check:

1. Each block has a valid PoW hash
2. Each block‘s PrevHash field points to the hash of the previous block

func (bc *Blockchain) IsValid() bool {
    for i := 1; i < len(bc.Blocks); i++ {
        prevBlock := bc.Blocks[i-1]
        currBlock := bc.Blocks[i]

        if !currBlock.Validate() {
            return false
        }

        if !bytes.Equal(currBlock.PrevHash, prevBlock.Hash()) {
            return false
        }
    }
    return true
}

Step 6: Handling a Distributed Network

So far we‘ve described interactions with a local blockchain. However, the real power of blockchains lies in their distributed nature. Blockchains operate over a peer-to-peer (P2P) network, with each node maintaining its own copy of the chain.

Key aspects of a distributing the blockchain include:

• Connecting to peer nodes – Using libraries like libp2p to form a P2P network
• Discovering new peers – Exchanging lists of known peers to keep the network connected
• Syncing the blockchain – Requesting missing blocks from peers to get the latest state
• Broadcasting new blocks and transactions – Informing peers when a new block is mined or a new transaction is made

Implementing a P2P network is a complex topic that could easily fill its own post. For a deeper dive, I recommend checking out this tutorial on building a blockchain P2P network in Go.

Step 7: Interacting with the Blockchain

The last piece of the puzzle is giving clients a way to interact with our blockchain node to query for information and submit new transactions.

A common way to do this is to expose a REST API over HTTP. We can use the net/http package to create API endpoints:

http.HandleFunc("/chain", func(w http.ResponseWriter, r *http.Request) {
    json.NewEncoder(w).Encode(blockchain.Blocks)
})

http.HandleFunc("/mine", func(w http.ResponseWriter, r *http.Request) {
    newTxs := blockchain.FlushTxPool() 
    blockchain.AddBlock(newTxs)
    w.WriteHeader(http.StatusCreated)
})

http.HandleFunc("/tx", func(w http.ResponseWriter, r *http.Request) {
    var tx Transaction
    json.NewDecoder(r.Body).Decode(&tx)
    blockchain.PendingTx = append(blockchain.PendingTx, tx)
    w.WriteHeader(http.StatusAccepted) 
})

log.Fatal(http.ListenAndServe(":3000", nil))

This sets up the API endpoints:

• GET /chain – Returns the full blockchain
• POST /mine – Tells the node to mine a new block
• POST /tx – Submits a new transaction to the pending pool

With this API in place, clients can fetch the current blockchain state, mine new blocks, and broadcast transactions to be included in the next block.

Challenges and Next Steps

Congratulations! By now you should have a working blockchain implementation in Go and an understanding of the core concepts involved. However, we‘ve only just scratched the surface.

Some of the key challenges and areas for further development include:

• Security – Our PoW implementation is susceptible to 51% attacks, better alternatives are needed
• Scalability – The basic blockchain structure doesn‘t scale well to high tx volumes, techniques like sharding, sidechains or DAGs could help
• Smart contracts – Executing arbitrary code on the blockchain opens up a world of possibilities but introduces new complexities
• Wallet management – Tools are needed for users to safely store and transact with their blockchain assets

I encourage you to extend the basic blockchain we‘ve built and experiment with enhancements like the ones above. There are endless opportunities to innovate in this space!

Conclusion

In this post, we‘ve taken a hands-on look at the foundational concepts of blockchains by building one from the ground up in Go. I hope this has equipped you with the knowledge and skills to start contributing to real blockchain projects or even launch your own.

The full source code is available on Github – feel free to clone the repo and play around. If you have any questions or feedback, let me know in the comments. Thanks for reading, now go forth and build the future of blockchains!