Kafka - How to Get Started

Table of Contents

1. Introduction
2. Login
3. Create a Cluster
4. Get Started with an Event-Driven Application with C#
5. Conclusion

Introduction

This article walks through getting started with Kafka, from signing up and creating a cluster to producing and consuming data using a C# client.

Login

You can try Kafka for 30 days for free, so just log in to Kafka with your email.

Create a Cluster

Once you register with a new email, you'll receive $400 credit as a free trial for 30 days.

Get Started with an Event-Driven Application with C#

Set Up a Topic

Here, we enter the topic name manually in the UI. We'll need to set up a cluster key and secret later in order to communicate from the C# client.

Click the Create Kafka cluster API key to get the API key.

Produce Data

The following is the C# code to send a message to Kafka as a string key and value:

    static void SendKafkaString(string topic, string key, string value)
    {
        IConfiguration configuration = readConfig();
        Message<string, string> message = new Message<string, string> { Key = key, Value = value };
        using (var producer = new ProducerBuilder<string, string>(configuration.AsEnumerable()).Build())
        {
            producer.Produce(topic, message,
              (deliveryReport) =>
              {
                  KafkaResult(deliveryReport, topic);
              }
            );

            producer.Flush(TimeSpan.FromSeconds(10));
        }
    }

SendKafkaString("topic_0", "key_2000", "test12");

This is the actual result shown in the Kafka UI.

var myJson = @"{
    ""str"": ""a string"",
    ""int"": 1,
    ""obj"": {
    ""key"": ""value""
    }
}";

SendKafkaString("topic_0", "key_2000", myJson);

Since the value type is string, we can also send JSON to Kafka.

The following is how to serialize an object as JSON and send it to Kafka with the C# client.

var objectTest = new ModelClass { Name = "yo" };
SendKafkaBytes("topic_0", "key_2100", System.Text.Json.JsonSerializer.SerializeToUtf8Bytes(objectTest));

This is the content of the code, which takes a byte array as a parameter.

static void SendKafkaBytes(string topic, string key, byte[] value)
{
    IConfiguration configuration = readConfig();
    Message<string, byte[]> message = new() { Key = key, Value = value };
    using var producer = new ProducerBuilder<string, byte[]>(configuration.AsEnumerable()).Build();
    producer.Produce(topic, message,
        (deliveryReport) =>
        {
            KafkaResult(deliveryReport, topic);
        }
    );

    producer.Flush(TimeSpan.FromSeconds(10));
}

This is the result showing how to send an object to Kafka.

Consume Data

Conclusion

Kafka provides a powerful event-driven messaging platform. With the Confluent cloud service, you can quickly set up a cluster, create topics, and start producing and consuming data. The C# client library makes it straightforward to integrate Kafka into .NET applications using both string and byte array serialization.