kafka-connect-github-source

The GitHub Source Connector streams issues and pull requests from a GitHub repository into a Kafka topic using the GitHub API.

Key behaviors:

• Update-based streaming: Issues are pulled based on the updated_at field, so any update to an issue or pull request triggers a new event in the stream.
• Log Compaction: Because it streams updates to existing entities, the resulting Kafka topic is an excellent candidate for demonstrating log compaction.
• Use Case: It is designed as an educational example for learning how to write non-trivial Kafka Source Connectors.

To run the connector locally for development, ensure you have Java 8 installed.

1. Configure config/worker.properties to point to your Kafka cluster.
2. Build the project using ./build.sh.
3. Run the connector using ./run.sh.

If you have Docker installed, ./run.sh will automatically pull a Dockerfile and run the connector in standalone mode.

To use the connector, provide a configuration file with the following properties. Note that github.owner and github.repo define the target repository, and since.timestamp determines the starting point for data retrieval.

It is highly recommended to provide auth.username and auth.password for authenticated access to the GitHub API.

name=GitHubSourceConnectorDemo
tasks.max=1
connector.class=com.simplesteph.kafka.GitHubSourceConnector
topic=github-issues
github.owner=kubernetes
github.repo=kubernetes
since.timestamp=2017-01-01T00:00:00Z
# I heavily recommend you set those two fields:
auth.username=your_username
auth.password=your_password