So I've been eager to use something easier to set up than generating those TLS certificates and such for development environment, yet still somewhat more secure than just hardcoded JWT inside a file in Pulsar VM (container in this case). If you're not familiar with JWT setup for Pulsar, it's not necessary, but you can think of it as lesser OAuth2 since they both rely mainly on verifying JWT but process for logging in with Brokers and Clients can be different.

KeyCloak setup

KeyCloak requires only client app to be set up for this. App needs to Service Accounts Enabled to be on which is basically client_credentials grant type in OAuth2. In order to see this option Access Type has to be set to "confidential".

Next thing to set it the "specialised" claim. This claim would be easy to set as realm role or client role but Apache Pulsar has issues reading complex types in JWT by default, and so to skip writing custom authentication resolver we must enforce claim inside JWT to be simple string. Problem is that role claims by default go into array even if specified as non-multivalued. So one could get something like

"role": "[admin]" or
"role": ["admin"]

but we need

"role": "admin"

for Pulsar to play nice with defaults.
which is unexpected given that it should be only 1 string. To skip this hassle go and set-up Hardcoded claim.

Token claim name can be role and value can be for this purpose superuser. These values are important for Pulsar setup so keep track of what was put in here. Pulsar will require you to set up role name for the superuser which is dedicated role for managing all Pulsar stuff. On the other hand other roles might be used but then you need to remember to setup tenant namespaces to be accessible by that role. This means one would have to manage that pulsar instance manually through REST or CLI or something else to setup tenant namespaces manageable by user role you wish to use by certain app.

Next, get the public key of the realm. Easy you might thing? Well no, again because of Pulsar. Go to Realm Settings and into Keys tab.

There you should find RS256 under active tab if all defaults were left as-is. If you wish to use different algorithm please also mark that as important and keep track of it as well as of the key value. If not then please continue and click on Public Key of that RS256 row.

You should get some Base64 text. It will look like gibberish of alphanumeric and some special characters.

Now copy that value and somehow convert it to bytes and store as file. Pulsar lib for java at 2.7 will try to read it as x509 which will fail as it needs to be decoded into bytes prior to this. Steps to do it are:

1. Copy base64 value
2. Decode it back to bytes using either something online or simply program your own mini script
3. Store bytes into file A Groovy example:

byte[] key = "MACakje21/adkjwp9qmk4231/ea\d;qwdq=="..decodeBase64()
File f = new File("yourfilename")
f.bytes = key

Why Groovy? Well it was fast to write and has all those .decode and .bytes and... you can execute it as script through IntelliJ - Tools -> Groovy Console. I was mainly using JVM things with pulsar and Python but Python fails to use KeyCloak settings because of some previous bug which was fixed for Java and C++ but apparently not for Python which should rely on C++ lib.

Pulsar image

After setting up KeyCloak and storing that key to some file we can build a customised image that uses OAuth2 in standalone mode.

Now create a file that you will later copy into image. Please visit Pulsar docs if you want more info. If not then just treat this as credentials file for OAuth2 client app so client ID and secret from KeyCloak (go to App then first tab contains ID and Credentials tab has the secret). Below is simple example

{
    "type": "client_credentials",
    "client_id": "pulsar",
    "client_secret": "dadada7a7a7-a7ad77ad7da-da7a7ad7ad77",
    "issuer_url": "https://host.docker.internal/auth/realms/test"
}

I named it oauth2.json. _Download files conf/standalone.conf and conf/client.conf from Pulsar repository. These files will be changed to configure Pulsar image to use Authentication.

Configure standalone.conf. Set properties in that file to something like in below example@

authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken
# here goes the role you've picked in case you want KeyCloak client app to behave in superuser or else just leave default
superUserRoles=superuser

#brokers need to login to other things so they must also be set up
brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2
#oauth2.json path should be set to wherever Dockerfile is copying it
brokerClientAuthenticationParameters={"issuerUrl": "https://host.docker.internal/auth/realms/test","privateKey": "/pulsar/oauth2.json","audience": "pulsar"}
#bytes from base64, file path must be as defined by Dockerfile
tokenPublicKey=file:///pulsar/oauth_public.key

This will tell brokers to use JSON passed as a parameter, extract file under privateKey and post it's content as payload to isssuerUrl. Thus generating the JWT and refreshing when necessary. brokerClientAuthenticationPlugin is different from the authenticationProviders because broker generates JWT every now and then by authenticating to KeyCloak while main provider will only verify that token is valid (non-expired and signature is good). tokenPublicKey will instruct which file to use to verify JWT signature. It's the same as for simple JWT setup without the OAuth2. What's different here is that we need public key from the server while JWT can have key pair generated for Pulsar only and used by it or symmetric key for the same purpose. OAuth2 should not share it's private keys thus symmetric key is not a good option. KeyCloak by default uses asymmetric so it helps quite a bit.

Alternatively, if KeyCloak is set to never expire tokens, one can be generated in advance for Pulsar brokers and then copied into Pulsar container. This could be used instead of AuthenticationOAuth2 and parameters for broker would be file path to JWT token file. I would not advice alternative approach as it forces you to use never-expiring tokens on all of your realm not just that particular app.

Don't forget to configure your CLI tools. These are needed to run, add tenants/namespaces..., test topics, and do any kind of administration through CLI.

authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2

authParams={"issuerUrl": "https://host.docker.internal/auth/realms/test","privateKey": "/pulsar/oauth2.json","audience": "pulsar"}

So same values are used as for broker configuration plugin and params.

And finally Dockerfile. I made something like this:

FROM apachepulsar/pulsar-standalone:2.7.0
WORKDIR /pulsar
#using workdir /pulsar we copy public key and it's path will be /pulsar/oauth_public.key and this value must be same as in properties for standalone
COPY oauth_public.key oauth_public.key
#note oauth2.json is being copied from into workdir /pulsar which is linked in brokerClientAuthenticationParameters
COPY oauth2.json oauth2.json
#client for CLI tools
COPY client.conf conf/client.conf
#
COPY standalone.conf conf/standalone.conf

If you used same naming then you should be good to go with this image. One note is that I'm relying on -standalone instead of just pulsar or pulsar-all. If you choose something else you might want to add start-up command at the end to run pulsar when you trigger container run from docker. Standalone version runs automatically but others I used didn't so optionally use at the end

CMD [ "/pulsar/bin/pulsar", "standalone"]

if you used other image as a base.

Build image and run

This should be sufficient to run it. I made something similar and tested it with CLI, Java, and Python clients where I've noticed that Python complains about .well-known/... which was exception I got when Java library was not fixed to respect path of Issuer URL. What was wrong with Java lib is that URL was stripped of path and only root part was used so if you have http://keycloak/realms/test it would only take http://keycloak/ so keep that in mind while testing.

Hope it goes well.