Goal : Send Magicxpi ActivityLog to OpenTelemetryCollector and export it to Elastic Cloud to build Dashboard with Kibana
Prerequisites :
- OpenTelemetry (OTEL) Collector installed
- Access to ElasticsearchCloud service
- IMM up and running with correct timezone
1/ OpenTelemetry Installation and configuration
Follow the steps on this link Quick start | OpenTelemetry
- sudo snap install docker
- sudo apt install golang -y
- export GOBIN=${GOBIN:-$(go env GOPATH)/bin}
- sudo docker pull otel/opentelemetry-collector-contrib:0.116.1
- go install github.com/open-telemetry/opentelemetry-collector-contrib/cmd/telemetrygen@latest
2/ On windows, Set a correct time zone in your “values.yaml” of your IMM (<Magicxpi Installation>\InMemoryMiddleware\deploy) and deploy your IMM.
For example, on the first line in the file, enter :
timeZone: &timeZone “Europe/Paris”
On the Linux machine verify your time zone by typing :
timedatectl status
If the value is different from the value in the file “values.yaml”, change it by typing (for my example)
sudo timedatectl set-timezone Europe/Paris
3/ After installing and configuring your OpenTelemetry Collector on the linuw machine and set the timezone, you can start the collector by typing :
sudo docker run -p 4317:4317 -p 4318:4318 -p 55679:55679 otel/opentelemetry-collector-contrib:0.116.1 2
(or sudo docker run -p 4317:4317 -p 4318:4318 -p 55679:55679 otel/opentelemetry-collector-contrib:0.116.1 2>&1 | tee collector-output.txt if you want to have an output file)
Now you need to identify the relevant OTEL config file (config.yaml)
you can enter : sudo ls -asl /var/snap/docker/common/var-lib-docker/overlay2/
you should several uuid directories
You can identify the relevant uuid directory according to the datetime when you have started the OTEL collector
if you don’t know, you can stop the collector (sudo docker stop <containerid>) and restart it
to get the container list and identify your containerid, you can enter : sudo docker ps -a
Your config directory file should be like this : /var/snap/docker/common/var-lib-docker/overlay2/f279d3a4964f6dbb133c189c93e119ce4cfde768020a671a65cb1c8d07fa2d97/diff/etc/otelcol-contrib/config.yaml
we will change the config file later on step 6
4/ Set the OpenTelemetry URL in the magicxpi Monitor
IP value “172.31.39.205” corresponds to my Linux machine where the OTEL collector is started.
5/ Start your magicxpi project and verify in the file “collector-output.txt” that you receive magicxpi activitylog.
The format should be like this :
{“_class”:”com.magicsoftware.xpi.info.data.ActivityLog”,”blobexists”:1,”bpid”:1,”category”:” “,”createTimeStamp”:”2025-01-08T13:59:20.995Z”,”createdTimeInNanoSec”:5 35067875665199,”dateOfWritingToSpace”:{},”extension”:”txt”,”filelocation”:” “,”flowid”:1,”flowrequestid”:15,”fsid”:15,”fsstep”:1,”messagestring”:”{\”message\”:\”this is a test 53960995\”}”,”messagetypeid”:14,”objectlevel”:3,”projectkey”:”Test”,”rootfsId”:15,”runId”:”77d5c62e-8282-4e10-9809-0d878d629d2b”,”serverid”:1,”severity”:0,”statuscode”:0,”t imeZone”:”Europe/Paris”,”userblob”:null,”usercode”:0,”userkey1″:” “,”userkey2″:” “,”versionkey”:” “}
5/ if it’s ok then you can change your OTEL configuration file to add Elastic exporter in other to get your magicxpi activity log in Elastic.
- You need to retrieve the Elastic APM Server URL and the authorization Bearer from the Elastic console (Integrations=>Elastic APM)
- Scroll down and copy the secret token
6/ You can now edit OTEL config.yaml file (step 3)
sudo vi /var/snap/docker/common/var-lib-docker/overlay2/f279d3a4964f6dbb133c189c93e119ce4cfde768020a671a65cb1c8d07fa2d97/diff/etc/otelcol-contrib/config.yaml
If you want to export only logs to Elastic (add otlphttp in the exporters section) and to have only 4318 as the recevier port, your config should be like this
7/ Stop your OTEL collector
sudo docker stop <containerid> and restart it by typing
sudo docker run -p 4317:4317 -p 4318:4318 -p 55679:55679 otel/opentelemetry-collector-contrib:0.116.1 2
8/ Create a simple magic xpi project with One flow with 2 steps One Save Message and One BAM step
Set a scheduler trigger and execute the flow (Monitor log level is set to User in the IFS.INI: [MAGIC_IBOLT]MonitorLogLevel=* User)
Switch to the Elastic console, you should see your activity log when your discover your index.
From now, you can build Dashboards and do other stuff inside ELK.
(*) Magic is sending the activitylog in incremental way.
