Published: Tue 31 October 2023
By "Carsten"
In Testing .
How to test data ingestion to Influxdb
I am currently developing a python package, which will enable me to monitor my internet connections uptime.
Or at least that is the end result - the project is called iconnmon
The desired end result is to add ping result data in Influxdb, from pinging selected ip's like my router, and the hop after my router on the ISP side. This way I will be able to make nice graphs of how my internet connection is faring. At least that is what I am chasing.
I am using TDD - Test Driven Development - on this project, so I get sharper on this topic, while getting familiar with Influxdb at the same time. SO I needed a way to both test locally, and as I push my code to Github (yeah I know... all my articles ranting about leaving github - it is apparently hard, as CI is having some rough patches on Codeberg , so for this project I am starting on Github).
Testing locally
The local testing is relatively simple for now, where I am just spinning up a docker container in the background, and I have a Influxdb token in my local .env file. The value then get loaded in my test using the os.environ.get('INFLUX_TOKEN')
I do however want to move towards Launching Containers before starting a debub session in VS Code - I just don't have that part up and running yet for pytest. I need to figure that out.
SO right now I just start an influxdb container:
docker run -d -p 8086 :8086 \
-v $PWD /influx/data:/var/lib/influxdb2 \
-v $PWD /influx/config:/etc/influxdb2 \
-e DOCKER_INFLUXDB_INIT_MODE = setup \
-e DOCKER_INFLUXDB_INIT_USERNAME = some-user \
-e DOCKER_INFLUXDB_INIT_PASSWORD = some-password \
-e DOCKER_INFLUXDB_INIT_ORG = some-org \
-e DOCKER_INFLUXDB_INIT_BUCKET = some-bucket \
-e DOCKER_INFLUXDB_INIT_ADMIN_TOKEN = verysecretoken \
influxdb:2.7.3
I then use my admin token to retrieve a token to use in my tests:
influx auth create \
--all-access \
--host http://localhost:8086 \
--org some-org \
--token verysecretoken
The token is just added to a .env file and placed under the tests folder:
Then loading the env. variables with the python-dotenv package:
from dotenv import load_dotenv
def test_influx_insert_data ():
"""
Initial test to add data to influxdb
"""
load_dotenv ()
...
Influxdb as service container in Github Actions
Github actions has the concept of service containers - and that really comes in handy here. The actions will wait for the container to be ready before executing the rest of the workflow.
With the following definition:
services :
influx :
image : influxdb:2.7.3
ports :
- 8086:8086
env :
DOCKER_INFLUXDB_INIT_MODE : setup
DOCKER_INFLUXDB_INIT_USERNAME : ${{ secrets.INFLUX_USERNAME }}
DOCKER_INFLUXDB_INIT_PASSWORD : ${{ secrets.INFLUX_PASSWORD }}
DOCKER_INFLUXDB_INIT_ORG : iconnmon
DOCKER_INFLUXDB_INIT_BUCKET : iconnmonseries
DOCKER_INFLUXDB_INIT_ADMIN_TOKEN : ${{ secrets.INFLUX_TOKEN }}
the influx service container will be spun up, and once up and running these steps will get the token for tests, and in turn expose it via an environment variable:
- name : Install influx cli
run : |
sudo apt update
sudo apt install -y wget
wget https://dl.influxdata.com/influxdb/releases/influxdb2-client-2.7.3-linux-amd64.tar.gz
tar xvzf influxdb2-client-2.7.3-linux-amd64.tar.gz
sudo cp influx /usr/local/bin/
- name : Get an access token
run : |
influx config create --config-name influx-config \
--host-url http://localhost:8086 \
--org iconnmon \
--token ${{ secrets.INFLUX_TOKEN }}\
--active
echo "INFLUXDB_TOKEN=$(influx auth create \
--all-access \
--host http://localhost:8086 \
--org iconnmon \
--token ${{ secrets.INFLUX_TOKEN }} | sed -n '2 p'| awk '{print $2}')" >> $GITHUB_ENV
echo $INFLUXDB_TOKEN
Things of note:
The way Github actions sets env. variables in the steps is a bit special - look in the "Get an access token" step how it is piped into the $GITHUB_ENV env. variable. Then some sort of magic happens afterwards
As always remember to keep your secrets secret, and put them in the secrets of the GH Actions flow.
This example is still a bit rough - it needs to be cleaned up, i.e. the version of influx should be put in a variable and other such things. So a little refactoring. Right now this article mostly serves as an example to myself, should I forget it ;-)