Using a Golang package in Python using Gopy

Back story

While working on a Python-based CLI tool, we came across a requirement to statically validate PromQL queries. But we couldn't find any Python package to do so. There were some packages that did this, but the set of rules used for the validation was not on par with the original validation performed by Prometheus itself.

This leaves us with two options:

1. Create a Python version of the Golang-based PromQL parser used in Prometheus.
2. Or use an existing Golang package itself.

The first option is not very desirable here. The obvious reason is that it’s “reinventing the wheel”. But also, any new package that mimics the existing Go package will always have to “keep up” with the original. This is an unnecessary maintenance overhead.

So, reusing the existing Go package is the more desirable choice. This will save a lot of time and we will be dealing directly with the source of truth of PromQL. Also, it will open up doors for a lot of other functionality of the Prometheus ecosystem (which is predominantly in Golang).

But for various reasons, we did not end up using Prometheus’s PromQL parser. Instead, we used VictoriaMetrics’s MetricsQL parser. VictoriaMetrics is another open-source TSDB similar to Prometheus and MetricsQL is its query language. MetricsQL is a backward-compatible extension of PromQL. So, This works for our use case – static validation of PromQL queries

Solution

When it comes to using Go code in a Python project, there are several ways to do it. One common approach is to generate a shared object (SO) file containing the compiled Go code, which can then be imported directly into Python.

This method is relatively straightforward for small scripts or quick automation. However, it can become more complicated when working with larger applications that need to be packaged for use by others. Additionally, this approach requires knowledge of both Go and C, which can be a steep learning curve for Python developers who are not familiar with these languages.

To address these challenges, there are several libraries available that simplify the process of incorporating Go code into Python projects. One of the most popular is Gopy, which allows Go code to be compiled into a Python module using a simple command-line interface.

Let us dive deep into the exact steps involved in using gopy. If you want to follow along – create an empty working directory, set up a new virtual environment, and activate it.

Steps to Follow

TL;DR

Nitty Gritty

1. Clone the metricsQL repo

git clone git@github.com:VictoriaMetrics/metricsql.git --depth 1
cd metricsql

This command will clone the metricsQL repository from GitHub to your local machine. The --depth 1 flag tells Git to only clone the most recent commit, which will make the cloning process a lot faster.

2. Install gopy and the required Python build dependencies

 go install github.com/go-python/gopy@latest
 pip3 install pybindgen wheel

pybindgen is internally used by gopy. We will use the wheel package later to build the .whl file.

3. Build the Python module

gopy build --output=py_metricsql -vm=python3 .

This command will build the Python module from the Go code in the current directory. The --output=py_metricsql flag tells gopy to output the built module to the py_metricsql directory. The --vm=python3 flag tells gopy to build the module for Python 3. This creates an SO file and some other artifacts.

4. Create a setup.py file with the following contents

import setuptools

setuptools.setup(
    name="py_metricsql",
    packages=setuptools.find_packages(include=["py_metricsql"]),
    py_modules = ["py_metricsql.metricsql"],
    package_data={"py_metricsql": ["*.so"]},
    include_package_data=True,
)

The setup.py file is used to build and install the Python package. It contains the boilerplate code to include the SO file created in the previous step.

5. Create an installable .whl file using the setup.py

python3 setup.py bdist_wheel

This creates a .whl file in a directory called dist

6. Pip install the .whl file

wheel_file=$(ls dist/*.whl | head -n1); pip3 install $wheel_file

Now metricsQL’s Python version should be installed in your environment. You can verify it by running the following code snippet in your Python shell:

from py_metricsql import metricsql

query = "sum(increase(request_count))"
metricsql.Parse(query)

If it worked, it should print the parsed query object.

Thats it! 🎉

The py_metricsql package will have all the functionality of the original Go metricsql package!

It’s worth noting that the error handling is done in the “Python way”. Where the Go version of the code returns err with every function call, the Python version raises an exception:

Things to keep in mind

While using a Go package in Python can provide powerful capabilities, it may also introduce some additional complexity and overhead.

• Compatibility: Go packages are typically compiled into machine code, which means they need to be compiled separately for each operating system and architecture they will be used on. This can create compatibility issues if the compiled binary is not compatible with the target system. If you are shipping this for external use, make sure to have separate builds for all the OS/Architectures you want to support.
• Testing: Testing a Python project that uses a Go package may require additional setup and configuration steps. In addition to testing the Python code, it may be necessary to test the interaction between the Python code and the Go package.

Using a Go package in Python can also introduce other potential issues related to performance, development overhead, and debugging. Therefore, it’s important to carefully evaluate whether the benefits of using a Go package in your Python project outweigh the potential costs and challenges.