LogQL
Video Lecture
Description
Now that we have a loki data source we can query it with the LogQL query language.
In this video, we will try out many LogQL queries in the systemd-journal stream selector that we just set up.
There are two types of LogQL queries:
- Log queries returning the contents of log lines as streams.
- Metric queries that convert logs into value matrixes.
A LogQL query consists of,
- The log stream selector
- Filter expression
We can use operations on both the log stream selectors and filter expressions to refine them.
Log Stream Selectors
Operators
- = : equals
- != : not equals
- =~ : regex matches
- !~ : regex does not match
Examples
Return all log lines for the job systemd-journal
{job="systemd-journal"}
Return all log lines for the unit ssh.service
{unit="ssh.service"}
Return all log lines for the job systemd-journal and the unit cron.service
{job="systemd-journal",unit="cron.service"}
Show all log lines for 2 jobs with different names
{job=~"sbcode/systemd-journal|systemd-journal"}
Filter Expressions
Operators
Used for testing text within log line streams.
- |= : equals
- != : not equals
- |~ : regex matches
- !~ : regex does not match
Examples
Return lines including the text "error"
{job="systemd-journal"} |= "error"
Return lines not including the text "error"
{job="systemd-journal"} != "error"
Return lines including the text "error" or "info" using regex
{job="systemd-journal"} |~ "error|info"
Return lines not including the text "error" or "info" using regex
{job="systemd-journal"} !~ "error|info"
Return lines including the text "error" but not including "info"
{job="systemd-journal"} |= "error" != "info"
Return lines including the text "Invalid user" and including ("paulo" or "redis") using regex
{job="systemd-journal"} |~ "Invalid user (paulo|redis)"
Return lines including the text "status 403" or "status 503" using regex
{job="systemd-journal"} |~ "status [45]03"
Range and Instance Vectors
The data so far is returned as streams of log lines. We can graph these in visualisations if we convert them to vectors. We can aggregate the lines into numeric values, such as counts, which then become known as instance or range vectors.
- count_over_time : Shows the total count of log lines for time range
- rate : Similar as count_over_time but converted to number of entries per second
- bytes_over_time : Number of bytes in each log stream in the range
- bytes_rate : Similar to bytes_over_time but converted to number of bytes per second
Examples
The count of jobs at 1 minutes time intervals
count_over_time({job="systemd-journal"}[1m])
The rate of logs per minute. Rate is similar to count_over_time but shows the entries per second.
rate({job="systemd-journal"}[1m])
Info
rate = count_over_time / 60 / range(m)
eg,
12 / 60 / 2 = 0.1
The count of errors at 1h time intervals
count_over_time({job="systemd-journal"} |= "error" [1h])
Aggregate Functions
An aggregate function converts a range vector result into a single instance vector.
- sum : Calculate the total of all instance vectors in the range at time
- min : Show the minimum value from all instance vectors in the range at time
- max : Show the maximum value from all instance vectors in the range at time
- avg : Calculate the average of the values from all instance vectors in the range at time
- stddev : Calculate the standard deviation of the values from all instance vectors in the range at time
- stdvar : Calculate the standard variance of the values from all instance vectors in the range at time
- count : Count the number of elements all all instance vectors in the range at time
- bottomk : Select lowest k values in all the instance vectors in the range at time
- topk : Select highest k values in all the instance vectors in the range at time
Note
bottomk and topk don't produce an instance vector, but a range vector of containing the k number of instance vectors in the time range.
Examples
Calculate the total of all instance vectors in the range at time
sum(count_over_time({job="systemd-journal"}[1m]))
Show the minimum value from all instance vectors in the range at time
min(count_over_time({job="systemd-journal"}[1m]))
Show the maximum value from all instance vectors in the range at time
max(count_over_time({job="systemd-journal"}[1m]))
Show only the top 2 values from all instance vectors in the range at time
topk(2, count_over_time({job="systemd-journal"}[1h]))
Aggregate Group
Convert an instance vector into a range vector organised by unit
Examples
Group a single log stream by unit
sum(count_over_time({job="systemd-journal"}[1m])) by (unit)
Group multiple log streams by job
sum(count_over_time({job=~"sbcode/systemd-journal|systemd-journal"}[1m])) by (job)
Group multiple log streams and specific unit by job
sum(count_over_time({job=~"sbcode/systemd-journal|systemd-journal", unit="cron.service"}[1m])) by (job)
Group multiple log streams by job and unit
sum(count_over_time({job=~"sbcode/systemd-journal|systemd-journal"}[1m])) by (job,unit)
Comparison Operators
Comparison Operators. Used for testing numeric values present in scalars and vectors.
- == (equality)
- != (inequality)
- > (greater than)
- >= (greater than or equal to)
- < (less than)
- <= (less than or equal to)
Examples
Returns values greater than 4
sum(count_over_time({job="systemd-journal"}[1m])) > 4
Returns values less than or equal to 1
sum(count_over_time({job="systemd-journal"}[1m])) <= 1
Logical Operators
These can be applied to both range and instance vectors
- and : Bother sides must be true
- or : Either side must be true
- unless : Return values unless value
Examples
Returns values greater than 4 or values less then or equal to 1
sum(count_over_time({job="systemd-journal"}[1m])) > 4 or sum(count_over_time({job="systemd-journal"}[1m])) <= 1
Return values between 2 and 5
sum(count_over_time({job="systemd-journal"}[1m])) > 2 and sum(count_over_time({job="systemd-journal"}[1m])) < 5
Arithmetic Operators
- + : Add
- - : Subtract
- * : Multiply
- / : Divide
- % : Modulus
- ^ : Power/Exponentiation
Examples
sum(count_over_time({job="systemd-journal"}[1m])) * 10
sum(count_over_time({job="systemd-journal"}[1m])) % 2
Operator order
Many Operators can be used at a time. The order follows the PEMDAS construct. PEMDAS is an acronym for the words parenthesis, exponents, multiplication, division, addition, subtraction.
Examples
A nonsensical example
sum(count_over_time({job="systemd-journal"}[1m])) % 4 * 2 ^ 2 + 2
# is the same as
((sum(count_over_time({job="systemd-journal"}[1m])) % 4 * (2 ^ 2)) + 2)
Proving that count_over_time / 60 / range(m) = rate
rate({job="systemd-journal"}[2m]) == count_over_time({job="systemd-journal"}[2m]) / 60 / 2
# is the same as
rate({job="systemd-journal"}[2m]) == ((count_over_time({job="systemd-journal"}[2m]) / 60) / 2)