Jq basics for testers

You might have worked some time with data formatted as JSON if you are involved in API testing or web applications’ data services testing. JSON has become a popular data format because it’s lightweight, it’s compatible with JavaScript, easy to read by humans  and easy to parse by computers.  

As a QA engineer, you might be interested in selecting part of the JSON data to check the product tested is behaving as expected (in this case, that the software is producing the desired data), and maybe storing that data to perform  assertions on it and automate your tests. 

The not so cool aspect of JSON is that working with it through the command line is not straightforward at all. Shells such as Bash cannot manipulate JSON data directly, so working with JSON with tools like sed and grep is not the optimal solution. 

jq

To solve this very problem we have jq, an awesome command line tool that will allow you to manipulate JSON data very flexibly from your terminal. I myself use jq to automate the parsing of metrics from logs and it’s proven to be a highly convenient and reliable solution. Let’s go over some of the jq basics to manipulate JSON data like a pro.

Quick installation

Pretty print a JSON

This is probably the most frequently used jq functionality. Suppose with have the following JSON with data about an employee and their skills.

{"name": "Jason","surname": "Miller","department": "IT","dateArrival": "2020-05-04","salary": 4000,"educationLevel": "phd","softSkills": [{"skill": "client-oriented","grade": 6},{"skill": "attention to detail","grade": 8},{"skill": "communication","grade": 10}],"hardSkills": [{"skill": "Java coding","grade": 9},{"skill": "shell scripting","grade": 8}]}

With jq you can make JSON more readable: jq ‘.’ your_file.json

jq ‘.’ employee.json

{
"name": "Jason",
"surname": "Miller",
"department": "IT",
"dateArrival": "2020-05-04",
"salary": 4000,
"educationLevel": "phd",
"softSkills": [{
"skill": "client-oriented",
"grade": 6
}, {
"skill": "attention to detail",
"grade": 8
}, {
"skill": "communication",
"grade": 10
}],
"hardSkills": [{
"skill": "Java coding",
"grade": 9
}, {
"skill": "shell scripting",
"grade": 8
}]
}

Retrieve data from JSON

This is the jq feature that saves me most of the headaches. Jq allows you to target specific keys and obtain their corresponding value. You can do this with jq ‘.target_key’ your_file.json

 jq ‘.surname’ employee.json

“Miller" 

You can also retrieve data without the double quotes (“”) using the -r flag.

$ jq -r ‘.surname’ employee.json

Miller

How about parsing multiple keys at once? 

You can save some time by separating your targeted keys with commas.

jq ‘.name, .surname, .department’ employee.json

"Jason"
"Miller"
“IT"

What if your targeted data is within an array?

If you need to get the values from keys that are within an array you just have to follow the same syntax but indicating jq you are accessing an array with “[]”. Let’s say we’re interested in knowing the grade of the second skill stored in the softSkills array, this is  the “attention to detail” skill.

jq '.softSkills[1].grade' employee.json

8

Note I indicated the index (softSkills[1]) for targeted data. In this case , since I want the second element of the array softSkills I indicate jq I want to retrieve data at the position 1. This is because the first position of an array is always 0. 

And filtering by keys?

Let’s get wild! If you know the structure of your JSON, you can use filters to retrieve all data you’re interested in. For example let’s say I want all grades from the hardSkills array. I would filter this in the following way:

jq '.hardSkills[] | .grade' employee.json 

9
8

Can you use jq with pipes?

Sure! You can redirect the stdout of your commands to jq using pipes to make your  life easier. For example, in my day to day I have to process tons of JSONs and thanks to output redirection with jq I can automate JSON processing from my logs by coding jq commands in shell scripts.  

Let’s make a simple example: I will redirect the the content of employee.json to jq and extract the “name” value using the cat command and the bash pipe. 

cat employee.json| jq -r ‘.name’

Jason

Want to delete something?

Maybe you want to delete some annoying keys. You can do it with ‘del({targeted_key})’. This will not actually delete the data in your original JSON but rather it will output your JSON without your deleted keys. If you want, you can save that output into a new file. 

In my case, I don’t want the data corresponding to soft skills. Let’s delete all data within the softSkills array and save it to a file

jq 'del(.softSkills[])' employee.json > employee_modified.json

cat employee_modified.json

{
  "name": "Jason",
  "surname": "Miller",
  "department": "IT",
  "dateArrival": "2020-05-04",
  "salary": 4000,
  "educationLevel": "phd",
  "softSkills": [],
  "hardSkills": [
    {
      "skill": "Java coding",
      "grade": 9
    },
    {
      "skill": "shell scripting",
      "grade": 8
    }
  ]
}

Conclusion 

That is all I have for jq basics for testing. There’s much more to it. So, if you want to master JSON manipulation feel free to run the man jq command and see what you can do with it.