Jq basics for testers
You might have worked some time with data formatted as JSON if you are involved in API testing or web applications’ data services testing. JSON has become a popular data format because it’s lightweight, it’s compatible with JavaScript, easy to read by humans and easy to parse by computers.
As a QA engineer, you might be interested in selecting part of the JSON data to check the product tested is behaving as expected (in this case, that the software is producing the desired data), and maybe storing that data to perform assertions on it and automate your tests.
The not so cool aspect of JSON is that working with it through the command line is not straightforward at all. Shells such as Bash cannot manipulate JSON data directly, so working with JSON with tools like sed and grep is not the optimal solution.

jq
To solve this very problem we have jq, an awesome command line tool that will allow you to manipulate JSON data very flexibly from your terminal. I myself use jq to automate the parsing of metrics from logs and it’s proven to be a highly convenient and reliable solution. Let’s go over some of the jq basics to manipulate JSON data like a pro.
Quick installation
Linux (Debian and Ubuntu):
sudo apt-get install jq
OS X (need Homebrew package manager)
brew install jq
Pretty print a JSON
This is probably the most frequently used jq functionality. Suppose with have the following JSON with data about an employee and their skills.
{"name": "Jason","surname": "Miller","department": "IT","dateArrival": "2020-05-04","salary": 4000,"educationLevel": "phd","softSkills": [{"skill": "client-oriented","grade": 6},{"skill": "attention to detail","grade": 8},{"skill": "communication","grade": 10}],"hardSkills": [{"skill": "Java coding","grade": 9},{"skill": "shell scripting","grade": 8}]}
With jq you can make JSON more readable: jq ‘.’ your_file.json
jq ‘.’ employee.json
{ "name": "Jason", "surname": "Miller", "department": "IT", "dateArrival": "2020-05-04", "salary": 4000, "educationLevel": "phd", "softSkills": [{ "skill": "client-oriented", "grade": 6 }, { "skill": "attention to detail", "grade": 8 }, { "skill": "communication", "grade": 10 }], "hardSkills": [{ "skill": "Java coding", "grade": 9 }, { "skill": "shell scripting", "grade": 8 }] }
Retrieve data from JSON
This is the jq feature that saves me most of the headaches. Jq allows you to target specific keys and obtain their corresponding value. You can do this with jq ‘.target_key’ your_file.json
jq ‘.surname’ employee.json
“Miller"
You can also retrieve data without the double quotes (“”) using the -r
flag.
$ jq -r ‘.surname’ employee.json
Miller
How about parsing multiple keys at once?
You can save some time by separating your targeted keys with commas.
jq ‘.name, .surname, .department’ employee.json
"Jason" "Miller" “IT"
What if your targeted data is within an array?
If you need to get the values from keys that are within an array you just have to follow the same syntax but indicating jq you are accessing an array with “[]”. Let’s say we’re interested in knowing the grade of the second skill stored in the softSkills array, this is the “attention to detail” skill.
jq '.softSkills[1].grade' employee.json
8
Note I indicated the index (softSkills[1
]) for targeted data. In this case , since I want the second element of the array softSkills I indicate jq I want to retrieve data at the position 1. This is because the first position of an array is always 0.
And filtering by keys?
Let’s get wild! If you know the structure of your JSON, you can use filters to retrieve all data you’re interested in. For example let’s say I want all grades from the hardSkills array. I would filter this in the following way:
jq '.hardSkills[] | .grade' employee.json
9 8
Can you use jq with pipes?
Sure! You can redirect the stdout of your commands to jq using pipes to make your life easier. For example, in my day to day I have to process tons of JSONs and thanks to output redirection with jq I can automate JSON processing from my logs by coding jq commands in shell scripts.
Let’s make a simple example: I will redirect the the content of employee.json to jq and extract the “name” value using the cat command and the bash pipe.
cat employee.json| jq -r ‘.name’
Jason
Want to delete something?
Maybe you want to delete some annoying keys. You can do it with ‘del({targeted_key})’
. This will not actually delete the data in your original JSON but rather it will output your JSON without your deleted keys. If you want, you can save that output into a new file.
In my case, I don’t want the data corresponding to soft skills. Let’s delete all data within the softSkills
array and save it to a file
jq 'del(.softSkills[])' employee.json > employee_modified.json
cat employee_modified.json
{ "name": "Jason", "surname": "Miller", "department": "IT", "dateArrival": "2020-05-04", "salary": 4000, "educationLevel": "phd", "softSkills": [], "hardSkills": [ { "skill": "Java coding", "grade": 9 }, { "skill": "shell scripting", "grade": 8 } ] }
Conclusion
That is all I have for jq basics for testing. There’s much more to it. So, if you want to master JSON manipulation feel free to run the man jq
command and see what you can do with it.