Read and Write JSON File Python 3

January 14, 2020February 12, 2020 Dave 0 Comments json, programming, python, scripting

JSON data is a pretty common format, especially if you work with API. Many popular APIs will give or expect to get data in JSON format. Here is how to read and write JSON data files with Python 3.

What is JSON?

In the most simple form, JSON is a text file containing data. You can however, have very complicated data structures!

List of JSON Data Formats

JSON can be any data format that you choose (some examples):

Strings
Lists/Tuples
Dictionaries
List of Lists
Lists of Dictionaries

Whatever your data structure is without JSON, it will remain the same with JSON. JSON is just a way to represent the data, in a text file.

Using Python 3 to Write JSON

Here are some examples of using python JSON module to write sample JSON for you (and their corresponding JSON examples).

Sample Code to Create JSON Python3 String

Creating a string as a JSON file is hardly worth the effort, but in case you wanted to script it, here it is:

# import the module
import json

# load up the data
example0 = 'The quick brown fox'

# write the data
with open ('ex_string.json', 'w') as outfile:
	json.dump(example0, outfile, indent=2)
	print("File Dumped as ex_string.json")

# import the module

import json

# load up the data

example0 = 'The quick brown fox'

# write the data

with open ('ex_string.json', 'w') as outfile:

json.dump(example0, outfile, indent=2)

print("File Dumped as ex_string.json")

String Sample Output:

"The quick brown fox"

1	"The quick brown fox"

Sample Code to Create JSON Python3 List

Here is the Python 3 Script for making a list, and writing it out to a JSON data file:

import json
example1 = ['orange', 'apple', 'pear', 'banana', 'kiwi', 'apple', 'banana']
with open ('ex_list.json', 'w') as outfile:
	json.dump(example1, outfile, indent=2)
	print("File Dumped as ex_list.json")

import json

example1 = ['orange', 'apple', 'pear', 'banana', 'kiwi', 'apple', 'banana']

with open ('ex_list.json', 'w') as outfile:

json.dump(example1, outfile, indent=2)

print("File Dumped as ex_list.json")

List Sample Output:

[
  "orange",
  "apple",
  "pear",
  "banana",
  "kiwi",
  "apple",
  "banana"
]

[

"orange",

"apple",

"pear",

"banana",

"kiwi",

"apple",

"banana"

]

Sample Code to Create JSON Python3 Tuple

Here is the Python 3 Script for making a tuple and writing it out to a JSON data file:

import json
example2 = 'foo', 'baz', 'bar'
with open ('ex_tuple.json', 'w') as outfile:
	json.dump(example2, outfile, indent=2)
	print("File Dumped as ex_tuple.json")

import json

example2 = 'foo', 'baz', 'bar'

with open ('ex_tuple.json', 'w') as outfile:

json.dump(example2, outfile, indent=2)

print("File Dumped as ex_tuple.json")

Sample output of a tuple:

[
  "foo",
  "baz",
  "bar"
]

[

"foo",

"baz",

"bar"

]

If you are paying attention, you’ll see that both a list and a tuple look the same in JSON. JSON doesn’t make it a list or a tuple, how you load and use it does!

Sample Code to Create JSON Python3 Dictionary

Dictionaries are quite common when working with JSON, especially as building blocks for more complicated data structures. Here is a python script to create a dictionary file in JSON:

import json
example3 = {'house1': '2042 Q', 'house2': '629 W'}
with open ('ex_dictionary.json', 'w') as outfile:
	json.dump(example3, outfile, indent=2)
	print("File Dumped as ex_dictionary.json")

import json

example3 = {'house1': '2042 Q', 'house2': '629 W'}

with open ('ex_dictionary.json', 'w') as outfile:

json.dump(example3, outfile, indent=2)

print("File Dumped as ex_dictionary.json")

Output sample for a JSON dictionary:

{
  "house1": "2042 Q",
  "house2": "629 W"
}

{

"house1": "2042 Q",

"house2": "629 W"

}

Sample Code to Create JSON Python3 List of Lists

If you wanted to make a list of lists (a matrix) in python and create a json data file, you could do something like this:

import json
example4 = [
	[1, 2, 3, 4],
	[5, 6, 7, 8],
	['a','b','c','d'],
]
with open ('ex_listoflists.json', 'w') as outfile:
	json.dump(example4, outfile, indent=2)
	print("File Dumped as ex_listoflists.json")

import json

example4 = [

[1, 2, 3, 4],

[5, 6, 7, 8],

['a','b','c','d'],

]

with open ('ex_listoflists.json', 'w') as outfile:

json.dump(example4, outfile, indent=2)

print("File Dumped as ex_listoflists.json")

Output example of JSON data for a list of lists in JSON:

[
  [
    1,
    2,
    3,
    4
  ],
  [
    5,
    6,
    7,
    8
  ],
  [
    "a",
    "b",
    "c",
    "d"
  ]

]

[

"a",

"b",

"c",

"d"

]

Sample Code to Create JSON Python3 Nested Tuple

Nested tuples can be created in JSON using python just as simple as any other data format:

import json
example2 = 'foo', 'baz', 'bar'
example4 = example2, (1,2,3)
with open ('ex_nestedtuple.json', 'w') as outfile:
	json.dump(example4, outfile, indent=2)
	print("File Dumped as ex_nestedtuple.json")

import json

example2 = 'foo', 'baz', 'bar'

example4 = example2, (1,2,3)

with open ('ex_nestedtuple.json', 'w') as outfile:

json.dump(example4, outfile, indent=2)

print("File Dumped as ex_nestedtuple.json")

Output of a nested tuple in JSON:

[
  [
    "foo",
    "baz",
    "bar"
  ],
  [
    1,
    2,
    3
  ]
]

[

"foo",

"baz",

"bar"

[

]

Using JSON Data Like a Table (List of Dictionaries)

For the final example, I’ll share how to manipulate JSON data that mimics a table. For this example, I am writing a script to talk to the API on a popular firewall. My script needs to know my current IP (a dynamic DNS). Once I know my IP, then I can open up the firewall. As a “first step”, I needed to have a rule base so I could manage my firewall. This script example is that “first step”.

Common ports I will be working with are 22,80,443:

tcp/22 (ssh)
tcp/80 (http)
tcp/443 (https)

Here is the sample data I want to load as an initial rule set using those common ports and a structure similar to what I will work with in the final script:

ID	Source IP	Dest IP	Service	Description
100	1.2.3.4	2.3.4.5	tcp/22	Data Center A
101	1.2.3.4	2.3.4.5	tcp/80	Data Center B
102	1.2.3.4	2.3.4.5	tcp/443	Data Center B

My list of dictionaries uses the same keys as part of my logic, like a table header (see example above). I wanted to represent a much more complicated rule set in a spreadsheet for easy human readable audits but use scripting to talk to the API.

In this example we are using a data structure in Python that is a list of dictionaries and I wasn’t quite sure what it was supposed to look like.

The quick solution (instead of looking up data structures), was to just print it out and let Python tell me what it was supposed to look like, so that’s what I did.

Here is the script that printed out my data and json for me using sample data:

import json
# sample data so I can see what json should look like
indata =  [
    {'id' : 100, 'sorc' : "x.x.x.1", 'dest' : "y.y.y.y", 'srvc' : "tcp/22", 'desc' : "comment 1" },
    {'id' : 101, 'sorc' : "x.x.x.2", 'dest' : "y.y.y.y", 'srvc' : "tcp/22", 'desc' : "comment 2"  },
]

with open ('testout.json', 'w') as outfile:
	json.dump(indata, outfile, indent=2)
	print("File Dumped as testout.json")

import json

# sample data so I can see what json should look like

indata = [

{'id' : 100, 'sorc' : "x.x.x.1", 'dest' : "y.y.y.y", 'srvc' : "tcp/22", 'desc' : "comment 1" },

{'id' : 101, 'sorc' : "x.x.x.2", 'dest' : "y.y.y.y", 'srvc' : "tcp/22", 'desc' : "comment 2" },

]

with open ('testout.json', 'w') as outfile:

json.dump(indata, outfile, indent=2)

print("File Dumped as testout.json")

Here is how the JSON should look for a list of dictionaries:

[
  {
    "id": 100,
    "sorc": "1.2.3.4",
    "dest": "5.4.3.2",
    "srvc": "tcp/22",
    "desc": "Data Center A"
  },
  {
    "id": 101,
    "sorc": "1.2.3.4",
    "dest": "5.4.3.2",
    "srvc": "tcp/80",
    "desc": "Data Center B"
  },
  {
    "id": 102,
    "sorc": "1.2.3.4",
    "dest": "5.4.3.2",
    "srvc": "tcp/443",
    "desc": "Data Center B"
  }
]

[

{

"id": 100,

"sorc": "1.2.3.4",

"dest": "5.4.3.2",

"srvc": "tcp/22",

"desc": "Data Center A"

{

"id": 101,

"sorc": "1.2.3.4",

"dest": "5.4.3.2",

"srvc": "tcp/80",

"desc": "Data Center B"

{

"id": 102,

"sorc": "1.2.3.4",

"dest": "5.4.3.2",

"srvc": "tcp/443",

"desc": "Data Center B"

}

]

Once I saw what it was supposed to look like I could then verify the rest of my code.

Reading JSON Data in Python

If we save this as a simple text file such as “rulesin.json” we can then read and update that file using Python. Reading is done withe ‘r’, just like writing is done with the ‘w’ flag. The following example loads my json file into ‘data’ variable. Because the JSON data is a list of dictionaries, ‘data’ will also be a list of dictionaries and can be used the same way.

import json

# open JSON file for reading
with open('rulesin.json', 'r') as infile:
    # put it into a dictionary called "data"
    data = json.load(infile)

import json

# open JSON file for reading

with open('rulesin.json', 'r') as infile:

# put it into a dictionary called "data"

data = json.load(infile)

Now all of the table is loaded into the “data” list of dictionaries. Here is how to iterate over each list member and access that data. Note, I am using rid, src, dst, svc, and des as my dictionary key names and accessing those named members using the “i” variable in a loop, of the enumerated “data” list of dictionaries in my example above. Your exact data structure doesn’t matter, as long as the JSON matches what you expect.

Looping Over a List of Dictionaries in Python 3

# loop over 1 row per line of indata
for i in data:
    rid = i['id']
    src = i['sorc']
    dst = i['dest']
    svc = i['srvc']
    des = i['desc']

# loop over 1 row per line of indata

for i in data:

rid = i['id']

src = i['sorc']

dst = i['dest']

svc = i['srvc']

des = i['desc']

Now we can print out the data if we needed (or do other operations with it). In the following example code I am using the \t (tab) to display my output on a command line:

    print("ID: ",rid,"\tSRC:",src,"\tDST:",dst,"\tSVC:",svc,"\tDESC:",des)

1	print("ID: ",rid,"\tSRC:",src,"\tDST:",dst,"\tSVC:",svc,"\tDESC:",des)

You could of course do anything you want to the data, munge it, consume it, etc. I wanted to update the source ip so I added another small function and some error checking:

import sys
from requests import get 	# for the get function only from requests

'''
must use external IP check as we are behind router
let's use ipify's API
'''
def getPublicIP():
	try:
		ip = get('https://api.ipify.org').text
		return ip
	except:
		print("Failed: ", ip)
		sys.exit()

# get our IP
myip = getPublicIP()

import sys

from requests import get # for the get function only from requests

'''

must use external IP check as we are behind router

let's use ipify's API

'''

def getPublicIP():

try:

ip = get('https://api.ipify.org').text

return ip

except:

print("Failed: ", ip)

sys.exit()

# get our IP

myip = getPublicIP()

Final Script: Get IP and Update JSON file

Combining all of it together is the script that will do what I wanted:

Read from rulesin.json file
Get my public IP
Update data with my Current IP
Write updated data to the rulesout.json file

import json
import sys			# argv stuff and sys.exit
from requests import get 	# for the get function only from requests

'''
must use external IP check as we are behind router
let's use ipify's API
'''
def getPublicIP():
	try:
		ip = get('https://api.ipify.org').text
		return ip
	except:
		print("Failed: ", ip)
		sys.exit()

# get our IP
myip = getPublicIP()

# open json file for reading
with open('rulesin.json', 'r') as infile:
    # put it into a dictionary called "data"
    data = json.load(infile)

# loop over 1 row per line of indata
for i in data:
	rid = i['id']
	src = i['sorc'] = myip
	dst = i['dest']
	svc = i['srvc']
	des = i['desc']

	# do api calls here, after printing to verify data works as expected
	print("ID: ",rid,"\tSRC:",src,"\tDST:",dst,"\tSVC:",svc,"\tDESC:",des)

# write json file output
with open ('rulesout.json', 'w') as outfile:
	json.dump(data, outfile, indent=2)
	print("File Dumped as rulesout.json")

sys.exit()

import json

import sys # argv stuff and sys.exit

from requests import get # for the get function only from requests

'''

must use external IP check as we are behind router

let's use ipify's API

'''

def getPublicIP():

try:

ip = get('https://api.ipify.org').text

return ip

except:

print("Failed: ", ip)

sys.exit()

# get our IP

myip = getPublicIP()

# open json file for reading

with open('rulesin.json', 'r') as infile:

# put it into a dictionary called "data"

data = json.load(infile)

# loop over 1 row per line of indata

for i in data:

rid = i['id']

src = i['sorc'] = myip

dst = i['dest']

svc = i['srvc']

des = i['desc']

# do api calls here, after printing to verify data works as expected

print("ID: ",rid,"\tSRC:",src,"\tDST:",dst,"\tSVC:",svc,"\tDESC:",des)

# write json file output

with open ('rulesout.json', 'w') as outfile:

json.dump(data, outfile, indent=2)

print("File Dumped as rulesout.json")

sys.exit()

The output of this program simply prints what it reads from the JSON file, after modifying the source IP:

And the updated json data:

[
  {
    "id": 100,
    "sorc": "76.1.2.2",
    "dest": "5.4.3.2",
    "srvc": "tcp/22",
    "desc": "Data Center A"
  },
  {
    "id": 101,
    "sorc": "76.1.2.2",
    "dest": "5.4.3.2",
    "srvc": "tcp/80",
    "desc": "Data Center B"
  },
  {
    "id": 102,
    "sorc": "76.1.2.2",
    "dest": "5.4.3.2",
    "srvc": "tcp/443",
    "desc": "Data Center B"
  }
]

[

{

"id": 100,

"sorc": "76.1.2.2",

"dest": "5.4.3.2",

"srvc": "tcp/22",

"desc": "Data Center A"

{

"id": 101,

"sorc": "76.1.2.2",

"dest": "5.4.3.2",

"srvc": "tcp/80",

"desc": "Data Center B"

{

"id": 102,

"sorc": "76.1.2.2",

"dest": "5.4.3.2",

"srvc": "tcp/443",

"desc": "Data Center B"

}

]

Common Problems with JSON

JSON is finicky. If you don’t have your commas right, spacing, or other characters just right it could fail to read. You can Validate JSON using any of the many free tools online. Another option is to use python to write out your JSON file data. Python will write JSON just like it sees the data – and sometimes this will reveal your typos. Keep these tips in mind:

Properly Nest all JSON Data
Use commas properly, especially at the end of a list
Do not mix tabs and spaces
Do not comment your JSON using # or // type of comments
Remember JSON is a text file, not a database!

The Wrong Way to Use JSON and Python

JSON is an established file format but should not be used for extensive data entries where tens of thousands of records are required or where you expect heavy disk I/O (obviously). For that type of data, look towards proper database setups! However, for smaller records it works great.

Problem Solving with Python

In this article, you have learned quite a bit about reading, writing and even validating json files, but you also learned about various data structures, looping, and getting your public IP using a definition.

Now that you know how to do these things in Python, what will you do with your programming skill? Share your ideas below!

Ruby Sash Consulting

Read and Write JSON File Python 3

What is JSON?

List of JSON Data Formats

Using Python 3 to Write JSON

Sample Code to Create JSON Python3 String

Sample Code to Create JSON Python3 List

Sample Code to Create JSON Python3 Tuple

Sample Code to Create JSON Python3 Dictionary

Sample Code to Create JSON Python3 List of Lists

Sample Code to Create JSON Python3 Nested Tuple

Using JSON Data Like a Table (List of Dictionaries)

Reading JSON Data in Python

Looping Over a List of Dictionaries in Python 3

Final Script: Get IP and Update JSON file

Common Problems with JSON

The Wrong Way to Use JSON and Python

Problem Solving with Python

Leave a Reply Cancel reply