The last time Hackerfall tried to access this page, it returned a not found error. A cached version of the page is below, or clickhereto continue anyway

An Introduction to Scientific Python - NumPy - Data Dependence

Oh the amazing things you can do with Numpy.

NumPy is a blazing fast maths library for Python with a heavy emphasis on arrays. It allows you to do vector and matrix maths within Python and as a lot of the underlying functions are actually written in C, you get speeds that you would never reach in vanilla Python.

Numpy is an absolutely key piece to the success of scientific Python and if you want to get into Data Science and or Machine Learning in Python, it’s a must learn. NumPy is well built in my opinion and getting started with it is not difficult at all.

This is the second post in a series of posts on scientific Python, don’t forget to check out the others too.

ARRAY BASICS

Creation

NumPy revolves around these things called arrays. Actually ndarrays, but we don’t need to worry about that. With these arrays we can do all sorts of useful things like vector and matrix maths at lightning speeds. Get your linear algebra on! (Just kidding we won’t be doing any heavy maths)

# 1D Array

a = np.array([0, 1, 2, 3, 4])

b = np.array((0, 1, 2, 3, 4))

c = np.arange(5)

d = np.linspace(0, 2*np.pi, 5)

 

print(a) # >>>[0 1 2 3 4]

print(b) # >>>[0 1 2 3 4]

print(c) # >>>[0 1 2 3 4]

print(d) # >>>[ 0.          1.57079633  3.14159265  4.71238898  6.28318531]

print(a[3]) # >>>3

The above code shows 4 different ways of creating an array. The most basic way is just passing a sequence to NumPy’s array() function; you can pass it any sequence, not just lists like you usually see.

Notice how when we print an array with numbers of different length, it automatically pads them out. This is useful for viewing matrices. Indexing on arrays works just like that of a list or any other of Python’s sequences. You can also use slicing on them, I won’t go into slicing a 1D array here, if you want more information on slicing, check out this post.

The above array example is how you can represent a vectorwith NumPy, next we will take a look at how we can represent matrices and more with multidimensional arrays.

# MD Array,

a = np.array([[11, 12, 13, 14, 15],

              [16, 17, 18, 19, 20],

              [21, 22, 23, 24, 25],

              [26, 27, 28 ,29, 30],

              [31, 32, 33, 34, 35]])

 

print(a[2,4]) # >>>25

To create a 2D array we pass the array() function a list of lists (or a sequence of sequences). If we wanted a 3D array we would pass it a list of lists of lists, a 4D array would be a list of lists of lists of lists and so on.

Notice how with a 2D array (with the help of our friend the space bar), is arranged in rows and columns. To index a 2D array we simply reference a row and a column.

A Bit of the Maths Behind It

To understand this properly, we should really take a look at what vectors and matrices are.

A vector is a quantity that has both direction and magnitude. They are often used to represent things such as velocity, acceleration and momentum. Vectors can be written in a number of ways although the one which will be most useful to us is the form where they are written as an n-tuple such as (1, 4, 6, 9). This is how we represent them in NumPy.

A matrixis similar to a vector, except it is made up of rows and columns; much like a grid. The values within the matrix can be referenced by giving the row and the column that it resides in. In NumPy we make arrays by passing a sequence of sequences as we did previously.

Multidimensional Array Slicing

Slicing a multidimensional array is a bit more complicated than a 1D one and it’s something that you will do a lot while using NumPy.

# MD slicing

print(a[0, 1:4]) # >>>[12 13 14]

print(a[1:4, 0]) # >>>[16 21 26]

print(a[::2,::2]) # >>>[[11 13 15]

                  #     [21 23 25]

                  #     [31 33 35]]

print(a[:, 1]) # >>>[12 17 22 27 32]

As you can see you slice a multidimensional array by doing a separate slice for each dimension separated with commas. So with a 2D array our first slice defines the slicing for rows and our second slice defines the slicing for columns.

Notice that you can simply specify a row or a column by entering the number. The first example above selects the 0th column from the array.

The diagram below illustrates what the given example slices do.

Array Properties

When working with NumPy you might want to know certain things about your arrays. Luckily there are lots of handy methods included within the package to give you the information that you need.

# Array properties

a = np.array([[11, 12, 13, 14, 15],

              [16, 17, 18, 19, 20],

              [21, 22, 23, 24, 25],

              [26, 27, 28 ,29, 30],

              [31, 32, 33, 34, 35]])

 

print(type(a)) # >>><class 'numpy.ndarray'>

print(a.dtype) # >>>int64

print(a.size) # >>>25

print(a.shape) # >>>(5, 5)

print(a.itemsize) # >>>8

print(a.ndim) # >>>2

print(a.nbytes) # >>>200

As you can see in the above code a NumPy array is actually called an ndarray. I don’t know why it’s called an ndarray, if anyone knows please leave a comment! My guess is that it stands for n dimensional array.

The shape of an array is how many rows and columns it has, the above array has 5 rows and 5 columns so its shape is (5, 5).

The ‘itemsize’ property is how many bytes each item takes up. The data type of this array is int64, there are 64 bits in an int64, 8 bits in a byte, divide 64 by 8 and you get how many bytes it takes up, which in this case is 8.

The ‘ndim’ property is how many dimensions the array has. This one has 2. A vector for example however, has just 1.

The ‘nbytes’ property is how many bytes are used up by all the data in the array. You should note that this does not count the overhead of an array and so the actual space that the array takes up will be a little bit larger.

WORKING WITH ARRAYS

Basic Operators

Just being able to make and retrieve elements and properties from an array isn’t going to get you very far, you will need to do maths on them sometimes too. You can do this using the basic operators such as+, -, /, etc.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

# Basic Operators

a = np.arange(25)

a = a.reshape((5, 5))

 

b = np.array([10, 62, 1, 14, 2, 56, 79, 2, 1, 45,

              4, 92, 5, 55, 63, 43, 35, 6, 53, 24,

              56, 3, 56, 44, 78])

b = b.reshape((5,5))

 

print(a + b)

print(a - b)

print(a * b)

print(a / b)

print(a ** 2)

print(a < b) print(a > b)

 

print(a.dot(b))

With the exception of dot() all of these operators work element-wise on the array. For example (a, b, c)+ (d, e, f) would be (a+d, b+e, c+f). It will work separately on each element, pairing the corresponding elements up and doing arithmetic on them. It will then return an array of the results. Note that when using logical operators such as < and > an array of booleans will be returned, which has a very useful application which we will go through later.

The dot() function works out the dot product of two arrays. This does not return an array, but a scalar (a value with just magnitude and no direction).

A Bit of the Maths Behind It

The dot() function is something called the dot product. The best way to understand this is to see how it is calculated.

Array Specific Operators

There are also some useful operators provided by NumPy for processing an array.

# dot, sum, min, max, cumsum

a = np.arange(10)

 

print(a.sum()) # >>>45

print(a.min()) # >>>0

print(a.max()) # >>>9

print(a.cumsum()) # >>>[ 0  1  3  6 10 15 21 28 36 45]

The sum(), min() and max() functions are pretty obvious in what they do. Add up all the elements and find the minimum and maximum elements.

The cumsum() function however is a little less obvious. It adds together every element like sum() but it does this by first adding up the first and the second and storing the result of that calculation in a list and adding that result to the third, which again is then stored in a list. This is done for all elements in the array, returning a running total of the sum of the array as a list.

Advanced Indexing

Fancy Indexing

‘Fancy indexing’ is a useful way of picking out specific array elements that you want to work with.

# Fancy indexing

a = np.arange(0, 100, 10)

indices = [1, 5, -1]

b = a[indices]

print(a) # >>>[ 0 10 20 30 40 50 60 70 80 90]

print(b) # >>>[10 50 90]

As you can see in the above example we index the array with a sequence of the specific indexes that we want to retrieve. This in turn returns a list of the the elements we indexed.

Boolean masking

Boolean masking is a fantastic feature that allows us to retrieve elements in an array based on a condition that we specify.

# Boolean masking

import matplotlib.pyplot as plt

 

a = np.linspace(0, 2 * np.pi, 50)

b = np.sin(a)

plt.plot(a,b)

mask = b >= 0

plt.plot(a[mask], b[mask], 'bo')

mask = (b >= 0) & (a <= np.pi / 2)

plt.plot(a[mask], b[mask], 'go')

plt.show()

The above example shows how to do boolean masking. All you have to do is pass the array a conditional involving the array and it will give you an array of the values that return true for that condition.

The example produces the following plot: We use the conditions to select different points on the plot. The blue points (which in the diagram also include the green points, but the green points cover up the blue ones), show all the points that have a value greater than 0. The green points show all points that have a value greater than 0 and that are less than half pi.

Incomplete Indexing

Incomplete indexing is a convenient way of taking an index or slice from the first dimension of a multidimensional array. For example if you had the array a = [[1, 2, 3, 4, 5], [6, 7, 8, 9, 10]], then a[3] would give you the element with index 3 in the first dimension of the array, which here would be the value 4.

# Incomplete Indexing

a = np.arange(0, 100, 10)

b = a[:5]

c = a[a >= 50]

print(b) # >>>[ 0 10 20 30 40]

print(c) # >>>[50 60 70 80 90]

Where

the where() function is another useful way of retrieving elements of an array conditionally. Simply pass it a condition and it will return a list of elements wherethat condition is true.

# Where

a = np.arange(0, 100, 10)

b = np.where(a < 50)

c = np.where(a >= 50)[0]

print(b) # >>>(array([0, 1, 2, 3, 4]),)

print(c) # >>>[5 6 7 8 9]

And that’s NumPy, not so hard right? Of course this post only covers the basics to get you going, there are many other things that you can do in NumPy that when you are comfortable with these basics, you should take a look at.

Jamal works as a developer for a Tokyo-based start up. He has a particular love for Python and likes to dabble in Data Science.

Don't forget to share and follow!

Remember, don't forget to share this post so that other people can see it too!Also, make sure you subscribe to this blog's mailing list, follow me on Twitter and add me on Google+so that you don't miss out on any useful posts!

I read all comments, so if you have something to say, something to share or questions and the like, leave a comment below!

Related

Continue reading on www.jamalmoir.com