Joining Lists in Python – How to Concatenate Lists Like a Pro

As a Python developer, you‘ll frequently need to combine multiple lists into one. Whether you‘re working with data from different sources, processing results in chunks, or reorganizing information, list concatenation is a common task.

Luckily, Python provides several ways to join lists together. In this in-depth tutorial, we‘ll explore the essentials of list concatenation in Python. You‘ll learn how to combine lists using built-in operators, how to merge lists with methods like extend() and append(), how to concatenate lists with comprehensions, and more.

We‘ll walk through practical code examples for each approach and discuss the performance implications and tradeoffs. By the end of this guide, you‘ll be well-equipped to efficiently concatenate lists in any situation. Let‘s get started!

What is List Concatenation?

In Python, concatenation refers to joining two or more objects together into a single object. When working with strings, concatenation would involve combining multiple strings into one longer string:

str1 = "hello"
str2 = "world"
full_str = str1 + str2 
print(full_str) # "helloworld"

When it comes to lists, concatenation means taking two or more lists and merging them together into one list that contains all the elements in order. For example, let‘s say we have two lists:

list1 = [1, 2, 3]
list2 = [4, 5, 6]

Concatenating them would give us a single list with all the elements:

concat_list = [1, 2, 3, 4, 5, 6]

The elements from the first list ([1, 2, 3]) come first, followed by the elements from the second list ([4, 5, 6]), in the same order they appeared in their original lists.

List concatenation comes in handy in many scenarios, such as:

• Combining data that has been split into multiple lists
• Joining results that were processed in separate batches
• Merging lists of keys and values into a list of tuples
• Reorganizing and reshaping list data structures
• Building up a list incrementally from other lists

Essentially, any time you need to combine lists, concatenation is the way to go. Python offers several techniques to concatenate lists, so let‘s look at each of them in detail.

Concatenating Lists with the + Operator

The simplest way to join two lists is to use the + operator. This creates a new list containing all the elements from the lists being combined. Here‘s a basic example:

list1 = [1, 2, 3]
list2 = [4, 5, 6]

concat_list = list1 + list2
print(concat_list) # [1, 2, 3, 4, 5, 6]

We simply add the two lists using + and assign the result to a new list concat_list. The elements are joined in the order of the operands – list1 elements come first, followed by list2 elements.

The + operator is convenient for quickly combining lists, but it does have some downsides. First, it requires creating a new list object in memory, which can be inefficient for very large lists. Second, the syntax becomes harder to read if you are concatenating many lists:

final_list = list1 + list2 + list3 + list4 + list5 # Gets unwieldy

So while the + operator is handy for joining a few small lists, it‘s not always the best tool for the job. Let‘s look at some other methods.

Repeating Lists with the * Operator

A related operation to concatenation is repetition – creating a new list by repeating an existing list a certain number of times. You can do this easily with the * operator:

list1 = [1, 2, 3]
repeated_list = list1 * 3
print(repeated_list) # [1, 2, 3, 1, 2, 3, 1, 2, 3]

Here we created a new list repeated_list by repeating list1 three times. The * operator is useful if you need to initialize a list with the same elements repeated a set number of times.

Keep in mind that * does not always give the same result as + when combining the same list:

list1 = [1, 2, 3]

repeated_sum = list1 * 2 # [1, 2, 3, 1, 2, 3]
sum_repeat = list1 + list1 # [1, 2, 3, 1, 2, 3]

print(repeated_sum == sum_repeat) # True

repeated_concat = list1 * 2
concat_repeated = (list1 + list1) * 2
print(repeated_concat == concat_repeated) # False

The sequential order of summing and repetition can lead to different final lists. Just something to watch out for.

Merging Lists with the extend() Method

While + and * are operators, Python also provides some list methods for concatenation. One of the most useful is the extend() method. It allows you to append the elements of one list to the end of another:

list1 = [1, 2, 3]
list2 = [4, 5, 6]

list1.extend(list2)
print(list1) # [1, 2, 3, 4, 5, 6]

Here we took list2 and extended list1 with its elements. Notice that list1 was updated in place – extend() modifies the list being extended rather than creating a new list.

This can make extend() more memory-efficient than +, since it avoids creating an additional list object. You can also use extend() to append any iterable, not just lists:

list1 = [1, 2, 3]
list1.extend(range(4,7))
print(list1) # [1, 2, 3, 4, 5, 6]

Here we extended list1 with the elements from a range object. Other iterables like tuples, strings, and sets can be used as well.

One thing to watch out for – because extend() alters the input list in place, the reference to that list will also change:

list1 = [1, 2, 3]
list2 = list1
list1.extend([4, 5, 6])
print(list2) # [1, 2, 3, 4, 5, 6]

Since list2 referenced the same object as list1, extending list1 also modified list2. Just something to keep in mind.

Appending Lists with a Loop

Another way to concatenate lists is to loop through one list and append each element to the other list using the append() method:

list1 = [1, 2, 3]
list2 = [4, 5, 6]

for item in list2:
    list1.append(item)

print(list1) # [1, 2, 3, 4, 5, 6]  

This loop iterates through each element in list2 and appends it to the end of list1. Like extend(), this modifies list1 in place.

The append() loop approach gives you more control over how the lists are joined. For example, you could filter the elements being appended with a conditional:

list1 = [1, 2, 3]
list2 = [4, 5, 6]

for item in list2:
    if item % 2 == 0:
        list1.append(item)

print(list1) # [1, 2, 3, 4, 6]

Now only the even elements from list2 are appending to list1. You could also perform operations on the elements before appending them:

list1 = [1, 2, 3]
list2 = [4, 5, 6]

for item in list2:
    list1.append(item ** 2)

print(list1) # [1, 2, 3, 16, 25, 36]

Here we squared each element from list2 before appending it to list1. Use cases like these make a loop more flexible than the previous methods.

However, loops can become inefficient for large lists, since we‘re calling append() many times. Luckily, there‘s a more streamlined approach.

Concatenating Lists with Comprehensions

List comprehensions are a concise way to create new lists in Python. We can also use them to join lists together efficiently. Consider our example from before:

list1 = [1, 2, 3]
list2 = [4, 5, 6]

concat_list = [item for sublist in [list1, list2] for item in sublist]
print(concat_list) # [1, 2, 3, 4, 5, 6]

Here we used a nested list comprehension to flatten list1 and list2 into a single list concat_list. The advantage of this approach is that it creates the concatenated list in a single line without needing an append() loop.

List comprehensions also make it easy to filter and transform the elements from the input lists:

list1 = [1, 2, 3] 
list2 = [4, 5, 6]

even_squared = [x**2 for sublist in [list1, list2] for x in sublist if x % 2 == 0]
print(even_squared) # [4, 16, 36]

This comprehension filters out the odd numbers from list1 and list2, squares the remaining even numbers, and combines them into a single list.

Comprehensions offer a good balance of performance and simplicity for joining lists. However, they can become hard to read if the logic gets too complex. Use judiciously.

Chaining Lists with Unpacking

Finally, let‘s consider some more advanced techniques. One powerful feature in Python is the ability to unpack iterables into lists using the * operator. This can also be used for concatenation:

list1 = [1, 2, 3]
list2 = [4, 5, 6]
list3 = [*list1, *list2]
print(list3) # [1, 2, 3, 4, 5, 6]

Here we unpacked list1 and list2 into a new list list3. We could achieve the same effect with the following steps:

list1 = [1, 2, 3]
list2 = [4, 5, 6]

list3 = []
list3.extend(list1)
list3.extend(list2)

But the unpacking syntax [list1, list2] is cleaner and creates the final list directly. We can even mix in individual elements:

list1 = [1, 2, 3]
list2 = [4, 5, 6]
list3 = [*list1, 42, *list2, 7]
print(list3) # [1, 2, 3, 42, 4, 5, 6, 7]

Unpacking provides a concise syntax for joining lists along with other elements. Iterable unpacking is a very flexible tool, so consult the Python reference for more use cases.

Chaining Iterables with itertools.chain()

The itertools module in Python‘s standard library provides a number of functions that operate on iterables. One of them, itertools.chain(), can be used to concatenate lists efficiently:

import itertools

list1 = [1, 2, 3]
list2 = [4, 5, 6] 
list3 = [7, 8, 9]

concat_list = list(itertools.chain(list1, list2, list3))
print(concat_list) # [1, 2, 3, 4, 5, 6, 7, 8, 9]

Here we passed our three lists to itertools.chain() which chained them together into a single iterable. We converted the result to a list to get our final concatenated list.

The advantage of itertools.chain() is that it operates lazily – it doesn‘t create the full concatenated list in memory, but rather yields each element one at a time. This makes it very memory-efficient, especially for large lists.

itertools.chain() can also chain any number of iterables together, not just lists:

import itertools

concat = list(itertools.chain(range(3), [3, 4], [5, 6, 7]))  
print(concat) # [0, 1, 2, 3, 4, 5, 6, 7]

Here we chained a range object, two lists, into a single list. If you‘re dealing with very large lists, definitely give itertools a look.

Conclusion

In this guide, we‘ve covered the main techniques for concatenating lists in Python:

• Using the + operator to add lists
• Using the * operator to repeat lists
• Using the list.extend() method
• Appending elements with a loop
• Combining lists with comprehensions
• Unpacking iterables into new lists
• Chaining iterables with itertools.chain()

Each approach has its strengths and weaknesses in terms of performance, memory usage, and readability. Let‘s summarize the key takeaways:

• For simply joining a few small lists, the + operator is usually sufficient
• For memory-efficient concatenation of lists, use extend() or itertools.chain()
• For joining lists with conditional logic or transformations, use a comprehension
• For very large lists, use itertools.chain() to avoid memory issues

In general, it‘s good to be familiar with all these methods so you can choose the right tool for the job. List concatenation is a common task, so it pays to understand the tradeoffs.

I hope this in-depth guide has been helpful! Let me know if you have any other questions. Happy coding!