How to Read File Line by Line From Bash Script

Fatmawati Achmad Zaenuri/Shutterstock

Information technology's pretty easy to read the contents of a Linux text file line by line in a beat script—equally long as you deal with some subtle gotchas. Here'southward how to do it the safe way.

Files, Text, and Idioms

Each programming language has a fix of idioms. These are the standard, no-frills means to accomplish a gear up of common tasks. They're the unproblematic or default manner to apply one of the features of the language the developer is working with. They become role of a programmer's toolkit of mental blueprints.

Actions like reading information from files, working with loops, and swapping the values of two variables are proficient examples. The developer volition know at to the lowest degree 1 way to attain their ends in a generic or vanilla fashion. Maybe that volition suffice for the requirement at hand. Or perhaps they'll embellish the code to make it more efficient or applicable to the specific solution they are developing. But having the building-block idiom at their fingertips is a groovy starting point.

Knowing and understanding idioms in i language makes it easier to selection upward a new programming linguistic communication, too. Knowing how things are constructed in i language and looking for the equivalent—or the closest thing—in another language is a adept way to capeesh the similarities and differences between programming languages you already know and the one you're learning.

Reading Lines From a File: The 1-Liner

In Fustigate, you can apply a while loop on the command line to read each line of text from a file and do something with information technology. Our text file is called "data.txt." It holds a list of the months of the year.

January February March . . October Nov December

Our simple one-liner is:

while read line; practise echo $line; done < data.txt

The while loop reads a line from the file, and the execution flow of the petty program passes to the body of the loop. The echo control writes the line of text in the terminal window. The read attempt fails when there are no more lines to be read, and the loop is done.

Ane neat flim-flam is the ability to redirect a file into a loop. In other programming languages, you lot'd need to open the file, read from it, and close it over again when you'd finished. With Fustigate, you can simply use file redirection and let the shell handle all of that low-level stuff for yous.

Of form, this one-liner isn't terribly useful. Linux already provides the cat command, which does exactly that for us. Nosotros've created a long-winded way to replace a 3-letter control. But information technology does visibly demonstrate the principles of reading from a file.

That works well enough, up to a betoken. Suppose we accept another text file that contains the names of the months. In this file, the escape sequence for a newline character has been appended to each line. We'll call information technology "data2.txt."

Jan\n February\due north March\n . . October\due north Nov\due north December\n

Let'due south use our one-liner on our new file.

while read line; do echo $line; done < data2.txt

The backslash escape character " \ " has been discarded. The result is that an "due north" has been appended to each line. Fustigate is interpreting the backslash as the start of an escape sequence. Ofttimes, nosotros don't desire Bash to interpret what information technology is reading. Information technology can be more convenient to read a line in its entirety—backslash escape sequences and all—and choose what to parse out or supplant yourself, inside your own code.

If we desire to practice any meaningful processing or parsing on the lines of text, nosotros'll demand to employ a script.

Reading Lines From a File With a Script

Here's our script. It'southward chosen "script1.sh."

                                          #!/bin/bash                                                          Counter=0                                            while                IFS=                ''                read                -r                LinefromFile                ||                [[                -north                "                ${LinefromFile}                "                ]];                exercise                                                          ((                Counter                ++                ))                                            echo                "Accessing line                                $Counter                :                                ${LinefromFile}                "                                                          done                <                "                $1                "                                    

We set a variable chosen Counter to zero, so we define our while loop.

The starting time statement on the while line is IFS='' . IFS stands for internal field separator. It holds values that Bash uses to place word boundaries. By default, the read command strips off leading and trailing whitespace. If nosotros desire to read the lines from the file exactly as they are, we need to set IFS to be an empty string.

Nosotros could set this once outside of the loop, only similar we're setting the value of Counter . Just with more complex scripts—especially those with many user-divers functions in them—it is possible that IFS could be set to different values elsewhere in the script. Ensuring that IFS is set to an empty string each fourth dimension the while loop iterates guarantees that we know what its behavior will be.

Nosotros're going to read a line of text into a variable called LinefromFile . Nosotros're using the -r (read backslash as a normal character) choice to ignore backslashes. They'll exist treated but like whatsoever other graphic symbol and won't receive any special treatment.

At that place are two weather condition that will satisfy the while loop and let the text to be candy past the torso of the loop:

• read -r LinefromFile : When a line of text is successfully read from the file, the read command sends a success signal to the while , and the while loop passes the execution flow to the torso of the loop. Note that the read command needs to encounter a newline graphic symbol at the end of the line of text in gild to consider it a successful read. If the file is not a POSIX compliant text file, the terminal line may not include a newline character. If the read control sees the end of file marking (EOF) before the line is terminated past a newline, it volition not care for it every bit a successful read. If that happens, the last line of text volition not be passed to the body of the loop and volition non exist processed.
• [ -n "${LinefromFile}" ] : We need to do some extra work to handle non-POSIX uniform files. This comparison checks the text that is read from the file. If it isn't terminated with a newline character, this comparison will still return success to the while loop. This ensures that whatever abaft line fragments are candy by the body of the loop.

These 2 clauses are separated by the OR logical operator " || " so that ifeither clause returns success, the retrieved text is processed past the body of the loop, whether there is a newline character or not.

In the body of our loop, nosotros're incrementing the Counter variable past 1 and using echo to send some output to the terminal window. The line number and the text of each line are displayed.

We can still use our redirection trick to redirect a file into a loop. In this case, we're redirecting $1, a variable that holds the name of the commencement control line parameter that passed to the script. Using this trick, we can easily pass in the name of the information file that nosotros want the script to work on.

Copy and paste the script into an editor and salvage it with the filename "script1.sh." Use the chmod command to make it executable.

chmod +x script1.sh

Let'south come across what our script makes of the data2.txt text file and the backslashes contained within it.

./script1.sh data2.txt

Every graphic symbol in the line is displayed verbatim. The backslashes are not interpreted equally escape characters. They're printed as regular characters.

Passing the Line to a Function

Nosotros're still just echoing the text to the screen. In a real-world programming scenario, we'd likely be almost to do something more interesting with the line of text. In most cases, it is a good programming exercise to handle the further processing of the line in another function.

Here's how we could practice information technology. This is "script2.sh."

                                                #!/bin/fustigate                                                                  Counter=0                                                  function                                      process_line()                  {                                                                  echo                  "Processing line                                    $Counter                  :                                    $1                  "                                                                  }                                                                  while                  IFS=                  ''                  read                  -r                  LinefromFile                  ||                  [[                  -n                  "                  ${LinefromFile}                  "                  ]];                  do                                                                  ((                  Counter                  ++                  ))                                                  process_line                  "                  $LinefromFile                  "                                                                  done                  <                  "                  $1                  "                                          

We define our Counter variable equally before, and and so we define a function chosen process_line() . The definition of a function must announced before the office is commencement chosen in the script.

Our function is going to be passed the newly read line of text in each iteration of the while loop. We can access that value within the part by using the $1 variable. If there were two variables passed to the function, we could access those values using $1 and $2 , and and so on for more variables.

The westhile loop is mainly the same. There is merely 1 alter within the body of the loop. The echo line has been replaced by a call to the process_line() office. Note that you don't need to utilise the "()" brackets in the proper noun of the function when you are calling it.

The name of the variable holding the line of text, LinefromFile , is wrapped in quotation marks when it is passed to the office. This caters for lines that have spaces in them. Without the quotation marks, the first word is treated as $ane past the role, the second word is considered to be $ii , so on. Using quotation marks ensures that the unabridged line of text is handled, altogether, as $ane. Note that this is not the same $ane that holds the same data file passed to the script.

Because Counter has been declared in the main torso of the script and non inside a function, it can exist referenced inside the process_line() function.

Copy or blazon the script above into an editor and salve it with the filename "script2.sh." Make it executable with chmod :

chmod +x script2.sh

Now we tin can run information technology and pass in a new information file, "data3.txt." This has a list of the months in it, and i line with many words on information technology.

January February March . . October November \nMore text "at the cease of the line" December

Our command is:

./script2.sh data3.txt

The lines are read from the file and passed one past i to the process_line() function. All the lines are displayed correctly, including the odd one with the backspace, quotation marks, and multiple words in it.

Building Blocks Are Useful

There's a train of idea that says that an idiom must contain something unique to that language. That'due south not a belief that I subscribe to. What's important is that information technology makes skilful apply of the language, is easy to remember, and provides a reliable and robust style to implement some functionality in your code.

littlelefordled.blogspot.com

Source: https://www.howtogeek.com/709838/how-to-process-a-file-line-by-line-in-a-linux-bash-script/

Share this post