[awk] The GNU Awk User's Guide

The GNU Awk User’s Guide

Next: Foreword, Up: (dir) [Contents][Index]

---

General Introduction

This file documents awk, a program that you can use to select particular records in a file and perform operations upon them.

Copyright © 1989, 1991, 1992, 1993, 1996–2005, 2007, 2009–2014
Free Software Foundation, Inc.

This is Edition 4.1 of GAWK: Effective AWK Programming: A User’s Guide for GNU Awk, for the 4.1.1 (or later) version of the GNU implementation of AWK.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with the Invariant Sections being “GNU General Public License”, the Front-Cover texts being (a) (see below), and with the Back-Cover Texts being (b) (see below). A copy of the license is included in the section entitled“GNU Free Documentation License”.

1. “A GNU Manual”
2. “You have the freedom to copy and modify this GNU manual. Buying copies from the FSF supports it in developing GNU and promoting software freedom.”

• Foreword:		Some nice words about this Web page.
• Preface:		What this Web page is about; brief history and acknowledgments.
• Getting Started:		A basic introduction to using awk. How to run an awk program. Command-line syntax.
• Invoking Gawk:		How to run gawk.
• Regexp:		All about matching things using regular expressions.
• Reading Files:		How to read files and manipulate fields.
• Printing:		How to print using awk. Describes the print and printf statements. Also describes redirection of output.
• Expressions:		Expressions are the basic building blocks of statements.
• Patterns and Actions:		Overviews of patterns and actions.
• Arrays:		The description and use of arrays. Also includes array-oriented control statements.
• Functions:		Built-in and user-defined functions.
• Library Functions:		A Library of awk Functions.
• Sample Programs:		Many awk programs with complete explanations.
• Advanced Features:		Stuff for advanced users, specific to gawk.
• Internationalization:		Getting gawk to speak your language.
• Debugger:		The gawk debugger.
• Arbitrary Precision Arithmetic:		Arbitrary precision arithmetic with gawk.
• Dynamic Extensions:		Adding new built-in functions to gawk.
• Language History:		The evolution of the awk language.
• Installation:		Installing gawk under various operating systems.
• Notes:		Notes about adding things to gawk and possible future work.
• Basic Concepts:		A very quick introduction to programming concepts.
• Glossary:		An explanation of some unfamiliar terms.
• Copying:		Your right to copy and distribute gawk.
• GNU Free Documentation License:		The license for this Web page.
• Index:		Concept and Variable Index.
• History:		The history of gawk and awk.
• Names:		What name to use to find awk.
• This Manual:		Using this Web page. Includes sample input files that you can use.
• Conventions:		Typographical Conventions.
• Manual History:		Brief history of the GNU project and this Web page.
• How To Contribute:		Helping to save the world.
• Acknowledgments:		Acknowledgments.
• Running gawk:		How to run gawk programs; includes command-line syntax.
• One-shot:		Running a short throwaway awk program.
• Read Terminal:		Using no input files (input from terminal instead).
• Long:		Putting permanent awk programs in files.
• Executable Scripts:		Making self-contained awk programs.
• Comments:		Adding documentation to gawk programs.
• Quoting:		More discussion of shell quoting issues.
• DOS Quoting:		Quoting in Windows Batch Files.
• Sample Data Files:		Sample data files for use in the awk programs illustrated in this Web page.
• Very Simple:		A very simple example.
• Two Rules:		A less simple one-line example using two rules.
• More Complex:		A more complex example.
• Statements/Lines:		Subdividing or combining statements into lines.
• Other Features:		Other Features of awk.
• When:		When to use gawk and when to use other things.
• Command Line:		How to run awk.
• Options:		Command-line options and their meanings.
• Other Arguments:		Input file names and variable assignments.
• Naming Standard Input:		How to specify standard input with other files.
• Environment Variables:		The environment variables gawk uses.
• AWKPATH Variable:		Searching directories for awk programs.
• AWKLIBPATH Variable:		Searching directories for awk shared libraries.
• Other Environment Variables:		The environment variables.
• Exit Status:		gawk’s exit status.
• Include Files:		Including other files into your program.
• Loading Shared Libraries:		Loading shared libraries into your program.
• Obsolete:		Obsolete Options and/or features.
• Undocumented:		Undocumented Options and Features.
• Regexp Usage:		How to Use Regular Expressions.
• Escape Sequences:		How to write nonprinting characters.
• Regexp Operators:		Regular Expression Operators.
• Bracket Expressions:		What can go between ‘[...]’.
• GNU Regexp Operators:		Operators specific to GNU software.
• Case-sensitivity:		How to do case-insensitive matching.
• Leftmost Longest:		How much text matches.
• Computed Regexps:		Using Dynamic Regexps.
• Records:		Controlling how data is split into records.
• Fields:		An introduction to fields.
• Nonconstant Fields:		Nonconstant Field Numbers.
• Changing Fields:		Changing the Contents of a Field.
• Field Separators:		The field separator and how to change it.
• Default Field Splitting:		How fields are normally separated.
• Regexp Field Splitting:		Using regexps as the field separator.
• Single Character Fields:		Making each character a separate field.
• Command Line Field Separator:		Setting FS from the command-line.
• Full Line Fields:		Making the full line be a single field.
• Field Splitting Summary:		Some final points and a summary table.
• Constant Size:		Reading constant width data.
• Splitting By Content:		Defining Fields By Content
• Multiple Line:		Reading multiline records.
• Getline:		Reading files under explicit program control using the getline function.
• Plain Getline:		Using getline with no arguments.
• Getline/Variable:		Using getline into a variable.
• Getline/File:		Using getline from a file.
• Getline/Variable/File:		Using getline into a variable from a file.
• Getline/Pipe:		Using getline from a pipe.
• Getline/Variable/Pipe:		Using getline into a variable from a pipe.
• Getline/Coprocess:		Using getline from a coprocess.
• Getline/Variable/Coprocess:		Using getline into a variable from a coprocess.
• Getline Notes:		Important things to know about getline.
• Getline Summary:		Summary of getline Variants.
• Read Timeout:		Reading input with a timeout.
• Command line directories:		What happens if you put a directory on the command line.
• Print:		The print statement.
• Print Examples:		Simple examples of print statements.
• Output Separators:		The output separators and how to change them.
• OFMT:		Controlling Numeric Output With print.
• Printf:		The printf statement.
• Basic Printf:		Syntax of the printf statement.
• Control Letters:		Format-control letters.
• Format Modifiers:		Format-specification modifiers.
• Printf Examples:		Several examples.
• Redirection:		How to redirect output to multiple files and pipes.
• Special Files:		File name interpretation in gawk. gawk allows access to inherited file descriptors.
• Special FD:		Special files for I/O.
• Special Network:		Special files for network communications.
• Special Caveats:		Things to watch out for.
• Close Files And Pipes:		Closing Input and Output Files and Pipes.
• Values:		Constants, Variables, and Regular Expressions.
• Constants:		String, numeric and regexp constants.
• Scalar Constants:		Numeric and string constants.
• Nondecimal-numbers:		What are octal and hex numbers.
• Regexp Constants:		Regular Expression constants.
• Using Constant Regexps:		When and how to use a regexp constant.
• Variables:		Variables give names to values for later use.
• Using Variables:		Using variables in your programs.
• Assignment Options:		Setting variables on the command-line and a summary of command-line syntax. This is an advanced method of input.
• Conversion:		The conversion of strings to numbers and vice versa.
• All Operators:		gawk’s operators.
• Arithmetic Ops:		Arithmetic operations (‘+’, ‘-’, etc.)
• Concatenation:		Concatenating strings.
• Assignment Ops:		Changing the value of a variable or a field.
• Increment Ops:		Incrementing the numeric value of a variable.
• Truth Values and Conditions:		Testing for true and false.
• Truth Values:		What is “true” and what is “false”.
• Typing and Comparison:		How variables acquire types and how this affects comparison of numbers and strings with ‘<’, etc.
• Variable Typing:		String type versus numeric type.
• Comparison Operators:		The comparison operators.
• POSIX String Comparison:		String comparison with POSIX rules.
• Boolean Ops:		Combining comparison expressions using boolean operators ‘||’ (“or”), ‘&&’ (“and”) and ‘!’ (“not”).
• Conditional Exp:		Conditional expressions select between two subexpressions under control of a third subexpression.
• Function Calls:		A function call is an expression.
• Precedence:		How various operators nest.
• Locales:		How the locale affects things.
• Pattern Overview:		What goes into a pattern.
• Regexp Patterns:		Using regexps as patterns.
• Expression Patterns:		Any expression can be used as a pattern.
• Ranges:		Pairs of patterns specify record ranges.
• BEGIN/END:		Specifying initialization and cleanup rules.
• Using BEGIN/END:		How and why to use BEGIN/END rules.
• I/O And BEGIN/END:		I/O issues in BEGIN/END rules.
• BEGINFILE/ENDFILE:		Two special patterns for advanced control.
• Empty:		The empty pattern, which matches every record.
• Using Shell Variables:		How to use shell variables with awk.
• Action Overview:		What goes into an action.
• Statements:		Describes the various control statements in detail.
• If Statement:		Conditionally execute some awk statements.
• While Statement:		Loop until some condition is satisfied.
• Do Statement:		Do specified action while looping until some condition is satisfied.
• For Statement:		Another looping statement, that provides initialization and increment clauses.
• Switch Statement:		Switch/case evaluation for conditional execution of statements based on a value.
• Break Statement:		Immediately exit the innermost enclosing loop.
• Continue Statement:		Skip to the end of the innermost enclosing loop.
• Next Statement:		Stop processing the current input record.
• Nextfile Statement:		Stop processing the current file.
• Exit Statement:		Stop execution of awk.
• Built-in Variables:		Summarizes the built-in variables.
• User-modified:		Built-in variables that you change to control awk.
• Auto-set:		Built-in variables where awk gives you information.
• ARGC and ARGV:		Ways to use ARGC and ARGV.
• Array Basics:		The basics of arrays.
• Array Intro:		Introduction to Arrays
• Reference to Elements:		How to examine one element of an array.
• Assigning Elements:		How to change an element of an array.
• Array Example:		Basic Example of an Array
• Scanning an Array:		A variation of the for statement. It loops through the indices of an array’s existing elements.
• Controlling Scanning:		Controlling the order in which arrays are scanned.
• Delete:		The delete statement removes an element from an array.
• Numeric Array Subscripts:		How to use numbers as subscripts in awk.
• Uninitialized Subscripts:		Using Uninitialized variables as subscripts.
• Multidimensional:		Emulating multidimensional arrays in awk.
• Multiscanning:		Scanning multidimensional arrays.
• Arrays of Arrays:		True multidimensional arrays.
• Built-in:		Summarizes the built-in functions.
• Calling Built-in:		How to call built-in functions.
• Numeric Functions:		Functions that work with numbers, including int(), sin() and rand().
• String Functions:		Functions for string manipulation, such as split(), match() and sprintf().
• Gory Details:		More than you want to know about ‘\’ and ‘&’ with sub(), gsub(), and gensub().
• I/O Functions:		Functions for files and shell commands.
• Time Functions:		Functions for dealing with timestamps.
• Bitwise Functions:		Functions for bitwise operations.
• Type Functions:		Functions for type information.
• I18N Functions:		Functions for string translation.
• User-defined:		Describes User-defined functions in detail.
• Definition Syntax:		How to write definitions and what they mean.
• Function Example:		An example function definition and what it does.
• Function Caveats:		Things to watch out for.
• Calling A Function:		Don’t use spaces.
• Variable Scope:		Controlling variable scope.
• Pass By Value/Reference:		Passing parameters.
• Return Statement:		Specifying the value a function returns.
• Dynamic Typing:		How variable types can change at runtime.
• Indirect Calls:		Choosing the function to call at runtime.
• Library Names:		How to best name private global variables in library functions.
• General Functions:		Functions that are of general use.
• Strtonum Function:		A replacement for the built-in strtonum() function.
• Assert Function:		A function for assertions in awk programs.
• Round Function:		A function for rounding if sprintf() does not do it correctly.
• Cliff Random Function:		The Cliff Random Number Generator.
• Ordinal Functions:		Functions for using characters as numbers and vice versa.
• Join Function:		A function to join an array into a string.
• Getlocaltime Function:		A function to get formatted times.
• Readfile Function:		A function to read an entire file at once.
• Data File Management:		Functions for managing command-line data files.
• Filetrans Function:		A function for handling data file transitions.
• Rewind Function:		A function for rereading the current file.
• File Checking:		Checking that data files are readable.
• Empty Files:		Checking for zero-length files.
• Ignoring Assigns:		Treating assignments as file names.
• Getopt Function:		A function for processing command-line arguments.
• Passwd Functions:		Functions for getting user information.
• Group Functions:		Functions for getting group information.
• Walking Arrays:		A function to walk arrays of arrays.
• Running Examples:		How to run these examples.
• Clones:		Clones of common utilities.
• Cut Program:		The cut utility.
• Egrep Program:		The egrep utility.
• Id Program:		The id utility.
• Split Program:		The split utility.
• Tee Program:		The tee utility.
• Uniq Program:		The uniq utility.
• Wc Program:		The wc utility.
• Miscellaneous Programs:		Some interesting awk programs.
• Dupword Program:		Finding duplicated words in a document.
• Alarm Program:		An alarm clock.
• Translate Program:		A program similar to the tr utility.
• Labels Program:		Printing mailing labels.
• Word Sorting:		A program to produce a word usage count.
• History Sorting:		Eliminating duplicate entries from a history file.
• Extract Program:		Pulling out programs from Texinfo source files.
• Simple Sed:		A Simple Stream Editor.
• Igawk Program:		A wrapper for awk that includes files.
• Anagram Program:		Finding anagrams from a dictionary.
• Signature Program:		People do amazing things with too much time on their hands.
• Nondecimal Data:		Allowing nondecimal input data.
• Array Sorting:		Facilities for controlling array traversal and sorting arrays.
• Controlling Array Traversal:		How to use PROCINFO["sorted_in"].
• Array Sorting Functions:		How to use asort() and asorti().
• Two-way I/O:		Two-way communications with another process.
• TCP/IP Networking:		Using gawk for network programming.
• Profiling:		Profiling your awk programs.
• I18N and L10N:		Internationalization and Localization.
• Explaining gettext:		How GNU gettext works.
• Programmer i18n:		Features for the programmer.
• Translator i18n:		Features for the translator.
• String Extraction:		Extracting marked strings.
• Printf Ordering:		Rearranging printf arguments.
• I18N Portability:		awk-level portability issues.
• I18N Example:		A simple i18n example.
• Gawk I18N:		gawk is also internationalized.
• Debugging:		Introduction to gawk debugger.
• Debugging Concepts:		Debugging in General.
• Debugging Terms:		Additional Debugging Concepts.
• Awk Debugging:		Awk Debugging.
• Sample Debugging Session:		Sample debugging session.
• Debugger Invocation:		How to Start the Debugger.
• Finding The Bug:		Finding the Bug.
• List of Debugger Commands:		Main debugger commands.
• Breakpoint Control:		Control of Breakpoints.
• Debugger Execution Control:		Control of Execution.
• Viewing And Changing Data:		Viewing and Changing Data.
• Execution Stack:		Dealing with the Stack.
• Debugger Info:		Obtaining Information about the Program and the Debugger State.
• Miscellaneous Debugger Commands:		Miscellaneous Commands.
• Readline Support:		Readline support.
• Limitations:		Limitations and future plans.
• General Arithmetic:		An introduction to computer arithmetic.
• Floating Point Issues:		Stuff to know about floating-point numbers.
• String Conversion Precision:		The String Value Can Lie.
• Unexpected Results:		Floating Point Numbers Are Not Abstract Numbers.
• POSIX Floating Point Problems:		Standards Versus Existing Practice.
• Integer Programming:		Effective integer programming.
• Floating-point Programming:		Effective Floating-point Programming.
• Floating-point Representation:		Binary floating-point representation.
• Floating-point Context:		Floating-point context.
• Rounding Mode:		Floating-point rounding mode.
• Gawk and MPFR:		How gawk provides arbitrary-precision arithmetic.
• Arbitrary Precision Floats:		Arbitrary Precision Floating-point Arithmetic with gawk.
• Setting Precision:		Setting the working precision.
• Setting Rounding Mode:		Setting the rounding mode.
• Floating-point Constants:		Representing floating-point constants.
• Changing Precision:		Changing the precision of a number.
• Exact Arithmetic:		Exact arithmetic with floating-point numbers.
• Arbitrary Precision Integers:		Arbitrary Precision Integer Arithmetic with gawk.
• Extension Intro:		What is an extension.
• Plugin License:		A note about licensing.
• Extension Mechanism Outline:		An outline of how it works.
• Extension API Description:		A full description of the API.
• Extension API Functions Introduction:		Introduction to the API functions.
• General Data Types:		The data types.
• Requesting Values:		How to get a value.
• Memory Allocation Functions:		Functions for allocating memory.
• Constructor Functions:		Functions for creating values.
• Registration Functions:		Functions to register things with gawk.
• Extension Functions:		Registering extension functions.
• Exit Callback Functions:		Registering an exit callback.
• Extension Version String:		Registering a version string.
• Input Parsers:		Registering an input parser.
• Output Wrappers:		Registering an output wrapper.
• Two-way processors:		Registering a two-way processor.
• Printing Messages:		Functions for printing messages.
• Updating ERRNO:		Functions for updating ERRNO.
• Accessing Parameters:		Functions for accessing parameters.
• Symbol Table Access:		Functions for accessing global variables.
• Symbol table by name:		Accessing variables by name.
• Symbol table by cookie:		Accessing variables by “cookie”.
• Cached values:		Creating and using cached values.
• Array Manipulation:		Functions for working with arrays.
• Array Data Types:		Data types for working with arrays.
• Array Functions:		Functions for working with arrays.
• Flattening Arrays:		How to flatten arrays.
• Creating Arrays:		How to create and populate arrays.
• Extension API Variables:		Variables provided by the API.
• Extension Versioning:		API Version information.
• Extension API Informational Variables:		Variables providing information about gawk’s invocation.
• Extension API Boilerplate:		Boilerplate code for using the API.
• Finding Extensions:		How gawk finds compiled extensions.
• Extension Example:		Example C code for an extension.
• Internal File Description:		What the new functions will do.
• Internal File Ops:		The code for internal file operations.
• Using Internal File Ops:		How to use an external extension.
• Extension Samples:		The sample extensions that ship with gawk.
• Extension Sample File Functions:		The file functions sample.
• Extension Sample Fnmatch:		An interface to fnmatch().
• Extension Sample Fork:		An interface to fork() and other process functions.
• Extension Sample Inplace:		Enabling in-place file editing.
• Extension Sample Ord:		Character to value to character conversions.
• Extension Sample Readdir:		An interface to readdir().
• Extension Sample Revout:		Reversing output sample output wrapper.
• Extension Sample Rev2way:		Reversing data sample two-way processor.
• Extension Sample Read write array:		Serializing an array to a file.
• Extension Sample Readfile:		Reading an entire file into a string.
• Extension Sample API Tests:		Tests for the API.
• Extension Sample Time:		An interface to gettimeofday() and sleep().
• gawkextlib:		The gawkextlib project.
• V7/SVR3.1:		The major changes between V7 and System V Release 3.1.
• SVR4:		Minor changes between System V Releases 3.1 and 4.
• POSIX:		New features from the POSIX standard.
• BTL:		New features from Brian Kernighan’s version of awk.
• POSIX/GNU:		The extensions in gawk not in POSIX awk.
• Feature History:		The history of the features in gawk.
• Common Extensions:		Common Extensions Summary.
• Ranges and Locales:		How locales used to affect regexp ranges.
• Contributors:		The major contributors to gawk.
• Gawk Distribution:		What is in the gawk distribution.
• Getting:		How to get the distribution.
• Extracting:		How to extract the distribution.
• Distribution contents:		What is in the distribution.
• Unix Installation:		Installing gawk under various versions of Unix.
• Quick Installation:		Compiling gawk under Unix.
• Additional Configuration Options:		Other compile-time options.
• Configuration Philosophy:		How it’s all supposed to work.
• Non-Unix Installation:		Installation on Other Operating Systems.
• PC Installation:		Installing and Compiling gawk on MS-DOS and OS/2.
• PC Binary Installation:		Installing a prepared distribution.
• PC Compiling:		Compiling gawk for MS-DOS, Windows32, and OS/2.
• PC Testing:		Testing gawk on PC systems.
• PC Using:		Running gawk on MS-DOS, Windows32 and OS/2.
• Cygwin:		Building and running gawk for Cygwin.
• MSYS:		Using gawk In The MSYS Environment.
• VMS Installation:		Installing gawk on VMS.
• VMS Compilation:		How to compile gawk under VMS.
• VMS Dynamic Extensions:		Compiling gawk dynamic extensions on VMS.
• VMS Installation Details:		How to install gawk under VMS.
• VMS Running:		How to run gawk under VMS.
• VMS GNV:		The VMS GNV Project.
• VMS Old Gawk:		An old version comes with some VMS systems.
• Bugs:		Reporting Problems and Bugs.
• Other Versions:		Other freely available awk implementations.
• Compatibility Mode:		How to disable certain gawk extensions.
• Additions:		Making Additions To gawk.
• Accessing The Source:		Accessing the Git repository.
• Adding Code:		Adding code to the main body of gawk.
• New Ports:		Porting gawk to a new operating system.
• Derived Files:		Why derived files are kept in the git repository.
• Future Extensions:		New features that may be implemented one day.
• Implementation Limitations:		Some limitations of the implementation.
• Extension Design:		Design notes about the extension API.
• Old Extension Problems:		Problems with the old mechanism.
• Extension New Mechanism Goals:		Goals for the new mechanism.
• Extension Other Design Decisions:		Some other design decisions.
• Extension Future Growth:		Some room for future growth.
• Old Extension Mechanism:		Some compatibility for old extensions.
• Basic High Level:		The high level view.
• Basic Data Typing:		A very quick intro to data types.

Short Table of Contents

• Foreword
• Preface
• 1 Getting Started with awk
• 2 Running awk and gawk
• 3 Regular Expressions
• 4 Reading Input Files
• 5 Printing Output
• 6 Expressions
• 7 Patterns, Actions, and Variables
• 8 Arrays in awk
• 9 Functions
• 10 A Library of awk Functions
• 11 Practical awk Programs
• 12 Advanced Features of gawk
• 13 Internationalization with gawk
• 14 Debugging awk Programs
• 15 Arithmetic and Arbitrary Precision Arithmetic with gawk
• 16 Writing Extensions for gawk
• Appendix A The Evolution of the awk Language
• Appendix B Installing gawk
• Appendix C Implementation Notes
• Appendix D Basic Programming Concepts
• Glossary
• GNU General Public License
• GNU Free Documentation License
• Index

Table of Contents

• Foreword
• Preface
  • History of awk and gawk
  • A Rose by Any Other Name
  • Using This Book
  • Typographical Conventions
    • Dark Corners
  • The GNU Project and This Book
  • How to Contribute
  • Acknowledgments
• 1 Getting Started with awk
  • 1.1 How to Run awk Programs
    • 1.1.1 One-Shot Throwaway awk Programs
    • 1.1.2 Running awk Without Input Files
    • 1.1.3 Running Long Programs
    • 1.1.4 Executable awk Programs
    • 1.1.5 Comments in awk Programs
    • 1.1.6 Shell-Quoting Issues
      • 1.1.6.1 Quoting in MS-Windows Batch Files
  • 1.2 Data Files for the Examples
  • 1.3 Some Simple Examples
  • 1.4 An Example with Two Rules
  • 1.5 A More Complex Example
  • 1.6 awk Statements Versus Lines
  • 1.7 Other Features of awk
  • 1.8 When to Use awk
• 2 Running awk and gawk
  • 2.1 Invoking awk
  • 2.2 Command-Line Options
  • 2.3 Other Command-Line Arguments
  • 2.4 Naming Standard Input
  • 2.5 The Environment Variables gawk Uses
    • 2.5.1 The AWKPATH Environment Variable
    • 2.5.2 The AWKLIBPATH Environment Variable
    • 2.5.3 Other Environment Variables
  • 2.6 gawk’s Exit Status
  • 2.7 Including Other Files Into Your Program
  • 2.8 Loading Shared Libraries Into Your Program
  • 2.9 Obsolete Options and/or Features
  • 2.10 Undocumented Options and Features
• 3 Regular Expressions
  • 3.1 How to Use Regular Expressions
  • 3.2 Escape Sequences
  • 3.3 Regular Expression Operators
  • 3.4 Using Bracket Expressions
  • 3.5 gawk-Specific Regexp Operators
  • 3.6 Case Sensitivity in Matching
  • 3.7 How Much Text Matches?
  • 3.8 Using Dynamic Regexps
• 4 Reading Input Files
  • 4.1 How Input Is Split into Records
  • 4.2 Examining Fields
  • 4.3 Nonconstant Field Numbers
  • 4.4 Changing the Contents of a Field
  • 4.5 Specifying How Fields Are Separated
    • 4.5.1 Whitespace Normally Separates Fields
    • 4.5.2 Using Regular Expressions to Separate Fields
    • 4.5.3 Making Each Character a Separate Field
    • 4.5.4 Setting FS from the Command Line
    • 4.5.5 Making The Full Line Be A Single Field
    • 4.5.6 Field-Splitting Summary
  • 4.6 Reading Fixed-Width Data
  • 4.7 Defining Fields By Content
  • 4.8 Multiple-Line Records
  • 4.9 Explicit Input with getline
    • 4.9.1 Using getline with No Arguments
    • 4.9.2 Using getline into a Variable
    • 4.9.3 Using getline from a File
    • 4.9.4 Using getline into a Variable from a File
    • 4.9.5 Using getline from a Pipe
    • 4.9.6 Using getline into a Variable from a Pipe
    • 4.9.7 Using getline from a Coprocess
    • 4.9.8 Using getline into a Variable from a Coprocess
    • 4.9.9 Points to Remember About getline
    • 4.9.10 Summary of getline Variants
  • 4.10 Reading Input With A Timeout
  • 4.11 Directories On The Command Line
• 5 Printing Output
  • 5.1 The print Statement
  • 5.2 print Statement Examples
  • 5.3 Output Separators
  • 5.4 Controlling Numeric Output with print
  • 5.5 Using printf Statements for Fancier Printing
    • 5.5.1 Introduction to the printf Statement
    • 5.5.2 Format-Control Letters
    • 5.5.3 Modifiers for printf Formats
    • 5.5.4 Examples Using printf
  • 5.6 Redirecting Output of print and printf
  • 5.7 Special File Names in gawk
    • 5.7.1 Special Files for Standard Descriptors
    • 5.7.2 Special Files for Network Communications
    • 5.7.3 Special File Name Caveats
  • 5.8 Closing Input and Output Redirections
• 6 Expressions
  • 6.1 Constants, Variables and Conversions
    • 6.1.1 Constant Expressions
      • 6.1.1.1 Numeric and String Constants
      • 6.1.1.2 Octal and Hexadecimal Numbers
      • 6.1.1.3 Regular Expression Constants
    • 6.1.2 Using Regular Expression Constants
    • 6.1.3 Variables
      • 6.1.3.1 Using Variables in a Program
      • 6.1.3.2 Assigning Variables on the Command Line
    • 6.1.4 Conversion of Strings and Numbers
  • 6.2 Operators: Doing Something With Values
    • 6.2.1 Arithmetic Operators
    • 6.2.2 String Concatenation
    • 6.2.3 Assignment Expressions
    • 6.2.4 Increment and Decrement Operators
  • 6.3 Truth Values and Conditions
    • 6.3.1 True and False in awk
    • 6.3.2 Variable Typing and Comparison Expressions
      • 6.3.2.1 String Type Versus Numeric Type
      • 6.3.2.2 Comparison Operators
      • 6.3.2.3 String Comparison With POSIX Rules
    • 6.3.3 Boolean Expressions
    • 6.3.4 Conditional Expressions
  • 6.4 Function Calls
  • 6.5 Operator Precedence (How Operators Nest)
  • 6.6 Where You Are Makes A Difference
• 7 Patterns, Actions, and Variables
  • 7.1 Pattern Elements
    • 7.1.1 Regular Expressions as Patterns
    • 7.1.2 Expressions as Patterns
    • 7.1.3 Specifying Record Ranges with Patterns
    • 7.1.4 The BEGIN and END Special Patterns
      • 7.1.4.1 Startup and Cleanup Actions
      • 7.1.4.2 Input/Output from BEGIN and END Rules
    • 7.1.5 The BEGINFILE and ENDFILE Special Patterns
    • 7.1.6 The Empty Pattern
  • 7.2 Using Shell Variables in Programs
  • 7.3 Actions
  • 7.4 Control Statements in Actions
    • 7.4.1 The if-else Statement
    • 7.4.2 The while Statement
    • 7.4.3 The do-while Statement
    • 7.4.4 The for Statement
    • 7.4.5 The switch Statement
    • 7.4.6 The break Statement
    • 7.4.7 The continue Statement
    • 7.4.8 The next Statement
    • 7.4.9 The nextfile Statement
    • 7.4.10 The exit Statement
  • 7.5 Built-in Variables
    • 7.5.1 Built-in Variables That Control awk
    • 7.5.2 Built-in Variables That Convey Information
    • 7.5.3 Using ARGC and ARGV
• 8 Arrays in awk
  • 8.1 The Basics of Arrays
    • 8.1.1 Introduction to Arrays
    • 8.1.2 Referring to an Array Element
    • 8.1.3 Assigning Array Elements
    • 8.1.4 Basic Array Example
    • 8.1.5 Scanning All Elements of an Array
    • 8.1.6 Using Predefined Array Scanning Orders
  • 8.2 The delete Statement
  • 8.3 Using Numbers to Subscript Arrays
  • 8.4 Using Uninitialized Variables as Subscripts
  • 8.5 Multidimensional Arrays
    • 8.5.1 Scanning Multidimensional Arrays
  • 8.6 Arrays of Arrays
• 9 Functions
  • 9.1 Built-in Functions
    • 9.1.1 Calling Built-in Functions
    • 9.1.2 Numeric Functions
    • 9.1.3 String-Manipulation Functions
      • 9.1.3.1 More About ‘\’ and ‘&’ with sub(), gsub(), and gensub()
    • 9.1.4 Input/Output Functions
    • 9.1.5 Time Functions
    • 9.1.6 Bit-Manipulation Functions
    • 9.1.7 Getting Type Information
    • 9.1.8 String-Translation Functions
  • 9.2 User-Defined Functions
    • 9.2.1 Function Definition Syntax
    • 9.2.2 Function Definition Examples
    • 9.2.3 Calling User-Defined Functions
      • 9.2.3.1 Writing A Function Call
      • 9.2.3.2 Controlling Variable Scope
      • 9.2.3.3 Passing Function Arguments By Value Or By Reference
    • 9.2.4 The return Statement
    • 9.2.5 Functions and Their Effects on Variable Typing
  • 9.3 Indirect Function Calls
• 10 A Library of awk Functions
  • 10.1 Naming Library Function Global Variables
  • 10.2 General Programming
    • 10.2.1 Converting Strings To Numbers
    • 10.2.2 Assertions
    • 10.2.3 Rounding Numbers
    • 10.2.4 The Cliff Random Number Generator
    • 10.2.5 Translating Between Characters and Numbers
    • 10.2.6 Merging an Array into a String
    • 10.2.7 Managing the Time of Day
    • 10.2.8 Reading A Whole File At Once
  • 10.3 Data File Management
    • 10.3.1 Noting Data File Boundaries
    • 10.3.2 Rereading the Current File
    • 10.3.3 Checking for Readable Data Files
    • 10.3.4 Checking For Zero-length Files
    • 10.3.5 Treating Assignments as File Names
  • 10.4 Processing Command-Line Options
  • 10.5 Reading the User Database
  • 10.6 Reading the Group Database
  • 10.7 Traversing Arrays of Arrays
• 11 Practical awk Programs
  • 11.1 Running the Example Programs
  • 11.2 Reinventing Wheels for Fun and Profit
    • 11.2.1 Cutting out Fields and Columns
    • 11.2.2 Searching for Regular Expressions in Files
    • 11.2.3 Printing out User Information
    • 11.2.4 Splitting a Large File into Pieces
    • 11.2.5 Duplicating Output into Multiple Files
    • 11.2.6 Printing Nonduplicated Lines of Text
    • 11.2.7 Counting Things
  • 11.3 A Grab Bag of awk Programs
    • 11.3.1 Finding Duplicated Words in a Document
    • 11.3.2 An Alarm Clock Program
    • 11.3.3 Transliterating Characters
    • 11.3.4 Printing Mailing Labels
    • 11.3.5 Generating Word-Usage Counts
    • 11.3.6 Removing Duplicates from Unsorted Text
    • 11.3.7 Extracting Programs from Texinfo Source Files
    • 11.3.8 A Simple Stream Editor
    • 11.3.9 An Easy Way to Use Library Functions
    • 11.3.10 Finding Anagrams From A Dictionary
    • 11.3.11 And Now For Something Completely Different
• 12 Advanced Features of gawk
  • 12.1 Allowing Nondecimal Input Data
  • 12.2 Controlling Array Traversal and Array Sorting
    • 12.2.1 Controlling Array Traversal
    • 12.2.2 Sorting Array Values and Indices with gawk
  • 12.3 Two-Way Communications with Another Process
  • 12.4 Using gawk for Network Programming
  • 12.5 Profiling Your awk Programs
• 13 Internationalization with gawk
  • 13.1 Internationalization and Localization
  • 13.2 GNU gettext
  • 13.3 Internationalizing awk Programs
  • 13.4 Translating awk Programs
    • 13.4.1 Extracting Marked Strings
    • 13.4.2 Rearranging printf Arguments
    • 13.4.3 awk Portability Issues
  • 13.5 A Simple Internationalization Example
  • 13.6 gawk Can Speak Your Language
• 14 Debugging awk Programs
  • 14.1 Introduction to gawk Debugger
    • 14.1.1 Debugging in General
    • 14.1.2 Additional Debugging Concepts
    • 14.1.3 Awk Debugging
  • 14.2 Sample Debugging Session
    • 14.2.1 How to Start the Debugger
    • 14.2.2 Finding the Bug
  • 14.3 Main Debugger Commands
    • 14.3.1 Control of Breakpoints
    • 14.3.2 Control of Execution
    • 14.3.3 Viewing and Changing Data
    • 14.3.4 Dealing with the Stack
    • 14.3.5 Obtaining Information about the Program and the Debugger State
    • 14.3.6 Miscellaneous Commands
  • 14.4 Readline Support
  • 14.5 Limitations and Future Plans
• 15 Arithmetic and Arbitrary Precision Arithmetic with gawk
  • 15.1 A General Description of Computer Arithmetic
    • 15.1.1 Floating-Point Number Caveats
      • 15.1.1.1 The String Value Can Lie
      • 15.1.1.2 Floating Point Numbers Are Not Abstract Numbers
      • 15.1.1.3 Standards Versus Existing Practice
    • 15.1.2 Mixing Integers And Floating-point
  • 15.2 Understanding Floating-point Programming
    • 15.2.1 Binary Floating-point Representation
    • 15.2.2 Floating-point Context
    • 15.2.3 Floating-point Rounding Mode
  • 15.3 gawk + MPFR = Powerful Arithmetic
  • 15.4 Arbitrary Precision Floating-point Arithmetic with gawk
    • 15.4.1 Setting the Working Precision
    • 15.4.2 Setting the Rounding Mode
    • 15.4.3 Representing Floating-point Constants
    • 15.4.4 Changing the Precision of a Number
    • 15.4.5 Exact Arithmetic with Floating-point Numbers
  • 15.5 Arbitrary Precision Integer Arithmetic with gawk
• 16 Writing Extensions for gawk
  • 16.1 Introduction
  • 16.2 Extension Licensing
  • 16.3 At A High Level How It Works
  • 16.4 API Description
    • 16.4.1 Introduction
    • 16.4.2 General Purpose Data Types
    • 16.4.3 Requesting Values
    • 16.4.4 Memory Allocation Functions and Convenience Macros
    • 16.4.5 Constructor Functions
    • 16.4.6 Registration Functions
      • 16.4.6.1 Registering An Extension Function
      • 16.4.6.2 Registering An Exit Callback Function
      • 16.4.6.3 Registering An Extension Version String
      • 16.4.6.4 Customized Input Parsers
      • 16.4.6.5 Customized Output Wrappers
      • 16.4.6.6 Customized Two-way Processors
    • 16.4.7 Printing Messages
    • 16.4.8 Updating ERRNO
    • 16.4.9 Accessing and Updating Parameters
    • 16.4.10 Symbol Table Access
      • 16.4.10.1 Variable Access and Update by Name
      • 16.4.10.2 Variable Access and Update by Cookie
      • 16.4.10.3 Creating and Using Cached Values
    • 16.4.11 Array Manipulation
      • 16.4.11.1 Array Data Types
      • 16.4.11.2 Array Functions
      • 16.4.11.3 Working With All The Elements of an Array
      • 16.4.11.4 How To Create and Populate Arrays
    • 16.4.12 API Variables
      • 16.4.12.1 API Version Constants and Variables
      • 16.4.12.2 Informational Variables
    • 16.4.13 Boilerplate Code
  • 16.5 How gawk Finds Extensions
  • 16.6 Example: Some File Functions
    • 16.6.1 Using chdir() and stat()
    • 16.6.2 C Code for chdir() and stat()
    • 16.6.3 Integrating The Extensions
  • 16.7 The Sample Extensions In The gawk Distribution
    • 16.7.1 File Related Functions
    • 16.7.2 Interface To fnmatch()
    • 16.7.3 Interface To fork(), wait() and waitpid()
    • 16.7.4 Enabling In-Place File Editing
    • 16.7.5 Character and Numeric values: ord() and chr()
    • 16.7.6 Reading Directories
    • 16.7.7 Reversing Output
    • 16.7.8 Two-Way I/O Example
    • 16.7.9 Dumping and Restoring An Array
    • 16.7.10 Reading An Entire File
    • 16.7.11 API Tests
    • 16.7.12 Extension Time Functions
  • 16.8 The gawkextlib Project
• Appendix A The Evolution of the awk Language
  • A.1 Major Changes Between V7 and SVR3.1
  • A.2 Changes Between SVR3.1 and SVR4
  • A.3 Changes Between SVR4 and POSIX awk
  • A.4 Extensions in Brian Kernighan’s awk
  • A.5 Extensions in gawk Not in POSIX awk
  • A.6 History of gawk Features
  • A.7 Common Extensions Summary
  • A.8 Regexp Ranges and Locales: A Long Sad Story
  • A.9 Major Contributors to gawk
• Appendix B Installing gawk
  • B.1 The gawk Distribution
    • B.1.1 Getting the gawk Distribution
    • B.1.2 Extracting the Distribution
    • B.1.3 Contents of the gawk Distribution
  • B.2 Compiling and Installing gawk on Unix-like Systems
    • B.2.1 Compiling gawk for Unix-like Systems
    • B.2.2 Additional Configuration Options
    • B.2.3 The Configuration Process
  • B.3 Installation on Other Operating Systems
    • B.3.1 Installation on PC Operating Systems
      • B.3.1.1 Installing a Prepared Distribution for PC Systems
      • B.3.1.2 Compiling gawk for PC Operating Systems
      • B.3.1.3 Testing gawk on PC Operating Systems
      • B.3.1.4 Using gawk on PC Operating Systems
      • B.3.1.5 Using gawk In The Cygwin Environment
      • B.3.1.6 Using gawk In The MSYS Environment
    • B.3.2 How to Compile and Install gawk on VMS
      • B.3.2.1 Compiling gawk on VMS
      • B.3.2.2 Compiling gawk Dynamic Extensions on VMS
      • B.3.2.3 Installing gawk on VMS
      • B.3.2.4 Running gawk on VMS
      • B.3.2.5 The VMS GNV Project
      • B.3.2.6 Some VMS Systems Have An Old Version of gawk
  • B.4 Reporting Problems and Bugs
  • B.5 Other Freely Available awk Implementations
• Appendix C Implementation Notes
  • C.1 Downward Compatibility and Debugging
  • C.2 Making Additions to gawk
    • C.2.1 Accessing The gawk Git Repository
    • C.2.2 Adding New Features
    • C.2.3 Porting gawk to a New Operating System
    • C.2.4 Why Generated Files Are Kept In git
  • C.3 Probable Future Extensions
  • C.4 Some Limitations of the Implementation
  • C.5 Extension API Design
    • C.5.1 Problems With The Old Mechanism
    • C.5.2 Goals For A New Mechanism
    • C.5.3 Other Design Decisions
    • C.5.4 Room For Future Growth
  • C.6 Compatibility For Old Extensions
• Appendix D Basic Programming Concepts
  • D.1 What a Program Does
  • D.2 Data Values in a Computer
• Glossary
• GNU General Public License
• GNU Free Documentation License
  • ADDENDUM: How to use this License for your documents
• Index