LISP - String Replacement

Modern Lisp: String Manipulation and Replacement

String manipulation is a cornerstone of most applications. In Common Lisp, strings are immutable sequences of characters. This means that any function that “replaces” part of a string actually returns a new string with the modifications. This guide covers both built-in functions and the modern, standard approach using regular expressions.

Basic Character Replacement with substitute

The simplest replacement tool is substitute. It replaces all occurrences of a specific character with a new one.

(substitute #\_ #\Space "a file name with spaces")
;=> "a_file_name_with_spaces"

(substitute #\o #\i "Lisp is interesting.")
;=> "Losp os onterestong."

You can also use its more powerful variants, substitute-if and substitute-if-not, with a predicate function.

;; Replace all digit characters with an asterisk '*'
(substitute #\* #\Space "user-id-12345-tmp"
            :test (lambda (item seq-item) (digit-char-p seq-item)))

;; A more idiomatic way using substitute-if
(substitute-if #\* #'digit-char-p "user-id-12345-tmp")
;=> "user-id-*****-tmp"

The Industry Standard: Substring Replacement with Regular Expressions

For anything more complex than single-character replacement, the de facto standard in the Common Lisp community is the cl-ppcre library. It provides powerful and efficient regular expression tools. While you can write manual substring replacement functions, it is rarely a good idea in production code due to poor performance and complexity.

Setting Up Your Environment with Quicklisp

Quicklisp is the package manager for Common Lisp. If you haven’t installed it, you should do so first. Once installed, you can load any library with one command.

;; To load the library for the current session:
(ql:quickload "cl-ppcre")

Example: Using regex-replace-all

The most common function you’ll use is cl-ppcre:regex-replace-all.

(ql:quickload "cl-ppcre")

;; Simple substring replacement
(cl-ppcre:regex-replace-all "brown" "A quick brown fox" "red")
;=> "A quick red fox"

;; Using regular expressions to replace multiple spaces with a single space
(cl-ppcre:regex-replace-all "\\s+" "hello     world   from Lisp" " ")
;=> "hello world from Lisp"

;; More complex replacement: reformatting a date from YYYY-MM-DD to MM/DD/YYYY
;; We use capture groups \1, \2, \3 to refer to the matched parts.
(cl-ppcre:regex-replace-all "(\\d{4})-(\\d{2})-(\\d{2})"
                              "Date: 2023-10-26"
                              "\\2/\\3/\\1")
;=> "Date: 10/26/2023"

For Learning: A Manual Substring Replacement Function

To understand the underlying logic, it can be useful to see how a manual replacement function works. The following function replaces all occurrences of a substring. It’s a good exercise but demonstrates why using a library like cl-ppcre is preferred for its robustness and performance.

(defun replace-all (string old new)
  "Replaces all occurrences of `old` with `new` in `string`.
   NOTE: For production code, use cl-ppcre:regex-replace-all."
  (with-output-to-string (s)
    (loop with old-len = (length old)
          with str-len = (length string)
          for start = 0 then (+ pos old-len)
          for pos = (search old string :start2 start)
          do (write-string string s :start start :end pos)
          when pos do (write-string new s)
          while pos)))

;; --- Test it ---
(replace-all "A quick brown fox jumps over the brown dog" "brown" "red")
;=> "A quick red fox jumps over the red dog"

Best Practices

• Default to cl-ppcre: For any non-trivial string replacement, use cl-ppcre. It’s fast, powerful, and a community standard.
• Understand Immutability: Remember that you are always creating new strings. In a tight loop, be mindful of memory allocation. Use with-output-to-string as shown in the replace-all example for efficient string building.
• Learn Basic Regex: A small investment in learning regular expressions will pay huge dividends in your ability to manipulate text effectively.