Project Record
PDF Parser Using sed Commands
Command-template workflow for extracting text from PDFs and normalizing output through sed filters.
Context
Many internal reports and disclosures are PDF-only. Extracting text for search, parsing, or downstream pipelines usually requires a reliable PDF-to-text step plus normalization (whitespace, line breaks, encoding). A command-template workflow using pdftotext and sed keeps the pipeline simple and auditable without heavy dependencies.
Problem
Raw PDF text extraction is messy: inconsistent spacing, stray line breaks, and non-ASCII characters. We need a repeatable pipeline that (1) extracts text with layout preserved where useful, (2) normalizes spaces and blank lines, and (3) optionally applies further sed-style rules for field extraction or cleanup. The solution should work in scripts and in an internal tool that previews or applies the same logic to pasted text.
Shell Pipeline
pdftotext -layout report.pdf - | sed -E 's/[[:space:]]+/ /g' | sed '/^$/d'First stage extracts text; subsequent sed steps collapse whitespace and remove empty lines. Additional rules can strip headers, normalize dates, or extract tables.
Implementation
- Command templates: parameterized by input path and optional sed script name.
- In the static-first app: the module generates command previews and can apply the same sed-like cleanup to user-provided text snippets so that output can be checked without running shell commands.
- Optional: store favorite sed presets (e.g. “earnings report”, “regulatory filing”) for one-click application.
Trade-offs
Layout extraction is best-effort; complex tables or multi-column PDFs may need dedicated tools (e.g. tabula, camelot). Sed is powerful but brittle for highly variable formats; for production parsing, consider structured extraction or ML once the pipeline is stable.