Web Harvesting Services - Library of Congress

Form / Attachment / Pricing Sheet

This document, "J1 - Description-of-Tech-Environment_20250912", provides a detailed description of the Library of Congress's technical environment for web archiving, specifically in the context of the FY26 Web Harvesting IDIQ Contract. It outlines the tools, formats, and processes used by the Library, which are relevant for potential bidders to understand the operational landscape and integration requirements.

• Curator Tool: The Library uses an in-house tool called Digiboard for managing seed URL nomination, permissions, scoping, and preparation of seed lists.
• Seed Lists: These are provided in plain text with scoping instructions in SURT format, posted on a collaborative web space via a ticketing system.
• Archiving Software: The Library's infrastructure supports open-source tools like Heritrix, Brozzler, OpenWayback (v2.4.0), pywb (v2.8.4), and OutbackCDX (1.0.0).
• Storage Format: Harvested content is stored in the WARC (Web ARChive) file format.
• Data Transfer: Amazon Web Services (AWS) S3 is used for transferring web archives from vendors.
• Crawl Reports: The Library utilizes detailed crawl reports for quality review and content inventorying. Key data elements include:
  • Harvested/non-harvested seeds
  • Hosts harvested
  • Documents per host and total documents
  • MIME type breakdown
  • HTTP response codes
  • Bandwidth and data sizes (raw and compressed WARC)
  • Crawl duration and depth
  • Specific reports for 3XX, 4XX, 5XX HTTP codes, PDFs, and multimedia content replayability.
• Custom Host Data Report: A detailed report per unique host, including URI counts, byte counts, robots.txt exclusions, remaining URIs, novel URIs/bytes, duplicate content indicators, and 304 status code counts.

This attachment is crucial for bidders to understand the Library's existing technical infrastructure, preferred data formats (WARC, SURT), reporting requirements, and the specific metrics the Library values for assessing crawl quality and completeness. Familiarity with these elements will aid in developing compliant and competitive proposals for the Web Harvesting IDIQ Contract.

Solicitation / RFP / RFQ

This Solicitation (RFP 030ADV26R0010) is issued by the Library of Congress, Contracts and Grants Directorate, for Web Harvesting Services. The purpose is to procure contract support for systematic, at-scale web content harvesting, including temporary access to content, crawl reports, and transfer of content to the Library for preservation and public access. The Library aims to enrich its digital collections and serve Congress and America's communities.

• Web Content Harvesting: Perform crawls based on Library specifications, seed lists, scoping instructions, and robots.txt. This includes comprehensive capture of various digital objects (HTML, images, PDFs, multimedia) to accurately replicate webpages, employing politeness factors to minimize server impact.

• Data Packaging & Transfer: Package captured content in valid WARC (Web ARChive) files (ISO 28500_2017 WARC file format) with 11-field CDX indexes for transfer to the Library's S3 bucket via secure internet (HTTPS). Single BagIt bags should not exceed 1 TB, with a target WARC file size of ~1 GB.

• Quality Review & Reporting: Provide an access tool for Library staff to review crawl results prior to transfer. Generate detailed reports (ASCII text and XML) within 5 days of crawl completion, including statistical information on captured content, resources, and crawl performance. Develop and maintain a Quality Control Program (QCP).

• Infrastructure & Security: Utilize US-based servers for crawling. Maintain reliable and secure data storage with two copies (access and backup). Provide a web-based communication tool (e.g., ticketing system) for collaboration. Adhere to strict information security policies, including restrictions on Generative AI use and mandatory IT Security Training.

• Key Personnel: The contractor must provide qualified Program Manager/Alternate, Crawl Engineer, and Quality Assurance Lead, meeting specified experience requirements.

• Contract Type: Indefinite-Delivery, Indefinite-Quantity (IDIQ) with firm-fixed-price Task Orders.

• Period of Performance: Base period from June 1, 2026, to May 31, 2031.

• Estimated Value: Minimum order of $300,000.00; Maximum order of $15,000,000.00.

• Place of Performance: Contractor's own facilities.

• Offer Due Date: February 24, 2026, at 5:00 PM EST.

• Questions Due Date: February 2, 2026, at 12:00 PM EST, to Jennifer Zwahlen (CO) and Colleen Daly (Contract Specialist).

• Proposal Submission: Electronically via email to both the CO (jzwa@loc.gov) and Contract Specialist (cdaly@loc.gov). No zipped files. Total email attachment size not to exceed 20MB. Proposals must be valid through June 6, 2026.

• Proposal Content: Must include four volumes: Technical Approach (including a sample web crawl), Corporate Experience and Capabilities (including Key Personnel resumes), Past Performance (using Attachment J3), and Price (using Attachment J4).

• Evaluation Criteria: Best-Value Trade-Off (BVTO) approach. Evaluation factors are in descending order of importance: Technical Approach, Corporate Experience and Capabilities, Past Performance, and Price. Non-price factors combined are significantly more or equally important to price. The Library may award without discussions.

• Set-Aside: Unrestricted.

• Contractor Employee Fitness: Personnel with access to Library facilities, IT systems, or non-public information will undergo background investigations and continuous vetting.

• Attachments: The solicitation references several attachments (J1-J8) including a Description of Tech Environment, Sample Web Crawl Requirements, Seed List, Past Performance Questionnaire, Price Schedule, Web Archiving FAQ, Heritrix 3 Configuration, ISO WARC file format, and a Sample Task Order SOW. Bidders should review these for detailed instructions and context.

• Invoicing: Invoices must be submitted electronically via https://www.ipp.gov.

Solicitation / RFP / RFQ

This document outlines a technical test crawl requirement for the Library of Congress as part of RFP 030ADV26R0010. It details the specific instructions for performing a one-time web crawl as a technical test for potential offerors.

• Perform a one-time crawl using a Library-provided seed list (Attachment J2b) within 48 hours.

• Do not respect robots.txt per Library permissions policies.

• The crawl must be conducted in the environment proposed by the Contractor (e.g., cloud).

• The technical test must be repeated for each distinct web harvest technology proposed.

• Deliver results in WARC format, along with CDX files and generated reports.

• Reports should include a brief description of the tools/software used.

• Indicate the size (or estimated size) of the crawl data and reports via email to the Contract Specialist (cdaly@loc.gov) and Contracting Officer (jzwa@loc.gov) by 5:00 PM Eastern Time on February 2, 2026.

• Files shall be transferred via SFTP using instructions provided by the Library.

• Confirm successful data transfer with the Contract Specialist and Contracting Officer.

• Not specified in this attachment.

• Offerors are not required to make results accessible in their access tool for Library review.

• Offerors may be required to demonstrate their access tool using the crawl results (Section C.3.2).

• Not specified in this attachment.

• This is Attachment J2a, "Sample Web Crawl Requirement."

• The RFP number is 030ADV26R0010.

Form / Attachment / Pricing Sheet

This document, "J2b - Sample_Web_Crawl_Seed_List_20250917", is an attachment to the "Web Harvesting IDIQ Competition" Request for Proposals (RFP 030ADV26R0010). It provides a list of "seeds" or URLs intended for web crawling. The list is specified to be crawled weekly as part of an ongoing contract.

• Frequency: Weekly
• Crawl Type: Contract - ongoing
• Total Records: 100
• The document contains a list of specific URLs, each associated with a "Record ID".
• For each primary URL, there are often secondary URLs listed with a '+' prefix, indicating inclusion, and sometimes URLs with a '-' prefix, indicating exclusion from the crawl.

This attachment defines a critical component of the web harvesting service being procured. Bidders should review this seed list to understand the scope and nature of the data sources that will be targeted for crawling. It provides insight into the types of websites (e.g., government, news, organizational, personal blogs) and specific domains that are expected to be included in the web harvesting efforts under this contract. This information is crucial for assessing the technical requirements and potential challenges associated with fulfilling the contract's objectives.

Form / Attachment / Pricing Sheet

This questionnaire is designed to gather information on a contractor's past performance to assess their suitability for providing services to the Library of Congress. It is critical for the proposal evaluation process.

• Contractor Information: Fields for the contractor's name.
• Performance Ratings: A scale from Outstanding to Unsatisfactory, with definitions for each level.
• Narrative Explanations: Required for Marginal, Unsatisfactory, or N/A ratings.
• General Information: Details about the contract being evaluated (Name, POC, Contract Number, Period of Performance, Dollar Value, Description).
• Specific Questions: A series of 10 questions covering contract management, quality of products/services, project management effectiveness, timeliness, report submission, sub-contractor performance, personnel competence, communication, performance within cost, and contract terminations/rescheduling.

This document is a Past Performance Questionnaire (PPQ). Bidders should be aware that their past performance is of considerable importance to this acquisition. Input from current or previous customers is highly valued. Contractors should ensure they have mechanisms in place to gather and submit this information accurately and promptly. The deadline for returning the completed questionnaire is noon (Eastern) on February 9, 2026, to Colleen Daly (cdaly@loc.gov) with a copy to Jennifer Zwahlen (jzw@loc.gov).

Form / Attachment / Pricing Sheet

This document is a price schedule for the Library of Congress Request for Proposals (RFP) 030ADV26R0010 for Web Harvesting IDIQ Competition. It outlines the structure for proposing firm fixed-priced amounts per one (1) terabyte.

• Unit Price - One (1) Terabyte: Bidders must propose a unit price for one terabyte of web harvesting services.
• Ordering Period: The schedule includes five ordering periods, each with a one-year duration:
  • Period I: 06/01/2026 - 05/31/2027
  • Period II: 06/01/2027 - 05/31/2028
  • Period III: 06/01/2028 - 05/31/2029
  • Period IV: 06/01/2029 - 05/31/2030
  • Period V: 06/01/2030 - 05/31/2031
• No ordering tiers or discounts: The pricing structure does not include tiers or discounts based on volume.

This price schedule is critical for bidders to understand the pricing structure required for the Web Harvesting IDIQ contract. Bidders must propose their firm fixed-priced amounts per terabyte for each of the five ordering periods. Failure to provide pricing in the specified format may impact proposal evaluation.

Questions & Answers / Clarifications

This document provides Frequently Asked Questions (FAQs) for site owners regarding the Library of Congress's Web Archiving Program. It aims to clarify the process, permissions, and implications of archiving websites.

• Real Request Verification: Emails from the Library of Congress regarding web archiving will come from "webcapture AT loc DOT gov" with the subject line "Inclusion of your Website in the Library of Congress Web Archives." Site owners can contact the Web Archiving Team to confirm legitimacy.
• Permissions: Granting permission allows the Library to crawl and archive the website for historical and cultural preservation. Denying permission or not responding means the site will not be preserved.
• Website Selection: Websites are selected by Library subject experts based on guidance and focus on specific subjects, themes, and events.
• Crawling Frequency & Duration: Crawling frequency varies (weekly, monthly, quarterly) based on site content changes. Archiving can be for a specific period or ongoing.
• Crawler Issues: The Library's agent crawls politely to minimize server impact. Site owners should contact the Library if problems occur.
• Password-Protected Content: Password-protected content is generally not archived unless special permission is granted.
• Robots.txt: The Library generally bypasses robots.txt exclusions to collect as much of the site as possible, but site owners can contact them with questions.
• Third-Party Services: For protection against bots, site owners can ask the Library for their crawler IP addresses and user agent to "allow-list" them.
• URL Changes: The Library monitors for changes, but site owners can provide updates.
• Researcher Access: Researchers can access archives via loc.gov, browsing metadata and performing URL searches. Full-text search is not available, and off-site display depends on granted permissions.
• Opting Out: Copyright owners can opt out of online access (outside Library premises) by contacting the Library and providing specific details.
• Hosting: The Library archives a copy; site owners remain responsible for hosting and maintaining their live websites.

This document does not appear to be directly related to a specific solicitation but rather provides general information for site owners participating in the Library of Congress's ongoing web archiving efforts. It clarifies the process and expectations for website owners whose sites may be selected for archiving.

Other / Unclassified

This document provides detailed technical configuration information for the Heritrix web crawler. It covers various aspects of setting up and customizing crawl jobs, including:

• Basic Job Settings: Configuring crawl jobs via crawler-beans.cxml files.
• Crawl Limits: Setting limits on download size, document count, and time duration.
• Crawl Scope: Defining which URIs are included or excluded using DecideRules.
• Politeness Settings: Configuring delays and bandwidth limits to manage crawl rate.
• Retry Policy: Setting parameters for retrying failed fetches.
• Authentication and Cookies: Handling HTTP basic/digest authentication, HTML form authentication, and loading cookies.
• Protocol Support: Details on configuring FTP, HTTP/2, HTTP/3, SFTP, and WHOIS fetching.
• Job Modification: Options for modifying running jobs via the Bean Browser or Scripting Console.
• Proxy Configuration: Setting up HTTP and SOCKS5 proxies, and DNS over HTTP (DoH).

This document is a technical guide for configuring the Heritrix web crawler. While it does not describe a specific government procurement opportunity, it provides in-depth information on how to set up and customize web crawling behavior. This would be relevant to bidders who are involved in web crawling, data collection, or digital archiving projects that utilize or are compatible with the Heritrix framework. Understanding these configurations could be crucial for bidders proposing solutions that involve large-scale web data acquisition and processing.

Other / Unclassified

This document is the International Standard ISO 28500:2017, which specifies the WARC (Web ARChive) file format. The WARC format is designed for storing both the payload content and control information from Internet application layer protocols (such as HTTP, DNS, and FTP). It also supports storing arbitrary metadata, data compression, record integrity, control information from harvesting protocols, results of data transformations, and duplicate detection events. The format is designed to be extensible without disrupting existing functionality and can support handling of long records through truncation or segmentation.

This standard defines a file format for archiving web content. If the SAM.gov opportunity involves web archiving, data collection, or management of large datasets from the internet, understanding this standard would be crucial for bidders to ensure compliance with data storage and formatting requirements. It outlines the technical specifications for how such data should be structured and stored.

Statement of Work (SOW) / Performance Work Statement (PWS)

This Statement of Work (SOW) outlines the requirements for contract support to systematically harvest web content based on instructions from Library of Congress (Library) staff. The goal is to provide temporary access to harvested content and crawl reports for quality review, and to enable the transfer of content to the Library for preservation and public access.

• Capture web content from Library-provided seed lists at specified frequencies.

• Provide temporary access to harvested content and required crawl reports for quality review.

• Transfer captured content to the Library for preservation and public access.

• The contractor shall collect an estimated 350-700 Terabytes (TB) of data.

• Weekly Crawls: Capture content from approximately 1,700 seed URLs weekly. 46 crawls are planned. Crawls last one week or until data caps are met. Deduplication against an initial undeduplicated crawl is required. Robots.txt is generally ignored unless otherwise instructed.

• Monthly Crawls: Capture content from approximately 4,000 seed URLs monthly. 11 crawls are planned over 11 months. Crawls last an estimated 4 weeks or until data caps are met. Deduplication is required. Robots.txt is generally ignored.

• Extended Crawls: Perform in-depth capture of priority content, including very large sites. 6-10 crawls are planned, each with up to 10,000 seed URLs and lasting an estimated 6-12 weeks or until data caps are met. Deduplication is required for each extended type. Robots.txt is generally ignored.

• US Election 2026 Weekly Crawls: Capture content from seed URLs related to the US Election 2026. 26 weekly crawls are planned. Seed URLs will incrementally increase up to 1,200-1,500 by Election Day, then stabilize or decrease. Crawls last one week or until data caps are met. Deduplication is required. Robots.txt is generally ignored.

• Reports required per C.3.3.1 of the IDIQ contract, due no later than 5 days after each crawl completion.

• Quality assurance results submitted per an approved Quality Assurance Plan.

• Invoicing will be monthly for accepted data.

• The period of performance will be identified at the time of award.

• Each task order includes approximately two weeks at the end for final quality assurance, acceptance, and data transfer.

• Seed lists are the property of the Library and are solely for collection purposes.