Web Harvesting Services - Library of Congress

Amendment / Modification

• The proposal due date has been extended by one week.

• The new proposal due date is March 3, 2026, at 5:00 PM Eastern Time.

• This extension does NOT affect the due date for the sample web crawl transfer information, which remains February 17, 2026, at 5:00 PM Eastern Time.

• Solicitation due date.

• The deadline for submitting proposals has been moved to March 3, 2026.

• Offerors and interested parties are responsible for monitoring SAM.gov for any further changes to the solicitation.

Amendment / Modification

This amendment provides a revised version of Attachment J9, dated 02/13/2026. This revised attachment includes answers to two late questions (numbered 31 and 32) pertaining to the sample web crawl. Revisions within the attachment are indicated in red text.

Attachment J9 (revised version dated 02/13/2026).

No changes to proposal deadlines, requirements, or evaluation factors are indicated, apart from the updated attachment content.

Questions & Answers / Clarifications

This document provides responses to questions submitted by potential offerors regarding the "Web Harvesting Services" solicitation (Opportunity ID: 030ADV26R0010).

• Sample Web Crawl Due Date: Information regarding the size of the sample web crawl is due by 5:00 PM Eastern Time on February 17, 2026. This is one week before the due date for all proposal materials.
• Past Performance Questionnaires (PPQs) Due Date: PPQs are due by noon Eastern Time on February 24, 2026. These must be sent directly from the past performance reference.
• User Agent for Sample Crawl: Offerors are not required to use a specific user agent for the sample crawl. However, they should not use a user agent indicating the Library of Congress for the test crawl.
• Cloud Infrastructure: The Library will not consider an approach where crawl data is written directly into cloud infrastructure owned and controlled by the Library.
• Estimated Crawl Volume: The anticipated crawl volume is a range of 300-700 TB per year, though the actual number may vary.
• Robots.txt Support: Implementation for respecting robots.txt can vary, depending on the contractor's web harvest tools and specifications.
• Test Crawl Duration: The sample web crawl must include all provided seeds within a single, continuous 48-hour period.
• Previous Contract Pricing: This solicitation is for a new, separate contract and is not related to previous contract pricing.
• Incumbent Information: The Library does not provide information about the incumbent contractor.
• Pricing Metric: The total compressed WARC size will be used to calculate the invoiced volume.

This document clarifies several key dates and technical requirements, including the due dates for sample web crawl data and past performance questionnaires. It also provides specific guidance on technical approaches and data handling, which may influence offerors' proposal preparation and technical solutions.

Amendment / Modification

This amendment provides an updated version of Attachment J9 – Q&A_030ADV26R0010, dated 02/03/2026. This attachment contains answers to questions received regarding the solicitation.

Attachment J9 (Questions & Answers) has been updated.

No other changes are made to the solicitation or its attachments. Offerors and interested parties are responsible for monitoring SAM.gov for any further changes.

Questions & Answers / Clarifications

This document provides responses to vendor questions regarding the "Web Harvesting Services" solicitation (030ADV26R0010).

• Data Due Dates: Information regarding the size of the sample web crawl is due February 17, 2026. Past Performance Questionnaires (PPQs) are due February 24, 2026.
• User Agent: Offerors are not required to use a specific user agent for the sample crawl and should not use one that indicates the Library of Congress.
• Cloud Infrastructure: The Library will not consider an approach where crawl data is written directly into cloud infrastructure owned and controlled by the Library.
• Estimated Crawl Volume: The anticipated crawl volume is 300-700 TB per year, though this may vary.
• Robots.txt Support: Implementations that check HTML pages disallowed in robots.txt or check all in-page resources may be acceptable, depending on the tools used.
• Test Crawl Duration: The sample web crawl must include all provided seeds and be completed within a single 48-hour period.
• Previous Contract: This solicitation is for a new, separate contract and is not related to previous contracts or RFIs.
• Generative AI: The Library does not currently distinguish between "Generative AI" and "Analytical/Agentic AI," but this field is emerging.
• Pricing Metric: The total compressed WARC size will be used to calculate invoiced volume.

This Q&A document clarifies several aspects of the solicitation, including deadlines, technical requirements, and data handling. Offerors should review these clarifications to ensure their proposals accurately address the requirements. No extension to the proposal submission deadline has been granted.

Amendment / Modification

• Updates the submission date for the Past Performance Questionnaire (PPQ) found in Attachment J3.

• Adds Attachment J9, which consolidates questions received regarding proposal document due dates.

• Attachment J3 (Past Performance Questionnaire)

• New Attachment J9 (Questions & Answers)

• The deadline for submitting the Past Performance Questionnaire has been updated.

• Attachment J9 provides clarifications on proposal document due dates.

Form / Attachment / Pricing Sheet

This questionnaire is designed to gather information on a contractor's past performance to assess their suitability for providing services to the Library of Congress. It is critical for the proposal evaluation process.

• Rating Scale: Provides definitions for Outstanding, Good, Satisfactory, Marginal, and Unsatisfactory performance.
• Contractor Information: Fields for the contractor's name, point of contact, telephone number, contract number, contract type, period of performance, approximate dollar value, and contract description.
• Performance Questions: A series of questions (1-10) requiring ratings on various aspects of contract management, quality of products/services, project management effectiveness, timeliness, reporting, sub-contractor efforts, personnel competence, communication, cost performance, and contract terminations/rescheduling.
• Narrative Explanations: Requires detailed explanations for Marginal, Unsatisfactory, or N/A ratings.
• Submission Instructions: Specifies returning the completed form via email to Colleen Daly (cdaly@loc.gov) and Jennifer Zwahlen (jzwa@loc.gov) by noon Eastern on February 24, 2026.

This document is crucial for bidders as it outlines the specific criteria and format for providing past performance feedback. Bidders should understand that past performance is of considerable importance to this acquisition, and input from current or previous customers is highly valued. Completing this form accurately and providing honest assessments will directly impact the evaluation of proposals for the digital collection web harvesting services.

Questions & Answers / Clarifications

This document provides clarifications and answers to vendor questions regarding the "Web Harvesting Services" opportunity (030ADV26R0010).

• Q1/A1: The due date for information regarding the size of the sample web crawl is February 17, 2026, by 5:00 PM Eastern Time. This information is due to the Library Contracting Team. The amended Attachment J2a, posted with Amendment 0001, contains this updated information.
• Q2/A2: Past Performance Questionnaires (PPQs) are due by noon Eastern Time on February 24, 2026. These questionnaires must be sent directly to the Contracting Team by the past performance reference. PPQs sent directly by the offeror to the Library will not be considered.

This Q&A document clarifies submission deadlines for specific attachments (sample web crawl size and PPQs), impacting proposal preparation and submission timelines.

Amendment / Modification

• This amendment (0001) updates the information submission date within Attachment J2a – Sample Web Crawl Requirement. The specific details highlighted in blue within Attachment J2a - Sample Web Crawl Requirement_20260123 indicate the revised submission date.

• Attachment J2a - Sample Web Crawl Requirement

• The proposal deadline or submission requirements may be affected due to the updated information submission date in Attachment J2a. Offerors should review the highlighted changes in the specified attachment.

Amendment / Modification

This amendment provides Attachment J2a, which details a sample web crawl requirement for a technical test. The key requirements for this test crawl are:

• Scope: Perform a one-time crawl using a Library-provided seed list (Attachment J2b) containing 100 seeds of varying types and complexity, plus scopes.
• Duration: The crawl must not exceed 48 total hours.
• Robots.txt: Do not respect robots.txt per Library permissions policies for web archives.
• Environment: The technical test crawl must be conducted in the environment proposed for the program (e.g., cloud if proposed).
• Technology: The technical test must be repeated for each distinct web harvest technology proposed.
• Deliverables: Results must be delivered in WARC format, along with CDX files and any reports generated (as outlined in SOW section C.3.3.1.2.2).
• Reporting: Offeror's report should briefly describe the tool(s)/software used.
• Notification: Upon completion, indicate the size (or estimated size) of the sample crawl data and reports via email to the Contract Specialist (cdaly@loc.gov) and Contracting Officer (jzwa@loc.gov) by 5:00 PM Eastern Time on February 17, 2026.
• Transfer Method: Files shall be transferred via SFTP. The Library will provide transfer instructions.
• Confirmation: After transfer, contact the Contract Specialist and Contracting Officer to confirm receipt.
• Access Tool Demonstration: Offerors are not required to make results accessible in their access tool, but may be required to demonstrate their access tool using the crawl results.

This amendment introduces Attachment J2a (Sample Web Crawl Requirement) and references Attachment J2b (Sample Web Crawl Seed List) and SOW section C.3.3.1.2.2.

This amendment introduces a specific technical test requirement and a deadline of February 17, 2026, for notifying the Library of the sample crawl data size. Bidders must prepare to conduct this test crawl and submit the required data and reports.

Form / Attachment / Pricing Sheet

This document, "J1 - Description-of-Tech-Environment_20250912", provides a detailed description of the Library of Congress's technical environment for web archiving, specifically in the context of the FY26 Web Harvesting IDIQ Contract. It outlines the tools, formats, and processes used by the Library, which are relevant for potential bidders to understand the operational landscape and integration requirements.

• Curator Tool: The Library uses an in-house tool called Digiboard for managing seed URL nomination, permissions, scoping, and preparation of seed lists.
• Seed Lists: These are provided in plain text with scoping instructions in SURT format, posted on a collaborative web space via a ticketing system.
• Archiving Software: The Library's infrastructure supports open-source tools like Heritrix, Brozzler, OpenWayback (v2.4.0), pywb (v2.8.4), and OutbackCDX (1.0.0).
• Storage Format: Harvested content is stored in the WARC (Web ARChive) file format.
• Data Transfer: Amazon Web Services (AWS) S3 is used for transferring web archives from vendors.
• Crawl Reports: The Library utilizes detailed crawl reports for quality review and content inventorying. Key data elements include:
  • Harvested/non-harvested seeds
  • Hosts harvested
  • Documents per host and total documents
  • MIME type breakdown
  • HTTP response codes
  • Bandwidth and data sizes (raw and compressed WARC)
  • Crawl duration and depth
  • Specific reports for 3XX, 4XX, 5XX HTTP codes, PDFs, and multimedia content replayability.
• Custom Host Data Report: A detailed report per unique host, including URI counts, byte counts, robots.txt exclusions, remaining URIs, novel URIs/bytes, duplicate content indicators, and 304 status code counts.

This attachment is crucial for bidders to understand the Library's existing technical infrastructure, preferred data formats (WARC, SURT), reporting requirements, and the specific metrics the Library values for assessing crawl quality and completeness. Familiarity with these elements will aid in developing compliant and competitive proposals for the Web Harvesting IDIQ Contract.

Solicitation / RFP / RFQ

This Solicitation (RFP 030ADV26R0010) is issued by the Library of Congress, Contracts and Grants Directorate, for Web Harvesting Services. The purpose is to procure contract support for systematic, at-scale web content harvesting, including temporary access to content, crawl reports, and transfer of content to the Library for preservation and public access. The Library aims to enrich its digital collections and serve Congress and America's communities.

• Web Content Harvesting: Perform crawls based on Library specifications, seed lists, scoping instructions, and robots.txt. This includes comprehensive capture of various digital objects (HTML, images, PDFs, multimedia) to accurately replicate webpages, employing politeness factors to minimize server impact.

• Data Packaging & Transfer: Package captured content in valid WARC (Web ARChive) files (ISO 28500_2017 WARC file format) with 11-field CDX indexes for transfer to the Library's S3 bucket via secure internet (HTTPS). Single BagIt bags should not exceed 1 TB, with a target WARC file size of ~1 GB.

• Quality Review & Reporting: Provide an access tool for Library staff to review crawl results prior to transfer. Generate detailed reports (ASCII text and XML) within 5 days of crawl completion, including statistical information on captured content, resources, and crawl performance. Develop and maintain a Quality Control Program (QCP).

• Infrastructure & Security: Utilize US-based servers for crawling. Maintain reliable and secure data storage with two copies (access and backup). Provide a web-based communication tool (e.g., ticketing system) for collaboration. Adhere to strict information security policies, including restrictions on Generative AI use and mandatory IT Security Training.

• Key Personnel: The contractor must provide qualified Program Manager/Alternate, Crawl Engineer, and Quality Assurance Lead, meeting specified experience requirements.

• Contract Type: Indefinite-Delivery, Indefinite-Quantity (IDIQ) with firm-fixed-price Task Orders.

• Period of Performance: Base period from June 1, 2026, to May 31, 2031.

• Estimated Value: Minimum order of $300,000.00; Maximum order of $15,000,000.00.

• Place of Performance: Contractor's own facilities.

• Offer Due Date: February 24, 2026, at 5:00 PM EST.

• Questions Due Date: February 2, 2026, at 12:00 PM EST, to Jennifer Zwahlen (CO) and Colleen Daly (Contract Specialist).

• Proposal Submission: Electronically via email to both the CO (jzwa@loc.gov) and Contract Specialist (cdaly@loc.gov). No zipped files. Total email attachment size not to exceed 20MB. Proposals must be valid through June 6, 2026.

• Proposal Content: Must include four volumes: Technical Approach (including a sample web crawl), Corporate Experience and Capabilities (including Key Personnel resumes), Past Performance (using Attachment J3), and Price (using Attachment J4).

• Evaluation Criteria: Best-Value Trade-Off (BVTO) approach. Evaluation factors are in descending order of importance: Technical Approach, Corporate Experience and Capabilities, Past Performance, and Price. Non-price factors combined are significantly more or equally important to price. The Library may award without discussions.

• Set-Aside: Unrestricted.

• Contractor Employee Fitness: Personnel with access to Library facilities, IT systems, or non-public information will undergo background investigations and continuous vetting.

• Attachments: The solicitation references several attachments (J1-J8) including a Description of Tech Environment, Sample Web Crawl Requirements, Seed List, Past Performance Questionnaire, Price Schedule, Web Archiving FAQ, Heritrix 3 Configuration, ISO WARC file format, and a Sample Task Order SOW. Bidders should review these for detailed instructions and context.

• Invoicing: Invoices must be submitted electronically via https://www.ipp.gov.

Solicitation / RFP / RFQ

This document outlines a technical test crawl requirement for the Library of Congress as part of RFP 030ADV26R0010. It details the specific instructions for performing a one-time web crawl as a technical test for potential offerors.

• Perform a one-time crawl using a Library-provided seed list (Attachment J2b) within 48 hours.

• Do not respect robots.txt per Library permissions policies.

• The crawl must be conducted in the environment proposed by the Contractor (e.g., cloud).

• The technical test must be repeated for each distinct web harvest technology proposed.

• Deliver results in WARC format, along with CDX files and generated reports.

• Reports should include a brief description of the tools/software used.

• Indicate the size (or estimated size) of the crawl data and reports via email to the Contract Specialist (cdaly@loc.gov) and Contracting Officer (jzwa@loc.gov) by 5:00 PM Eastern Time on February 2, 2026.

• Files shall be transferred via SFTP using instructions provided by the Library.

• Confirm successful data transfer with the Contract Specialist and Contracting Officer.

• Not specified in this attachment.

• Offerors are not required to make results accessible in their access tool for Library review.

• Offerors may be required to demonstrate their access tool using the crawl results (Section C.3.2).

• Not specified in this attachment.

• This is Attachment J2a, "Sample Web Crawl Requirement."

• The RFP number is 030ADV26R0010.

Form / Attachment / Pricing Sheet

This document, "J2b - Sample_Web_Crawl_Seed_List_20250917", is an attachment to the "Web Harvesting IDIQ Competition" Request for Proposals (RFP 030ADV26R0010). It provides a list of "seeds" or URLs intended for web crawling. The list is specified to be crawled weekly as part of an ongoing contract.

• Frequency: Weekly
• Crawl Type: Contract - ongoing
• Total Records: 100
• The document contains a list of specific URLs, each associated with a "Record ID".
• For each primary URL, there are often secondary URLs listed with a '+' prefix, indicating inclusion, and sometimes URLs with a '-' prefix, indicating exclusion from the crawl.

This attachment defines a critical component of the web harvesting service being procured. Bidders should review this seed list to understand the scope and nature of the data sources that will be targeted for crawling. It provides insight into the types of websites (e.g., government, news, organizational, personal blogs) and specific domains that are expected to be included in the web harvesting efforts under this contract. This information is crucial for assessing the technical requirements and potential challenges associated with fulfilling the contract's objectives.

Form / Attachment / Pricing Sheet

This questionnaire is designed to gather information on a contractor's past performance to assess their suitability for providing services to the Library of Congress. It is critical for the proposal evaluation process.

• Contractor Information: Fields for the contractor's name.
• Performance Ratings: A scale from Outstanding to Unsatisfactory, with definitions for each level.
• Narrative Explanations: Required for Marginal, Unsatisfactory, or N/A ratings.
• General Information: Details about the contract being evaluated (Name, POC, Contract Number, Period of Performance, Dollar Value, Description).
• Specific Questions: A series of 10 questions covering contract management, quality of products/services, project management effectiveness, timeliness, report submission, sub-contractor performance, personnel competence, communication, performance within cost, and contract terminations/rescheduling.

This document is a Past Performance Questionnaire (PPQ). Bidders should be aware that their past performance is of considerable importance to this acquisition. Input from current or previous customers is highly valued. Contractors should ensure they have mechanisms in place to gather and submit this information accurately and promptly. The deadline for returning the completed questionnaire is noon (Eastern) on February 9, 2026, to Colleen Daly (cdaly@loc.gov) with a copy to Jennifer Zwahlen (jzw@loc.gov).

Form / Attachment / Pricing Sheet

This document is a price schedule for the Library of Congress Request for Proposals (RFP) 030ADV26R0010 for Web Harvesting IDIQ Competition. It outlines the structure for proposing firm fixed-priced amounts per one (1) terabyte.

• Unit Price - One (1) Terabyte: Bidders must propose a unit price for one terabyte of web harvesting services.
• Ordering Period: The schedule includes five ordering periods, each with a one-year duration:
  • Period I: 06/01/2026 - 05/31/2027
  • Period II: 06/01/2027 - 05/31/2028
  • Period III: 06/01/2028 - 05/31/2029
  • Period IV: 06/01/2029 - 05/31/2030
  • Period V: 06/01/2030 - 05/31/2031
• No ordering tiers or discounts: The pricing structure does not include tiers or discounts based on volume.

This price schedule is critical for bidders to understand the pricing structure required for the Web Harvesting IDIQ contract. Bidders must propose their firm fixed-priced amounts per terabyte for each of the five ordering periods. Failure to provide pricing in the specified format may impact proposal evaluation.

Questions & Answers / Clarifications

This document provides Frequently Asked Questions (FAQs) for site owners regarding the Library of Congress's Web Archiving Program. It aims to clarify the process, permissions, and implications of archiving websites.

• Real Request Verification: Emails from the Library of Congress regarding web archiving will come from "webcapture AT loc DOT gov" with the subject line "Inclusion of your Website in the Library of Congress Web Archives." Site owners can contact the Web Archiving Team to confirm legitimacy.
• Permissions: Granting permission allows the Library to crawl and archive the website for historical and cultural preservation. Denying permission or not responding means the site will not be preserved.
• Website Selection: Websites are selected by Library subject experts based on guidance and focus on specific subjects, themes, and events.
• Crawling Frequency & Duration: Crawling frequency varies (weekly, monthly, quarterly) based on site content changes. Archiving can be for a specific period or ongoing.
• Crawler Issues: The Library's agent crawls politely to minimize server impact. Site owners should contact the Library if problems occur.
• Password-Protected Content: Password-protected content is generally not archived unless special permission is granted.
• Robots.txt: The Library generally bypasses robots.txt exclusions to collect as much of the site as possible, but site owners can contact them with questions.
• Third-Party Services: For protection against bots, site owners can ask the Library for their crawler IP addresses and user agent to "allow-list" them.
• URL Changes: The Library monitors for changes, but site owners can provide updates.
• Researcher Access: Researchers can access archives via loc.gov, browsing metadata and performing URL searches. Full-text search is not available, and off-site display depends on granted permissions.
• Opting Out: Copyright owners can opt out of online access (outside Library premises) by contacting the Library and providing specific details.
• Hosting: The Library archives a copy; site owners remain responsible for hosting and maintaining their live websites.

This document does not appear to be directly related to a specific solicitation but rather provides general information for site owners participating in the Library of Congress's ongoing web archiving efforts. It clarifies the process and expectations for website owners whose sites may be selected for archiving.

Other / Unclassified

This document provides detailed technical configuration information for the Heritrix web crawler. It covers various aspects of setting up and customizing crawl jobs, including:

• Basic Job Settings: Configuring crawl jobs via crawler-beans.cxml files.
• Crawl Limits: Setting limits on download size, document count, and time duration.
• Crawl Scope: Defining which URIs are included or excluded using DecideRules.
• Politeness Settings: Configuring delays and bandwidth limits to manage crawl rate.
• Retry Policy: Setting parameters for retrying failed fetches.
• Authentication and Cookies: Handling HTTP basic/digest authentication, HTML form authentication, and loading cookies.
• Protocol Support: Details on configuring FTP, HTTP/2, HTTP/3, SFTP, and WHOIS fetching.
• Job Modification: Options for modifying running jobs via the Bean Browser or Scripting Console.
• Proxy Configuration: Setting up HTTP and SOCKS5 proxies, and DNS over HTTP (DoH).

This document is a technical guide for configuring the Heritrix web crawler. While it does not describe a specific government procurement opportunity, it provides in-depth information on how to set up and customize web crawling behavior. This would be relevant to bidders who are involved in web crawling, data collection, or digital archiving projects that utilize or are compatible with the Heritrix framework. Understanding these configurations could be crucial for bidders proposing solutions that involve large-scale web data acquisition and processing.

Other / Unclassified

This document is the International Standard ISO 28500:2017, which specifies the WARC (Web ARChive) file format. The WARC format is designed for storing both the payload content and control information from Internet application layer protocols (such as HTTP, DNS, and FTP). It also supports storing arbitrary metadata, data compression, record integrity, control information from harvesting protocols, results of data transformations, and duplicate detection events. The format is designed to be extensible without disrupting existing functionality and can support handling of long records through truncation or segmentation.

This standard defines a file format for archiving web content. If the SAM.gov opportunity involves web archiving, data collection, or management of large datasets from the internet, understanding this standard would be crucial for bidders to ensure compliance with data storage and formatting requirements. It outlines the technical specifications for how such data should be structured and stored.

Statement of Work (SOW) / Performance Work Statement (PWS)

This Statement of Work (SOW) outlines the requirements for contract support to systematically harvest web content based on instructions from Library of Congress (Library) staff. The goal is to provide temporary access to harvested content and crawl reports for quality review, and to enable the transfer of content to the Library for preservation and public access.

• Capture web content from Library-provided seed lists at specified frequencies.

• Provide temporary access to harvested content and required crawl reports for quality review.

• Transfer captured content to the Library for preservation and public access.

• The contractor shall collect an estimated 350-700 Terabytes (TB) of data.

• Weekly Crawls: Capture content from approximately 1,700 seed URLs weekly. 46 crawls are planned. Crawls last one week or until data caps are met. Deduplication against an initial undeduplicated crawl is required. Robots.txt is generally ignored unless otherwise instructed.

• Monthly Crawls: Capture content from approximately 4,000 seed URLs monthly. 11 crawls are planned over 11 months. Crawls last an estimated 4 weeks or until data caps are met. Deduplication is required. Robots.txt is generally ignored.

• Extended Crawls: Perform in-depth capture of priority content, including very large sites. 6-10 crawls are planned, each with up to 10,000 seed URLs and lasting an estimated 6-12 weeks or until data caps are met. Deduplication is required for each extended type. Robots.txt is generally ignored.

• US Election 2026 Weekly Crawls: Capture content from seed URLs related to the US Election 2026. 26 weekly crawls are planned. Seed URLs will incrementally increase up to 1,200-1,500 by Election Day, then stabilize or decrease. Crawls last one week or until data caps are met. Deduplication is required. Robots.txt is generally ignored.

• Reports required per C.3.3.1 of the IDIQ contract, due no later than 5 days after each crawl completion.

• Quality assurance results submitted per an approved Quality Assurance Plan.

• Invoicing will be monthly for accepted data.

• The period of performance will be identified at the time of award.

• Each task order includes approximately two weeks at the end for final quality assurance, acceptance, and data transfer.

• Seed lists are the property of the Library and are solely for collection purposes.